This post includes an update on my plans to do expressions with VRoid characters for the purpose of creating short animated video clips using Unity and VRoid Studio characters.
Note: The main value of this post may be the summary list of blendshapes and how they move parts of the face below. The tables can be used to more quickly find the right blendshapes to use to create a particular expression. I plan to have a later post that lists the expressions I settled on and the blendshapes (and strengths) I ended up with.
What are blendshapes vs blendshape clips?
VRM files from VRoid Studio support a set of standard expressions (Angry, Blink, Fun, Joy, Sorrow, Surprised, A, E, I, O, U and more), plus allow additional custom expressions to be defined. Many apps offer buttons or similar to trigger the expressions. These expressions are captured in “blendshape clips”.
Blendshape clips are defined using a set of strength settings for blendshapes (also known as Shape Keys in Blender). Blendshapes define how to warp the mesh of the face. For example, you can define an expression such as anger using a blendshape clip to narrow the eyelids (the first blendshape), bring the eyebrows down in the middle (another blendshape), and raise the lips in the corner of the mouth in a snarl (a third blendshape). The blendshape clip keeps the strength settings for the three blendshapes, as needed.
Some blendshapes defined by VRoid Studio control one part of the face (BRW = eyebrow, EYE = eye, MTH = mouth, HA = fangs), others control many parts at once (ALL). The former is useful if the latter does not provide fine enough control.
There are an additional set of blendshapes defined by ARKit which some apps can use. The following is a subset of the additional blendshapes available. I am using HANA_Tool to provide the blendshapes for these standard names.
You can also define your own blendshapes (which I have done a little). These can be created using Blender (with some learning effort), or in my case I created some simple blendshapes \using C# code.
VRoid blendshape clips are defined in files starting with “BlendShape” (just to make things a bit confusing).
The “BlendShape” file above references the different blendshape clips in the directory. The standard expressions include:
Unity Animation Clips
I have previously said that I plan to use animation clips in Unity by recording blendshape strengths directly, rather than use blendshape clips, because it also allows me to fade skin textures using a custom shader. I am however planning to move across to blendshape clips as there are more and more VRM tools becoming available, and if I use blendshape clips then I can use those tools with my avatars.
I still plan to use animation clips inside Unity, but each clip will record a single property, the strength of the blendshape clip. This is because when you drag an animation clip into a Unity Cinemachine Timeline, you get to see the animation clip name. If you do direct recordings of properties, it’s a bit harder to see what the expression is at different time points. (The following shows examples three tracks using animation clips and two tracks recording properties directly.)
Note: if you want more technical details on how to create characters using advanced features, a great YouTube channel I came across is by Suvdriel. I am going to be using many of the techniques she describes, to avoid using a custom shader (my current approach) to do expressions such as blushing. There are the VRoid tricks like using hair layers with special hair materials, but she also describes how to tile multiple alternate images and do some color layer adjustments. (I discovered her content the day after I worked out some of these myself, but she describes it all very well on her channel so I am not going to repeat it here.)
How to animate expressions in Unity
VRoid characters imported using the UniVRM package for Unity have a component that provides access to the set of available blendshape clips. The only problem is it only exposes the clips during “play” time, not when editing. This makes development harder.
Here is the component in edit mode. The blendshape clips are not shown or accessible.
Here is the same component in play mode.
The problem is you normally use edit mode to create animations as some settings are lost when you exit play mode and return to edit mode. So the properties are not available for animation when you need them.
To overcome this problem, I plan to create my own component with a hard coded list of properties (the strength of each expression I will support). Hard coding the list into the component source code is not great, but it makes expressions easier to animate using an animation clips.
The available blendshapes
I want to create simple animated cartoons, so expressions are important to me. I want lots of different ones. I also don’t want to only rely on live recordings using a face tracking camera where I have to create expressions on my own face. It may be fine for live YouTube stream recordings where people expect a bit of choppiness, but I find not ideal for a recorded cartoon. (Call me fussy!) So I am reading up on how to create different facial expressions (I am not an artist!!). Articles I have been reading talk about how to raise tilt eyebrows, adjust eye lids, and so forth for different expressions. (They also suggest a really powerful and radical approach to understanding expressions involving a mirror.)
To make things a bit faster, I decided to create a rough index of the blendshapes so I know which ones may be appropriate in different circumstances. The following the list of available blendshapes (with abbreviated names) and a brief mention of how they move the face.
EYEBROWS
| Blendshape | Outer | Middle | Inner | Horizontal | Notes |
| browDownL/R | Down | Down | Down | ||
| Face.BRW_Sorry | Down a bit | Down | Down a bit | Sorry, sad | |
| noseSneerL/R | Down | Inwards | Concern, thoughtful | ||
| Face.BRW_Angry, alansBrowInnerDownL/R | Down | ||||
| browOuterUpL/R | Up | Down | |||
| alansBrowOuterUpL/R | Up | ||||
| Face.BRW_Fun, Face.BRW_Joy, Face.BRW_Surprise, browInnerUp | Up a bit | Up | Quizzical |
EYELIDS
| Blendshape | Upper outer | Upper middle | Upper Inner | Lower | Notes |
| Face.EYE_Close[_L/R] & eyeBlinkL/R | Down | Down | Down | Up a tiny bit | 30% bored, 100% demure |
| Face.EYE_Joy(_L/R) | Down | Down | Down | Up a lot | Happy closed |
| eyeSquintLeft/R | Down | Down | Down | Up | Squint, wary, suspicious |
| eyelidUpperSorrowL/R | Down | Sorry | |||
| Face.EYE_Sorrow | Down | Up | Sorrow | ||
| Face.EYE_Angry, eyelidUpperAngryL/R | Down | Angry | |||
| eyeWideLeft/R | Up | Up | Up | Surprised | |
| Face.EYE_Surprise | Up | Up | Up | Iris shrinks too | |
| eyelidLowerSquintLeft/R | Middle up | ||||
| Face.EYE_Extra | > < eyes |
IRIS
| Blendshape | Notes |
| irisMoveBack | the iris moves backwards into the skull! |
| irisReduceLeft | Iris gets smaller |
| eyeHighlightReduceL/R | Iris highlight gets smaller |
MOUTH
| Blendshape | Outer | Inner | Width | Notes |
| alansMouthLowerL/R | Down | |||
| Face.MTH_Down | Down | Down | Moves whole mouth down | |
| Face.MTH_Angry | Down | Narrower | ||
| mouthFrownL/R & mouthStretchL/R | Down a bit | left edge down, out a little | ||
| mouthLowerDownL/R | Down | Lower lip down (does not look good in middle unless both sides done the same) | ||
| Face.MTH_Neutral | lips close a little, outer curls up a little | |||
| Face.MTH_Up | Up | Up | Moves whole mouth up | |
| Face.MTH_Fun | Up | Wider | Joy | |
| cheekSquintL/R | Up | Wider | ||
| alansMouthUpperL/R | Up | Snarl outer edge moves, center does not | ||
| mouthSmileLeft | Up a bit | Wider | left edge up, out a little | |
| mouthUpperUpL/R | Up | lift upper lip (does not look good in middle unless both sides done the same) | ||
| Face.MTH_Joy | Open wide | Teethy move, opens with wide “D” shape | ||
| Face.MTH_Surprise | Open wide | Oh, upside down D shape | ||
| jawOpen | Open wide | Does not move jaw or teeth so can look strange without teethExtend | ||
| Face.MTH_Sorrow | Narrower | narrows, upside down D shape | ||
| mouthPucker | Narrow | |||
| mouthL/R | Wider | |||
| mouthDimpleL/R | Slightly wider | |||
| mouthFunnel | sticks lips out forwards | |||
| mouthShrugUpper/Lower | stick out upper/lower lip | |||
| mouthClose | rolls lips in a bit | |||
| mouthPressL/R | compress lip and raise slightly |
TEETH
| teethUpperUp/Down | upper rack of teeth go upwards/down |
| teethLowerUp/Down | Lower rack of teeth up / down |
| teethExtend | teeth get taller (meeting in middle at same point) |
TONGUE
| tongueOut | |
| tongueUp | tip of tongue curls upwards |
| tongueDown | tip of tongue curls downwards |
| tongueLeft/Right | extends to left and right |
CHEEKS
| cheekPuff | As stated |
JAW
| alansJawOpen | Bottom 3rd of face extends downward in jaw open |
Expressions
If you want an introductory video on how to draw expressions, I came across this one. Earlier videos on their channel seemed to talk about how to draw. Later ones seemed to have drift more into entertainment than learning.
Searching on the web also brings up various cheat sheets of expressions, for example https://www.animeoutline.com/12-anime-facial-expressions-chart-drawing-tutorial/ includes a chart then goes through how to draw the different expressions in some detail.
NEUTRAL
First, for reference, here is the default neutral expression.
ANGRY
The above post describes “angry” as:
- Eyebrows down and together
- Eyes squint
- Mouth upside down arch
In this case there is an “angry” blendshape built in.
Using the above table brings up another option. “noseSnearLeft” and “noseSnearRight” indicates the brows are moved together. Combine that with “eyeSquintLeft” and “eyeSquintRight” at 33% and “mouthFrownLeft” with “mouthFrownRight” and you end up with the following.
| noseSnearLeft | 100% |
| noseSnearRight | 100% |
| eyeSquintLeft | 33% |
| eyeSquintRight | 33% |
| mouthFrownLeft | 100% |
| mouthFrownRight | 100% |
Or you could try an non-symmetrical expression
| Face.M_F00_000_00_Fcl_EYE_Angry | 100% |
| eyeSquintLeft | 20% |
| noseSnearLeft | 100% |
| noseSnearRight | 80% |
| mouthFrownLeft | 100% |
| mouthFrownRight | 100% |
Or even
| Face.M_F00_000_00_Fcl_EYE_Angry | 100% |
| Face.M_F00_000_00_Fcl_BRW_Angry | 20% |
| Face.M_F00_000_00_Fcl_MTH_Angry | 50% |
| noseSnearLeft | 100% |
| noseSnearRight | 80% |
| mouthFrownLeft | 33% |
| mouthFrownRight | 33% |
| alansMouthUpperUpLeft | 100% |
| alansMouthUpperUpRight | 100% |
| alansMouthLowerDownLeft | 100% |
| alansMouthLowerDownRight | 100% |
| jawOpen | 33% |
CONTENT / HAPPY
The best eyes for happy are JOY. There are several ways to do the smiling mouth. Using JOY for the mouth is wide open. Fun is a more traditional smile. Note the eyes might need fine tuning depending on the eye shape as I often find the eye lashes merge into the face of the skin.
| Face.M_F00_000_00_Fcl_EYE_Joy | 100% |
| Face.M_F00_000_00_Fcl_MTH_Fun | 75% |
However I think the mouthSmile and mouthDimple creates a nicer curved smile.
| Face.M_F00_000_00_Fcl_EYE_Joy | 100% |
| mouthSmileLeft | 100% |
| mouthSmileRight | 100% |
| mouthDimpleLeft | 100% |
| mouthDimpleRight | 100% |
More of an open mouth grin can be achieved by combining an open mouth with teeth extension (unless you want to see the gums… which I think looks eerie/strange.
| Face.M_F00_000_00_Fcl_EYE_Joy | 100% |
| mouthSmileLeft | 100% |
| mouthSmileRight | 100% |
| mouthDimpleLeft | 100% |
| mouthDimpleRight | 100% |
| mouthUpperUpLeft | 100% |
| mouthUpperUpRight | 100% |
| mouthLowerDownLeft | 100% |
| mouthLowerDownRight | 100% |
| teethExtend | 50% |
You can add mouthLeft/Right of 33% to widen the mouth a bit more, but the teeth do not move so if increased too much it does not look as good. You can also add some jawOpen 20% to make the mouth a bit wider (going too far looks ugly, and you may need to increase teethExtend beyond 50%).
Note: you can use the similes and grins with open eyes as well, joy at 50% closes the eyes somewhat top and bottom (the bottom rising simulates your cheek pushing upwards at the bottom of the eyes).
GLOATING
Take a grin with normal eyes and curve the eyebrows in a U shape. There are two ways to do this with slightly different shapes: using BRW_Sorrow or using browInnerUp + browOuterUpLeft/Right. I show a bigger grin here to compare to the happy/content look above (but think its overkill), with a mix of the two brow approaches. I am not sure this looks like gloating to me, so I added a bit of EYE_Close which reduces the energy of the expression (a little bored/superior).
| Face.M_F00_000_00_Fcl_BRW_Fun | 10% |
| Face.M_F00_000_00_Fcl_BRW_Sorrow | 75% |
| browInnerUp | 50% |
| browOuterLeftUp | 100% |
| browOuterRightUp | 100% |
| Face.M_F00_000_00_Fcl_EYE_Close | 20% |
| jawOpen | 33% |
| mouthSmileLeft | 100% |
| mouthSmileRight | 100% |
| mouthDimpleLeft | 100% |
| mouthDimpleRight | 100% |
| mouthUpperUpLeft | 100% |
| mouthUpperUpRight | 100% |
| mouthLowerDownLeft | 100% |
| mouthLowerDownRight | 100% |
| teethExtend | 100% |
| mouthLeft | 20% |
| mouthRight | 20% |
LOVE
The recommendation is to draw the pupils larger than normal. There is an irisShrink blendshape, but no enlarge, so this would currently require different artwork for the eyes. This is not necessarily a problem however as it also recommends to put more highlights (stary look( in the eyes at the same time.
SURPRISED / CONFUSED
There is the default surprised expression built in which you could use at 50% strength for surprised and 100% for shocked. If you want to follow the article however it uses an “oh” shape for the mouth. This uses a few blendshapes to reduce mouth size appropriately. There are also separate eyeWideLeft/Right and irisReduceLeft/Right settings if you want to have finer control over how much the iris shrinks. For example, I prefer the iris to shrink only a little, but the brows move near full strength (leaving a bit more for a shocked expression).
| Face.M_F00_000_00_Fcl_BRW_Surprised | 100% |
| Face.M_F00_000_00_Fcl_MTH_Angry | 33% |
| Face.M_F00_000_00_Fcl_MTH_O | 20% |
| mouthPucker | 100% |
| eyeWideLeft | 100% |
| eyeWideRight | 100% |
Confused might adjust one of the eyebrows so they are not symmetrical. I created my own alansBrowOuterUpLeft (50%) which you might combine with browOuterDownRight (50%).
UPSET / SAD
The recommended look for upset is not quite the same as Face.M_F00_000_00_Fcl_ALL_Sorrow (which I might use at 50%). The eyes looking downwards cannot be done here (would be done by eye tracking). To get close to this article, the brows might need to bend more than the default Sorrow blendshape (it depends on your artwork). For example,
| Face.M_F00_000_00_Fcl_ALL_Sorrow | 50% |
| browInnerUp | 50% |
| browOuterUpLeft | 100% |
| browOuterUpRight | 100% |
GRINNING / PLOTTING / SNEAKY
A sneaky/plotting look includes a squint and eyebrows lowered in the middle. The nose snear I current have modifies the eye brows, which I am not sure is good to rely on (might not port to other blendshape definitions as well).
| eyeSquintLeft | 25% |
| eyeSquintRight | 25% |
| cheekSquintLeft | 100% |
| cheekSquintRight | 100% |
| noseSnearLeft | 100% |
| noseSnearRight | 100% |
SCARED
I would use surprised at 100%, but to show a more extreme variation you can use the individual controls in combination with a half strength surprised expression. This stacking makes the expression even more extreme. If you need scared vs terrified, different strengths could be used. You could also make the face skin paler.
| Face.M_F00_000_00_Fcl_ALL_Surprised | 50% |
| Face.M_F00_000_00_Fcl_BRW_Surprised | 100% |
| eyeWideLeft | 100% |
| eyeWideRight | 100% |
| irisReduceLeft | 75% |
| irisReduceRight | 75% |
| mouthFrownLeft | 100% |
| mouthFrownRight | 100% |
EMBARRASSED
Embarrassed in the article is like upset/sad above, but with a smile instead. I would consider adding a blush to the cheeks as well. Looking down as well (which is not a part of the blendshape).
PUZZLED
An “oh” mouth shape, like above, with eyebrows raised, and with eyes looking upwards a bit.
SMILE
Described above, I don’t think need to repeat it.
Conclusions
I have built some proof of concept scripts and made sure everything works. My next step is to go back and rework my VRoid Studio models (AGAIN!) to only use blendshape clips. I then need to decide on the final list of expressions to support as that will be hard coded into the script. The above is an example starting point, but there are certainly more that can be done. I am concerned that adding more expressions later may cause problems with previously serialized objects in scenes.
Extra Ordinary, the Series 