Example Expressions from Blendshapes with VRoid Studio, HANA_Tool, and Unity

This post includes an update on my plans to do expressions with VRoid characters for the purpose of creating short animated video clips using Unity and VRoid Studio characters.

Note: The main value of this post may be the summary list of blendshapes and how they move parts of the face below. The tables can be used to more quickly find the right blendshapes to use to create a particular expression. I plan to have a later post that lists the expressions I settled on and the blendshapes (and strengths) I ended up with.

What are blendshapes vs blendshape clips?

VRM files from VRoid Studio support a set of standard expressions (Angry, Blink, Fun, Joy, Sorrow, Surprised, A, E, I, O, U and more), plus allow additional custom expressions to be defined. Many apps offer buttons or similar to trigger the expressions. These expressions are captured in “blendshape clips”.

Blendshape clips are defined using a set of strength settings for blendshapes (also known as Shape Keys in Blender). Blendshapes define how to warp the mesh of the face. For example, you can define an expression such as anger using a blendshape clip to narrow the eyelids (the first blendshape), bring the eyebrows down in the middle (another blendshape), and raise the lips in the corner of the mouth in a snarl (a third blendshape). The blendshape clip keeps the strength settings for the three blendshapes, as needed.

Some blendshapes defined by VRoid Studio control one part of the face (BRW = eyebrow, EYE = eye, MTH = mouth, HA = fangs), others control many parts at once (ALL). The former is useful if the latter does not provide fine enough control.

There are an additional set of blendshapes defined by ARKit which some apps can use. The following is a subset of the additional blendshapes available. I am using HANA_Tool to provide the blendshapes for these standard names.

You can also define your own blendshapes (which I have done a little). These can be created using Blender (with some learning effort), or in my case I created some simple blendshapes \using C# code.

VRoid blendshape clips are defined in files starting with “BlendShape” (just to make things a bit confusing).

The “BlendShape” file above references the different blendshape clips in the directory. The standard expressions include:

Unity Animation Clips

I have previously said that I plan to use animation clips in Unity by recording blendshape strengths directly, rather than use blendshape clips, because it also allows me to fade skin textures using a custom shader. I am however planning to move across to blendshape clips as there are more and more VRM tools becoming available, and if I use blendshape clips then I can use those tools with my avatars.

I still plan to use animation clips inside Unity, but each clip will record a single property, the strength of the blendshape clip. This is because when you drag an animation clip into a Unity Cinemachine Timeline, you get to see the animation clip name. If you do direct recordings of properties, it’s a bit harder to see what the expression is at different time points. (The following shows examples three tracks using animation clips and two tracks recording properties directly.)

Note: if you want more technical details on how to create characters using advanced features, a great YouTube channel I came across is by Suvdriel. I am going to be using many of the techniques she describes, to avoid using a custom shader (my current approach) to do expressions such as blushing. There are the VRoid tricks like using hair layers with special hair materials, but she also describes how to tile multiple alternate images and do some color layer adjustments. (I discovered her content the day after I worked out some of these myself, but she describes it all very well on her channel so I am not going to repeat it here.)

How to animate expressions in Unity

VRoid characters imported using the UniVRM package for Unity have a component that provides access to the set of available blendshape clips. The only problem is it only exposes the clips during “play” time, not when editing. This makes development harder.

Here is the component in edit mode. The blendshape clips are not shown or accessible.

Here is the same component in play mode.

The problem is you normally use edit mode to create animations as some settings are lost when you exit play mode and return to edit mode. So the properties are not available for animation when you need them.

To overcome this problem, I plan to create my own component with a hard coded list of properties (the strength of each expression I will support). Hard coding the list into the component source code is not great, but it  makes expressions easier to animate using an animation clips.

The available blendshapes

I want to create simple animated cartoons, so expressions are important to me. I want lots of different ones. I also don’t want to only rely on live recordings using a face tracking camera where I have to create expressions on my own face. It may be fine for live YouTube stream recordings where people expect a bit of choppiness, but I find not ideal for a recorded cartoon. (Call me fussy!) So I am reading up on how to create different facial expressions (I am not an artist!!). Articles I have been reading talk about how to raise tilt eyebrows, adjust eye lids, and so forth for different expressions. (They also suggest a really powerful and radical approach to understanding expressions involving a mirror.)

To make things a bit faster, I decided to create a rough index of the blendshapes so I know which ones may be appropriate in different circumstances. The following the list of available blendshapes (with abbreviated names) and a brief mention of how they move the face.

EYEBROWS

BlendshapeOuterMiddleInnerHorizontalNotes
browDownL/RDownDownDown  
Face.BRW_SorryDown a bitDownDown a bit Sorry, sad
noseSneerL/R  DownInwardsConcern, thoughtful
Face.BRW_Angry, alansBrowInnerDownL/R  Down  
browOuterUpL/RUp Down  
alansBrowOuterUpL/RUp    
Face.BRW_Fun, Face.BRW_Joy, Face.BRW_Surprise, browInnerUpUp a bit Up Quizzical

EYELIDS

BlendshapeUpper outerUpper middleUpper InnerLowerNotes
Face.EYE_Close[_L/R] & eyeBlinkL/RDownDownDownUp a tiny bit30% bored, 100% demure
Face.EYE_Joy(_L/R)DownDownDownUp a lotHappy closed
eyeSquintLeft/RDownDownDownUpSquint, wary, suspicious
eyelidUpperSorrowL/R Down  Sorry
Face.EYE_Sorrow Down UpSorrow
Face.EYE_Angry, eyelidUpperAngryL/R  Down Angry
eyeWideLeft/RUpUpUp Surprised
Face.EYE_SurpriseUpUpUp Iris shrinks too
eyelidLowerSquintLeft/R   Middle up 
Face.EYE_Extra    > < eyes

IRIS

BlendshapeNotes
irisMoveBackthe iris moves backwards into the skull!
irisReduceLeftIris gets smaller
eyeHighlightReduceL/RIris highlight gets smaller

MOUTH

BlendshapeOuterInnerWidthNotes
alansMouthLowerL/RDown   
Face.MTH_DownDownDown Moves whole mouth down
Face.MTH_AngryDown Narrower 
mouthFrownL/R & mouthStretchL/RDown a bit  left edge down, out a little
mouthLowerDownL/R Down Lower lip down (does not look good in middle unless both sides done the same)
Face.MTH_Neutral   lips close a little, outer curls up a little
Face.MTH_UpUpUp Moves whole mouth up
Face.MTH_FunUp WiderJoy
cheekSquintL/RUp Wider 
alansMouthUpperL/R Up  Snarl outer edge moves, center does not
mouthSmileLeftUp a bit Widerleft edge up, out a little
mouthUpperUpL/R Up lift upper lip (does not look good in middle unless both sides done the same)
Face.MTH_Joy Open wide Teethy move, opens with wide “D” shape
Face.MTH_Surprise Open wide Oh, upside down D shape
jawOpen Open wide Does not move jaw or teeth so can look strange without teethExtend
Face.MTH_Sorrow  Narrowernarrows, upside down D shape
mouthPucker  Narrow 
mouthL/R  Wider 
mouthDimpleL/R  Slightly wider 
mouthFunnel   sticks lips out forwards
mouthShrugUpper/Lower   stick out upper/lower lip
mouthClose   rolls lips in a bit
mouthPressL/R   compress lip and raise slightly

TEETH

teethUpperUp/Downupper rack of teeth go upwards/down
teethLowerUp/DownLower rack of teeth up / down
teethExtendteeth get taller (meeting in middle at same point)

TONGUE

tongueOut 
tongueUptip of tongue curls upwards
tongueDowntip of tongue curls downwards
tongueLeft/Right extends to left and right

CHEEKS

cheekPuffAs stated

JAW

alansJawOpenBottom 3rd of face extends downward in jaw open

Expressions

If you want an introductory video on how to draw expressions, I came across this one. Earlier videos on their channel seemed to talk about how to draw. Later ones seemed to have drift more into entertainment than learning.

Searching on the web also brings up various cheat sheets of expressions, for example https://www.animeoutline.com/12-anime-facial-expressions-chart-drawing-tutorial/ includes a chart then goes through how to draw the different expressions in some detail.

NEUTRAL

First, for reference, here is the default neutral expression.

ANGRY

The above post describes “angry” as:

  • Eyebrows down and together
  • Eyes squint
  • Mouth upside down arch

In this case there is an “angry” blendshape built in.

Using the above table brings up another option. “noseSnearLeft” and “noseSnearRight” indicates the brows are moved together. Combine that with “eyeSquintLeft” and “eyeSquintRight” at 33% and “mouthFrownLeft” with “mouthFrownRight” and you end up with the following.

noseSnearLeft100%
noseSnearRight100%
eyeSquintLeft33%
eyeSquintRight33%
mouthFrownLeft100%
mouthFrownRight100%

Or you could try an non-symmetrical expression

Face.M_F00_000_00_Fcl_EYE_Angry100%
eyeSquintLeft20%
noseSnearLeft100%
noseSnearRight80%
mouthFrownLeft100%
mouthFrownRight100%

Or even

Face.M_F00_000_00_Fcl_EYE_Angry100%
Face.M_F00_000_00_Fcl_BRW_Angry20%
Face.M_F00_000_00_Fcl_MTH_Angry50%
noseSnearLeft100%
noseSnearRight80%
mouthFrownLeft33%
mouthFrownRight33%
alansMouthUpperUpLeft100%
alansMouthUpperUpRight100%
alansMouthLowerDownLeft100%
alansMouthLowerDownRight100%
jawOpen33%

CONTENT / HAPPY

The best eyes for happy are JOY. There are several ways to do the smiling mouth. Using JOY for the mouth is wide open. Fun is a more traditional smile. Note the eyes might need fine tuning depending on the eye shape as I often find the eye lashes merge into the face of the skin.

Face.M_F00_000_00_Fcl_EYE_Joy100%
Face.M_F00_000_00_Fcl_MTH_Fun75%

However I think the mouthSmile and mouthDimple creates a nicer curved smile.

Face.M_F00_000_00_Fcl_EYE_Joy100%
mouthSmileLeft100%
mouthSmileRight100%
mouthDimpleLeft100%
mouthDimpleRight100%

More of an open mouth grin can be achieved by combining an open mouth with teeth extension (unless you want to see the gums… which I think looks eerie/strange.

Face.M_F00_000_00_Fcl_EYE_Joy100%
mouthSmileLeft100%
mouthSmileRight100%
mouthDimpleLeft100%
mouthDimpleRight100%
mouthUpperUpLeft100%
mouthUpperUpRight100%
mouthLowerDownLeft100%
mouthLowerDownRight100%
teethExtend50%

You can add mouthLeft/Right of 33% to widen the mouth a bit more, but the teeth do not move so if increased too much it does not look as good. You can also add some jawOpen 20% to make the mouth a bit wider (going too far looks ugly, and you may need to increase teethExtend beyond 50%).

Note: you can use the similes and grins with open eyes as well, joy at 50% closes the eyes somewhat top and bottom (the bottom rising simulates your cheek pushing upwards at the bottom of the eyes).

GLOATING

Take a grin with normal eyes and curve the eyebrows in a U shape. There are two ways to do this with slightly different shapes: using BRW_Sorrow or using browInnerUp + browOuterUpLeft/Right. I show a bigger grin here to compare to the happy/content look above (but think its overkill), with a mix of the two brow approaches. I am not sure this looks like gloating to me, so I added a bit of EYE_Close which reduces the energy of the expression (a little bored/superior).

Face.M_F00_000_00_Fcl_BRW_Fun10%
Face.M_F00_000_00_Fcl_BRW_Sorrow75%
browInnerUp50%
browOuterLeftUp100%
browOuterRightUp100%
Face.M_F00_000_00_Fcl_EYE_Close20%
jawOpen33%
mouthSmileLeft100%
mouthSmileRight100%
mouthDimpleLeft100%
mouthDimpleRight100%
mouthUpperUpLeft100%
mouthUpperUpRight100%
mouthLowerDownLeft100%
mouthLowerDownRight100%
teethExtend100%
mouthLeft20%
mouthRight20%

LOVE

The recommendation is to draw the pupils larger than normal. There is an irisShrink blendshape, but no enlarge, so this would currently require different artwork for the eyes. This is not necessarily a problem however as it also recommends to put more highlights (stary look( in the eyes at the same time.

SURPRISED / CONFUSED

There is the default surprised expression built in which you could use at 50% strength for surprised and 100% for shocked. If you want to follow the article however it uses an “oh” shape for the mouth. This uses a few blendshapes to reduce mouth size appropriately. There are also separate eyeWideLeft/Right and irisReduceLeft/Right settings if you want to have finer control over how much the iris shrinks. For example, I prefer the iris to shrink only a little, but the brows move near full strength (leaving a bit more for a shocked expression).

Face.M_F00_000_00_Fcl_BRW_Surprised100%
Face.M_F00_000_00_Fcl_MTH_Angry33%
Face.M_F00_000_00_Fcl_MTH_O20%
mouthPucker100%
eyeWideLeft100%
eyeWideRight100%

Confused might adjust one of the eyebrows so they are not symmetrical. I created my own alansBrowOuterUpLeft (50%) which you might combine with browOuterDownRight (50%).

UPSET / SAD

The recommended look for upset is not quite the same as Face.M_F00_000_00_Fcl_ALL_Sorrow (which I might use at 50%). The eyes looking downwards cannot be done here (would be done by eye tracking). To get close to this article, the brows might need to bend more than the default Sorrow blendshape (it depends on your artwork). For example,

Face.M_F00_000_00_Fcl_ALL_Sorrow50%
browInnerUp50%
browOuterUpLeft100%
browOuterUpRight100%

GRINNING / PLOTTING / SNEAKY

A sneaky/plotting look includes a squint and eyebrows lowered in the middle. The nose snear I current have modifies the eye brows, which I am not sure is good to rely on (might not port to other blendshape definitions as well).

eyeSquintLeft25%
eyeSquintRight25%
cheekSquintLeft100%
cheekSquintRight100%
noseSnearLeft100%
noseSnearRight100%

SCARED

I would use surprised at 100%, but to show a more extreme variation you can use the individual controls in combination with a half strength surprised expression. This stacking makes the expression even more extreme. If you need scared vs terrified, different strengths could be used. You could also make the face skin paler.

Face.M_F00_000_00_Fcl_ALL_Surprised50%
Face.M_F00_000_00_Fcl_BRW_Surprised100%
eyeWideLeft100%
eyeWideRight100%
irisReduceLeft75%
irisReduceRight75%
mouthFrownLeft100%
mouthFrownRight100%

EMBARRASSED

Embarrassed in the article is like upset/sad above, but with a smile instead. I would consider adding a blush to the cheeks as well. Looking down as well (which is not a part of the blendshape).

PUZZLED

An “oh” mouth shape, like above, with eyebrows raised, and with eyes looking upwards a bit.

SMILE

Described above, I don’t think need to repeat it.

Conclusions

I have built some proof of concept scripts and made sure everything works. My next step is to go back and rework my VRoid Studio models (AGAIN!) to only use blendshape clips. I then need to decide on the final list of expressions to support as that will be hard coded into the script. The above is an example starting point, but there are certainly more that can be done. I am concerned that adding more expressions later may cause problems with previously serialized objects in scenes.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s