Extra Ordinary, the Series

I have been exploring iFacialMocap, an iOS app that uses the iPhone “ARKit” library to convert facial expressions into a series of movements. These can be sent to Unity so your character inside Unity follows what is being recorded on your iPhone.

A challenge however is the iOS ARKit library uses more blendshapes than come with a VRoid Studio character. This blog lists the ARKit blendshapes and possible mappings to VRoid character blendshapes. My goal is to see how far I can go without having to use a tool such as Blender to manually create blend shapes.

Creating Blend Shapes in Unity

A blend shape consists of a series of delta vectors from the “basis”, the standard neutral pose of a character. For every vertex on the mesh of the face, it records a delta to apply. For example, to make a smile, all the vertices around the mouth may change a little, with the edges of the mouth moving upwards more. The rest of the vertices on the face would not move (the delta would be zero). Blend shapes then have a strength applied, which is multiplied to all of the deltas, adjusting the strength of the expression (50% creates a half smile).

Unity has sufficient power to create new blend shapes programmatically. I have found it fairly easy to copy and adjust blend shapes making simple transformations in the past.

Below I present a table describing how to create most of the desired blendshapes from existing blend shapes. In other experiments I have tried to generate blend shapes purely from code, but I found it difficult to get right because the position and size of a character’s eyes and mouths can be changed in VRoid Studio. I found it too difficult to get it right. By building upon the blendshapes provided in the VRM file by VRoid Studio this is simplified, but not all ARKit facial expressions can be done.

The primitive operations needed for the changes in the table below are as follows:

• Copy existing blend shape verbatim.
• Copy existing blend shape, multiplying by strength factor (e.g. 25% of original blendshape). This involves scaling all of the vectors in the blend shape by a constant.
• Take left or right half of original blendshape. This involves finding the horizontal mid-point (X = 0) and then replacing deltas to the left or right of that point with zero vectors. This allows a blend shape moving both eyebrows to be split into two blend shapes to move the two eyebrows individually.
• Take existing blendshape, but negate the Y deltas, so all upward movements become downward movements. This is implemented by multiplying Y offsets by a negative constant, such as -1.0.
• Take blendshape, but scale from 0% on left to the specified % on right so a proportional amount is kept. This can be useful for some mouth expressions where ARKit wants to control half the mouth at a time, but if you do a hard cutoff in the middle of the mouth, it does not line up smoothly.
• For the “O” blend shape, keep only the upward movements (upper lip) or downward movements (lower lip) with a proportion of the left or right side. The goal is to convert the “O” open mouth position into mouthUpperUpLeft, mouthUpperUpRight, mouthLowerDownLeft, and mouthLowerDownRight blend shapes below.
• Add a proportional Y offset to everything in bottom quarter of the face to do jaw open movement. This would be added to the “O” blendshape so the lower jaw drops.
• An important case is the VRoid Studio “Surprised” blend shape combines both the upper eye lid moving upwards and the iris shrinking. This is separated into two blend shapes using the material submesh for the iris to identify which vertex deltas are shrinking the iris. See the next section for more details.

Iris Shrinking Blendshape

In one case we want to raise the upper eyelid, like surprised, but without shrinking the iris. The iris is on a separate submesh in front the eyeball submesh. (A submesh is part of a mesh that shares a material. The “Face” in VRoid uses 9 materials to create the face, including one for the eye iris.) The iris is shrunk in the VRoid “Surprised” expression. We want to separate the iris shrinkage from the other submeshes so we can do these two changes separately.

To identify the iris submesh, one approach is to look at the array of materials in the “Skinned Mesh Renderer” component. For VRoid characters, the material for the iris is called “F00_000_00_EyeIris_00_EYE”. The index in this materials array is the submesh index.

The following shows the iris mesh shrinking in Blender (it is a bit hard to see, but the upper eyelid is also moving upwards.

The default eye and iris:

The shrunk iris (and widened eyelid):

With materials included, the default eye:

The shrunk iris (and widened eyelid):

The algorithm used to separate the iris from the eyelid movement is to copy the existing “surprised” blend shape, but to replace all delta’s on the iris submesh with zero vectors so the iris does not shrink (and vice versa to keep the shrinking iris as a separate blend shape so it can be added later by hand).

The Final Transform Function

Most of the transformations can be generalized into a single function (see AddBlendShape() in AddIFacialMocapBendShapes.cs) with a set of parameters.

• Vector magnitude scale factor (%).
• Y value scale factor (%), which can be negative.
• Keep left/right/both of blend shape (zeroing values not kept).
• Keep upward/downward/both of blend shape (zeroing values not kept).
• Adjust scale factor left-to-right or right-to-left from zero to specified scale factors.
• Adjust scale factor based on distance from X=0 so smooth transition from middle.
• Include/exclude specified submesh.

For now I have hard code the blend shapes to create, but it would be possible to create an array of instructions that users can modify for each character to make individual character adjustments.

Note that two additional functions were created to support the “jawOpen” blend shape and the outer eyebrow movements (which VRoid Studio does not supply). But the above function was used for all the rest of the blend shapes to be created.

ARKit Blend Shapes

Here are the ARKit / iFacialMocap blend shapes with how VRoid Studio blend shapes were used to generate them from. The blend shapes are shown with a sample face provided with iFacialMocap. A character created in VRoid Studio is also shown where a different blend shape was created. (For many expressions I use the same blend shape as an approximation of the desired final result, to save effort.)

Note that when there are separate left and right poses (e.g. the left and right eyebrow), only one is shown as they are perfect mirror images for the left and right side of the face.

Also note that the VRoid character expression often has less detail as the face, especially around the nose, has less detail than the sample character.

The following images of a VRoid character were created applying the script to a VRoid character directly imported into Unity. The script is by no means perfect, it can take between 5 to 10 minutes to run to generate the missing blend shapes, but you can judge the results yourself. My main objective was to avoid an additional step of exporting to Blender to add the missing blend shapes.

The Basis Pose

The default pose, a reference point for all the following blend shapes.

browDownLeft & browDownRight

One eyebrow lowers at a time, moving straight down (no twist).

Currently I use Face.M_F00_000_00_Fcl_BRW_Angry, split into left and right sides. This is not exactly the same as the existing blend shape does not lower eyebrow equally (it is on an angle).

browInnerUp

Tilt the eyebrows so the inner part moves upwards (the outer edge stays still).

The closest is Face.M_F00_000_00_Fcl_BRW_Surprised (Fun and Joy are also possibilities).

browOuterUpLeft & browOuterUpRight

Cannot use Face.M_F00_000_00_Fcl_BRW_Angry in this case, so instead computed a Y offset for each vertex in the eyebrow mesh algorithmically, where vertices near the center of the face move upwards a short distance, and vertices near the outer edge of the face move upwards more. See the AddOuterUpBlendShape() function in AddIFacialMocapBendShapes.cs.

cheekPuff

Expand both cheeks on the face.

I have given up on this expression for now as much of the face (including around the mouth) changes and there is no similar existing blend shape.

cheekSquintLeft & cheekSquintRight

The lower lid raises and the upper lid lowers a bit. Blinks in comparison do not raise the lower lid. Cheek moves a bit too.

Face.M_F00_000_00_Fcl_EYE_Joy at 25% strength is close. EYE_Joy at 25% is an alternative, but the top eyelid moves more than lower eyelid. Too hard to do the cheek movement for now.

eyeBlinkLeft & eyeBlinkRight

Top eyelid closes to close eye.

Face.M_F00_000_00_Fcl_EYE_Close is the appropriate existing blend shape, but for some characters I find 80% is better than 100% which lowers the lid too far. I may need to change the script to allow different characters specify different values for how far to close the eyelid. Almost but not quite closing the eye does not look good, but closing the eye too far also does not look good.

eyeLookDownLeft & eyeLookDownRight

This and following blend shapes is not for moving the iris to look in a particular direction, but rather to adjust the eyelids. I have not implemented any of these movements at this stage.

The upper and lower eyelid both move downwards, with eye closing somewhat (like a squint).

eyeLookInLeft & eyeLookInRight

The left eye narrows, shrinking towards the middle of the face. The inner edge of the eye does not move. The outer edge of the eye moves inwards slightly. This causes the top of the arc of the eyelid to move inwards as well.

eyeLookOutLeft & eyeLookOutRight

The inner edge of the eye does not move, the outer edge of the eye moves outward further. This causes the mid-point of the eye to move outwards a bit.

eyeLookUpLeft & eyeLookUpRight

Upper eyelid moves upwards, causing a “surprised” like expression, but without the iris shrinking.

eyeSquintLeft & eyeSquintRight

The lower eyelid moves upwards.

Face.M_F00_000_00_Fcl_EYE_Joy at 25% strength is close. (Same as cheekSquintLeft and cheekSquintRight above.)

eyeWideLeft & eyeWideRight

The upper lid goes up. The lower lid goes down a touch.

Face.M_F00_000_00_Fcl_EYE_Surprised is best option, but iris shrinks as well which is undesirable. So introduced an argument to discard the vertex deltas on the iris mesh (as described above).

jawForward

Juts the lower jaw forwards. Lower part of face looks more flush (flat).

Gave up on this one for now. Set them to zero deltas to avoid warning messages about missing blend shapes when using the iFacialMocap provided script for Unity.

jawLeft & jawRight

The jaw moves to the left with appropriate mouth adjustments.

Gave up on this for now.

jawOpen

Big mouth shape, jaw drops to open up the mouth.

Nothing moves the Jaw, so wrote a special function for this one based on Face.M_F00_000_00_Fcl_MTH_O. For the bottom section of the face, all vertices have an addition Y delta applied, relative to how far down the face the vertex is.

mouthClose

Bottom lip moves upward to overlap upper lip, all the way up to the nose!

Might be able to take mouth position, drop upper half where things move upwards, invert downward movement of bottom half, then also push out forwards slightly. But gave up as too hard for now.

mouthDimpleLeft & mouthDimpleRight

The outer edge of the mouth pulls to the side and upwards.

Take Face.M_F00_000_00_Fcl_MTH_Fun and split in half for left and right. Could reduce strength a bit. For now, not implemented.

mouthFrownLeft & mouthFrownRight

Left outer edge of mouth goes down (and the cheek around it a bit to stretch and stay in place).

Closest is Take Face.M_F00_000_00_Fcl_MTH_Fun and split in half for left and right, then invert Y for all movements.

mouthLeft & mouthRight

Whole mouth moves to the left, with the left edge moving upwards.

Take Face.M_F00_000_00_Fcl_MTH_Fun and split in half for left and right. Could increase strength a bit.

mouthPressLeft & mouthPressRight

The left edge of the mouth moves upwards a bit, with a slight stretch to the left. The other edge of the mouth retains its position.

Face.M_F00_000_00_Fcl_MTH_Fun is the closest, but it adds curvature as well. Maybe take Fun but proportionally adjust the Y weight so one side does not move and the other moves at 50% strength. For now I just approximate this, sharing the same shape as many of the other left/right smile/mouth-press/etc movements.

mouthSmileLeft & mouthSmileRight

Right edge of mouth does not move. Everything else proportionally moves up to the left, creating a slight dimple.

Take Face.M_F00_000_00_Fcl_MTH_Fun and split in half for left and right. Basically ignore the dimple and copy mouthLeft and mouthRight as close enough.

mouthStretchLeft & mouthStringRight

Left side stretches out and down, deforming the cheek so it ticks out a bit more.

Take Face.M_F00_000_00_Fcl_MTH_Fun, invert all Y movements, and split in half for left and right. Basically ignore the dimple and copy mouthFrownLeft and mouthFrownRight as close enough.

mouthUpperUpLeft & mouthUpperUpRight

Upper right lip opens mouth slightly. Bottom lip does not move.

Same technique as mouthLowerDownLeft & mouthLowerDownRight, but use upper movements instead of lower movements.

mouthLowerDownLeft & mouthLowerDownRight

Lower lip moves downward. No other face movement, but existing movement affects the jaw line a little bit.

Takes an open mouth movement, discards the upward movements (which would be the top lip), then proportionally adjust downward movement relative to center of mouth. Need to pick a movement that does not adjust mouth width much, like Face.M_F00_000_00_Fcl_MTH_O.

noseSneerLeft & noseSneerRight

Left eyebrow lowers a bit and left edge of nose is pulled upwards a bit.

Too hard, ignore. (The nose is not very clear in VRoid.)

mouthFunnel

Mouth narrows. Lips stick outward (forwards) a bit. Nose flattens inwards a bit.

Can use Face.M_F00_000_00_Fcl_MTH_Sorrow, or maybe 75% of Face.M_F00_000_00_Fcl_MTH_U.

mouthPucker

Mouth narrows, lips stick further forwards (puckering for kiss).

Maybe 50% of Face.M_F00_000_00_Fcl_MTH_U. Nothing really makes lips stick out.

mouthRollLower

Bottom lip moves up and curls inward (no movement to upper lip).

Not much lip movement in VRoid, so skip this one.

mouthRollUpper

Top lip moves downward, curling inward (so bottom lip does not move).

Not much lip movement in VRoid, so skip this one.

mouthShrugLower

Bottom lip and bottom jaw move upwards a bit, causing lower lip to go slightly above upper lip.

Could use 25% of Face.M_F00_000_00_Fcl_MTH_Fun but invert Y movements.

mouthShrugUpper

Upper lip lifts upward slight, opening mouth a little bit.

The end result

tongueOut

There is one additional blend shape that can be generated by iFacialMocap that is not from ARKit, which is to stick the tongue out. This is too hard to achieve for now.

Conclusion

Using the above you can generate some facial expressions and head movements. This can be useful, however I suspect in practice I will use the additional facial expressions supported directly from animation clips. The reason for this is I find these sorts of apps are a bit jittery. That is fine for say a live stream, but looks lower quality when in an animated video clip.