Extra Ordinary, the Series

My goal is to create characters in VRoid Studio and then use them in an animated video created in Unity. A part of this challenge however is I want to add more expressions to the character than the default.

• I tried to just use the existing VRoid BlendShapes – there were not enough for all the facial expressions I want
• I tried to use Blender to add my own BlendShape – it looked like hard work in a new tool, so I kept looking
• I tried to use the Blend Shape Builder package in Unity – it looks even harder work (but does work)

Then I found blend shapes someone had added to another VRoid character, so I thought maybe I could copy them across to my character! This blog are notes from my journey.

TL;DR: Copying sounds simple, but as soon as you put the character through import/export tools, things get more complex. The challenge is different tools can adjust meshes behind the scenes. I am coming to the opinion that learning something like Blender is the best solution. Then once you create a blend shape on your character, it is not hard to copy verbatim to another character as long as it went through the same tool chain (and so has had identical modifications made).

A Quick Recap on Terminology

The face of a character in VRoid Studio has a mesh of triangles. The bitmap artwork of the character’s face is mapped onto those triangles.

VRoid Studio uses 10 submeshes for a face – the main face, the ears, inside the mouth, eyebrows, eye lashes, the line around the eye, the irises, eye highlights, the eye whites, and an “extra” mesh for doing the strange anime “> <” eye expression. 9 materials (the main face and ears share the same material) allow you to define artwork for each of those submeshes separately.

A “blend shape” defines how to warp the mesh to make a different expression, such as raising eyebrows, narrowing the eyes, or opening the mouth wider. It does this by specifying an array of deltas to apply to each vertex of all submeshes. For example, to smile, the vertices for the mesh near the middle of the mouth may not move much, but the vertices near the corners of the mouth would move upwards more.

VRoid Studio generates a number of default blend shapes for each of the expressions listed with the character. Each blend shape strength can be between 0 and 100, allowing a half smile (set smile to 50 instead of 100). You can also combine different blend shapes for more expressions.

The problem is the default set of blend shapes included by VRoid Studio cannot express all the expressions I wanted (although it is pretty good). So how to get more blend shapes added?

iFacialMocap.com

I came across ifacialmocap.com in my travels, an iPhone app that generates a stream of Apple’s ARKit blend shape commands. ARKit drives the Apple emoji experience (the little animals that mimic your face expression). It also included links to some blend shape definitions that experts had created for adding support for ARKit blends to a VRoid character. A series of blog posts (in Japanese! Google Translate is your friend!) walks you through the steps to add the blend shapes to your character using a combination of Unity and Blender. Eureka!

That is, this app made two things possible. First, you can look at your phone and make an expression and a character copies it. The sequence of blend shape strengths is sent to Unity (over Wifi), or you can email a FBX file with an animation clip in it. I tried this with the demo face provided and it worked well. Second, I thought I should be able to copy the blend shape definitions from the default characters to my VRoid characters.

Someone reported they did get the set of instructions in the blog post to work, but it failed for me. I probably have the wrong versions of software installed. Specially, installing the Blender extension to copy blend shapes from one character to another did not work for me. So I went exploring for another solution. (The blog post may have been a better solution in practice – I may go back and try it again, but that is not the focus of this blog post.)

The blog posts had useful findings regardless. Part 2 of the series includes downloads of the different base characters in Blender format. I loaded these files into Blender, exported as FBX, then loaded the FBX files into Unity. That level of Blender I know how to do!  (Import File and Export File!) From there I planned to use a C# script in Unity to copy the blend shapes from the downloaded Face to my character.

As a side note, one thing I discovered is ARKit does not support angry or sad expressions! This means I will keep the default VRoid blends and add the new ARKit ones to it. That should allow me to merge VRoid angry with ARKit expressions. When use the ifacialmocap software, I will not be able to generate angry expressions from the camera, but I may use some keyboard triggers for ‘angry’ etc and merge them that way when recording a clip.

My conclusion is ifacialmocap might be a useful tool for live streaming, but for a quality animation I found that it (like pretty well every camera package I have tried) introduces a bit too much jitter as the software attempts to pick up eye movements, eye brow movements, etc correctly. (I will try some more as the latest version has some “smoothing” settings in it now. The Unity connection code also includes source code, so I might try thresholding movements in the code as well to eliminate small jitters and smooth them out.) I suspect in practice for final animations I plan to create (yeah, if I ever get there!) I won’t use this tool because I want finer grain control. I did find it useful for animating natural looking head tilts etc easily. We shall see.

Copying Blend Shapes Between Characters

CopyBlenShapes.cs was an early experiment on copying BlendShapes from one Unity “Face” mesh to another. The code runs when I hit play (it’s just a quick hack). But the end result is it did successfully copy blend shapes from one character to another.

One challenge with the above code however is it assumes the source and target face mesh have identical mesh structure. This is actually generally true, except I found some of the import/export tools between formats would make slight adjustments to the mesh. For example, one tool merged the ear and face submeshes (they share the same material and the two submeshes touch, so it makes sense). Another I think re-ordered the submeshes. None of these are wrong for such tools to do, but the above code assumes the array of mesh triangles are in an identical order.

A More Advanced Blendshape Copy

As soon as an import/export utility made any change to the mesh, the above code can end with spectacularly bad results. For example if the triangles in the mesh are reordered, copying a blend shape across will move the wrong triangles on the face. This can result in some very “interesting” visual results.

So I got ambitious. Instead of copying the blendshape definition blindly, I decided to write a script that would run through each vertex in my VRoid character and find the closest vertex in the template with the blendshapes I wanted. This means that multiple vertices in the final VRoid character can share the same blend vertex.

The next problem however was that VRoid allows you to change the character, such as moving the eyes up/down, closer together, and so on. So the source and target faces had different coordinates in the meshes. As a result, I introduced code to try and adjust a coordinate on one character with the corresponding position on the other character. E.g. if the eyes are lower, then the warp code would take that into account.

The current solution I have uses a 3×3 matrix with the bounding box of the face and 4 points inside for the left eye, right eye, left mouth corner, and right mouth corner. I picked these points to make sure any animations regarding the shape of the eyes or mouth were correctly adjusted. (You can move the eyes and mouth up/down, wider/narrower, etc in VRoid.)

This strategy worked pretty well for the eye animations, but frankly did not work reliably enough for the mouth. Some vertices of the bottom or top lip would sneak onto the other side, causing the open mouth animations to not work acceptably. (See the video below.)

A copy of the script I used is here. Note that it takes over an hour (!!!) to run on my laptop to copy all 50 odd blendshapes. It is experimental code only, not intended for production usage. It can definitely be made a lot more efficient, but I was just trying to see how it looked.

Here is a sample video showing it working with ifacialmocap.

Conclusions

I hope to have many VRoid characters to bring into Unity, so I want to automate the process as much as possible. I want also be able to update a character in VRoid Studio and reimport into Unity with the least possible effort. Copying blend shapes seems like a possibly useful step in this process, even if it’s creating blend shapes once in Blender by hand, then copying them over each time I import the character again from VRoid Studio.

But to really have good quality there is nothing like hand tuning the blend shape for a specific character in a tool like Blender. For example, in the video above, the eye lashes and eye lids do not align quite right (this is a challenge with VRoid Studio when you adjust the face proportions). Using Blender to adjust the mesh for each blend shape may be necessary to get the best final result.

I suspect I will need to import the VRoid characters into Blender, add the blend shape, export to FBX (or VRM) to load into Unity. Then if I update the character, import into Blender again, export to FBX, then run a script to copy the blendshapes from the old character to the new copy. The meshes will hopefully be identical if the characters go through the same tool chain.