Extra Ordinary, the Series

Some expressions involve changing the face structure (a smile is created by a blend shape) while others involve changing the texture of the face (like blushing). The following describes my current plans to fade between textures for blushing and other expressions that work by changing the texture of a material.

Multiple Textures

In order to implement blushes (and other expressions), the approach I have gone with is to blend between the standard texture of a layer and a hand drawn alternative texture with the expression.

Here is an example of an original texture for a face.

Here is a modified face texture with blushing (crudely!) added.

The first challenge was then to get Unity to cross fade from one texture to the other.

Shaders

Shaders in Unity control how materials and textures are rendered. Without going into too much depth, think of a material like dealing with shiny metallic or cloth materials, and textures being a bitmap image with the paint/colors. A shader is a graphics card algorithm used by the material to render itself on the screen.

When you use UniVRM to import a VRM file into Unity, it uses the VRM/MToon shader by default. This is to make it feel more like anime.

In recent years, Unity has reworked the rendering pipeline, introducing a “Light Weight Render Pipeline” (LWRP), a “High Definition Render Pipeline” (HDRP), and more recently a “Universal Render Pipeline” (URP) which has replaced the LWRP. I have tried to use these new pipelines, but I am still learning Unity. I have hit so many problems trying to upgrade old assets to the new rendering pipelines that I am giving up for now and sticking with the older approach.

Why this is relevant is the new pipelines support “Shader Graph” which includes a fancy graphical UI for designing a shader instead of writing code. Some of the tutorials really look impressive. I may give this another go in the future, but for now I am going to stick with the older shader approach.

So instead I modified the VRM/MToon shader to support a main texture and a secondary texture with a “lerp” (linear interpolation) value to blend between the two textures (0 = main texture, 1 = secondary texture). I basically added a new property similar to _MainTex called _SecondaryTex as well as a new float property for _TextureLerp. I then used the following line of code to interpolate between the two textures.

half4 mainTex = (_TextureLerp == 0.0f)
    ? tex2D(_MainTex, mainUv)
    : lerp(tex2D(_MainTex, mainUv),
           tex2D(_SecondaryTex, mainUv),
           _TextureLerp);

This will blend between the main and secondary textures.

Animation Clips

Shaders (as I understand them) run in the GPU on your machine. Shader algorithms are not as flexible as a general purpose programming language. As such, I can blend between two textures, but I cannot have an array of textures. Instead, my approach is to use a Script to change the secondary texture to one of several secondary textures and then lerp between the main texture and the desired expression .

I originally wrote a script that took an array of names (such as “blush”) and secondary textures, with the intent to animate the expression name and the lerp strength to fade it in. By using a name I was hoping the animation clip could be shared by multiple characters (each character would map the “blush” name to a different texture suitable for that character). However this approach failed as animation clips cannot animate string properties.

So instead I have gone with an approach to hard code in a set of expressions (blush, sweat, etc) into a script using pairs of properties for a texture and lerp (float) value. This script only permits one Lerp value to be greater than zero at a time, swapping the secondary texture of the shader as required. This makes creating animation clips simple – they just have to animate the appropriate lerp property.

The code is on GitHub in FaceTextureBlend.cs.

The negative is if a new facial expression is required, the code has to be extended to hard code in another texture name and lerp value. It does not feel very generic, but in practice this is not a big deal as the script code is pretty short.

Also the script is designed to work with a single material (the face skin in this case). Separate scripts are required for each material to be modified in the same way. For example, to animate the texture of the eye iris (e.g. to make them glow red when angry, or go dull when confused), another script would have to be created for the F00_000_00_EyeIris_00_EYE material with the appropriate expressions and textures as desired.

Conclusions

It is a little clunky, but it works. Adding new properties does not affect any previous recordings.

Also when animating face blend shapes, the “Neutral” clip records values for all of the blend shape strengths. This is not necessary for the lerp variations because as soon as you change one lerp value (e.g. fade blush on), the other lerp values are immediately set back to zero. This also means adding new properties later will not cause problems with older recordings.

My main concern is whether it is flexible enough to support all of the required expressions. For example, what if you want to be nervous (sweating) and blushing at the same time? That might require a different approach, such as superimposing a layer with transparency over the other layer.

One final benefit of the approach is that if I later upgrade to use one of the newer render pipelines, previous animation clips will be not be affected as they animate script properties. To change the rendering approach, the script needs to change (not the animation clips). The script acts as a proxy between the animated properties and the shader, avoiding this problem.