Extra Ordinary, the Series

I am trying to create a comic series using 3D VRoid characters. So far it has been more learning than creating content. I have slowly been fleshing out my cast of characters in VRoid Studio and creating locations in Unity leveraging purchased assets. I have explored blend shapes using AR Kit for facial expressions. I have found useful tools for doing hand poses. I have worked out in Unity how to layer the different clips together so I can combine a body pose or animation clip, overlaid with hand poses, facial expressions, and so on.

But what is taking me a long time still is body poses. For example, having a person sitting in a chair with crossed arms.

Note: I am going for a lightly animated comic style rather than full animation. For example, some panels have a little bit of movement like eyes narrowing, but most are completely static. From previous efforts voice actors were just too hard and full animation with speech bubbles feels a bit weird to me. So I am simplifying my search for now for how to efficiently do, not full animation.

Facial Expressions

For facial expressions there are tools like VSeeFace, Waidayo, iFacialMocap, etc that allow you to use the VMC protocol to stream into Unity and then capture the data. You need a good set of blend shapes (I am using AR Kit ones because they are available via HanaTool, plus a few of my own). I am not good enough at Blender to create my own efficiently.

However my current plan is to create my own facial expressions just creating animation poses inside Unity using a tool such as UMotion. Drag a few sliders, get the expression you want, save the clip away. Using this I am getting a reasonable number of good expressions for not too much effort.

I have also added to the blend shapes to textures to do things like blushing cheeks, blank eyes, dark expressions, and so forth. Suvidriel has a superb YouTube channel with tutorials for all the options you have with VRoid characters.

Of course then I saw a recent tweet by @MuRo. So good! (And so depressing comparing to my work!)

Hand Poses

Hand poses are pretty easy – there are only so many of them. I have looked at video camera trackers and the LeapMotion tracker, but for my needs I think I am going to stick with HumanoidHandPoseHelper. It is good enough and it is reliable compared to the trackers I have tried so far. Unity makes it easy to layer hand poses over the top of say full body poses.

Head Poses

Head poses are pretty easy. I wrote some scripts in Unity to turn the head directly towards a look target, but frankly a few simple Left Head Turn, Right Head Turn, etc animation scripts have been good enough most of the time for simple scenes.

The negative however is you often want the head position to align with the rest of the body movement, to reinforce an expression. So I am not sure whether separate head poses are the right solution.

That brings me to body poses.

3D Body Poses

The next challenge that is unsolved for me is how to *efficiently* do body poses. There is a lot of emotion you can express in the position of the body. Since voice cannot express emotion, facial expressions and body poses become more important.

Here are some options I have been exploring. (Note, each tool has a range of equivalent tools. I have looked a few, but they tend to be variations on a theme rather than a radically different approach. So I only mention one or two tools per category here.)

• Mixamo – there are a number of libraries of animation clips, but many have not worked well on VRoid characters when I tried them (e.g. feet twisting or glitches in the smoothness of the animation), and there is not enough variation for different characters. Different characters walk differently! A swagger, plodding feet, etc.
• UMotion – there is IK support so you can move just the hands and feet, and all the other joint angles in between are worked out for you. This has been useful, but it does not always work well. For example, the arms cross image above I have failed to do well with IK – there is just not fine enough control to get the elbows in the correct position with respect to the body.
• DeepMotion, @yukihiko_a (and more) – converting a video into a full body animation clips look really promising, but still a bit new. Arm crossing again makes them struggle.
• VSeeFace – does a good job for head and shoulders, as long as the arms stay in front of the body.
• Full body MoCap suits look pretty cool, but beyond my budget for a hobby! A few thousand dollars seemed the lowest price I could find. I considered gloves, but found hand poses were relatively easy to do and reuse, so it did not seem worth the expense for my needs.
• PlayAnimaker and AniCast Maker – using VR controllers to position the character looks really interesting. I am seriously wondering if this will give me the fine control I need at times. I just find a mouse to change angles too unintuitive. I am not sure they will allow me to create and reuse animation clips. I am creating a series, so want to reuse the same pose multiple times over different episodes. Unfortunately they seem Oculus based and I have a HTC Vive, so I have not tried yet. I think I recall some Unity tools on GitHub, but it all takes time to try out each tool.

So far all approaches I have tried have all struggled when I was after precise control. For example, sitting on a chair with arms crossed, because of the crossed arms, it is hard for video approaches to see what is going on exactly. I do want to create poses in Unity so I can reuse them. It might be time to see if I can script something up with HTC Vive controllers (annoying because the computer the base stations are attached too are not the computer I normally work on). Not sure I am willing to buy a new VR set up just to experiment.

In my case I am going for static poses, not full animation. There are some very cool VTuber tools around live video streaming (like VSeeFace).

While great for live streaming, I have found them less useful than I hoped at times because they are generally not full body. I have struggled to do things like cross my arms, or put a hand on my head. The LeapMotion camera cannot see everything, so you have to restrict your body movements to those that the software can handle.

Conclusions

I look forward to the advancements in full body pose capture from simple cameras or videos. The VR tools also look really interesting.

For myself, I suspect I just need (again) to aim a bit lower. Crossed arms are too hard? Come up with a different body pose then, or crop the camera shot so hands in the wrong position are not visible. Basically take short cuts.

The good news however is as things standardize (like VRM files for characters), I can use the same characters in different tools. The end comic should not look that different if I change my tool chain as I proceed.