How To Use Cubism Lipsync in Naninovel Live2D Characters

Posted: 12 Jul 2022 07:26
by vapidfire

Would you like to use Cubism's lipsync capability in Naninovel? It takes a little programming to make it work. At least, that's what we found. We haven't found any instructions on how to do it, so we figured out something that works and I am providing here step-by-step instructions and the scripts needed. This is for Unity.

I plan to use this for all my game development, so if folks find better ways, I'd be excited to update this HOW-TO. Thanks.

How to create a character that uses Cubism Lipsync:

1. Import the entire Live2D model folder directly into Assets in the folder of your choosing by dragging it directly from the file explorer into the unity assets window. The folder name will be automatically capitalized for your convenience.

2. Select the prefab file (blue cube). Drag the animation controller (green and grey boxes) into the prefab's Animator component in the Controller field.

3. Scale the Transform component to taste. Start with Scale X, Y and Z set to 10 and experiment from there. This determines how pixelated your character ends up being.

4. Add Cubism components to the prefab:
-- Cubism Mouth Controller. Set blending mode to Override with Mouth Opening set to zero.
-- Cubism Audio Mouth Input. Set Gain and Smoothing to taste (Gain = 5, Smoothing = 0.3 to start) . Leave audio input blank.
-- Cubism Look Controller

5. Add Naninovel Live2D Support
-- Render Canvas – Change the size to something reasonable for the model so it doesn't clip. Size X = 16, Y = 24 is a starting point.
-- Live 2D Controller. Turn Control Mouth off. The Naninovel lipsync is flap only, so we turn that off and will use the better Cubism support.
-- Live 2D Lipsync Fixer – a component we wrote that fixes things to make lipsync work at runtime.
-- Audio Source – Set the Output Mixer Group to what voices should be in your game.

6. Edit the Animator by double clicking the animator file to provide support for motions defined in the model. to make the model work, you need to create a custom set of named triggers in the parameters that cause transitions that make the character move. Then use these names in the naninovel scripting so you can control the movements of the model.

The motions folder in the model folder contains all the animations that the model supports.
-- Drag all motions from the folder into the animator (drag middle mouse button to pan the editing area).
-- Right click initactor motion if you have one and Set as Default State. This make it run immediately when the model arrives. Initactor initializes the model to a known good state.
-- Select the parameters tab and enter custom triggers based on the needs of Naninovel scripting.
-- Create transitions in your animator that use the trigger parameters as necessary to animate the character to desired effect. Very often you probably want to create a transition of every possible trigger from the AnyState state to animate the character. Its very simple way to support everything to start.

7. Add the character to Naninovel configuration in the Project Settings panel.
-- Select Live2DCharacter as the implementation.
-- Drag the prefab into the Resource list at the bottom.
-- Drag the prefab again into the voice source.

That's it. The character should be ready and lipsync should work. Here is the fixer script:

Code: Select all

using Live2D.Cubism.Framework.MouthMovement;
using UnityEngine;

namespace Your.Namespace
{
    public class Live2DLipSyncFixer : MonoBehaviour
    {
        void Start()
        {
            if (gameObject.name.Contains("(Clone)")) // the character clone
            {
                // The Naninovel Live2d support clones the character. Unfortunately, the clone side has
                // the audio source that gets used while the original's audiosource is the one that moves the
                // mouth.  We work around this by detecting the clone and copying its audio source into the 
                // original so that the cubism mouth support will work.
                transform.parent.GetChild(0).GetComponent<Live2D.Cubism.Framework.MouthMovement.CubismAudioMouthInput>().AudioInput = 
                    GetComponent<AudioSource>();
            }
            else // The original, not the clone.
            {
                // Some models dont have the mouth parameters flagged as CubismMouthParameters. This hides
                // them from the CubismMouthController.  We don't understand why this is happening but we can
                // work around it here. We check if parameters are broken and fix them, and then get the
                // CubismMouthController refreshed, thus reloading parameters again.

                var parmlist = transform.parent.GetChild(0).GetComponent<Live2D.Cubism.Core.CubismModel>().Parameters;
                if (parmlist == null)
                    return;  // should never happen. fail silently if it does.

                bool changeMade = false;
                foreach(var parm in parmlist)
                {
                    if (parm.Id == "ParamMouthOpenY" || parm.Id == "ParamMouthForm")
                    {
                        if (parm.gameObject.GetComponent<CubismMouthParameter>() == null)
                        {
                            Debug.Log($"Fixed cubism parameter {parm.Id}");
                            parm.gameObject.AddComponent<CubismMouthParameter>();
                            changeMade = true;
                        }
                    }
                }
                if (changeMade)
                {
                    var mouthController = transform.parent.GetChild(0).GetComponent<Live2D.Cubism.Framework.MouthMovement.CubismMouthController>();
                    if (mouthController != null)
                        mouthController.Refresh();
                }
            }
        }
    }
}