[WIP] Cheshire Lip-Sync

Huzzah! After quite a bit of work, the Cheshire Unity plug-in is available for download on the Unity Asset Store. It is quite rough, but it's in a usable state. I will be fielding questions about the component here and in a thread I plan on creating in the Asset Store board.

You can find documentation about Cheshire on the Mad Wonder website. There are no message boards there yet, there just isn't nearly enough content or interest to warrant them.

I would appreciate whatever feedback the user-base here on the Unity boards could provide. I am very new at this, and everything is extremely rough and seat-of-my-pants. There's still a lot of documentation I have to do.

[UPDATE: 03/13/2015]
After a bit of a hiatus, I've actually produced some new visuals. Here's a model I'm working on to serve as the new "mascot" and example for the plug-in.

It's a kitty! I felt this would be an appropriate choice given the name of the plug-in. Also, I like cats.

 

I would be interested in this. One question. What requirements will a mesh need to use it.

 

Does this use regular unity blendshapes? And how much are you intending to ask for it? It seems i'll be pouring money into this subject since i'd like something a lot sooner than the end of may but i'd also like something that uses visemes like it seems yours does, i'm assuming you're using audio analysis. Ahh well, looks good, if it turns out well i might pick it up, cause why not hey

 

At the moment I don't have any plans to charge for it. The phoneme timing is being calculated using an open-source command-line program that was released by Annosoft back in 2005. It's free to use that program (open source), and last year I wrote a basic GUI front-end for it that makes it a bit more user-friendly.

The current requirement is that the model in question have 9 valid shape keys. That's pretty much it. When you assign the script to a model, it automatically detects if the model has a Skinned Mesh Renderer and whether or not there are 9 shape keys. If those criteria aren't met, the "Create Animation" button is inactivated. Under the "Mouth Shapes" section you just select which shape key you want paired up with which standard phoneme. When you click the "Create Animation" button, it pops up a little editor window where you can choose the text data file and Audio Clip that you want to use for your animation. The script then takes the timing data from the data file, and creates a Mecanim-standard Unity animation file. The animation file is just like any other standard Unity animation, and can be altered in the standard Unity animation window.

 

That's pretty cool, what's left for you to do? Does it need any battle testing? *hint hint haha* I'm using daz models and the genesis system onwards has a pretty strong set of of visemes, well, that's what they seem to be labelled as, a quick wiki reveals a viseme can be the visual appearance of several phonemes but i guess the point is i'd find such a thing very useful, so i'll watch out for this in future.

 

Great, does it synch with text only input too ?

My game does not have voice for every text shown for example

 

Unfortunately, no. This is not a text-to-speech kind of thing.

However, if you have the patience, you could do the voice work yourself, and then just input the timing data without using any audio. I am allowing for that scenario. The audio playback is handled as an animation event at the beginning of the animation tied to a function in the script. If you don't provide an audio clip during the creation process, the animation will be created without that event, and no audio clip will be required. I threw that in just in case anyone wanted their characters to "mouth" the words without actually playing any audio.

This would still require you to record some audio and extract the timings from it, but the audio would never have to be imported into Unity. Once you had the text file with the timing data, you could forget about the original audio if you don't need it for your project.

 

That would be perfect for me, really, i'd like some audio for the voice work but i didn't really want proper human voices, initially i was thinking of the typical animal crossing style sounds or something like simlish, but i dont think myself and some pals would mind at all if we could record the voice acting, process it with your tool then process the audio so it sounds a great deal stranger. Mind, I imagine this might work with the voice just being kind of gibberish anyways? Either way i love the idea of the voices being able to be used this way then process them for interesting voice effects, gibberish with vocoders but with convincing lip sync is a fun sounding start to me. I'd really like to have a good shot at this when it's available cheers! I've got to deal with facial expressions too but there's another nice free tool i could adapt for this with unity

 

wow cool

 

Yes, actually, it probably would work with gibberish. The command-line program that extracts the timings operates on a set of basic vocals sounds. As long as the gibberish features some of those sounds, it should still work. It works better with a text transcription, but it doesn't require that transcription to be proper English.

Before the weekend I was able to iron out a major bug that I had run across. Unfortunately, I was busy over the weekend so I wasn't able to get much work finished. (re-caulking my shower)

 

I'm getting a lot closer to being ready to go.

Unity Lip Sync Demo


Here's a quickie Unity Web-Player showing the finished results of animations created using this tool. All of these animations are unaltered. I didn't go back to clean them up after the fact. (though you can via Unity's standard animation window) It will hopefully provide an idea as to what I'm shooting for.

All of these animations took only a minute or two to create. It took longer to record and process the audio to my satisfaction than it did to get the animations running in Unity. For anyone who's curious, the spoken text is the first paragraph of H. P. Lovecraft's "Polaris." It's better to stick with the public domain for demo purposes. Star Wars quotes would have been more fun, but you never know when the lawyers might be looking. Besides, more complicated sentence structure makes for a better stress-test.

 

This looks great, and very professional. Hopefully there will be a video tutorial on how to set your model up for this.

Any thoughts on adding Eyebrow, Cheek and Eyelid movement for added expression? Would help with some of the stiff characters I still see in modern video games.

Best of luck!

 

I will touch on it briefly, and may indeed have a basic video tutorial for Blender. (the 3D program I'm using) The actual 3D model creation isn't the focus of this plug-in. Also, setting up models for this thing is a fairly straight-forward process. All you need to do is create about 9 blend shapes in your 3D program of course. That's pretty much it. The script detects the given names of the blend shapes from the imported 3D model, and lets you match them up with the various mouth sounds. So there are no specific names you have to use. Set-up is extremely basic. I wanted things to be as simple for the end-user as possible.

The script just uses 9 basic mouth blend shapes. It isn't designed to add any other animations, just the mouth movements. But since the end-product of the script is a Unity animation file, it is possible to open the animation in the animation editor afterwards, and do whatever you please. If you want to have more than 9 blend shapes on your model for more complex facial animations, that's up to you. You can create the mouth anim file, and then go in and add whatever further facial animation that you want. I would personally recommend some blinking, smiling, frowning, and eyebrow blend shapes. Additional blend shapes can be combined with the mouth animations to create more nuanced performances, they just won't be generated automatically.

After a little research, I decided that XML would not be the way to go. The parsing for XML in C# is frankly a bit of a pain. It just seems like it would be more trouble than it is worth, and overkill for the purposes of this project. I'll be sticking to basic CSV files. I still have a bit more testing, but I'm going to be working on the tutorials and documentation this week.

 

It's awesome, but the face in the demo is like one of those story teller toys haha

 

Yes, I can't pre-program different expressions into the face. That will have to be left up to the end-users. The nice thing is that they can easily add their own additional blend shapes to put a bit more "life" into the 3D models. (smiles, blinking, etc...)

I've been pre-occupied with personal matters recently, so I haven't made as much progress as I would like. Also, I've reached the point where I'm working on documentation, which is far more tedious than the fun of problem-solving. But I am still working on this project, and trying to package it up for an Asset Store release.

 

Quick update. I'm still at it. I've finished up about 50% of the documentation. Last night I went back to my example 3D model and optimized the UV mapping. Today I'm going to try to re-paint the basic skin for it. I'm not going to get too fancy, it's just there for an example.

I still have to go through the submission guidelines and produce some of the graphics to go along with the submission. But I'm in the home stretch now.

 

Whooo hooo! My very first Unity Asset Store component! Cheshire, the Unity lip-sync component, has made its debut on the Unity Asset Store.

It's free, so give it a download if you're interested and take it for a spin. Documentation can be found at the Mad Wonder website, as well as the Windows application that goes along with it.

Post any questions, complaints, or requests for more documentation here on the Unity boards.

 

I just got it off the asset store today and everything looks good until I try to actually create the animation. I get IndexOutOfRangeException: Array index is out of range.

And it takes me to every line like this: processCurves[_targetCat.GetAdjustedBlend(freshSymbols)].AddKey(freshFrame);

I'm not a programmer so I'm not sure what to do about it. Overall everything else looks wonderful! I attached the example animation to my character and it worked great!

 

freshSymbols is an array that stores the various phonemes extracted from the text file for processing. It's supposed to be referenced in the code like this...

Code (csharp):

1. freshSymbols[i]

I'm not sure what use-case would result in the line of code you've posted. I checked through it and couldn't find any point where the array access was not being used. Can you give me more details, and can you post the text file you're trying to process? I suspect there might be some conflict there that might be causing the plug-in to hang.

Never mind, I just found out why you didn't bother to put that little snippet. The BB code for this message board automatically interprets it as "itallic." Very annoying. Anyway, I would still like to see the text file you're trying to turn into an animation. It's possible that it's feeding in a symbol that the plug-in doesn't recognize. Also, is there a specific number of times it repeats the stated error? That information could help me narrow down where the bug is occurring.

 

My mistake, the lines it takes me to is:
processCurves[_targetCat.GetAdjustedBlend(freshSymbols)].AddKey(freshFrame);

and if I comment the first one out then it just takes me to the next one that has that is the same as the first. And only 1 error every time I press Create Animation (the one after you plug in the audio and txt files).

I'm using the example audio and .txt files. Watched the video posted in the 2nd review and did it the same way as them but I just keep getting the error IndexOutOfRangeException: Array index is out of range.

 

My first thought is that it is searching for a shape key that is no longer there. You didn't, by any chance, alter your 3D model after the script had already been attached to it, did you? I don't think I took that sort of event into account in my coding.

The other possibility I can think of is that it is running into a use-case with the text file that it wasn't designed to handle. I'm not exactly certain what that might be though. Is it possible that you selected the "Annosoft" option for exporting the timings from Cheshire?

 

Hello! Congratulations for your work.
I have used the program Cheshire but does not translate well the audio file. the resulting text does not capture well the phonemes. Then when I do the animations do not match the audio. Even if I use in Cheshire the example sound "01northWindow.ogg", it does not do well. The resulting content is a completely different "txt" file that you use in the example (01northWindowSync.txt), and if a use this file for creating an animation the result is´t good.

Can you help me to run the program Cheshire correctly? Do I have to configure something in the "sapi_lipsync" or "Cheshire"?

Thanks you.

 

The Cheshire stand-alone program has to be used with a WAV file, not an OGG file. Try using it again with a WAV file.

 

Why is it wav only? Seems like wav wouldn't be the first choice because of compression.

 

It's a limitation of the sapi_lipsync command-line program. Also, the WAV requirement is limited only to the stand-alone application. Once the timing is extracted, you can convert the audio into whatever you please. But the Cheshire standalone application needs the WAV format. For importing the audio into Unity, you can use MP3, OGG, WAV, whatever floats your boat.

The examples provided in the Cheshire Unity plugin were only intended for use with Unity. Using the OGG with the Cheshire stand-alone application would not work. If you converted that OGG to a WAV file, it probably would work.

 

I've always used WAV files. Of course the first thing I did was convert the OGG to WAV format, but nothing. I've caught "internet wav files free", I saved my voice,... all in WAV format, but neither works. Always creates a txt file that is not well represented phonemes....

 

Can you help me to run the program Cheshire correctly? Do I have to configure something in the "sapi_lipsync" or "Cheshire"?

Thanks you.

 

Love the idea, and I'd be really excited to use this with my projects. Sadly my sound files always seem to not generate correct animations, or rather any animation at all:

Code (CSharp):

1. %% Mad Wonder standard CSV Lip Sync data file %%
2. P,rest,0
3. W,0.38,0.52,in
4. W,0.52,1.07,either
5. W,1.25,1.37,the
6. W,1.37,1.62,letter
7. W,1.62,1.89,ends
8. W,1.89,2.37,led
9. W,2.37,2.62,the
10. W,2.88,3.2,I
11. W,3.38,3.66,can
12. W,3.66,3.95,do
13. W,4.3,4.54,it
14. W,4.69,5.38,all
15. W,5.38,5.83,emailinc
16. W,5.83,6.06,that's
17. P,rest,17.89388

The files actually longer, but from my parsing the file it seems like it only shows W. I tried looking at the documentation but so far I couldn't really find anything as to why. I'm using a 16-bit WAV file that I exported in Audacity. The Unity plug-in however seems fine. Any pointers? Thanks!

 

I encountered this error also and did a little tweaking. If you add this check at the top of the if else block, it will bypass any loops that would generate an error. I can't tell you what exactly the problem is, but this sorta fixes it.

Code (CSharp):

1.  
2. if(_targetCat.GetAdjustedBlend(freshSymbols[i])<0) { //insert this check to fix indexoutofbounds error
3.     continue;
4. }
5.  
6. if(freshTimings[i] == 0.0f) { //"CheshireCreate.cs" line 104
7.     Keyframe freshFrame = new Keyframe(0.0f, modelAnimStrength, 0.0f, 0.0f);
8.     freshFrame.tangentMode = 0;
9.     processCurves[_targetCat.GetAdjustedBlend(freshSymbols[i])].AddKey(freshFrame);
10. }
11. if(freshTimings[i] - leadTime <= 0.0f) {
12.     Keyframe freshFrame = new Keyframe(0.0f, modelAnimStrength, 0.0f, 0.0f);
13.     freshFrame.tangentMode = 0;
14.     processCurves[_targetCat.GetAdjustedBlend(freshSymbols[i])].AddKey(freshFrame);
15. }
16. else {
17.     Keyframe freshFrame = new Keyframe(freshTimings[i] - leadTime, modelAnimStrength, 0.0f, 0.0f);
18.     freshFrame.tangentMode = 0;
19.     processCurves[_targetCat.GetAdjustedBlend(freshSymbols[i])].AddKey(freshFrame);
20. }
21.  

Anyway, I want to say I am 110% happy with this plugin, my project has hundreds of lines of dialogue and with Cheshire, my 3d modeller just has to conform to the preston-blair 10 phoneme blendshapes and our whole pipeline is set up. Thank you RichardKain!

 

I really appreciate the feedback, byrne. It's always great to hear that someone is getting some use out of my work. Please let me know as soon as your project is finished, I would love to see the final version.

 

It looks like some manner of error in the parsing. I will perform a little testing, but I suspect that the punctuation you put in the transcription file might be to blame. A phonetic transcription should always be what you're shooting for. Commas should be okay, but I usually avoid things like apostrophes.

 

How do features/performance of the Cheshire plugin compare with existing Unity lip-sync solutions? (I'm particularly thinking of Tagarela, which is what I currently use)

 

Well, as near as I can tell, Tagarela is not on the Unity Asset Store. It is available from Github, but there is a certain amount of convenience from being able to snag plug-ins from the Asset Store, so there is that.

In terms of features/performance, there are a few items I can speak to. Tagarela is a more complete editing suite. It has a waveform preview, and allows the user to create keyframes as they see fit. This is definitely an advantage.

On the other hand, Tagarela requires that you create the keyframes yourself. One of the biggest advantages of Cheshire is that it uses a basic stand-alone application to rip the lip-sync timings out of a wav file. There is no real need to create the animations yourself by hand. Just feed a wav file and a text transcription into the stand-alone application, and it will spit out a text file with all of your symbols and timings already figured out. Import that text file using the Unity plug-in, and it will create your animation. The big advantage here is a savings in time. While Tagarela would allow for finer-tuning of specific animations, Cheshire would have an edge if a large number of animations needed to be created quickly.

A big difference between the two solutions is how they are intended to function. Tagarela creates custom animation events. Cheshire is specifically designed to work within Unity. The animations created by Cheshire are standard Unity animation files. There are advantages and disadvantages to both approaches.

Cheshire's Unity animations can be altered, played back, and edited like any standard Unity animation file. If you are already familiar with Unity's animation window, you can open up a Cheshire-created animation and edit or alter it however you please. You can use Unity's default animation controllers to play Cheshire-created animation files as well. It's all native to the Unity editing environment.

However, there are drawbacks to that approach, and advantages to what Tagarela provides. One thing to note in particular is that Tagarela has true "syncing" for audio files. Because Tagarela doesn't use native Unity animations, it can skirt around some of the limitations of Unity's standard system. Tagarela uses the time of the currently playing audio file for triggering it's animation events. This allows it to keep the animations completely synced to the audio. One use case where this comes in handy is mobile development. Mobile deployments often have trouble keeping standard animations synced to audio, due to the necessary decompression. Tagarela's approach would avoid such issues.

At the moment, neither solution is perfect, and they both have their strengths and weaknesses. Cheshire would be better for large-scale syncing on desktop projects, while Tagarela would be best for small-scale syncing projects targeting mobile devices.

 

Thankyou for the really considered, detailed response! Will definitely be checking out Cheshire for a project in the future.

 

I'm currently planning on updating Cheshire in the near future. I would love to hear about some of the features people would want, and would be useful for their projects.

In particular, I would like to separate the list of phoneme shapes from the editing features. I was planning on creating a different script to hold each of the various possible lip-sync objects. One for 3D skinned models, one for 2D sprites, and one generic one for function calls. The function call option would give developers the flexibility to script their own custom actions for phoneme shapes, and then have them occur based on the provided timing.

Also, I was wondering how important a custom animation system would be. While creating Unity-standard animations is useful, there is also quite a bit of utility to other approaches. I didn't bother attempting the kind of audio syncing that Tagarela employs because the version of the API that I was using didn't expose the timing data for audio files being played. I would have had to step back to a legacy version of the audio system to make that kind of syncing work. But it's possible that with the new 4.6 upgrade that some of those features may be available again.

I will do some more research and testing tomorrow.

 

Hi Mr Kain,

I have the exact same problem than netravelr. When i process lipsync timing in cheshire, the .txt he generate me is totally wrong. Even if i put the text with it. So my question is, did u have some more specific requirements to process lipsync timing? WAV quality? I am french, how do i set english language recognized ? The programm recognize french even i f i put english wav, or even i f i put my windows in english. Did u have more documentation about the sync.txt and the signification oflanguage it use? About the way we must write the text transcription ?

Thanks a lot

 

Sadly, there is not much I can do about the internationalization of the Cheshire program. The vast majority of the functionality that it features is bound up in the command-line program that it is piggy-backing off of. And that command-line program is an open-source release that I am not technically proficient enough to edit. (I've tried) If you knew a C or C++ programmer who knew their way around lipsync programming, they could go in and alter the application for you. I kept the command-line program separate from the GUI application.

In theory, it should run with the French language. The program is identifying basic vocal sounds, and separating them into pre-defined symbols. It's general-purpose enough to work with most languages. I would recommend making your text transcriptions as phonetically-focused as possible. If you want the best recognition results, spell your transcriptions the way it sounds, not necessarily with the correct spelling. Also, I'm not certain how it will respond to a transcription in a foreign language. You may need to use English-phoenetic in your transcriptions. (or get a programmer to switch the command-line program's dictionary over to French)

The WAV file is a more likely offender in your case. For starters, you want to feed the program a mono WAV file. If your source is stereo, convert a copy to mono for lipsync processing. You will want to use PCM compression, and use a lower compression quality. Not the lowest, but pretty low. Speech is commonly transferred at lower compression because there aren't that many different sounds associated with it. It's easier for a computer to interpret speech when the compression is closer to what you would hear over a phone line.

Obviously, you shouldn't delete or overwrite your original audio. Create a separate copy for processing, and keep the original high-quality audio. Once you have the timings, the software doesn't care what file you use for playback.

 

Just a quick up-date. I'm in the middle of re-factoring the script for the Skinned Mesh Renderer support. (Blend Shapes) I've added the list of Preston-Blair standard blend shapes, but it occurred to me that it might be a good idea to support a more flexible approach to the problem as well.

As such I'm also going to make a series of Lists that will also be mapped to the Preston-Blair phoneme shapes, and allow the user the option of swapping between the two approaches. The Lists will offer the user the option of mapping multiple blend shapes to a particular Preston-Blair phoneme, and adjusting the intensity of each of those mappings as desired. This would allow the user to employ a more "Valve-esque" approach to blend shape mapping. (where you create multiple small, subtle blend shapes and mix them together)

I can personally see a lot of utility in a feature like this, despite the slightly increased complexity. I think offering it as an option is worth the trouble.

 

Still at it. I know I haven't had much to show recently, but that is just because I've been working on re-factoring the back end. While progress is slow, I've been learning a lot of great tricks and approaches for improving the project. A lot of what's been holding me up is just the particulars of coding the interface. Editor coding still requires the use of the legacy UI system. I don't get to use the snazzy new in-game UI library for making editor-specific tools.

At first, I was a bit concerned about the increased support for the Blend-shape approach that I was planning. I wasn't sure how I would be able to incorporate this into a system for playing back the animations based on when the audio is playing. (as opposed to just a basic Unity animation) But after doing a bit more research into the AnimationCurve class, I am confident that I have my answer. I'm going to have to go back and check on what versions I will be able to support, but incorporating AnimationCurves I believe I will be able to provide everything the end-user is going to need for nice, smooth blend shape animations.

 

Whooof, custom GUI Editor scripting really takes it out of me. Its pretty much all procedural, and not all of it can be done through for loops. I put in a good 3-4 hours tonight. Managed to finish the GUI Editor coding for all of the standard Blend Shapes. Even performed some quick testing. Worked like a charm. Next I need to tackle the Blend Shape Lists. Those will pose their own special challenges, but at least I'll be able to loop most of the actual GUI code.

I'm thinking I'll take a break from the custom inspector monotony tomorrow and rig up one of the other features I've been meaning to try. I wanted to rig up the command-line program so that it can be run through Unity directly, instead of relying on a stand-alone GUI application. I'm pretty sure the terms of its license allow me to redistribute it, so I should be able to bundle it with the plug-in when I'm finished. If I can just get the output into a string variable in Unity, I'll be able to handle all the parsing and testing in the Unity scripting environment. This will be a lot easier than trying to struggle with Windows-native application development.

 

I actually have a visual image to share for once!

It's the new mascot for the Cheshire Plug-In! I took a break from coding last night, rolled up my sleeves, and made this quick sculpt. I was reminded of how rusty I am with 3D modeling. I really need to get some more practice in Sculptris, and maybe bust out my Wacom Bamboo pad.

But for now this little guy should do as a new example model for Cheshire. While the previous Block-Head model was decent, it definitely wasn't my best work. And it certainly doesn't show very well in the preview image on the Asset Store page. I think a fuzzy kitty with a big grin will be a lot better! I've already started the retopology for the in-engine model.

It wasn't all 3D art last night either. I made great strides in getting the sapi command-line program playing nice with Unity's editor. My previous experience developing the stand-alone GUI application came in handy here. It didn't take me long to get the output from the command-line program feeding back into Unity as a string variable. I even tested a little parsing to insure that the correct output is being retrieved. (to insure that the user isn't attempting to run the wrong external program. Development continues to be smooth.

 

Still at it, but I only did a little bit of work over the weekend. I was catching up on some TV shows that I had been putting off, and a few chores I had been meaning to get to. One very pleasant task was the new SSHD I had ordered that arrived on Sunday. I got that installed, partitioned, formatted, and moved my Steam install over to it. Now I have a full Terabyte to play around with for game storage, instead of the 180 GB I was using on my SSD. Windows and most of my major programs are staying on the SSD, for obvious performance reasons. But I've managed to free up 70+ Gigs, so I have a lot more wriggle room on my boot drive.

As to Cheshire, I started work on the primary editing window. At first I was thinking that I would separate out the command-line loader that I prototyped into it's own window. But the more I think about it, the less sure I am that it is the proper approach. I definitely need to comment the command-line loader out in the case of platforms other than Windows. But I can do that pretty much anywhere, a separate window isn't necessary. Now I'm thinking maybe a "Load Data" foldout at the top of the editing editing window might be more appropriate. I could put the text-file loading options in there as well. Possibly even have a checkbox that lets the user toggle between text loading and command-line loading. Yeah, I like that.

 

Looking interesting will you add unity 5 support?

 

Yes, actually! The latest upgrade is already being developed in Unity 5. I'm also going to be going backwards, and insuring compatibility with versions as far back as 4.0. The new upgrade is going to feature a "method-mapping" component. This feature will be compatible with much earlier versions of Unity, so we will create a version of the plug-in that will only feature this approach. That version of the plug-in will be made available to any developers who are still working on much earlier versions of Unity that don't have support for Sprites or Blend shapes. While not an ideal solution, it will add a great deal of flexibility. It's important to give developers freedom, as well as user-friendliness. Balancing those two elements is a constant challenge.

The initial release of Cheshire sought to maximize the user-friendliness, while scaling back on flexibility. A lot of the Cheshire users felt limited, and were interested in more features. The new upgrade is going to scale back the user-friendliness, but provide more documentation and tutorials to compensate.

I haven't gotten much done this week, I've just been immensely busy in my off hours. I am expecting to get quite a bit done over the weekend, though. I'll have more to say and show come Monday.

 

I took a brief break from working on Cheshire. I wanted to pick up an old project that I had dropped some time ago. But after less than two weeks of work, I've completed all the features of my aspect ratio tool, and now I'm back to working on Cheshire with renewed vigor.

Last night I experimented a little with loading and running the SAPI program from Unity, and discovered a way to uniquely identify which program it is that you are running. If you run the SAPI program without arguments, the standard output is empty, and can't be used for reliable identification. But the standard error is another matter. When run without arguments, the program returns a unique error message that can be used to positively identify the software in question.

This weekend, I'm going to rig up a class with some static functions for performing those tests anywhere in the software.

 

Boom! Nailed it! Last night I put the finishing touches on the static functions for managing the command-line program from within Unity. Did some testing as well, to make sure everything worked as desired. Now I have all the code for finding, testing, verifying, and running the command line program in one convenient location. And I can access it from anywhere in my plug-in, which makes it easier to use it in multiple editor windows.

I also finished coding the settings window that lets the user select the command line program from somewhere on their hard drive. If they select the wrong executable, it throws a little error dialog that tells them to keep looking. If they select the correct executable, it automatically stores that path in the Editor Preferences, and informs the user that they are ready to go.

I also began work on coding the editor window for loading in animation data, and editing animations. That one is going to take a while. It accounts for a healthy portion of the plug-in's functionality. It is also going to feature some of the most custom scripting and drawing.

Progress continues to be positive. I am pleased.

 

This is great news. I love how easy Cheshire is to use, and it sounds like it's only going to get better.

Do you still use your cheshire.exe program or does it go straight to sapi from the editor?

 

I have plans to re-build my cheshire.exe program, I found a few bugs in it that need correcting.

But the new version of the Cheshire Unity Asset will not require cheshire.exe on Windows. It will be able to access SAPI directly from the Unity editor, no middle-man. That particular feature is already tested and ready to go. I went over the license for the SAPI command-line program again, and determined that it would be legal for me to distribute the executable along with the Cheshire Asset. So the next upgrade will include a copy of that command-line program, so you won't have to download it separately.

Right now I'm working on the code to parse the text output from the SAPI program, as well as a couple of other lip-sync formats. I'm planning on supporting the SAPI output, the CSV Cheshire output, DAT Moho files, and the native format from Papagayo. That way Mac users won't be entirely locked out of the fun.

 

I'm still at it. Things slowed down a bit over the Easter holiday weekend. But I've been good about getting at least a little time in each day to work on this project. It helps to keep me motivated and thinking about this, even if I only make a little progress.

Recently, I've been grinding through the text parsing phase. The CSV Cheshire parsing was easy, and the Moho DAT file was also a walk in the park. The SAPI output was a bit more challenging, but it wasn't my first time parsing it, so I got through that without too much trouble.

The Papagayo format is definitely proving to be the most troublesome. This makes sense, it's almost certainly the most complex of the four. I think I may have hit on the proper method though. Gonna do a little more testing tonight and see if I can't lock it down.

 

Whooof! It took a little while, but I managed to nail down the Papagayo file parsing. I had to do a few mental gymnastics to sort that one out. It's not structured in a way that makes it easy to cut apart with the standard C# string tools. And of course, you constantly have to put up with the headache of converting a key-frame based environment to one based on decimal seconds.

But I powered through it, and tested it with multiple generated files. Unlike the exported Moho switch files, the Papagayo files store the framerate you entered in the application, so that's one less variable I have to ask the user to provide. I won't be able to get anymore serious development in until the weekend. I'll be thinking about which feature I want to tackle next. With all the data loading finished, it might be a good idea to jump right into automatic animation generation.