[RELEASED] LipSync Pro and Eye Controller - Lipsyncing and Facial Animation Tools

Thank you for your help trasher :3

I believe I did read about the author of this asset is working on providing LipSync without the need for an audio file?
Correct me if I am wrong about this.

The good news is that we are currently working on base models and it will take some time, so finding support from an asset to do what we are achieving is beneficial while we get to know our base meshes and the technical considerations we have to make to accommodate what we are looking for.

Currently, I believe support for Mega-Fiers and non-audio sources would be a huge benefit for us.

There is no doubt we intend to use LipSync Pro for LipSync, that much is obvious because it is so awesome.
However, we also want to dive right into making our artwork come alive when its ready instead of hunting for solutions, which is why I am looking for one.

At this point, I would enjoy getting a response from the author as well for clarity.

 

@Janet Merai Sorry for the late reply! I think @trasher258 is right, LipSync Pro probably isn't the right package for that. It doesn't do any manual mesh deformation on its own, it controls blend shapes and bones based on animations usually (but not necessarily) synced to audio.

 

Hi we found a little bug that happend, when we enlarge the clip editor while we are zoomed in, unity crashes

 

Thanks, I hadn't come across that before. I'm putting the fix for it into the next patch.

In the meantime, go into LipSyncClipSetup.cs and find this line:

Code (CSharp):

1. waveform = AudioUtility.GetWaveForm(clip, 0, (int)(fileLength * (position.width / (viewportEnd - viewportStart))), (position.height - waveformHeight) - 18);

then replace it with this one:

Code (CSharp):

1. waveform = AudioUtility.GetWaveForm(clip, 0, (int)Mathf.Clamp(fileLength * (position.width / (viewportEnd - viewportStart)), 0, maxWaveformWidth), (position.height - waveformHeight) - 18);

That will stop the waveform preview from going over the limit set in the clip editor settings page and crashing.

 

I can't seem to download the Flux extensions. It says server 500 error?

 

I am trying to implement a custom blend system.
I have already sent you an email and I am waiting for an answer.
I am trying to make a custom 2d sprite system with more than 1 sprite when going from one pose to another.
But I get some weird SetBlendableValue logs.
It should increment one mouth phoneme and then decrements it and go to the other one right?
It doesn't do that, it increments 3 phonemes at the same time???

 

I see a lot of our blog posts lost due to the unsuccessful attempt of forums redesign.

 

I'm using MakeHuman to make characters at the moment, and using a plugin to import the mhx2 into blender to get the blend shapes to work. His is then exported to fbx.

I have a problem when I use both the base mesh, and teeth as an optional mesh. Instead of appending the blend shapes from the teeth mesh to the list of blend shapes, it adds this to id 0 (brow_mid_down_left). Moving the teeth also moves the left eyebrow. Looks kinda funky

Anyone know a way to fix this?

 

LipSync Pro 1.2 is now done! The update has been submitted to the asset store and will be available soon. In the meantime, you can get the update from the LipSync website now.
​

Here's the change list for this release:

Features
- AutoSync Language Models can now contain their own phoneme mapper, which will be used instead of the default.
- (Unity 5.3 +) New tool for creating a mesh with blend shapes from several normal meshes.
- Improvements to how Blend Systems work:
- Different components on the same object can now use different Blend Systems.
- Blend System selection, command buttons etc are now standardised.
- Unused Blend Systems now remove themselves immediately - no more phantom components.
- Blend Systems can now inform BlendSystemUsers when the list of reported blendables has changed.
- Added ResetEmotion method to LipSync for blending back to neutral after setting a custom emotion.
- Clip Editor Keyboard Shortcuts are now fully re-bindable from the settings page.
- Clip Editor toolbar can be customised from external scripts for use in extensions.
- Clip Editor now supports rect selection of markers.
- SpriteManager components now display a message when not used by any SpriteBlendSystem.

Fixes
- Fixed stack overflow from recursive GUI rendering when inspecting multiple components using pose editors.
- Fixed detection of differences in emotions between characters and project settings. New ones are now always correctly added and missing ones are marked as such.
- Fixed bug where old LipSyncData files without a set length would cause ArrayOutOfBounds exceptions in the inspector.
- Fixed Pose Extractor Wizard's dependency on being used from a LipSync component.
- Fixed playing LipSync clips with a delay when the clip does not contain audio.
- Fixed NullReference on the reset delegate when first adding a LipSync component.
- Eye Controller's keepEyesClosed bool now functions correctly with PoseBased mode.
- Added warning to prevent errors when trying to export outside the assets folder.

Changes
- Merged all Unity versions into single package using preprocessor directives. No more re-downloading when changing Unity versions!
- Added sorting options to the Extensions window.
- Added dummy "BonesOnly" Blend System as a convenience for bone rigs where you don't need any blend system features.
- Gestures animator layer is now additive by default.
- Removed unused 'using' declarations in scripts.

I'm afraid not, the blendshape blend system expects all meshes to have the same set of blendshapes on them. I'll look into this for the next patch though - see if there's a simple way to allow a different blendshape to be picked for each mesh.

 

This looks interesting! Can you expand on this item (or link me to a post or page with more details)? I'm wondering if this is something similar to what I've done before in Maya LT, but in an automated fashion in the Unity editor. I will be thrilled, if that's the case.

 

Sure! It's actually a feature I was planning from about a year ago, but Unity's Mesh API didn't support it back then. It's an editor window that lets you take a base mesh and a number of other meshes and save out a new mesh with all the secondary meshes as blend shapes.

​

The mesh it saves works pretty much exactly like any other (but only in Unity, as it's a .Asset file). There are a few limitations from how the API works, so all the meshes have to have the same number of vertices, normals and tangents for it to work, and all the vertices have to be in the same order, otherwise it'll go kinda crazy when blending between them!

 

@Rtyper this is awesome! I'm not an artist, so I was only paying my Maya LT subscription for this purpose. Being able to do it via LipSync Pro will be so much easier/cheaper/better. Thank you!

 

Excellent. I'll see if I can fix this on my side in the meantime. Haven't really worked with blendshapes before.

 

So... I got a problem, I'm actually using SLATE extension and using the 'Speech' functionality, so that actually works, but Emotion doesn't do anything at all, and also in Lipsync Clip Editor the emotions made in using the option 'Create Pose form AnimationClip' doesn't work (only those made with 'Bone Transform')... the problem is that there isn't a error message in console. So, I don't really know if I'm actually doing something wrong with the setting of Lipsync. Do you have any documentation about SLATE extesion setting or Cinema Director (I can't do that it works)

 

LipSync Pro 1.2 and 1.201 (minor patch) have just gone live in the Asset Store!
Release notes for 1.2 are a couple of posts up, here's 1.201:

1.201
- Improved Clip Editor toolbar customisation: delegate now includes reference to the editor.
- Fixed emotion marker blend in/outs displaying the wrong colour when highlighted.
- Fixed realtime preview to update when settings on the target character change.

The Slate integration was actually created by Paradox Notion, so I'm not 100% familiar with how it all works, but what I can say for sure is that there shouldn't be any difference between poses made by hand and those made using create pose from animation clip. I'll have a look into this myself, but in the meantime I'd recommend downloading the latest version of LipSync Pro from the asset store, and making sure you've got the newest Slate integration from here: http://slate.paradoxnotion.com/downloads/

 

I've got this error message when mixing emotions in the Clip Editor, the character stops speaking (not always) and the emotions doesn't start.

Code (CSharp):

1. KeyNotFoundException: The given key was not present in the dictionary.
2. System.Collections.Generic.Dictionary`2[System.String,RogoDigital.Lipsync.EmotionShape].get_Item (System.String key) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/Dictionary.cs:150)
3. RogoDigital.Lipsync.LipSync.ProcessData () (at Assets/Rogo Digital/LipSync/Components/LipSync.cs:952)
4. LipSyncClipSetup.UpdatePreview (Single time) (at Assets/Rogo Digital/LipSync/Editor/LipSyncClipSetup.cs:2568)
5. LipSyncClipSetup.OnModalGUI () (at Assets/Rogo Digital/LipSync/Editor/LipSyncClipSetup.cs:704)
6. RogoDigital.ModalParent.OnGUI () (at Assets/Rogo Digital/Shared/Editor/ModalParent.cs:22)
7. System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Reflection/MonoMethod.cs:222)

I'll wait for reply about you can find testing Slate extension.

 

It looks like there's an emotion you're using in the clip that hasn't been added to the character. This error should go away after just closing the clip editor, selecting the character so the LipSync component's editor shows up in the inspector and then re-opening the clip editor. It probably shouldn't cause an error like that though, I'll make this a bit friendlier in the next update.

 

Woah! I was about to buy Maya LT, since I was having such a difficult time with this in Blender….
You just saved me $240/yr!!!

This could probably be a stand alone asset. I have been keeping my eye out for such a tool.

 

I'm getting this error on lite, could you help me?

Spoiler: Error 1

NullReferenceException: Object reference not set to an instance of an object
RogoDigital.AudioUtility.GetWaveForm (UnityEngine.AudioClip clip, Int32 channel, Single width, Single height)
LipSyncClipSetup.LoadFile (System.String path)
LipSyncClipSetup.ShowWindow (System.String loadPath, Boolean newWindow, System.String oldFileName, System.String oldLastLoad, Int32 oldMarkerTab, Single oldSeekPosition)
LipSyncClipSetup.ShowWindow (System.String loadPath, Boolean newWindow)
LipSyncDataEditor.OnProjectWindowItem (System.String itemGUID, Rect itemRect) (at Assets/Rogo Digital/LipSync/Editor/LipSyncDataEditor.cs:251)
UnityEditor.ObjectListArea+LocalGroup.DrawItem (Rect position, UnityEditor.FilterResult filterItem, UnityEditor.BuiltinResource builtinResource, Boolean isFolderBrowsing) (at C:/buildslave/unity/build/Editor/Mono/ObjectListLocalGroup.cs:843)
UnityEditor.ObjectListArea+LocalGroup.DrawInternal (Int32 beginIndex, Int32 endIndex, Single yOffset) (at C:/buildslave/unity/build/Editor/Mono/ObjectListLocalGroup.cs:190)
UnityEditor.ObjectListArea+Group.Draw (Single yOffset, Vector2 scrollPos) (at C:/buildslave/unity/build/Editor/Mono/ObjectListGroup.cs:144)
UnityEditor.ObjectListArea.HandleListArea () (at C:/buildslave/unity/build/Editor/Mono/ObjectListArea.cs:1370)
UnityEditor.ObjectListArea.OnGUI (Rect position, Int32 keyboardControlID) (at C:/buildslave/unity/build/Editor/Mono/ObjectListArea.cs:565)
UnityEditor.ProjectBrowser.OnGUI () (at C:/buildslave/unity/build/Editor/Mono/ProjectBrowser.cs:1744)
System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Reflection/MonoMethod.cs:222)

Spoiler: Error 2

NullReferenceException: Object reference not set to an instance of an object
RogoDigital.AudioUtility.GetWaveForm (UnityEngine.AudioClip clip, Int32 channel, Single width, Single height)
LipSyncClipSetup.Update ()
System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Reflection/MonoMethod.cs:222)
Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation.
System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Reflection/MonoMethod.cs:232)
System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[] parameters) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Reflection/MethodBase.cs:115)
UnityEditor.HostView.Invoke (System.String methodName, System.Object obj) (at C:/buildslave/unity/build/Editor/Mono/HostView.cs:187)
UnityEditor.HostView.Invoke (System.String methodName) (at C:/buildslave/unity/build/Editor/Mono/HostView.cs:180)
UnityEditor.HostView.SendUpdate () (at C:/buildslave/unity/build/Editor/Mono/HostView.cs:245)
UnityEditor.EditorApplication.Internal_CallUpdateFunctions () (at C:/buildslave/unity/build/artifacts/generated/common/editor/EditorApplicationBindings.gen.cs:217)

 

Thanks! Maya's tool for this probably works a little better though It's mostly just a simple GUI for the functionality in Unity's inbuilt Mesh API, so it can be a bit finicky!

I'm guessing you're using 5.3.6? There's an update coming later this month to fix that Unity version. Otherwise if you've got access to Unity 5.4 as well, you could download it in that and then copy the files into your other project, which should fix it.

 

ok, LOL I will try it before I burn my Autodesk acct…
But from the description this is exactly the tool I needed so I will report back.

**edit** hrmmm. Looks like you are right. My meshes did not build… Ahh well.

 

Thanks for this great asset! Is it possible to save the emotion intensity (e.g. happy 50%) in the xml?

 

Aw, that's a shame! I'll see if there's anything I can do to improve the tool, but unfortunately it's most likely just how Unity's API is built.

Sorry Faikus, it's not right now, that was a bit of an oversight! I'll add that to the XML exporter and send you the patch.

 

Thanks! I'm actually creating the xml myself and would need the setting to be read from the xml by Salsa at runtime.

 

I'm now using the plugin Manuel Bastionilab for Blender to create the basis for my characters. The Blend shapes for these models are easier to use with LipSync Pro, so it's working pretty well now. Just some tweaking of the different phonemes left. It does seem like AutoSync is creating inserting too many phonemes, which makes the mouth transition a bit too quickly at times. Is this something I should just tweak manually afterwards, or is there some way to prevent it?

The random blinking is looking good, but the random looking isn't working. The plugin only makes everything in (eyes, teeth etc.) a single model, so I have to use blend shapes to control eye movement. When I check Random Looking, hit PoseBased, configure the blend shapes and hit play, the eyes don't move

Any tips?

 

At the moment, no - you'll need to tweak this. Phoneme detection is a pretty inaccurate field in general, so results from AutoSync can vary wildly. I am working on a higher quality mode for version 1.3 though, which ends up looking a lot better, but can take considerably longer to process.

I'd guess the blendshape values are being overridden by something. Sometimes models get exported with animation tracks for blend shapes even when there isn't any actual animation on them. Try disabling the Animator component on your character and see if it's still not working.

 

Excellent. It's not usually a problem. I think I've only had to remove a phoneme once so far.

Didn't think of that. I'll check it out.

 

I have a strange bug. I have a bit of silent on this one audio clip, but yet auto sync addes a bunch of phonemes... so now his mouth is moving but yet he isn't talking. Please help.

 

Also having this issue with clips longer than 30 seconds

 

When i open the Clip Editor and add an Audio Clip I get a Object reference not set to and instance of an object error.

Im running:
Unity 5.4.1
LicSync Lite 1.1 from the Asset Store

NullReferenceException: Object reference not set to an instance of an object
RogoDigital.AudioUtility.GetWaveForm (UnityEngine.AudioClip clip, Int32 channel, Single width, Single height)
LipSyncClipSetup.Update ()
System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Reflection/MonoMethod.cs:222)
Rethrow as TargetInvocationException: Exception has been thrown by the target of an invocation.
System.Reflection.MonoMethod.Invoke (System.Object obj, BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Reflection/MonoMethod.cs:232)
System.Reflection.MethodBase.Invoke (System.Object obj, System.Object[] parameters) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Reflection/MethodBase.cs:115)
UnityEditor.HostView.Invoke (System.String methodName, System.Object obj) (at C:/buildslave/unity/build/Editor/Mono/HostView.cs:187)
UnityEditor.HostView.Invoke (System.String methodName) (at C:/buildslave/unity/build/Editor/Mono/HostView.cs:180)
UnityEditor.HostView.SendUpdate () (at C:/buildslave/unity/build/Editor/Mono/HostView.cs:245)
UnityEditor.EditorApplication.Internal_CallUpdateFunctions () (at C:/buildslave/unity/build/artifacts/generated/common/editor/EditorApplicationBindings.gen.cs:222)

 

Is your clip longer than 30 seconds, @gamegirl1984? If so, I've just fixed the problem both of you were having. If you can send an email to contact@rogodigital.com with your invoice number, I'll send you the link to the patch that fixes it. Obviously, it'll also be in 1.3 when that comes out next month.

LipSync Lite isn't currently compatible with 5.3.6 and higher. This'll be fixed soon bringing it up-to-date with LipSync Pro.

 

Quick update:
A new blend system for using PolyMorpher in place of blend shapes is now available from the Extensions window - thanks a lot @kalagaan for the help!

 

It's me (again) I tried using Blendshapes system (I always had used bones transforms). With shapes I have this bug.
When I use either phonemes in Lipsync Pro or blinking in EyeController, it happens that only the firts blendshape I add and modify its values works, I mean... any other blendshape I add after the first will not actully modify the mesh and when I delete the shapes after the bug had started I cannot used the phoneme panel to add anything else (blendshapes, bones transforms or animations clips)

Code (CSharp):

1. NullReferenceException: Object reference not set to an instance of an object
2. RogoDigital.Lipsync.BlendshapeBlendSystem.SetBlendableValue (Int32 blendable, Single value) (at Assets/Rogo Digital/LipSync/BlendSystems/BlendshapeBlendSystem.cs:55)
3. EyeControllerEditor.OnInspectorGUI () (at Assets/Rogo Digital/LipSync/Editor/EyeControllerEditor.cs:465)
4. UnityEditor.InspectorWindow.DrawEditor (UnityEditor.Editor editor, Int32 editorIndex, Boolean rebuildOptimizedGUIBlock, System.Boolean& showImportedObjectBarNext, UnityEngine.Rect& importedObjectBarRect) (at C:/buildslave/unity/build/Editor/Mono/Inspector/InspectorWindow.cs:1235)
5. UnityEditor.DockArea:OnGUI()
6.  

Also, when I use the Classic options of Blinking in Eyecontroller (blendshapes)... well, the characters doen't blink when I click Play, and this next bug happens cyclically.

Code (CSharp):

1. NullReferenceException: Object reference not set to an instance of an object
2. RogoDigital.Lipsync.BlendshapeBlendSystem.SetBlendableValue (Int32 blendable, Single value) (at Assets/Rogo Digital/LipSync/BlendSystems/BlendshapeBlendSystem.cs:55)
3. RogoDigital.EyeController.LateUpdate () (at Assets/Rogo Digital/LipSync/Components/EyeController.cs:259)
4.  

I haven't had any other issue with Bone Transforms... It only happened that I want to used blendshapes

 

Have you got an empty item in the "Optional Other Meshes" list? It looks like there's at least one item in the list, but one or more of them is null. I'll make this fail a bit more gracefully in 1.3!

 

Update time!

LipSync Pro 1.3 is nearly finished! I'll include the change list at the end as usual (bear in mind, it might change slightly before the final version), but first here's a quick preview of the new Emotion Mixer feature:

​

Secondly, we're launching two new programmes: LipSync Betas, and Powered by LipSync:

With LipSync Betas, you can get new versions of LipSync Pro before they're released in the Asset Store or on the site. In return for feedback on new features, usability or bugs, you'll get the chance to receive new Rogo Digital products, tools and games for free! More info on this coming soon - you'll have the chance to sign up on the website in a few days' time.

​

Powered by LipSync is a new section of the website highlighting interesting projects that make use of LipSync Pro or Lite.​

The aim is to bring some extra exposure to games and software using LipSync - anything and everything from small one-person game jam projects to serious games and training tools are welcome. We'll also be doing promotion on YouTube and social media.
You can sign up for Powered by LipSync now (and you can choose how and where your project will be shown) by filling out this Google form: https://goo.gl/forms/KrUEthZ2kqUov40n1

1.3 Change List:

Features
- AutoSync Improvements:
- New High Quality preset trades off longer processing time for better results.
- More advanced settings are exposed for customisation (Make sure you read up on PocketSphinx before changing these!)
- Automatic Cleanup option can do a basic 'thinning out' of markers on the timeline.
- Emotion Mixer - New type of Emotion marker allows you to blend multiple Emotions together simultaneously.
- Modifiers for more natural motion:
- Continuous variation (Emotions) - create subtle, automatic variance over the course of an emotion.
- Randomness (Phonemes) - add random variations between multiple identical phoneme markers.
- Phoneme Sets - You can now customise the set of phonemes used. Standard Preston Blair set is included by default.
- Rest phonemes can now be added directly to LipSyncData files. (Automatic rests are still used as well).
- Transitions out of animations are now smooth (e.g. when Stop is called).
- Blendables can now be selected using a fuzzy search as well as selection from the drop-down.
- XML Update Tool - run from the Window/Rogo Digital/LipSync Pro menu, will update any LipSync XML files to work with 1.3.
- New Import Manager - Simplifies the process of importing LipSync Pro into an existing project

Fixes
- AutoSync now merges split clips correctly (Files longer than 30 seconds now work)
- (Regression in 1.2) Dragging markers too quickly no-longer results in a rect selection instead.
- Marker intensities now correctly affect bones as well as blendables.
- Marker sustain is now correctly re-loaded in the Clip Editor.
- XML Export:
- Intensity is now correctly exported.
- Emotions element is properly ended now.
- No more NullReferenceException when exporting with no AudioClip.
- XML Import (Editor):
- No-longer fails when loading old XML files missing some newer attributes or elements.
- All metadata is now loaded.
- AudioClip is now optional.
- Improved Project Settings duplicate name detection. Now shouldn't be possible to have two identically named emotions or gestures.
- Preview in the LipSyncData inspector now plays at the correct speed with no AudioClip.
- File auto-update message now only appears when data has actually been changed.
- Additional renderers in the BlendShapeBlendSystem that are not assigned are now ignored instead of triggering a NullReferenceException.

Changes
- The 'Rest' pose (if present) is now used in place of neutral (blendable value 0, or bone's default position) in animations.
- Phoneme enums are now obsolete. Use phonemeName or phonemeNumber fields (where appropriate)
- Enabled multi-threading support in AutoSync where available.
- Simplified runtime XML parser - no longer supports loading of pre-1.3 XML files, these must be updated first.
- 'Audio Playback' timing mode is no-longer selectable when using WebGL to prevent confusion.
- Moved creation of new Blend Systems to Create/LipSync Pro menu.
- Warning text on shared blend systems now names the components sharing it.
- Changed the data loss warning when creating a new file to be consistent with the others.

 

Hey @Rtyper any ETA or teases to show off for LipSync Live? Like I said, this would be a useful tool to have. Also what would a likely price be? (Pls not over $60 )

 

@Rtyper Looking forward to version 1.3!

I'm still having problems with random looking (PoseBased). Nothing seems to be overriding the blend shapes. Any ideas? Alternatively, could I send you the model, so you could take a quick look?

I'm not at home now, but it says that some part of random looking doesn't work when PoseBased is selected. Is this something that will be changed?

As I said in an earlier post I'm using the ManuelBastioniLabs plugin for blender as the base for my characters (http://www.manuelbastioni.com).

 

Quick question: is this fast enough (on desktop) to work off a microphone, or does it need a lot of preprocessing?

I want to animate a player avatar based on in-game chat.

 

It needs LipSync Live to generate off a microphone. That's what I'm waiting for - it's coming "Q4 2016" - source is here

Matt

 

No, I'm afraid not. The system is geared towards creating higher quality animation than is possible through automatic solutions. There's no way to auto-detect the emotions, only phonemes (and even then, not perfectly), so clips have to be created in the editor beforehand.

As for realtime solutions, the main one currently available is SALSA. LipSync Live, as @Stormy102 mentioned was going to be our own solution for it, but SALSA improved a lot since LipSync Live was announced, so I don't really see much of a gap for it to fill unless we come up with something big that sets it apart. I'll be removing references to it from the site in the next day or two when it gets updated, sorry for not making this clearer earlier!

Sorry to hear that, Corvwyn! If you could send the model, that'd really help with looking into the problem. Could you maybe export the model, along with the character's setup as a prefab to a .unitypackage and send that to contact@rogodigital.com? The more info I have about the issue, the better!

The message about pose-based mode in Eye Controller is referring to the look at target functionality, which needs the two bones from classic mode to point at a target, so it has to be switched off in pose mode.

 

Shame... Competition usually gets the best out of Unity Asset Store publishers.
I was hoping to get LipSyncLive and Eye Controller bundled together. Guess I'll buy EyeController separately. Glad you made it purchasable as a separate product Looks pretty damn good!

 

@Rtyper, is there anything preventing you from adding optional lower quality realtime sync to LS Pro then? It could be this "something big that sets it apart"

 

The new update to lipsync.rogodigital.com went live earlier! You can now sign up to beta versions or access your downloads by clicking on the "Account" button in the top left, or click the "Documentation" button to check out the new, online documentation for LipSync. Bear in mind it's still being worked on, so not all pages have been created yet!
The 1.3 update was also submitted to the store yesterday, so with a bit of luck that'll be available in the next few days.

It's not really possible, unfortunately. The system LipSync Pro uses always has some delay to it, and it has to have the complete audio available (which rules out working on mic input). I'll post here about any developments though!

 

Hi, I am using Daz3d characters in unity. I tried LipSync lite to test the functionality but was unsuccessful. Does LipSync work with Daz3d characters? If so can you tell me what export settings I need to have.

Thanks

 

@Rtyper : thanks for the pointer to SALSA. Yours has higher quality, but even SALSA is better than what I do now (which is to randomly toggle between consonant and vowel whenever the volume is on -- looks odd when someone goes "hmmm" and the character mouths "banana").

 

any news about the new version?

 

Afraid not - it's ready and been submitted, but we're just waiting on Unity to approve it now. I'll try to get it up on lipsync.rogodigital.com later today, so if it's not in the Asset Store, check back there.

 

@Goldrake Scratch that, UT must be psychic!

LipSync 1.3 is now available from the Asset Store! The final change list can be found at the end (there's a couple of differences from the earlier one).

Apologies to anyone who tried to get on the website or access the extensions or download systems late last week! We had some trouble with our webhost and the server was down for the best part of a day - everything should be back to normal now, but let me know if you're still having problems.

Cheers!

1.3 Final Change List

Features
- AutoSync Improvements:
- New High Quality preset trades off longer processing time for better results.
- More advanced settings are exposed for customisation (Make sure you read up on PocketSphinx before changing these!)
- Automatic Cleanup option can do a basic 'thinning out' of markers on the timeline.
- Emotion Mixer - New type of Emotion marker allows you to blend multiple Emotions together simultaneously.
- Modifiers for more natural motion:
- Continous variation (Emotions) - create subtle, automatical varience over the course of an emotion.
- Randomness (Phonemes) - add random variations between multiple identical phoneme markers.
- Phoneme Sets - You can now customise the set of phonemes used. Standard Preston Blair set is included by default.
- Rest phonemes can now be added directly to LipSyncData files. (Automatic rests are still used as well).
- Transitions out of animations are now smooth (e.g. when Stop is called).
- Clips with audio can now be shortened from the Clip Settings window (relies on SoX for trimming the AudioClip).
- Blendables can now be selected using a fuzzy search as well as selection from the drop-down.
- XML Update Tool - run from the Window/Rogo Digital/LipSync Pro menu, will update any LipSync XML files to work with 1.3.

Fixes
- AutoSync now merges split clips correctly (Files longer than 30 seconds now work)
- (Regression in 1.2) Dragging markers too quickly no-longer results in a rect selection instead.
- Marker intensities now correctly affect bones as well as blendables.
- Marker sustain is now correctly re-loaded in the Clip Editor.
- XML Export:
- Intensity is now correctly exported.
- Emotions element is properly ended now.
- No more NullReferenceException when exporting with no AudioClip.
- XML Import (Editor):
- No-longer fails when loading old XML files missing some newer attributes or elements.
- All metadata is now loaded.
- AudioClip is now optional.
- Fixed bug causing gestures becoming out-of-sync between components and project settings.
- Corrected Eye Controller pose-based looking mode so smoothing works correctly.
- Improved Project Settings duplicate name detection. Now shouldn't be possible to have two identically named emotions or gestures.
- Preview in the LipSyncData inspector now plays at the correct speed with no AudioClip.
- Extensions Window now works even when Rogo Digital folder has been moved.
- File auto-update message now only appears when data has actually been changed.
- Additional renderers in the BlendShapeBlendSystem that are not assigned are now ignored instead of triggering a NullReferenceException.
- Gesture Setup Wizard no longer changes default state in existing layers.
- Fixed edge-case Emotion marker rendering issues (editor)
- Behind-the-scenes Project Settings inspector improvements.

Changes
- The 'Rest' pose (if present) is now used in place of neutral (blendable value 0, or bone's default position) in animations.
- Phoneme enums are now obsolete. Use phonemeName or phonemeNumber fields (where appropriate)
- Enabled multi-threading support in AutoSync where available.
- Simplified runtime XML parser - no longer supports loading of pre-1.3 XML files, these must be updated first. -
'Audio Playback' timing mode is no-longer selectable when using WebGL to prevent confusion.
- Moved creation of new Blend Systems to Create/LipSync Pro menu.
- Warning text on shared blend systems now names the components sharing it.
- Changed the data loss warning when creating a new file to be consistent with the others.

 

Hello - I have a question about using LipSync Pro with Morph 3D. My question is there's a preset for the Phenoms for Adobe Fuse but do not see one in there for Morph3D - even though I downloaded the Morph3D extension for LipSync Pro. I've tried to get the phenoms to match the examples, however the final result does not look natural at all. The Lincoln example is adequate with what I'm looking for in accuracy but when trying to setup the phenoms with Morph3D, its even strange when accessing the blendshapes to use.

Are there any presets we can use for Morph3D or how can this be a smoother process to get the phenoms for this to look good like it does with the Maximo defaults? Getting Morph3D to look good out of the box would help tremendously!

 

Sorry if this has been asked, but I can't find a solution. I'm using LP1.3, Unity5.4.1f and Morph3d MCSFemale Pro. In the clip editor, after using AutoSync everything seems fine. But when I go to Preview with the character, I get 999+ errors of the following:

I can save the asset and use it, it just won't preview without this error. Any help?