Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Discussion in 'Assets and Asset Store' started by Crazy-Minnow-Studio, Apr 23, 2014.

  1. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    We have submitted the end of our current sale to the Unity Asset Store, last chance to add procedural lip sync, random and programmatic eye movement, eye tracking, and emotional expressions(3D), to your 2D and 3D characters for only $10. Thanks to everyone who participated in the sale already!

    The Crazy Minnow Team
     
    Last edited: Oct 14, 2014
  2. atmuc

    atmuc

    Joined:
    Feb 28, 2011
    Posts:
    1,152
    is it compatible with mobile devices?
    is it possible to record animation to use it non realtime?
    do you give full source code?
     
  3. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello atmuc. Both assets work with mobile devices. Of course, your mileage will vary. Just as with anything else, you have to optimize your models and scenes. But SALSA and RandomEyes both run fine.

    Both SALSA and RandomEyes are animation playback systems. They do not record animations. Assuming you're asking this question with mobile processing in mind, both SALSA and RandomEyes are pretty lightweight as far as processing is concerned.

    The assets are provided as compiled libraries, offering the benefit of using whichever scripting library you would like to use with them. Each has a full API available, with a broadcasting system for notification.

    Hope that helps!
     
  4. atmuc

    atmuc

    Joined:
    Feb 28, 2011
    Posts:
    1,152
    i consider audio processing cost on mobile devices. i tested your demo scene with my note 3. it works fine but note 3 is 8 core 3 gb ram device. you mean audio processing cost is low.

    compiled libraries have problems when compiling multiplatforms.they require more support. when an asset has full code i can easily fix it without waiting publisher.

    For example your asset is not compatible with Windows Store Apps. for this reason i quit using Playmaker. what platforms have you tested your asset?

    Reference Rewriter: Error: method `System.Reflection.Assembly System.Reflection.Assembly::GetExecutingAssembly()` doesn't exist in target framework. It is referenced from SALSA.dll at UnityEngine.Texture2D CrazyMinnow.SALSA.SalsaUtility::LoadDllResource(System.String,System.Int32,System.Int32).

    UnityEngine.Debug:LogError(Object)
    PostProcessMetroPlayer:RunReferenceRewriter(MetroSDK, String, String, String, String) (at C:/BuildAgent/work/d63dfc6385190b60/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:483)
    PostProcessMetroPlayer:postProcessSDKSpecific(MetroSDK, BuildTarget, BuildOptions, String, String, ProjectImages) (at C:/BuildAgent/work/d63dfc6385190b60/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:1001)
    PostProcessMetroPlayer:postProcess(BuildTarget, BuildOptions, String, String, String, String) (at C:/BuildAgent/work/d63dfc6385190b60/PlatformDependent/MetroPlayer/Extensions/Managed/PostProcessMetroPlayer.cs:734)
    UnityEditor.HostView:OnGUI()
     
  5. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello atmuc. Please send your invoice number to our support email and we'll try to get this fixed for you. assetsupport@crazyminnow.com
     
  6. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Current status on the Windows platform error: we provided a package for Atmuc to test shortly after we received contact information. A build was successfully tested on a Lumia phone. We were able to successfully test against Win Phone and Apps. Atmuc did have an issue building to the Windows App Store and is performing some additional testing to confirm.
     
  7. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    The latest information we received from Atmuc is that builds are successful on Windows mobile platforms. We're submitting a fix to the Asset Store that includes the patch we provided to Atmuc for testing on the 15th. In our testing and his, this resolves the errors originally posted.
     
  8. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    1,409
    Thanks for your helpful tutorials! I left a review.

    Here's my first attempt. Sorry, the eyes aren't hooked up, and the text is a bit of a yawn, but this was easier to do than I thought it would be!

     
  9. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    wetcircuit,

    Awesome example, and a great looking model. We love BlendShapes too, they make it so much easier to add a lot of expressions. We're glad our tutorials helped, and we can't wait to see more with the eyes hooked up. As you mentioned in your review, amplitude is definitely an important factor, a broad spectrum of high peaks and low valleys in the wav form provide a more dynamic performance. Thanks for the review!

    The Crazy Minnow Team
     
    Last edited: Nov 10, 2014
    wetcircuit likes this.
  10. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    1,409
    Hi, I am trying to test SALSA in Unity5beta.... I get some warnings that SALSAEditor.dll and SALSA.dll use obsolete Unity API..., and Unity5 offers to "upgrade" them.

    What do you advise? Have you had a chance to test with U5 yet?
     
  11. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Assuming it's only API obsolescence warnings, it should not prevent testing, but we have not tested it with Unity 5 yet. Is it working in v5? Any API code Unity deprecates in v5 will be updated in SALSA and RandomEyes once v5 is final. Can you post the warnings and/or errors?

    Thanks
     
  12. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    1,409
    Well, I am not a Unity expert so I'm not sure I am the best person to trust...., but loading a fresh asset into a new scene seems to work fine (yay)! At least SALSA3D and RandomEyes3D, that's all I tested.

    Opening an old scene from U4 did not work, and there were warnings. The scene opened but the mouth did not move.



    Loading a fresh asset in U5 gave this warning:

    next part is odd.... I think I tried to add the component from the ADD COMPONENT button but it was just the title. No interface (maybe this is the wrong way to load it):


    Later after fussing around in the scene (open close, etc) I tried adding the component from the Component menu and it is working.
     
  13. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Thanks for the info. We're glad to hear it works with a fresh asset in a new v5 scene. I would not recommend opening your previous Unity projects with a beta version of Unity unless you're just testing and have backups of these projects. Adding the component from the Add Component button or the Component menu are both correct, this issue may be a v5 incompatibility. It Looks like Unity has definitely made some big API changes. We will update SALSA with RandomEyes to reflect the changed API once Unity 5 is official. In the meantime, we recommend using SALSA with RandomEyes with any release version after 4.3.
     
    wetcircuit likes this.
  14. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    We're launching a new marketing campaign to demonstrate the quality of SALSA with RandomEyes for lip sync, eye movement, and facial animation compared to AAA games already on the market.

    Check out our first SALSA vs AAA Lip Sync promotion video.
     
  15. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    The good folks over at Pixel Crushers have released version 1.4.0 of Dialogue System for Unity and they have some very nice words about their assessment of SALSA with RandomEyes. They liked it enough to add 3rd party support for SALSA into their product! Read their blog post about this release.

    If you check out the product page of their awesome app, you'll notice that it's garnered much praise and lots of (★★★★★) ratings! As you can imagine, we're super excited about the integrated support in Dialogue System for Unity. It's an awesome pairing of assets that give indie projects a AAA feel.​
     
    Last edited: Jan 19, 2015
  16. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    We've had quite a few inquiries on the subject of using a microphone as a real-time input source for SALSA. Previously, we pointed customers to a 3rd-party asset called MicControl to get it all working. Unfortunately, using the MicControl asset required some fiddly editing their script to get it all to link up with SALSA. When the last update to MicControl somewhat broke our how-to doc, we decided it was time to implement our own helper-script for real-time microphone input. Read more about it and get the new scripts.
     
    Last edited: Jan 19, 2015
  17. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    SALSA with RandomEyes version 1.2.1 has been released on the Asset Store. This is a fix/tweak release with no new functionality. This release does include modifications which should allow for successful builds on Windows Store and Windows Phone 8 platforms.
     
  18. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    We've put together a tutorial video (below) for integrating Mixamo's Fuse Character Creator models into the SALSA with RandomEyes workflow. Please visit our website to download the support resources.

     
  19. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Last edited: Jan 19, 2015
    wetcircuit likes this.
  20. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398

    PlayMaker Actions for SALSA with RandomEyes

    If you've already purchased SALSA with RandomEyes, we'd like to thank you with this free Playmaker add-on that includes 45 custom Playmaker Actions. If you haven't already purchased SALSA with RandomEyes, we'd like to sweeten the deal with this free Playmaker add-on that includes 45 custom Playmaker Actions. ;)

    http://crazyminnowstudio.com/posts/playmaker-actions-for-salsa-with-randomeyes/
     
    wetcircuit likes this.
  21. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
  22. steveR

    steveR

    Joined:
    Jul 14, 2013
    Posts:
    33
    Hi:

    This looks like a great solution especially for people with a Mac. The other solution is Papgayo, but requires manually placing the words with the audio. I am using AdventureCreator , which is a terrific product -- amazing really. The developer has added Papagayo support , but this looks like a more powerful solution. The developer is continuously adding great features. It would be great if there could be support for this product - as discussed in the forums. Perhaps you could work with the developer.

    Thanks
     
  23. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello steveR,

    Thanks for the tip, and the links, we'll communicate with the Adventure Creator developer and evaluate future integration. Our solution is definitely focused on making lip sync as simple as possible, and does not require manual phoneme mapping to audio.
     
  24. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    TonyLi likes this.
  25. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
  26. llde_chris

    llde_chris

    Joined:
    Aug 13, 2009
    Posts:
    205
    Hi Crazy Minnow.

    There's a bug with the latest version of SALSA on Unity 5.
    We have a large scene, a 3 km city with tens of thousands of GOs, and we noticed that it suddenly became very slow to edit. We've traced it to SALSA's UpdateIcon calls doing GetComponent calls every frame at edit time and heavily stalling the editor. I've attached the Editor Profiler screenshot.

    It works fine at runtime, but our scene is uneditable while SALSA is installed.

    Do you have a fix for this?

    Thanks.
     

    Attached Files:

  27. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hello llde_chris,

    Sorry for the inconvenience, but thanks for the detailed report. We'll look into this, see what we can find, and get back to you soon.

    Thanks,
    The Crazy Minnow Team
     
    Last edited: Jan 30, 2015
  28. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    llde_chris,

    We have identified the problem. It's in the editor code, which is why it's not presenting a problem at runtime. Please send us an email at assetsupport@crazyminnow.com with your invoice number and include a Gmail address, if you have one, so that we can share a patched version with you once it's ready.

    Thanks,
    The Crazy Minnow Team
     
  29. llde_chris

    llde_chris

    Joined:
    Aug 13, 2009
    Posts:
    205
    Awesome.

    Sent you email with our invoice number. Didn't notice you preferred Gmail though, should I resend it again with gmail?
     
  30. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Gmail might make it easier to share a patched download with you over GDrive. Thanks!
     
    Last edited: Jan 30, 2015
  31. llde_chris

    llde_chris

    Joined:
    Aug 13, 2009
    Posts:
    205
    Done. Thanks for the quick replies!
     
  32. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Chris,

    I have sent a patched version to your gmail address. Please let us know if this solves your issue and we'll include it in the next Unity Store update.

    Thanks,
    The Crazy Minnow Team
     
  33. llde_chris

    llde_chris

    Joined:
    Aug 13, 2009
    Posts:
    205
    The last update did the trick. Editor performance is back up to 60FPS.

    Great work guys!

    Thanks,
    Chris
     
  34. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    That's great news Chris! Thanks for your patience, and helping us get to the bottom of the issue with the detailed diagnostics.
     
  35. AlteredPlanets

    AlteredPlanets

    Joined:
    Aug 12, 2013
    Posts:
    455
    Hello,

    Anyone knows how to prepare a default biped for SALSA?
     
  36. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    For 2D characters you'll need four mouth shape sprites: rest, small, medium, large.
    For 3D characters you'll need three BlendShape mouth shapes: small, medium, large, rest is the basis shape.

    Please see this forums first post for an overview, or see our online manual for more details.
    http://crazyminnowstudio.com/projects/salsa-with-randomeyes-lipsync/manuals/

    If you're not familiar with BlendShapes, we have a three part video series covering the topic in Blender.
    http://crazyminnowstudio.com/categories/blender/

    We also have a quick start video series on youtube.
    http://m.youtube.com/#/playlist?list=PLcVmXGedVLuYwkvHkqWYUu913q8YSsBzq
     
  37. AlteredPlanets

    AlteredPlanets

    Joined:
    Aug 12, 2013
    Posts:
    455
    what about blend shapes in 3ds max?
     
  38. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    As long as the blend shapes are recognized by Unity, they should be good to go. They do not need to be created in Blender.
     
  39. kkmartin

    kkmartin

    Joined:
    Feb 11, 2015
    Posts:
    3
    Hi Crazy Minnow,

    I followed a tutorial in your website called "Unity3D Lip Sync Using and Mixamo Fuse Character Creator" for lip and eye movement to a character model. Using Blender 2.73, I attached and ran the python script for the mouth shapes then exported the model as a .fbx package and a .blend format. When I imported the .fbx and .blend package to Unity, the character model's skinning was changed, resulting to deformation near some joints when animating (ex. the armpit of my character model collapsed and stretched). If I were to fix the shapes in Maya, I would end up losing the blend shapes. Is there an alternative on how I can attach the lip movement script to a character model?
     

    Attached Files:

  40. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hi kkmartin,

    Thank you for your purchase! Let me start with a little background on the python script. Sorry if some of this is repetitive from our website, but I'm not sure how much of it you've seen. As you know, the Fuse models have a lot of BlendShapes associated with them. Unfortunately, two of the SALSA shapes (saySmall and sayMedium) aren't represented well with the standard Fuse shapes. However, the SALSA shapes can easily be blended from derivative mixes of the existing shapes, this is what the python script does automatically. Because you have Maya, you could skip the Blender/Python step and use the BlendShape value mix listed below to create SALSA's saySmall, sayMedium, and sayLarge shapes directly in Maya. This also allows you to fine tune the SALSA shapes to your own preference. We recommend always creating the shapes with a lot of enunciation, since you can reduce the range of motion from within the SALSA Inspector, and through the API. It provides more flexibility to increase the range of motion when a character is yelling and reduce it when they're talking or whispering. This should get you up and running, however, we'd still like to have a look at your pre-Blender/Python model if you don't mind sharing it so we can try to determine the issue when applying the python script. If you decide to send the model, please send it and your invoice number to assetsupport@crazyminnow.com.

    Thanks,
    The Crazy Minnow Team

    Fuse BlendShape mixes to create SALSA shapes:

    saySmall
    shape: MouthNarrow_Left
    value = 0.25
    shape: MouthNarrow_Right
    value = 0.25
    shape: MouthWhistle_NarrowAdjust_Left
    value = 1
    shape: MouthWhistle_NarrowAdjust_Right
    value = 1
    shape: UpperLipOut
    value = 0.25
    shape: Jaw_Down
    value = 0.25

    sayMedium
    shape: UpperLipIn
    value = 1
    shape: UpperLipUp_Left
    value = 0.65
    shape: UpperLipUp_Right
    value = 0.65
    shape: UpperLipDown_Left
    value = 0.65
    shape: UpperLipDown_Right
    value = 0.65
    shape: Smile_Left
    value = 0.45
    shape: Smile_Right
    value = 0.45
    shape: Jaw_Down
    value = 0.1
    shape: MouthUp
    value = 0.3
    shape: MouthOpen
    value = 0.25
    shape: MouthDown
    value = 0.25
    shape: TongueUp
    value = 0.4

    sayLarge
    shape: MouthOpen
    value = 0.75
     
    Last edited: Feb 11, 2015
  41. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hi kkmartin,

    We haven't heard from you, so we're just checking to see if our previous post provided a sufficient work around to your issue, or if you have additional questions?

    Thanks,
    The Crazy Minnow Team
     
  42. kkmartin

    kkmartin

    Joined:
    Feb 11, 2015
    Posts:
    3
    Hi Crazy Minnow,

    Thank you for the help but I seem to have another problem. After editing the existing model to add the SALSA shapes in Maya, when I export the new model to Unity it added prefixes to different blend shapes so it gives the parts a new name. With the new name, the SALSA shape have "duplicates" of some sort because of the different name labels. For example, the blendshapes of the eyebrow part of my model, the blink_left becomes ncl1.blink_left while its called ncl2.blink_left in the body.
    How can I export the new model without having prefixes in its labels?
     
  43. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    Hi kkmartin,

    We want to help you as much as possible, but we don't own or use Maya, so Maya-specific assistance is going to be limited. That being said, we have a few ideas on how we might be able to solve the problem. Please send an email to assetsupport@crazyminnow.com and include your invoice number, so we we can discuss in more detail through email.

    Thanks,
    The Crazy Minnow Team
     
    Last edited: Feb 14, 2015
  44. kkmartin

    kkmartin

    Joined:
    Feb 11, 2015
    Posts:
    3
    Alright thanks!
     
  45. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    1,409
    I was looking at FUSE since it's on sale, and then checking your blog for Fuse compatibility.... Sorry to be dense, but is it still necessary to take the Fuse model into Blender to create the 2 mouth blendshapes?

    If so, I think I have a feature request: "blend groups"

    They way it would work is you can manipulate the blendshapes, then create a "blend group" which can substitute for individual shapes. It works just like an individual morph but instead fires all the required morphs to make your custom shape.

    for instance:
    blend group "smile": (mouth left corner up, mouth right corner up, lips part, eyes squint, etc)
    or
    blend group "mouth small": (your blend shape formula for the custom Fuse shape that is added by the python script)

    In SALSA and RealEyes you would be able to substitute a custom blend group for any blendshape. This would allow RealEyes to trigger full expressions, rather than just quirky tics and disconnected morphs.
     
  46. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    We have been talking a lot about this subject lately. In our free Playmaker addon, we provide a custom action called RandomEyes3dExpression that does exactly this, and we have been considering either integrating a feature like this in RandomEyes or in a free workflow script. Our primary consideration is to remove the need for Blender interaction and gain the ability to drive custom shape scenarios directly. Thanks for your input, we'll keep the forum updated as we progress on this topic.
     
    wetcircuit likes this.
  47. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,398
    By the way, if you're using our API, you can already create your own blend groups by wrapping multiple custom shape function calls into your own function. This doesn't address a blend group scenario for Salsa shapes, but it does for manual control over Custom Shapes. Here's a simple example:

    Code (CSharp):
    1. void Smile(float duration)
    2. {
    3.     randomEyes.SetCustomShapeOverride("mouth left corner up", duration);
    4.     randomEyes.SetCustomShapeOverride("mouth right corner up", duration);
    5.     randomEyes.SetCustomShapeOverride("lips part", duration);
    6.     randomEyes.SetCustomShapeOverride("eyes squint", duration);
    7. }
     
  48. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    1,409
    Ahh, I see. This similar for your Playmaker expressions? I downloaded the actions but have not been working on a character project, have not had a chance to explore what they can do....

    Thank you so much! Anything that can help me avoid Blender is a wonderful gift. ;)
     
  49. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    1,409
    Thanks for the advice with the Fuse figures... They look pretty good with some realeyes expressions....

    But I am having a problem... I am not able to assign a new audioclip to SALSA with Playmaker...

    Set Property seems to be the most direct solution, which does change the clip in the inspector but I cannot hear it (and the mouth doesn't move). Actually, the only way I can get Salsa to work is to preset the audioclip and set the audiosource to Play on Wake...

    Do you have any advice?



    webplayer is here:
    http://www.3d.wetcircuit.com/Unity3D/HipChick/HipChick.html
     
  50. wetcircuit

    wetcircuit

    Joined:
    Jul 17, 2012
    Posts:
    1,409
    Also, when I unzip your Playmaker actions I get a folder with many folders inside, not the expected unityPackage...