1. Help us improve the editor usability and artist workflows. Join our discussion to provide your feedback.
    Dismiss Notice
  2. We're looking for feedback on Unity Starter Kits! Let us know what you’d like.
    Dismiss Notice
  3. We’re giving 2017.1 beta testers a chance to win t-shirts and a Nintendo Switch. Read more on the blog.
    Dismiss Notice
  4. We want to know how you learned Unity! Help us by taking this quick survey and have a chance at a $25 gift card
    Dismiss Notice
  5. Are you an artist or level designer going to Unite Europe? Join our roundtables there to discuss artist features.
    Dismiss Notice
  6. Unity 5.6 is now released.
    Dismiss Notice
  7. Check out all the fixes for 5.6 on the patch releases page.
    Dismiss Notice

SALSA with RandomEyes - Lip sync and expression system

Discussion in 'Assets and Asset Store' started by Crazy-Minnow-Studio, Apr 23, 2014.

  1. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    store_big_image_unity_forum_banner.jpg


    Website: CrazyMinnowStudio.com
    Product site: SALSA with RandomEyes
    Support email: assetsupport@crazyminnow.com
    Youtube: videos

    Version 1.5.0
    Available Now: Asset Store
    Release Notes

    Works with:
    Includes two comprehensive products to simplify lipsync, eye movement, and facial expressions:

    SALSA
    Simple Automated Lip Sync Approximation provides high quality, language-agnostic, lip sync approximation for your 2D and 3D characters using Sprites or BlendShapes. Audio dialog files are processed in real-time to automate the lip sync process. Results are superior to basic jaw bone animations, without labor intensive phoneme mapping.

    RandomEyes
    Included with SALSA, RandomEyes provides random and programmable eye movement, blinking, and eye tracking, on any number of eyes for your 2D and 3D characters using Sprites or BlendShapes.

    RandomEyes for 3D characters also includes:
    Custom Shapes - A procedural facial animation system that allows fully automated or manually controlled emotional expression.
    Shape Groups - Combine multiple BlendShapes into complex, multi-shape, facial expressions right in the editor.

    Both systems include options to send event-based responses for speech and custom shapes to specified GameObjects.

    Custom component inspectors provide code-free configurable options, while our simple API provides code accessibility from any Unity supported scripting language. Manuals, sample scripts (C# and UnityScript), and several demo scenes are included.

    Supports Unity & Unity Pro. Tested on Windows, Mac, Linux, legacy Web-Player, Android, IOS, Windows Store (8 & 10 .NET Core), WP8, and WP10.
     
    Last edited: Mar 29, 2017
    Jingle-Fett likes this.
  2. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    Some questions before I need to sign up for your forums:

    When you say you support morph targets from Blender, Maya, Maya LT, Modo, etc is there a means of assigning the proper morph targets in the FBX model to your SW control or does you SW search and find the morph targets because they all the same names and identifying information?

    I'll have morph targets exported both from Blender and Maya LT. No problem?

    Do you have pictures of how we should do our morph targets (shape keys / blend shapes) do have minimal fuss using your SW?

    Thanks.
     
  3. nasos_333

    nasos_333

    Joined:
    Feb 13, 2013
    Posts:
    6,071
    Can it synch lip motion by text ?

    If there is no voice available for a dialog for exaple
     
  4. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    Goat,

    SALSA achieves quality results using four defined mouth positions: sayRest, saySmall, sayMedium, sayBig (these shapes are outlined in the forums and covered in detail in the manual). When you import your FBX with these BlendShapes into a Unity scene, Unity adds a Skinned Mesh Renderer component to you models GameObject. All of your BlendShapes are in this component. For 3D characters, you add our components to the same GameObject as the Skinned Mesh Renderer. SALSA lets you link to these four shapes by entering their index number into the corresponding index field in our custom inspector. These four indexes and an audio clip are all that's required for SALSA to work.

    RandomEyes works the same way, but uses five positions: lookUp, lookDown, lookLeft, lookRight, and blink.

    As for morph target support, as long as you can see the BlendShapes in the Skinned Mesh Renderer component after importing into a Unity scene, then SALSA and RandomEyes can link to those indexes. The order of the indexes doesn't matter, since you can set the index values in our custom inspectors.

    I hope I answered all of your questions.
     

    Attached Files:

    Last edited: Apr 23, 2014
  5. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    nasos_333,

    SALSA performs lip sync approximation by analyzing audio only.
    Sorry, no text analysis.
     
  6. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    Hello Goat!
    Here's a little more clarification on some of your questions. To minimize your workflow efforts with SALSA, the component will automatically assign indexes for shape keys as follows (matching the mouth positions identified in the screenshot of our previous reply):
    - sayRest: 0
    - saySmall: 1
    - sayMedium: 2
    - sayBig: 3

    Likewise, RandomEyes automatically assigns indexes 4, 5, 6, 7, and 8 as:
    - LookUp: 4
    - LookDown: 5
    - LookLeft: 6
    - LookRight: 7
    - Blink: 8

    Keep in mind, these automatic assignments are easily changed to match your model information. We'll try to post up some more details and screenshots later today.

    There also is no issue with using exports from Blender and MayaLT. They should work together just fine.

    NOTE: When working with BlendShapes (at least with Blender), make sure to apply any modifiers *prior* to creating your BlendShapes. Additionally, modifying the mesh (adding/removing verticies) after creating your shapes, generally results in broken BlendShapes.
     
  7. adamz

    adamz

    Joined:
    Jul 18, 2007
    Posts:
    765
    This looks really good! I was looking for another solution instead of the pricey FaceFX.

    A couple of questions; can you create additional morph targets to achieve more accurate results?

    Can this also be setup with using text-to-speech from this post? If you can get your system to work with that I'm all over it!
     
  8. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    This does indeed look a lot lottt cheaper than the big name alternatives, like faceshift and such, and a lot less hassle - might not be accurate but 4 blendshapes instead of 1-2 dozen is so much easier and lighter and in this case, a hell of a lot cheaper. I'm planning to use daz/poser models as the art is getting excellent and very flexible, so i'm assuming i do my decimation in daz studio rather than whatever i'm importing it to. I suppose i could always test.
     
  9. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    Thanks. I'll be buying.

    The 4 lip syncs is good if you are not learning lip reading. This is an old project on Blendswap that uses all the phenomes of speak (InfameDavid) or there is, I think it's called targarela to base off of.

    The other question I have need clarification for although I think you answered it above is when I make Blender models (I am still learning Maya) I often use subdivision to help guide me into using good topology however I leave subdivide unapplied when exporting as FBX because it makes the polygons too high. So to work with Salsa I will need to leave subdivision either turned off or delete that modified before exporting as an FBX.

    In the future I'd like to see a whole gamut of emotions expressed such as the attached picture shows but honestly I'll have to create all the blendshapes and see how big that would make the file. It might make it too big complex for mobile.
     

    Attached Files:

    Last edited: Apr 23, 2014
  10. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    Hi Goat,
    Yes, SALSA's main goal is to provide great results with as little effort as possible. Making all of the phoneme shapes necessary for super-realistic mapping is very daunting and is not required of most game projects. We're trying to keep it simple but awesome!

    As for creating your BlendShapes in Blender. Our experience is all modifiers (that modify geometry) need to be applied *prior* to creating *any* BlendShapes. The reason being, BlendShapes are very fussy about vertex counts. Even one vertex add/subtract usually prohibits the ability to add or change existing shapes. And most likely will prevent the BlendShapes from working properly, if at all, once exported to Unity.

    Emotions/expressions are something that we are toying with for the future. ;)
     
  11. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    Thanks. You are right, I forgot the power of 'control-Z'. I've only created blend shapes with Blender so far and haven't tried them in Unity so your advice is helpful. I look forward to your emotion extensions.
     
  12. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    Hey adamz!
    Currently, SALSA only supports up to 4 shapes. You can use fewer shapes, but the max is four.

    As for the text-to-speech option (which is very cool, btw), the tts module would need to produce an actual audio object that could then be linked into SALSA. After taking a brief look at this tts solution, it doesn't appear to do this; therefore it wouldn't work as an integrated solution with SALSA in its current form. We'd have to take a look and see if it's possible to link up with this sort of dynamic output. Obviously, using tts output files does work since there is an actual audio object to link to.
     
  13. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    Hi lazygunn!
    I'm only mildly familiar with the capabilities of Daz, but decimation is one of those processes that you could really use any tool to perform. The biggest considerations would be quality and output format. Blender certainly has decimation and re-topo features as do most full modeling packages. Of note to keep in mind is keeping vertex count/instance static while working with BlendShapes. While using Blender, any changes to vertex counts or instances (i.e. adding a vertex and removing another) will break or cause lots of problems with BlendShapes...pretty much creating a need to start over... So, creating BlendShapes should be the last step in the workflow pipeline.
     
  14. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    lazygunn needs to know that DAZ FBX exporter isn't exporting the morphs of all the old Poser CR2 files I licensed. I haven't licensed all the DAZ owned content (as that's another $500) so I haven't tried Genesis morphs as an FBX export but I don't think they are supported either. You need MegaFiers/MegaDAZ ($150/$30) to import the morphs. MegaDAZ does not yet support Poser CR2s as he is working on that support but it does support the newer DAZ Genesis format.

    Blender does a good job of decimating DAZ/Poser characters too although if you'll notice it's the areas that have the morphs that tend to come out the worse in decimation (DAZ Decimator or Blender Decimator).

    Process for the Genesis is documented at west-racing.com and on his YouTube channel.
     
  15. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    Awesome goat!
    Thanks for the update! Automated decimation is never a perfect science. It's certainly getting better and better. For Blender, it is possible to take some control in the decimation process on where/what is decimated.
    1. In edit mode, select the vertices that you want decimated, use CTRL+G to assign the selection to a group.
    2. Add a Decimate modifier to the mesh and select the vertex group to apply the modifier to.
    3. When the modifier is applied, it will operate only on the selected vertices.
    $ScreenShot_20140423_153918.png
     
    Last edited: Apr 23, 2014
  16. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    Thanks for that. I'll have to look more closely at DAZ and see if it has similar capabilities.
     
  17. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    I've updated the original post with some new overview information and images to help everyone get an idea of how the system works and what it's requirements are. Bear in mind that this is still just an overview, the manuals and example code go much deeper.
     
  18. ZJP

    ZJP

    Joined:
    Jan 22, 2010
    Posts:
    2,593
    @Goat.
    [​IMG]
     
  19. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    goat, genesis models export their blendshapes just fine with studio's fbx exporter, you just need to set up the morph export rules correctly, i havent triied the older poser formats yet. In the case of setting up the rules, for me it has been the case for me to set a match for anything based on the character you're using (your base fbm) and perhaps anything else youre using as specifically as you can and it will export the genesis model and the items its wearing perfectly fine, the bottom rule 'anything else' will have everything not specified baked and not available as a blendshape

    I did actually buy mega daz only a few days ago not realising i didn't actually need it, i've contacted chris west and see if he's amenable to a refund as it would greatly aid in buying another of his products that id find extremely useful.

    You have to be careful what you specify as the rules in the morph export however, keeping it as minimal as possible, or the model can end up extremely heavy and will crash something like max, and sometimes screw up the rigging too. On the other hand it's also been very amenable to putting on several layers of clothes to a figure and binding them all to the genesis skeleton in the exporter, which obviously makes the model heavier but means you can very simply swap out clothes as needed. If you wish to customise the entire model with blendshapes, you can, just add them as rules, but i'v found changing the absolute width or height of the figure will have it mismatch its skeleton

    The export, i'll add, is perfect for things like this btw, as you can supply the shapes asked for easily - but you can also export full sets of phonemes or visemes if you like. Main thing ive noticed that dont export properly (or at all) are mcms which help non-standard fbms express properly cause their dimensions are different humans

    To prove my point, however, here's the ribbit character in Unity, exported using the Daz Studio fbx exporter:

    https://dl.dropboxusercontent.com/u/12598787/Showcase/Toady1.jpg


    Sorry for the slight derail minnow, but i hope this information helps people in future
     
  20. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    Not at all a derail lazygunn, BlendShape support in Unity only started in 4.3.3, so any information about BlendShape workflows is right on the rails! ;)
     
  21. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    I'm prevaricating on having voice acting in my game, at least 'real' voices, was maybe thinking more kinda simlish or the random gibbery nintendo style, but having a really strong manager for expressions would be brilliant, particularly with the workflow i mentioned above - the daz fbx exporter can easily supply various specified expressions as morph export rules too and being able to blend those in and maybe create triggerable macros and wotnot to automate them would be a godsend

    Maybe out of your remit but to me, a cert buy, without a doubt - i think a formula of: create an expression definition, add the blendshapes for the expression(s) and their blending amount each, there being more than one blendshape possible for 'custom' expressions that the speech blendshapes could be modified by (to so some percent?), and that triggerable.. heaven

    Supplying the blendshapes is made extremely easily by daz studio and daz3d's art used to be laughable and intractible for games, but now you can use it even as an in-game body customisation system with blendshapes it seems worth plugging, perfect marriage with something like this. I'd imagine you COULD update to include visemes etc but i neednt see the exact reason, if you add customisable expressions that are additive then you can modify the speech blendshapes by script or if youre feeling brave, some anim editor or curves then you can blend in lip movement and everything else to the speech without messing with the core system.

    That is just an idea however, chances are with the next 'payday' as such coming along, it's a buy, i just do hope you keep updating this software as it offers a great deal in a really nice straightforward way
     
  22. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    We have been discussing expressions outside the existing four speech shapes since the beginning, and we have a design plan that should allow an unlimited number of expressions that function independent of the speech shapes.

    SALSA for 3D, and RandomEyes for 2D/3D already expose [Blend Speed] and [Range of Motion] options, so any expression system we would build would likely include similar type of control. We have worked very hard to make the system as easy to use as possible, with the smallest amount of requirements, while still exposing powerful programmatic control over it. We even built and included the basic dialog system that is used in the online demo scene so we could run the system through its paces and demonstrate one approach to a real-world implementation.

    This doesn't get you integrated expressions today, but I hope it shows our commitment to the product.
     
  23. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    Thanks for the reply, sounds perfectly fine! Facial animation may be a bit further on for me anyways, hence i havent got this yet, but even now it wouldn't be too hard to make a little thing that did what i described myself, it would just be a bit of a bodge job and only suit my project. So no worries, nothing expected this instant, but i'll probs grab this sometime soon anyways just to make development more fun if nothing else (mute characters with little animation to no animation while communicating isnt very enthusing to work with). Cheers
     
  24. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    We decided to have a little fun with our second promo video, let us know what you think of the video.

     
  25. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    Looks good. I have a simple game that needs these.
     
  26. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    I think Chris West's MegaDAZ is needed because typically you need to decimate the character, export without the morphs, and then use MegaDAZ to import the needed morphs. As I need MegaDAZ to import Poser CR2s not the newer DAZ Genesis format I haven't tested this.

    If I were to PM you could you take a look at a comple of older CR2 format files for me? I'll put them in a drop box. As of you, I've not seen an model in the asset store with blend shapes :-(

    Thanks.
     
  27. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    Well I have a simple game that needs the expressions more than the speech for now so maybe I can 'lie' and send expression blend shapes instead of speech blend shapes until you have finished.

    ...but that requires me getting CR2 morphs imported into an fbx and working....
     
  28. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    I've got a bit of v4 and m4 stuff, no worries, i'll test them out now, but i'd genuinely ask you to consider Genesis instead, it's a notable move upwards in flexibility and quality. Still the usual gamut of blow up doll costumes for the lonely people but a lot of really strong art and lovely lovely blendshapes. Where i post the results is up to you but it's something that's worth being common knowledge right now (particularly any differences between cr2 and the Genesis format)

    To the thread author, don't worry, not off-topic, as far as I can tell, Daz is easily the best place to get decent affordable stock with blendshapes, modelling them can be a pain and generally far more expensive in money-hours
     
  29. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    Oh my are toon style characters. The licenses were expensive in total, over $500...they were licenses directly from the artist not DAZ. I'll PM you the link. I have an ideal for a game that I can program in one week (not counting the time to release it) I want to make.
     
  30. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    Bloody hell, you money to spare! aha, no i understand when you've invested a lot and it proves to be a pain in the arse

    I did own megadaz, im in the process of a refund as chris wests stuff is great (i have megafiers and megashapes), but mega daz is pretty irrelevant for at least genesis stuff now. PM me I guess, i'll look into it
     
  31. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    The link I sent cost less to license then Genesis, but not much. :)
     
  32. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    You should think about mileage! There's toon morphs for genesis that are pretty affordable.. although then again some aspects of these morphs i dont know how to export, like the mcms (morph controlling morphs) i have no idea how to link up in max etc so some very exagerated full body morphs relying on normal person shaped morphs through an mcm look a bit odd
     
  33. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    I think you misunderstand how DAZ Game licensing works. Those DAZ prices only entitle you to use those characters as 2D sprites (very doable for some characters) and not as 3D meshes included in a released game. And certainly not for resale as a standalone mesh. To use DAZ in a game it costs $500. Of course you don't need to pay for the license until you see if you can actually get them to work and release the game to the public, something I should have done with what I licensed.

    Example: There is a Chibi Girl Boy that I really like but that would be $500 to DAZ for their license (the Genesis base mesh) + whatever the man that make them wants for his male female. He might not even want to license his characters outside of thee 2D DAZ advertising print world.

    What I need to figure out is out to access the morphs from the CR2 once exported from DAZ Studio as an FBX that I licensed as that has plenty of cartoon characters I can generate from the morphs...perhaps it's time I try to give the UMA base meshes a cartoon look by re-texturing them.
     
    Last edited: Apr 29, 2014
  34. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    @goat
    What you are describing would potentially be possible using RandomEyes. It does not require an audio file to be linked in and does have independent, programmable access to call each slot. You would have to call them by the intended action tho. So, if you loaded a smile into the lookUp slot, you'd have to call it using the lookUp designation. That would give you 4 slots to load, 5 if you count the blink slot, but it is a timed slot.
     
  35. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    Thanks, I think I could load blank audio files too. I'll buy something this week if I can figure out how to get morphs exported with the CR2 in the FBXes otherwise it'll be another week or two while I get another model source...
     
  36. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    SALSA may be more difficult to work with in that respect since it's main goal is to analyze a waveform and map to the appropriate shape key. RandomEyes is definitely the one to use for this. In fact, the new custom shape key system will be added to RandomEyes, since they share a lot of the same underpinnings. FWIW, making shape keys in Blender is pretty easy. I've started working on a video tutorial today, demonstrating how the Boxhead shape keys were put together for the asset. Will prob have it completed in another day or two. If you're just making expression shapes for toon characters, they should be pretty easy to do. I guess that's all relative, depending how many models you're using and how many expressions they should have. ;)
     
  37. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    Thanks then I'll use Random Eyes....and if the CR2 don't export nice I'll edit some meshes in Blender in the next week or two...
     
  38. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    Short story: Poser stuff decimates and exports just fine via Daz Studio.

    Longer: You'll need to clean the model up somewhere, Blender's probably fine, just Daz models always seem to be broken up into a ton of mat ids that need condensing and fixing the object names would make it nicer to work with too

    Unless the clothing you're using has the same blendshapes as your base mesh, it wont follow your base mesh, so in a case like that have your body shape set in daz studio before export. The Genesis stuff i've used has always been pretty flexible with this but what i worked with in poser's formats wasn't so with the clothing i gave it. This might be a rare case though.

    In short though, if you're motivated by the idea of an asset like this, it' should be very functional with the kind of products you'll find at www.daz3d.com and similar so if the licenses dont scare you and the idea of making your own blendshapes does, you have a huge resource there. And randomeyes implementing expression blendshapes (or any blendshape on the model) sounds great, especially if there's some nice level of control. You could count me in.
     
  39. goat

    goat

    Joined:
    Aug 24, 2009
    Posts:
    5,168
    Thanks a lot for your help, as it so happens the clothing, hair, and accessories do conform and there is a decent size library too. I do want to learn to model and I have somewhat but I want to use what's I've licensed as well. They can be used in different niche games.

    Thanks again.
     
  40. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    Feature request time! Is it possible that instead of using the 4 positions you be able to supply a good set of visemes instead? I don't know how more complicated or less complicated that makes things for you but the visemes are quite easy to get with daz's Genesis model system and seem comprehensive
     
  41. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    Hey lazygunn,
    You could export visemes that you like, that roughly correspond to the recommended mouth shapes. Keep in mind, SALSA is *not* a phoneme mapping system. It's goal is to provide a great look without mapping complexity. Exporting the viseme morph targets that correspond to the recommendations sounds like a great way to make the simple workflow even simpler.
     
  42. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    You're quite right I suppose! I've set myself a mission for the whatever amount of time i'm up for and this is a bit of a good test bed for one part of it. Fingers crossed paypal likes me should I come to it
     
  43. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    Hey all! We posted a quick-tip on our devBlog, explaining how to use SALSA with fewer mouth shapes if desired.
    You can check it out here!
     
  44. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    Hey Everyone! We've got another QUICK-TIP posted on our devBlog today. This post is the perfect follow-up to yesterday's QT. Limit RandomEyes movement to specific quadrants. Great for keeping natural, random movement while limiting the general "look" direction.
    You can check it out on our devBlog!
     
  45. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    I'll make no bones about it, i'm strongly influenced by the game in development, Cradle, the interactions in this video sum up much of what i'd like from a system https://www.youtube.com/watch?v=2aFjdYxoCfs

    With adding extra visemes to the selection i'd have more control over lip and jaw movement inherent to the sound being made, hardly expecting some massive support for this kind of thing but id wonder if you might consider broadening the allowed amount of shapes used during the process in future development, if possible with how you produce the movement. I'm very close to purchasing this but I do know there are free tools coming that suit this kind of thing, its just that will be later and i'd quite like to start learning now, so i guess i'm just wondering what you have on your roadmap, if you have one

    Another feature for SALSA that would be very useful would be the option to 'modify' one mouth shape with another shape, maybe at a random point within a set range of the modifying shape each time the mouth makes the necessary normal shape. This would mean that you could add individual personality and characteristics to how the character talks, visually, say with a sneer or something like that, i'd probably also use it to group similar visemes together and when the mouth was in a particular shape, modify it to express slightly more variety. If this could be set programatically that could be great, as I could add tags as markup in any dialogue system I used for text on screen to prompt the expression. This might be a nice way to add expressions full stop, so long as you have audio as a cue. Or if one were to generate audio for quite a surreal effect (Which is something i'm considering), it could follow the written dialogue much more closely and expressions commonly added in the text as tags or markup or however you wish to define it. Theres a free unity tool for mapping expressions to parts of the audio which works on adding keys to audio, i guess having several options in how to implement things is always good

    I'm going to investigate the system I intend to use for dialogue anyways and see how that takes care of it, or if the option is there to introduce these kind of cues

    The next suggestion was very much aimed at random eyes, your examples are amusing and show the system works, i think with refinement you could get some very nice subtle effects it however, especially if the model has eyelid blendshapes. The Cradle video is a very nice example of how personality is being conveyed through body language and eye movement, the two characters you see having very different character. To me it seems your system at the moment could well get as nuanced as that and much like in the video and the tip you just gave, while not confining the look to a certain direction, weighting it towards particular points - say you had a grid and white vertices were strong weights for where someone might look and they got darker as they lose priority, This would be a great way to procedurally convey a variety of mood and character, and would, in the video seen, show a great replication of how strong that looks. The girl's weighting is a broad spread focussed on looking down and to the side (shy) with a very small but strong spread in the center - sudden looks towards you then back down again.

    The old guy just keeps looking at you, you get a really particular idea of what kind of person he is just because he looks right at you (Eager, genuine, very self assured)

    Then I guess you could modify weighting based on changing moods or development of characters, it would be fascinating to me to have characters whose body language and facial expression significantly convey how the events of the game have changed their demeanour, and that be done completely procedurally (FinalIK would be great for the overall body movement)

    I'm maybe going way out of your original intended scope or you already have this functionality in there, but what i've suggested seems natural progression from what you already have, and as i'm feeling i dont have a lot of choice, currently in this area, your asset seems the most likely buy, if it were to continue to be developed that would be great, body language and facial expressions are very much undercooked in games and you're already sat on the foundations of something potentially pretty powerful there

    I'm going to go and have a look at your website and a few other things to see what my requirements are exactly, i'm currently going through a very long process of sorting out the visuals for some characters who could well use this system, the first you can't see her eyes so i'd be interested in how salsa works with her, then the next will my pal mr frog if possible and i'd love to see how randomeyes does with him.

    If you do have any kind of roadmap be sure to share it and if you dont think my suggestions quite fit into the aim of this particular product, or i'm being ignorant and it's all there already, then no worries, but it seems like you have a good couple of products here that are somewhat interlinked. I think if I was writing dialogue i would very much like to hook them both into it, but as said, something like the level of nuance of Cradle would be my direction ultimately

    If that's a bunch more work, double your price, put Pro after the name and if you make it i'm sure folk can make some compelling implementations
     
  46. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    I should add your system looks good and can no doubt get lipsync and facial animation up and running, with broad abilities to do as I describe. However I am interested in a system with a deal more granularity for a good deal of control and very particular nuanced effects, although the grid idea just came off the top of my head.
     
  47. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    Much of what you're describing can be achieved to varying degrees in the current 1.0 release, and additional capabilities are being added now for our upcoming 1.1 release. I'll explain more about the new features below, but first I'll give some detail on current capabilities. Salsa3D and RandomEyes3D both use a fixed number of index slots that link to the applicable indexes of the referenced skinned mesh renderer. You are free to swap these indexes out programmatically if you wish, but our speech and look systems are designed to get the most accurate speech and broadest range of eye movement with the lowest number of Blendshapes, and without the need to manually map phoneme shapes. This is how we provide high quality with minimal effort.

    We also designed the system to be programmatically accessible, with great documentation and tons of examples. For users who want to influence the results, we expose functions and properties to adjust shape change triggers, blend speed, range of motion, and a SALSA event system that fires off details about the Salsa3D instance, talk status, talker name, and clip name, at the beginning and end of each clip so you can programmatically respond to a dialog sequence. Here are the inspectors for Salsa3D and RandomEyes3D with some descriptions.

    $Salsa3D.jpg

    $RandomEyes3D.jpg

    So you can have multiple sets of mouth shapes and you can dynamically swap them out as a response to a talk begin or talk end event, add in the ability to control blend speed and range of motion, and you get a lot of control. With RandomEyes, you can also control range of motion, the max interval between random eye look positions, the max interval between random blinks, blink duration, you can toggle between random and programmatically controlled look positions, and random and programmatically controlled blinking.

    Here's a teaser shot of what's coming down the pipe for RandomEyes3D in the next update.

    $RandomEyes3DNewFeatures.jpg

    The fixed number of shapes used for speech and eye movement are an intentional design decision so that we can ensure automated quality with minimal fuss. The independently controlled unlimited custom shapes, with individual shape controls, coming to RandomEyes3D in the next update should fill in any gaps for users looking to implement highly emotionally expressive characters.

    We are currently working on a video tutorial series, quick-tip blog posts, and the 1.1 update. For the latest information visit our website:
    CrazyMinnowStudio.com
     
  48. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    That's looking really good there, the ability to add any number of your own shapes into the system is great. I suppose the final thing i'd like would be to have a way of controlling randomeyes during an audio clip. I'm not that familiar with the animation editor in Unity but would it be straightforward to have a curve linked to the id of a randomeyes blendshape and use it to drive the shape over time? In terms of syncing it with salsa, an image and timeline of the audio would be handy, but with a few curves you could make it as simple or complex as you want - you have the basic 4 shape system, which is easy to understand, as is randomeyes, if i have the choice to do the above its quite straightforward, i would think, to add as much nuance as one would like, without the core idea being further complicated whatsoever. Well, lets see if my bank does still like me afterall, and maybe i'll look at putting in the curve thing myself if it comes to it

    The improbable happened and my bank was agreeable afterall. I'm doing quite a lot at the moment to get my whole character making workflow (i hate that word) in order, but i should be able to have a go with this in the next few days, as it's now part of it. Do you have a general idea of when the update will land and if possible a suggestion of how challenging the curve thing might be to implement - im not alien to programming but i'm not particularly adept either
     
    Last edited: May 2, 2014
  49. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    399
    @lazygunn - That's great! We really appreciate your purchase! RandomEyes can be fully controlled programmatically, but currently isn't linked to audio or SALSA directly. However, there is a broadcast system that can be used to trigger actions between the components. For example, once a character begins talking, a msg is sent to RandomEyes and triggers a raiseEyebrows shape (v1.1 - although could still be implemented with v1.0 if you weren't using the eyes).

    RandomEyes shapes can be triggered and given a speed and range of motion. Shapes can be triggered on and off. And (in v1.1), RandomEyes is getting broadcast events. In the current release (v1.0), the range-of-motion and speed can be programmatically set, so you could drive progression of a shape in that manner. However, as a caveat, the range-of-motion is clamped and setting the position of the shape via the range will immediately return the shape to the max range. For example, you could set the range to 0, then change it to 70 and the shape will progress to 70%. However, if you then set the range back down to 20, the shape will immediately return to 20% and will not smoothly animate down to that level.

    As for how difficult implementation will be will be greatly determined by how complex you want your control to be. At their core, SALSA and RandomEyes are designed to give really nice output with very little complex programming required (outside of the programmer triggering the components when they want them to activate). We've got some pretty good examples included in the product that are completely documented.

    v1.1 is going to greatly expand capabilities, it should be ready very soon -- finishing up code base and bug-checking now. Without putting an actual date on it, maybe a week or two unless we run into problems or decide to include more functionality. And we're already working on enhancement possibilities for v1.2. We don't have a roadmap per se since much of what is currently being added is based on user feedback and requests and our own imaginations.
     
  50. lazygunn

    lazygunn

    Joined:
    Jul 24, 2011
    Posts:
    2,749
    Hmm, well the curve idea isnt really adding to the complexity, which is why i suggested it as a feature, it's something to uncover if you want more, and would pretty much make it for me. Is there any reason why the shape snaps back? I understood the diagrams i'm a grown up (you don't need italics either, just saying), and i like that the range is clamped because the feasible ranges can be large but i'd typically want them to end at 100, i'll try to explain exactly what I mean by controlling the randomeyes shapes this way, and i suppose there doesnt need to be any explicit link between the audio file and randomeyes, only that a randomeyes blend value be controlled by a curve over time

    Well, having just looked at the docs and fiddled with the editor, it seems it'll probably be faster to just do it myself for now. But to explain in case you consider adding a far better implemented solution than i'll manage because i can guarantee the following explanation could surely be wrapped up in a more elegant manner that fitted in smoothly with the system, if you have a gameobject (maybe the skinned mesh itself) with a script on it with say, a public array of blendweights, the array index correspond to a blendshape on the skinnedmeshrenderer, opening the animation window will let you add a curve which can control any public values on the gameobject. Add a curve initially to create an animation and for it and further curves add for each blendshape you want to change by the blendweight in the array, add the animclip to an array in the script that matches the clips to the audio, attach the playing of the animation to the playing of the corresponding audio file, salsa does its thing, randomeyes does its thing and extra expressions during the lip sync be blended with these curves

    Clunky as hell as I have it there but it makes sense to keep randomeyes as procedural as possible. Really so long as everything's exposed then i dunno why i'm going on about it! My one feature request now is to have the above in an elegant form, but i'll survive i think