Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Word Detection - Verbal Commands

Discussion in 'Made With Unity' started by theylovegames, Sep 23, 2012.

  1. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    176
    I'll have to add it to my todo list. From this package you would only need to use FourierTransform.cs if you don't already have an FFT algorithm.

    You might want to look at the Reason SDK to get the level of frequency detection you are looking for. If you've played RockSmith it wouldn't be such a fun game if the pitch detection was less accurate.

     
    Last edited: Dec 22, 2012
  2. Foriero

    Foriero

    Joined:
    Jan 24, 2012
    Posts:
    584
    Thank you. I will look forward to your PitchDetection.cs. Human voice is not so easy to detect. You know there are lots overtones in human voice and the algorithms are not very precise. We have FFT and AutoCorrelator algorithms and both of them are not very precise for human voice. OR at least we don;t know how to get correctly the data from FFT so that we get the right fundamental pitch of when a human is requested to sing one single tone.
     
  3. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    176
    I'd like to find a good auto-tuning algorithm. Right now you record a command at a pitch and have to repeat at the same pitch. If I could always shift words to the same pitch, it would improve matching.
     
  4. Yda

    Yda

    Joined:
    Dec 4, 2012
    Posts:
    1
    I purchase this item but mic do not working

    I Use Unity version 4.0 and mic that built in Logitech c210 webcam

    In Mac(unity3d 3.5) is working, but In Windows7 (unity 4.0 and 3.5(another computer)) do not working

    I don't know What's the problem

    Help me
     
  5. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    Ah yes that sounds like a Unity hardware problem. The best thing is to submit a Unity bug with a repro project and tell them that your hardware is not working. I've had this problem on mobile devices. When you have the case #, go ahead and send that to me and I'll post on the beta list.

    Thanks,

    ~Tim
     
  6. michellejean

    michellejean

    Joined:
    Jan 8, 2013
    Posts:
    1
    I was wondering if the verbal commands were tuned to the speaker or if I could set them and another user could use them. I'm working on a language learning app and I would like to add functionality for a user to repeat a word and my program to check their pronounciation. Would that be possible using this plug-in. Sorry if this was answered and I didn't see it.
    thanks,
    Michelle
     
  7. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    Currently it's not doing the overhead of learning syllables and parsing out words and tones. It works by recording a sound and finding the closest match in your dictionary of recorded sounds.

    You could make an app where the child sits down with the teacher, the teacher has the child say the word correctly. And then the child can practice trying to say the word correctly again matching the sound.

    It could be a useful language learning tool. Just like memorizing a list of 10 words for spelling as a kid.

    So the short answer is no. Only the person that recorded the word can match the word right now. Since you have to match the pitch and tone that you used previously.
     
  8. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    176
  9. ZJP

    ZJP

    Joined:
    Jan 22, 2010
    Posts:
    2,649
    Hi,

    It is possible to use your tools with an another sound source? Like the internal Audio Listener.

    JP
     
  10. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    The package is full source. You'll see it pulls in input from the microphone. As long as you can get the raw wave data from the audio clip the same algorithm would work.

    You would take the WordDetection class and duplicate that to change the inputs. Or add a virtual interface for where it's getting the wave input data.
     
  11. ZJP

    ZJP

    Joined:
    Jan 22, 2010
    Posts:
    2,649
    Thx for the reply
     
  12. rstorm000

    rstorm000

    Joined:
    Aug 19, 2010
    Posts:
    229
    I put up a simple test of testing No and Yes. I made 5 recordings of Yes and 5 Recordings of No. And very often it will read a spoken Yes as no's. This is on a macbook pro mic. Any reason for this?
     
  13. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    Sure you want to make the dictionary of words distinct. Words that are too similar in sound will match. The demo has a play button next to the recording so you can say it exactly the same way.

    These sounds aren't limited to spoken words. You can use a drum, thumping the mic, playing an instrument as methods for getting unique sounds that you can apply to your game.

    It's also possible that your mic is picking up a lot of background noise and that would match another recording with similar background noise.

    I've had the best results using a mic headset with noise cancelling built-in.
     
  14. Skribble

    Skribble

    Joined:
    Oct 29, 2012
    Posts:
    16
    Just a question.. will this work on mobile Android?
    im interested!
     
  15. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    Yes it works on iOS and Android. You'll want to disable the texture plots on mobile devices for optimal performance.
     
  16. Skribble

    Skribble

    Joined:
    Oct 29, 2012
    Posts:
    16
    awesome! thanks for quick response.
    ima add this to my must-buy list!
     
  17. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    176
    I know that the documentation is a little bit light. But now that there's a steady stream of interest, I can put together a PDF to document the process so far. There are a few videos, levers, and knobs but you folks need a guide for how this stuff works. I also intend on adding a controller example where your words can move a character around. I realize there are some precision settings, but you need a guide to maximize accuracy.
     
  18. Play_Edu

    Play_Edu

    Joined:
    Jun 10, 2012
    Posts:
    722
    it's work with ios or android also and unity 4.2 supported because i want to buy this pkg. plase tell me or pm me.

    regrds,

    play_edu
     
  19. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    176
    I'll send you a demo. PM with your email and you can try it out in 4.2.
     
  20. Play_Edu

    Play_Edu

    Joined:
    Jun 10, 2012
    Posts:
    722
    thank you very much for the help.
     
  21. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    176
  22. Play_Edu

    Play_Edu

    Joined:
    Jun 10, 2012
    Posts:
    722
    Thanks for the help it's working:grin::grin::grin: and i'll buy very soon thanks.:p

    regards,
    play_edu
     
  23. BrUnO-XaVIeR

    BrUnO-XaVIeR

    Joined:
    Dec 6, 2010
    Posts:
    1,687
    Looks cool!
     
  24. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    In his case he had to install the RelTek drives to get Unity to recognize his microphone.
     
  25. Kaemalux

    Kaemalux

    Joined:
    Aug 1, 2013
    Posts:
    45
    Hello!
    Your project is awesome. I need some info though, can i simply use it with every kind of game, after setting up the asset, or there are limitations or restrictions? I'd like specifically to do some speech recognition on iOS and Android game, is it doable or the package is still in development for those?

    Thanks ^^
     
  26. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    There is a save mechanism and that same profile could be copied to a server or something. Basically the user has to record the word to be detected which creates a profile. It's probably best to record the profile on each device since the microphone will be different. And it's important to make each work distinct. You can try the demo to see if your microphone is compatible.
     
  27. John-Higuera

    John-Higuera

    Joined:
    Oct 14, 2012
    Posts:
    20
    We have a question, how to do, if you want to save for example a blow to the mic, and use this as a detection event in your iOS app without doing the user to record the blow, but instead of that you could use the one you recorded from Unity editor?
     
  28. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    If you look for SaveProfile and LoadProfile there are a couple versions that save to file or bytes. You can save these bytes to the Assets\StreamingAssets folder and load the profile at runtime on iOS.
     
  29. John-Higuera

    John-Higuera

    Joined:
    Oct 14, 2012
    Posts:
    20
    Thanks for your answer,

    We found LoadProfilesPrefs and LoadProfiles, which one should do the work?

    Any way we where looking on both, but not sure where are allocating the profiles right now, neither how to changes it location for reading and saving.

    Could you give us an advise about this.

    Thanks
     
  30. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    LoadProfiles and SaveProfiles handles file IO.

    If you place the profile in Streaming Assets you can stream it in during runtime..
    http://docs.unity3d.com/Documentation/Manual/StreamingAssets.html

    Or you can take a binary file and Base64 it to a string and save it into a TextAsset. Text Assets can be referenced by the scene.

    It's all about how you want to load it.
     
  31. cybersoft

    cybersoft

    Joined:
    Feb 12, 2014
    Posts:
    107
    Very interesting great work!

    I have a few questions:

    Can I use it with sentences not only words?

    So, could commands be sentences too like below?

    I am from Belgium.
    I am from Las Vegas.
    I am from New York.
    I am from Europe.

    Can I use longer sentences? If yes is that possible to show the issue in sentence like this:

    "This is my first speech recognition" where word speech in the sentence is with a bad pronunciation?

    Is there somewhere a working demo version?
    Do I need the PRO version or Unity Free is enough?

    Edit:

    As I see Verbal Commands is not for sentences only words? And I do not know how good does work this plugin and what about accuracy?


    Thank you
     
    Last edited: Feb 12, 2014
  32. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    It's really more intended for short words/sounds/grunts like commands. What you want is a speech processor and this doesn't do that.

    Unity free should be enough.
     
  33. cybersoft

    cybersoft

    Joined:
    Feb 12, 2014
    Posts:
    107
    Thank you for the answer.

    I wanted to have commands not only words but some sentences too (maybe only 2-4 words).

    Does it work if my command would be: "post office"?
    I want to have a game and the player have to decide where to go for example: go to post office.

    Let's assume we have following limited words: school, teacher, bus, post office, car, table.
    The player should select one of them from the screen using your plugin. What about the accuracy?
    Does it work?

    Dictation tools in Unity would be great. Maybe you could consider to use OpenSource CMU Sphinx for Speech recognition.

    Look at the video: http://www.youtube.com/watch?v=uItCqkpMU_k&t=172
    It works great!
     
  34. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    The accuracy completely depends on the uniqueness of the sounds. The longer the phrases I would suspect the less accurate it gets. But you just have to try.

    Here's the demo:
    http://theylovegames.com/WordDetection_1_3.html

    And when you set the profiles, you can say any words you want if they fit within 1 second. Profiles don't need to be 1 second, it's just what I used for the demo. Make sure you set the noise profile without talking.
     
  35. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    176
    Last edited: Feb 17, 2014
  36. BenoitFreslon

    BenoitFreslon

    Joined:
    Jan 16, 2013
    Posts:
    163
    Hello,

    I have one question about this plugin.

    Is it possible to assign audio assets in the Assets folder instead record my own voice?

    Thanks.
     
  37. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    No, it's not really designed for that. It's better to record the profiles in the environment that they will be detected in to match more closely.

    That doesn't mean you can't do it that way.

    There are methods in the API to load profiles from preferences or from a byte array. So you could save the profiles to StreamingAssets and load them on the fly.
     
  38. Jacob-Williams

    Jacob-Williams

    Joined:
    Jan 30, 2009
    Posts:
    267
    Hey tgraupmann - This plugin looks like it might work for my project, but I am going to have quite a few commands. Is it possible for words to have "parent" words that must be spoken first?

    For example: Mike - Show Map, Jim - Attack. This would allow me to give specific commands to certain people, and help with detection accuracy, as each person will have a unique set of commands.
     
  39. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    Word detection works from a list of words. So all you would need to do is have a list of the "parent" words and when a parent word is detected, swap in the list of "child" words, and vice-versa. You would just do this in an Update event and it's as simple as modifying a generic List of words.
     
  40. ngorgiev

    ngorgiev

    Joined:
    Mar 19, 2014
    Posts:
    1
    I am interested in buying this asset, but is there any simple tutorial how to use is to create a simple demo where you record a single comand, and triger event when that word is said?
     
  41. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    Examples 1-3 displace the microphone data in a visualizer. The rest of the examples record a sound and detect that sound. There's press to record a sound for 1 second. There's hold to record the sound. And then there's the example of recording sounds that playback mechanim animations.
     
  42. shenyi

    shenyi

    Joined:
    Mar 14, 2014
    Posts:
    3
    Hi, I'm deciding to buy your asset. But once I opened these demos in your thread, some GUI were missing.
    Like this $QQ??20140331231413.jpg
    So I can't control them. My browser is Chrome and I have tried to clear the cache, but it still didn't work. Could you give me some suggestions?
     
  43. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    That's odd. Can Unity not detect your Microphone? Usually the first screen that opens let's you select the Microphone.

    How about Firefox? It's possible when you started the app, you didn't use the dialog to allow Microphone access.

    So you are using this demo?
    http://theylovegames.com/WordDetection_1_6.html

    When you get the package, this will be an example scene. Any errors would be in the console log.

    The only GUI used is Unity GUI.
     
  44. shenyi

    shenyi

    Joined:
    Mar 14, 2014
    Posts:
    3
    Thank you for reply! Yes, the demo I tried is 1_6. At the beginning, the website asked me if I allowed the game to use my microphone, I chose Allow. But it still didn't show any button for me to control. I changed to IE but had the same problem.
    Then, I have to say that I'm not good at coding, I don't know whether I can use your asset sufficiently. Do you have a simple tutorial inside your asset?
     
  45. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    I have created example scenes. I don't necessarily have step by step examples just yet.

    I'm thinking you may not have a Unity compatible microphone. I'll PM you with a demo.
     
  46. shenyi

    shenyi

    Joined:
    Mar 14, 2014
    Posts:
    3
    That's great, thank you! My email is shenyi216@qq.com. Since I can record my voice in Unity, I guess the problem may be in Webplayer rather than microphone. It must be solved once I download the demo, I think.
     
  47. WilliamLv

    WilliamLv

    Joined:
    Feb 27, 2014
    Posts:
    4
    I have same problem like shenyi,both chrome and firefox can't show operate button.
     
    Last edited: Apr 14, 2014
  48. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    This issue has happened before and to solve the user installed their sound card drivers. In the past IIRC, it was the RealTek drivers. But of course it depends on your sound card or Mic drivers. Send me your email to support@theylovegames.com and I'll send you the next update which includes example 8. Example 8 let's you save a word profile to disk with the save/load dialog.
     
  49. WilliamLv

    WilliamLv

    Joined:
    Feb 27, 2014
    Posts:
    4
    On demo 7,speek a word will match more than one action , I want make this only match one action? how to do this?
     
  50. tgraupmann

    tgraupmann

    Joined:
    Sep 14, 2007
    Posts:
    828
    I've added some new word detection examples including using multiple word sets to increase accuracy using smaller sets of words that are context specific.

    I'll upload the new examples to the store. If you want early access, email me and I'll send you the new package.

    I created a new set of examples while supporting students at the University of Florida.