[Package] Dialog Editor, including speech recognition support for 3 services! Demo available

Hello all,

I hereby present to you a package to implement dialogs and speech recognition at the same time!
I've been working on this as part of a graduation project (University of Applied Sciences Rotterdam) and I'd love for you guys to give me some feedback. I hope it'll make speech and dialog implemantations easier for everyone.

Feature list:

• Dialog editor, for normal dialogs or dialogs with user options.
• Dialogs to implement cutscenes or cutscenes with user options
• Option to use different grammar files for each dialog (Sphinx 4)
• Export to Json & Load Json
• Typewriter animations
• Delay before a next dialog shows!
• Out of the box support for 3 speech recognition systems. Google Speech, Wit.ai and Sphinx 4.
• Automatic selection of dialog answers on speech result. Let's calculate that accuracy!
• Audio input analyzer. When did the user start talking and when did he stop? Let's cut that audio out, and recognize it!
• Voice Activation Volume adjuster
• Callbacks for timers, automatic answer selection and current state of audio input (no audio, listening & analyzing speech)
• Includes a working Sphinx 4 Server, and text to language model tool. (out of the box support for English, Dutch and German)
• Automatic grammar files to dictionary to decrease server load
• Docs available in source and here: https://hespen.net/Portfolio/UnityDialogEditor/annotated.html

The project can be found here:

https://github.com/Hespen/Speech-recognition-dialog-editor-and-speech-server-

It contains a Unity Package for easy implementation, the Unity project, a Sphinx 4 Server and a Sphinx 4 Text to Language Model Tool.

To make things easier for you to test, I've added keys for the Google Speech API and Wit.ai API. (Wit.ai has an English key, Change the language of Google in the code) Select the speech system you'd like to use on the main camera object

I advise you to take a look at Sphinx though. Just import it into your IDE, Gradle Make it, And run the Base object.

Remember this is not a finished product, as there still are some bugs in the dialog editor. And I haven't had the time to make it beautifull yet. I did implement this in a VR game, but as that is part of a company, I can't share that one.

I'd love to know what you guys think of it!

Screenshots:

https://i.gyazo.com/b0c3ca5ff76fd45afc6b2b9461629f95.png
https://i.gyazo.com/669bbbd74bd62a209c44438ad8f35ee5.png


Demo video (crappy quality, no audio):

How to use:
Enable Microphone Setup Object and run it and toggle the button for like 5 seconds while being silent. Toggle it off, and speak. When you speak the square should become green. (saved in prefs automatically)

Stop the game, disable the microphone setup object. Select the speech system you'd like to use on the Main Camera object. Press enter to start the dialog.

Remember: Google en Wit.ai are really slow, use Sphinx 4 for the fastest result! I did research on the implemented speech recognition systems and their accuracy and speed. Sphinx is the fastest with an average of 200ms recognition (external server)! Where Google and Wit.ai will need atleast 2-5 seconds. (tested with 3200 audiofiles, 2 languages)


The editor:
Windows -> Nodes Editor. Right click to create new nodes or load the json file. Demo Json in Resources folder. Middle click to drag, scroll to zoom. Right click to export to json. You can attach the json to the main camera!

• Keywords are used for speech recognition! They determine the accuracy.
• Delay in Seconds before dialog is shown
• Time until next node is a delay before the next node is shown. This one starts counting after the first delay has passed.

Last thing: The used google key in this project is attached to a trial account. If I've spend my cash, it won't work anymore. You can however in that case, set up your own trial account for free on the Google website.

 

Great job, I'm going to have a look on it.!!
Could you please make a tutorial how to install and run Sphinx 4 Server?
Can we run it on a virtual machine or just a simple hosting?

 

Thanks for trying it out. Love to hear what you think!

As for Sphinx, installation is pretty easy. Here is a video of booting it up:

If you want to run it externally, just make sure you change the IP and Port on Unity's side. I actually tested it speed by doing that, and is was still extremely fast.

 

Better demo available at:

 

Cant' wait to try this out. It seems to be the solution that I have been look for long time.
Thanks for your effort time @Hespen

 

I tried run SphinX server on MAC and get some errors here (screenshot). Please help!
Look like I missed junit.Assert. Where can I import it

 

The Test class is actually generated by Gradle, and has nothing to do with the main sphinx server. You can easilty remove that class, and everything should work fine.
Even with the class you should be able to run it without any problems, atleast I could. But completely removing it can be the first step

 

I tried follow exactly every single step and run Base.java then those things show up. What should I do?

 

You should look at the imports, I guess you didn't enable Auto-Import as soon as you imported the project in IntelliJ.
Open the 'Gradle View' and click refresh. If you can't open a Gradle view, reimport the project. There should be a popup, which says, would you like to enable automatic import.

 

I attached here video please have a look. I don't know what wrong!!

 

Yeah it actually didn't download the graddle right. Could you open the Gradle Tool Window? And press refresh?

 

Please have a look, Gradle window open and also (auto-import enabled) and I click refresh many times

 

Have you clicked the refresh button? The two blue arrows forming a circle?

 

YES, I did click on it. it also pop up a warning.

The following repositories used in your gradle projects were not indexed yet: https://mvnrepository.com/artifact/commons-io/commons-io If you want to use dependency completion for these repositories artifacts, Open Repositories List, select required repositories and press "Update" button.

--- > I also tried to click update button for each repository. You can see the middle one is turn red. That might cause problem. it is "https://mvnrepository.com/artifact/commons-io/commons-io"

 

I tried to run Unity Client and also follow your video tutorial and this warning appear. I have a jar server run local as well

System.Net.Sockets.SocketException: Connection refused
at System.Net.Sockets.Socket.Connect (System.Net.EndPoint remoteEP, Boolean requireSocketPolicy) [0x00000] in <filename unknown>:0
at System.Net.Sockets.Socket.Connect (System.Net.EndPoint remoteEP) [0x00000] in <filename unknown>:0
at System.Net.Sockets.TcpClient.Connect (System.Net.IPEndPoint remote_end_point) [0x00000] in <filename unknown>:0
at System.Net.Sockets.TcpClient.Connect (System.Net.IPAddress[] ipAddresses, Int32 port) [0x00000] in <filename unknown>:0
UnityEngine.Debug:Log(Object)
Job:SphinxRecognition() (at Assets/Plugins/dialogreader/Speech Recognition Systems/SpeechRecognition.cs:204)
Job:ThreadFunction() (at Assets/Plugins/dialogreader/Speech Recognition Systems/SpeechRecognition.cs:101)
ThreadedJob:Run() (at Assets/Plugins/dialogreader/Tools/ThreadedJob.cs:60)

 

Well Sphinx uses port 85 on localhost. So just make sure no other application uses that. (or change the port numbers in Sphinx and Unity)

But I made a video of setting up the Server. It's the package from GitHub, downloaded as Zip.

 

Thanks, I was able to run the Sphinx server. Then try to run sample scene. But this error comes up on IntelliJ.
It seems that savwav did not trim the wave file. The audio.wav save in the project always is always at 5s length with silence area at the beginning and the end.

-----------

java.io.IOException: Unexpected readed size. current: 0, excepted: 4

at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:521)

at com.gdc.sphinx4.server.main.SpeechServer$SpeechServerHandler.run(SpeechServer.java:87)

at java.lang.Thread.run(Thread.java:745)

error

CLOSE

-----------