Using Unity 3 and SILVIA in practice

This conversation was started in another thread.

I'm attempting to move it to a more appropriate place.

Copied from: http://forum.unity3d.com/threads/61151-SILVIA-Conversational-AI-Platform/page3

<snip>
We are heavily involved in using SILVIA and Unity right now. We are developing a demo of SILVIA and Unity whereby we have an avatar that will be an "expert" on a certain subject. The user will be able to talk to the avatar using voice recognition and have the avatar respond in an informative and intelligent way. The demo will showcase SILVIA's capabilities and Unity's aesthetic values.

We have run into a couple of snags, so far, with exchanging data with SILVIA and Unity. Nothing that can't be overcome, though. Maybe I should outline the problem here and see if the community has some suggestions to resolve them.

Jerry
Northrop Grumman
</snip>



<snip>
I have a couple of statements from our programmers:

<jeff>

I'm having an issue with integrating Unity and the SAPI functionality within windows. I have created some sample code that initializes the voice recognition system and loads a grammar list. This works when loaded with a WinForm but when I load it from unity with a c# script, unity closes with a generic error. A little more detail, the dll that I have created to interface with SAPI seems to be crashing at the point where it sets the call back function for the speech recognition event. The Specific line of code is:

SpeechLib.SpSharedRecoContext m_RecoContext = new SpeechLib.SpSharedRecoContext();
m_RecoContext.Recognition += new _ISpeechRecoContextEvents_RecognitionEventHandler( Reco_Event);

</jeff>

<jason>

I am spending time currently converting wav file to ogg files so I can load that file in Unity3d at runtime. Since our files are being created dynamically at runtime the only way I could find to load a file was by using their WWW scripting class. I am dealing with some other issues as well, but they are really not Unity related.

</jason>

I think Jason's issue is that Unity only allows streaming for .ogg. Therefore he's having to convert the .wav files generated into .ogg files, then stream those. It seems messy...

(we may want to move our conversations to a programming thread)

Jerry
Northrop Grumman
</snip>

 

Hi Jerry,

I'm working with Jeff on your SAPI issue, and I think I can help him with this, using components from our voice module. Our voice module was designed for AI interactions, both listening and playback, so this could give you a jump-start, so you don't have to build something that already exists on our side.

I think as a side benefit, you may not have to deal with the MP3/WAV to .ogg conversion from the SILVIA Server renders, as you could use a modified version of our live voice module to render AI output directly in realtime.

 

OK thanks. Looks like we are moving forward.

Jerry
Northrop Grumman

 

@Jerry: I think as more unity developers start using the SILVIA SDK and getting accustomed and familiar with the toolsets, API (self included) you will see alot more interaction and feedback on this thread.

Great to see the different ways Unity is being stretched and expanded as a platform with the various plugins that enable developers to also stretch their ideas and imagination.

 

Yes.

As it is, I think we are leading the experimentation with SILVIA for computer games (as far as integration with Unity). We now have our 3D avatar in conversation with us and have lip syncing working - fairly well.

It made my heart race a little to see it work the first time. It's VERY neat.

 

Is there any public demo of SILVIA+Unity integration?

 

That's awesome!

In a recent article about the origins and development life cycle of Microsoft's Project Natal, one of the heads behind the tech said of the question "that players might have full conversations with on-screen characters?" -- “Realistically, if you want a one-to- one conversation with an AI-driven character, that’s at least five years away,” Andreas says. “But thinking about where that tech is going,”"

Haha. Now think of Silvia/Unity running in Google's Native client on Google TV + a Gesture recognition Unity plugin like Softkinetic's Iisu 2.5 + 3D webcam.

Umm - that day maybe closer than they think.

 

It is closer. It's just about - now.

We are trying to make SILVIA an expert on a certain subject. The user will be able to quiz her or ask questions about this subject and SILVIA will respond with intelligent and informative answers.

It's already fun just to interact with SILVIA out of the box. But, we have to make her "know" some things about certain subject matter.

So far - so good.

 

Thank you for sharing this, Jerry. This is gratifying and exciting for us as well.

 

Sure thing. I'll post our findings as we go.

 

@jwaugh - Have you presented any of your team's Silvia integrated Projects in a public forum yet?

 

Hi KITT,

I can't speak for Jerry, for several reasons including the fact that Cognitive Code is under a non-disclosure where Northrop Grumman is concerned.

I can say, because it's already public knowledge, that Northrop Grumman has already developed and deployed commercial SILVIA-enabled products. However these are not public-facing products.

I can also say that Cognitive Code and Northrop Grumman continue to work together on various applications, some of which may or may not be public facing.

However, if you go to our site, we do have some updated news for Android users and game developers that may be of interest.

http://cognitivecode.com

Cheers!