[Demo] Flash webcam assisted Augmented Reality

Hi.

I was fooling around with the FLAR library yesterday and pulled together a simple proof of concept of integrating Flash and Unity to provide augmented reality.

Have a look at:
http://www.zerofractal.com/fractalreality/augmentedReality.html

Mac users, if you are having problems accessing your webcam, right click on flash, go to settings...camera. The built in camera is usually called USB Video device or something like that.

This is an initial approach. Now I intend to pass the video feed into a video texture. Any ideas are welcome. I was thinking pushing bitmap data though an external interface into unity, but I don't know if the browser javascript is up to the task, or if the SendMessage() supports arrays at all.

Crits and comments are welcome!

 

Pretty cool.

My printer hasn't been used for a while and I'm almost out of ink apparently, so the print out was pretty ugly. But it seemed to track it okay sometimes. And when it did, the scene moved and rotated very nicely.

I was trying to think of what this could be used for, when I remembered this video:
http://www.youtube.com/watch?v=Jd3-eiid-Uw

Yes, I would be geeky enough to glue your logo to my forehead to see a scene in 3d

 

Very coool! Been playing with AR in Director a little too... would be great if you could connect to camera with unity. Processing video images for AR should be faster than using Flash I think...

 

I remember w while back someone had something similar, but just Flash. Had a little green guy that jiggled around when you moved the camera.


But now, with Unity integration, it opens up worlds of possibilities.



Very cool man! I hope we see something more than a camera control tool out of this, though. What about taking the 3D model from Unity, and blue screening it into the flash video? That would be really cool!

(Oh, and lame computer framerate issues. I think it was the Unity scene you used. My Web R/C, right?)

 

Dakata. I´m using the exact same library as the green guy. I believe we can get full integration in two different directions:

1. by integrating the flash video feed into unity as a video texture. I´m scribbling on that one. This way I could overlay the 3d content on the video feed.

2. by developing through opencv ways to feed the video into unity plus building the AR libraries there. This one should be much more efficient, but I believe wonk work on the web, would it?

As for it being a camera tool it should be far from that. In the demo the camera is actually still. I´m transforming actually the object. I think some kick ass things can grow from here.

 

Totally!


The possibilities for Flash integrated augmented reality (even if all Flash does is get the video feed) with Unity are quite open.


I see this as on the forefront of the latest game innovation.


We'll see what people can do with it.

 

MacBookPro (Safari and Firefox), can't change the flash camera setting to Built-In as the dropdown list of Cameras is cropped within the webpage frame.

Would love to check this out though... can you change the page layout?

 

Done. Gave it 100+ pixels. Thx for your observation.

I also uploaded a new guide that yields better results.

 

@zerofractal, yep that works fine now.

... and it is very, very cool!

 

+1 for cool!

 

Very cool. I would be perfectly willing to tape your symbol to my forehead for headtracking btw (hint hint).

 

I like this much more for head tracking:

http://www.youtube.com/watch?v=Jd3-eiid-Uw

Looks more confortable.

 

What is the bug? I don't see anything on this page. The screen remains black after powering the cam.

 

Finally Flash began to work, but what must appear in Unity window and what I must do?

 

if you point the paper to the camera, you should see a house...

 

It's impressive!! What is the latest progress?

 

I'm stuck in transferring the webcam image from flash to unity on the fly, nor unity supports a wmode = transparent to simply overly unity on top of flash, so I'm now looking at building a Artoolkit based plugin for the stand alone unity projects :S

 

Hi there.
Could you please tell me what are you camera settings in Unity and if you are adjusting the values coming from flash in any way ? I'm trying to set up something similar but having problems matching the position.
Really appreciate you help !
Thanks !

 

check out the NyArtoolkit, its already written in c#,
http://nyatla.jp/nyartoolkit/wiki/index.php?NyARToolkitCS

i tried to do this but not enough time or will

I would send you super magical luckyness 10,000! if you got this to work

 

AR in Flash is indeed possible but not that fast and its an extra step.

I have heard of a library called OpenCV which should make this kind of stuff possible too.


The easiest way to do it is of course to use a WiiMote (be it an original or a wiimote where you desoldered the sensor and made your own electronics project with.

Wiimote can go 200fps for 4 blob tracking.
Unfortunately, it involves more work then printing out a tracker page.


I was allready investigating this WiiMote sensor to do for example motion capture. If it can capture 4 blobs at 200 fps, then with clever tricks it should be possible to track 4x4=16 markers at 50fps or 32markers at 25fps.

Cheap mocap possible ???

 

OpenCV was around but I don't know anything about it anymore there was a guy who used it to control boxes with your hands or something like that. Nice job, zerofractal!

 

Thanks. The big problem is actually feeding the camera to unity. I tried doing it via javascript but could get 1 512x512 canvas in no less than 3 seconds using AS3...which is not bad to pass 1 textures but simply unacceptable for feeing video. I guess a 320x240 feed should go at about 1 frame each 2 seconds or so...

The thing about opencv or such is not being to be able to do the webcam interface in a web player due to plugin restrictions. Theoretically you can do it for stand alone games. The same goes to wii motes. We are stating to use wiimotes to do head tracking but.... no web players.

as for the camera settings, the camera is not actually calibrated, but should be a deal of just sending the transforms and applying them to the unity camera. I'm passing off tracker info into unity through js.

On flash I did this:

Code (csharp):

1.  
2. private function _onEnterFrame(e:Event = null):void {
3.     this._capture.bitmapData.draw(this._video);
4.            
5.     if (this._detector.detectMarkerLite(this._raster, 80)  this._detector.getConfidence() > 0.5) {
6.         this._detector.getTransformMatrix(this._resultMat);
7.         this._baseNode.setTransformMatrix(this._resultMat);
8.         this._baseNode.visible = true;
9.         this._renderer.render();
10.  
11.         if (ExternalInterface.available) {
12.             var baseRotation = Matrix3D.matrix2euler(_baseNode.transform);
13.             ExternalInterface.call("GetUnity().SendMessage", "sandbox", "argPosX", -_baseNode.x); //mirror mode (make + for unmirrored)
14.             ExternalInterface.call("GetUnity().SendMessage", "sandbox", "argPosY", _baseNode.z);
15.             ExternalInterface.call("GetUnity().SendMessage", "sandbox", "argPosZ", -_baseNode.y);
16.             ExternalInterface.call("GetUnity().SendMessage", "sandbox", "argRotX", baseRotation.x);
17.             ExternalInterface.call("GetUnity().SendMessage", "sandbox", "argRotY", -baseRotation.z); //mirror mode (make + for unmirrored)
18.             ExternalInterface.call("GetUnity().SendMessage", "sandbox", "argRotZ", -baseRotation.y); //mirror mode (make + for unmirrored)
19.         }
20.     }
21. }
22.  

And in unity I did this:

Code (csharp):

1.  
2. var xAng : float = 0;
3. var yAng : float = 0;
4. var zAng : float = 0;
5.  
6. function argPosX(value){
7.     transform.position.x = value;
8. }
9.  
10. function argPosY(value){
11.     transform.position.y = value;
12. }
13.  
14. function argPosZ(value){
15.     transform.position.z = value;
16. }
17.  
18. function argRotX(value){
19.     xAng = value;
20.     transform.rotation.eulerAngles = Vector3(xAng,yAng,zAng);
21. }
22.  
23. function argRotY(value){
24.     yAng = value;
25.     transform.rotation.eulerAngles = Vector3(xAng,yAng,zAng);
26. }
27.  
28. function argRotZ(value){
29.     zAng = value;
30.     transform.rotation.eulerAngles = Vector3(xAng,yAng,zAng);
31. }
32.  

 

Thanks a lot for the info. So you are aplying transform to the camera in Unity?? I thought you would apply it to an object an keep the camera stationary.... Interesting.
I think I got the image feeding working, but the tracking is off. Once I have it figured out , I'll post the code. If you can post the unity file that would be great.
Thanks again!

 

You aré right!! The transformation is applied to an object. AR is all about the object

 

And is your camera sits at 0,0,0 pointing in Z direction ?

 

Hi, we're developing an application similar to ZeroFractal's, but it seems more like a workaround to the fact that Unity can't access a camera.

At first we tried flartoolkit (pv3d), but to achieve a "regular" quality it hugely compromises the performance (getting 3fps with a 4000 triangles model)... and that's why we moved to Unity.
Well application is running with a far more detailed model (over 30k triangle)... and it's much faster.

But we still have a problem. It works fine on PC, but on a Mac the axis coordinates seems to be messed up. Flash (FLAR) sends the same transformation matrix but model coordinates on unity are completely different and "unstable" (changing values all the time). Any ideas on how to fix this?

ps: since this thread is inactive for a while, any news on showing the camera inside Unity?

Thanks in advance for the help.

 

We ran into sort same problem. But it's consistent on pc and mac. Do you transform the angles or axises in any way?

 

we figured out how to fix the problem. we're trying to calculate the transform data "manually" on flash side and send a string with it to Unity, at first we had a problem converting the numbers (flash uses a dot for the fraction part of the float, and unity uses comma... no big deal, but you have to replace '.' for ',' ). we believe this should be causing the same inconsistency on mac side (i don't know for sure because i've never used a mac for more the 5min).

but the important thing, how we solved. just like zerofractal's. using Papervision's Matrix3d functions, and sending it to unity.

 

Could you please help me out. I tried to use the same exact code as zero fractal but the axises and seem off.
Also, what camera settings did you use in Unity ? Did you change "Field of view " settings or did you stick with 60 ? Any way you can post the code ??

 

Very interesting thing. For the record: I didn't print the guide, but displayed it on my iPhone. Works well as long as there isn't much angle, then it loses it due to reflections.

 

For the record too:
We was running out of ink in the printer and I'd start to draw the marker with a pilot. It didn't worked. But while complaining about the not working after that helpless rustic process, the app started to work using my head as a marker.

That was freaking AWESOME even knowing that is probably a calibration issue. That has inspired us a lot here :wink:

 

Hi,
I'm trying to put an AR app inside the iPhone using Unity, do you thing this is possible?

I was reading that the frame rate to put the image of the camera as a texture in Unity was not acceptable, have you found a fix to this issue?

Thanks in advance for your help.

 

Hello to all,

I need call a externall function (javascript) on my unity, i know that in flash i can to use this.

import flash.external.ExternalInterface, after ExternalInterface.call

How i to do this on Unity?

thanks