Augmented reality with the Kinect, managing occlusion and manipulation with the hands

Hi,
We work on augmented reality (AR) where a virtual object is added to a live video stream as if it was really part of the scene. Last year, we developped unity3d plugin that is able to create the AR illusion. Basically it grabs the webcam image and compute the camera position wrt. a marker. Unity then only shows the webcam image superimposed with a render of the virtual object from the correct view point. This video shows what it looks like:
http://youtu.be/npuubLGpU1A

Then we wanted to allow the user to manipulate the state of the object, rotate and resize, with his hands. The Kinect is perfect for that, the skeletal tracking measure where the user is relative to the camera (hand, head, etc). While doing so, we were confronted with the occlusion problem. That is, if the hand is closer to the camera than virtual object, the hand must be seen (we must not render the virtual object over the hand).

Thus, we used the Kinect (with the Microsoft sdk) to manage occlusion and manipulate the virtual objects interactively.
Here is a video showing the results (winter 2012) :
http://youtu.be/OTmHtAaQD_c

Rendering is done in Unity3d free, occlusion is managed in a unity shader (this is very fast since computation is done on the graphic card), manipulation is done in Unity3d from skeletal informations from the Kinect.

We are interested in comment from the community. While the technology is cool, usefull applications are difficult to find.

 

This is a cool development! Interaction when using the mobile AR can get a little awkward when the user can't place anything between the game models and the camera. I could see a greater immersion for AR games utilizing the kinect with being able to cull items. One example I can think of would be hiding an ingame object with the player's hand from an AI entity. But I'm sure there are a lot better implications that just that.

 

This project done by the HITLab New Zealand might be of interest to you then.

 

How did you achieve the occlusions? I am attempting something similar, but with mobile/web cams not Kinect.

 

Hi,

I am trying to write a similar AR application which will use the Microsoft Kinect with Unity Free.

Would you mind explaining how you manage to link the two technologies togeher?
If possible I would want this to run within a WPF application.

Thanks

 

Is there a tutorial for that?