Kinect Calibration with Unity

Hello!

Im currently working on a project where I need to display a pointcloud using Kinect, which I already achieved. The result can be seen at the uploaded picture.

The problem is that the RGB camera and the Depth camera are not calibrated, so the the pixels from each camera arent corresponding to each other at the moment.

After searching at the Kinect SDK documentation, I saw that there is this method called
KinectSensor.MapDepthFrameToColorFrame ( http://msdn.microsoft.com/en-us/library/microsoft.kinect.kinectsensor.mapdepthframetocolorframe.aspx ), which does exactly what I need to match both streams.

But then again I cant seem to find this method within Unity Kinect Plugin ( http://wiki.etc.cmu.edu/unity3d/index.php/Microsoft_Kinect_-_Microsoft_SDK ). Is such method implemented? Is there another easy way to achieve the same result?

I know that I can calibrate it myself by multiplying Projection and Modelview Matrixes, but that would be a bummer.

Thank you all for the help!

 

If you check the header files from the Kinect SDK directory for the function/parameters, you can add the function yourself inside the KinectInterop class

 

After checking the documentation, here is my try based on your suggestion:

At KinectInterop.cs, within NativeMethods class, I did this:

Code (CSharp):

1. [DllImportAttribute(@"C:\Windows\System32\Kinect10.dll", EntryPoint = "MapDepthFrameToColorFrame")]
2.         public static extern int MapDepthFrameToColorFrame( DepthImageFormat depthImageFormat,  ref DepthImagePixel[] depthPixels,  ColorImageFormat colorImageFormat,  ref ColorImagePoint[] colorPoints);

The structs are defined before NativeMethods class and goes as follow:

Code (CSharp):

1. [StructLayout(LayoutKind.Sequential)]
2.     public struct ColorImagePoint
3.     {
4.         public int X;
5.         public int Y;
6.     }
7.  
8.     [StructLayout(LayoutKind.Sequential)]
9.     public struct DepthImagePixel
10.     {
11.         public Int16 Depth;
12.         public bool IsKnownDepth;
13.         public Int16 PlayerIndex;
14.  
15.     }
16.  
17.     public enum ColorImageFormat : int
18.     {
19.         RGB = 0,
20.         YUV = 1,
21.         Bayer = 2,
22.         Undefined = 3,
23.     }
24.  
25.     public enum DepthImageFormat : int
26.     {
27.         Resolution320x240Fps30 = 0,
28.         Resolution640x480Fps30 = 1,
29.         Resolution80x60Fps30 = 2,
30.         Undefined = 3
31.     }

Now at the main file:

Code (CSharp):

1. private Kinect.ColorImagePoint[] mappedDepthLocations;
2. private Kinect.DepthImagePixel[] depthStreamPixel;
3.  
4. void Start () {
5. mappedDepthLocations = new Kinect.ColorImagePoint[depthImageHeight*depthImageWidth];
6. depthStreamPixel = new Kinect.DepthImagePixel[depthImageHeight*depthImageWidth];
7. }
8.  
9. void Update () {
10.  
11.         if (depthStream.pollDepth () && colorStream.pollColor()) {
12.  
13.  
14.         for ( int i = 0; i < depthImageWidth; i++ )
15.             {
16.                 for ( int j = 0; j < depthImageHeight; j++ )
17.                 {
18.                     int index = i+j*depthImageWidth;
19.                     depthStreamPixel[index].Depth = depthStream.depthImg[index];
20.                     depthStreamPixel[index].IsKnownDepth = true;
21.                     depthStreamPixel[index].PlayerIndex = 0;
22.                 }
23.             }
24.  
25.             Kinect.NativeMethods. MapDepthFrameToColorFrame ( Kinect.DepthImageFormat.Resolution320x240Fps30, ref depthStreamPixel, Kinect.ColorImageFormat.RGB, ref mappedDepthLocations );
26.         }
27. }

This is giving me the following error: EntryPointNotFoundException.

After double checking the Kinect SDK reference ( http://msdn.microsoft.com/en-us/library/jj883690.aspx ), Ive seen that the class defined is within the CoordinateMapper class, which is not defined, but should it really be?

If thats not the problem, what could it be? Im kinda lost here, is the first time Im working with these DllImportAttribute methods.

 

if you look here : http://msdn.microsoft.com/en-us/library/microsoft.kinect.kinectsensor.mapdepthframetocolorframe.aspx

(MSDN has some good documentation on Kinect SDK btw)

It says: Obsolete starting with version 1.6; replaced by the CoordinateMapper.MapDepthFrameToColorFrame method. In versions prior to 1.6, this method maps a depth frame to a color frame.

So I guess you are using an SDK version > 1.6, use the CoordinateMapper.MapDepthFrameToColorFrame instead

 

So, should I just use this instead?

Code (CSharp):

1. [DllImportAttribute(@"C:\Windows\System32\Kinect10.dll", EntryPoint = "CoordinateMapper.MapDepthFrameToColorFrame")]
2.         public static extern int MapDepthFrameToColorFrame( DepthImageFormat depthImageFormat,  ref DepthImagePixel[] depthPixels,  ColorImageFormat colorImageFormat,  ref ColorImagePoint[] colorPoints);

 

hi thrmotta
i had same problem like u
did you find a correct way to map color to depth?

 

Its been awhile, but Microsoft released an official asset that supports Kinect on Unity. Here's the link: http://wiki.etc.cmu.edu/unity3d/index.php/Microsoft_Kinect_-_Microsoft_SDK

This asset allowed me to map color to depth correctly.