Kinect v2 with MS-SDK

Hi @roumenf ,
I am trying to do human projection mapping,
My camera is placed near the floor(1m) but my projector is placed near the ceiling.
I need to match the perspective of the output so that the usersLblTex appears over the player in the real world.
Any tips with how to go about doing this, or implement/get data from room alive toolkit?

 

First, set the sensor height and angle settings of KinectManager, as described here: http://rfilkov.com/2015/01/25/kinect-v2-tips-tricks-examples/#t13 This will fix the 3d joints' coordinates. Then set the main camera so that it matches the projector's location and angle. This may do the trick, as to me. The RoomAlive-project is also worth exploring.

 

Hi @roumenf ,

I have set the sensor height and angle correctly, and also set the main camera, but this will solve the problem only if there is a 3Dmesh/object.
The usersLblTex is a Texture, so wherever the main camera is wont affect the perspective.

We somehow have to get the point cloud/mesh rendered from a different view point.
(an example of this is seen in Kinect Studio; when view is set to 3D, you can move the camera around and get a different perspective)


If we are able to control the view point then I think it should be possible to run calibration in room-alive and apply the matrix in unity.

I somehow have a feeling that it could be solved by using MapDepthToSpaceCoords to compute the usersLblTex, could you please explain where and how it is currently used?
That way we might be able to get results by manipulating the kinectToWorld matrix.

Thanks for you time.
Apologies for the long post.

 

Try to add a Quad-3d object to the scene and set KinectScripts/Samples/PointCloudView.cs as a component to this object. This will show the point cloud you probably need. KinectManager.Instance.MapDepthToSpaceCoords()-function is not used anywhere in the demo-scenes at the moment, but you are free to use it, if you like.

 

hey，I have a problem about kinect and unity. I have used Kinect v2 with Ms-SDK to develop a application. It can work in pc platform（no problem as .exe）. But it can not work in webplayer platform. It seems like the error is because it can not find some components of kinect. Do you have any experience like me? PLEASE help me.

 

This is the desired behavior "by design". Let me explain. The web player as all web components is limited in using your local computer resources, like cameras, sensors, even local files (except cookies). And this is the way it should be, as to me. Otherwise, just imagine someone over the Internet, able to use your Kinect-sensor and utilize its cameras and body tracking features to see and track you in your room, whenever this web page is open in your browser.

 

I start the scene then i got these errors which i can't fixed:

1.KinectForUnity cannot be loaded. Please check the Kinect SDK installation.
UnityEngine.Debug:LogError(Object)

2.DllNotFoundException: KinectForUnity

Please help me.

 

Make sure you have copied the Assets/Resources to your Unity project, too. It contains the needed native libraries, shaders and resources.

 

@roumenf

Hi Roumen!

First of all - your asset is great and extremely helpful! I have a small problem though and I'm not sure where do I make a mistake. I wan to control pre-designed 3D humanoid model. It is placed in simple scene - plane and cube. However, after few seconds, my model is ´sinking´ down so that it is visible only from, more or less, the waist up. Your model is working fine, but maybe you have idea what should I do.

Thank you in advance

 

Hello, Roumen!
First of all, thank you very much for the asset. It's great and super helpful. I'm new to Kinect and Unity, and right now I'm using your AvaterControllerClassic to control a 2d character. It works any way. But I don't know how to constrain the rotation of x-axis and y-axis, and constrain the position of z-axis, so that the puppet can move just in two dimensional space. Can you give me some ideas about what I should do?

 

Hey Roumen,
I am having trouble with the speech manager in v2, it is not giving me any errors but never triggers the isListening. So for some reason the sensor is not detecting speech. but i can still use the kinect microphone for cortana and recognizes my speech.

 

Some how missed installing the speech run time sdk I thought i installed everything but now it works

 

@3esak , have you properly set the Kinect camera height and angle values? Also perhaps you have enabled auto detection which might be picking a surface that is not the floor. You can check by looking at the value in the inspector while 'playing'.

 

I suppose this is related to the physics and rigid body attached to the model. I don't remember how exactly I solved a similar issue some time ago. Try to disable the Vertical-movement setting of AvatarController, attached to the model and compare the other settings with the settings of the working model.

 

Open Assets/KinectScripts/Interfaces/Kinect2Interface.cs, find PollBodyFrame()-function and before 'jointData.kinectPos = ...', insert:
'float zPos = (j != 0) ? bodyFrame.bodyData.joint[0].kinectPos.z : joint.Position.Z;'
Then modify the next line like this, to use the new zPos:
'jointData.kinectPos = new Vector3(joint.Position.X, joint.Position.Y, zPos);'

 

Hello, Roumen,


Everything works fine, but I would like to know how to accurately detect the position of the eyes.
On the other hand there are some filters to prevent ( smooth ) tremors detected objects .
Thank you,

 

The settings for the height and angle are taken from the demo scene and the auto detection is disabled in my project. I think the problem was that I didn't freeze all the translations and rotations. This solved the problem but now I have a new one. I added textures to the floor plane and now my model is not sinking suddenly but gradually when I get closer to Kinect (but I'm still in working field). Before adding the material everything was working fine.

 

Hi @roumenf
Thank you for the great job you did with this asset.

General (maybe trivial) question to everybody:
Unity and scripting in general are quite new to me - right now my scene in Unity depends on where the human is detected (right now my model is at (0,0,0)). I would like to know if it is possible to somehow set the dependency between the real position of kinect and the virtual surroundings (in other words if I enter the room from the right side my model will be in the right side of the scene and not in the center).

Thanks in advance!

 

While waiting for kinect 2 may I test it with my old kinect 1?

 

Regarding the position of the eyes, take a look at KinectScripts/Samples/GetFacePointsDemo.cs
Which filters do you mean?

 

Please try some combinations of rigid body settings (freeze xyz-rotations and/or xyz-positions) and the vertical-movement setting of the AvatarController. If the problem persists, zip the project and send it to me per e-mail next week, so I could take a closer look.

 

There is a setting of AvatarController, called 'Pos relative to camera'. Set it to be the main camera (drag and drop the MainCamera-object over this setting). Then the position of the avatar in the scene will match the real user position with respect to the sensor.

 

Of course But if you have installed Kinect SDK 2.0 already, look at this tip first: http://rfilkov.com/2015/01/25/kinect-v2-tips-tricks-examples/#t21

 

hi can i get only the texture 2d of the closest user detected and render it ?
usersLblTex

 

http://rfilkov.com/2015/01/25/kinect-v2-tips-tricks-examples/comment-page-2/#comment-2132

 

May I ask you if the Windows cable adaptor for the kinect 1 is the same for kinect 2?

 

No, it's different and you need to have USB-3 port on your machine, to be used exclusively by Kinect 2. Here is the adapter: http://www.microsoftstore.com/store/msusa/en_US/pdp/Kinect-Adapter-for-Windows/productID.308803600

 

Hi roumanf
I faced one problem with this assets in unity5.3. Can you help me with this?

Assets/KinectScripts/Interfaces/Kinect1Interface.cs(569,64): error CS1061: Type `System.IO.FileInfo' does not contain a definition for `Length' and no extension method `Length' of type `System.IO.FileInfo' could be found (are you missing a using directive or an assembly reference?)

 

Yes. Unity 5.3 is not quite finished yet, as to me. This error is because the Web-player platform is selected by default. Open menu 'File / Build Settings', select 'PC, Mac & Linux Standalone'-platform and click 'Switch Platform'-button. The target should be Windows / x86 or x86_64. If the Windows standalone platform is not installed (vague described as 'No xxx module loaded'), don't invoke the Module Manager, but start the Unity-Download-Assistant again, and from the list of components select and install 'Windows Build Support'.

FYI: After running the asset for the 1st time in Unity 5.3 editor, it crashed while instantiating some prefabs (cubeman bone lines) that had nothing to do with the sensor or its drivers, but it's pure Unity stuff. After that all scenes worked normally. But if you don't have a specific reason to use Unity 5.3 at the moment, I would recommend to get back to Unity 5.2.x, just to protect yourself from some other surprises

 

Hi roumenf
Thanks for your advice. I done it. I am learning "How to make virtual dressing room" from the fitting room scene. I need your help with that. Actually when we turned in front of kinect clothes are again come to originally position. It's not showing the back of that model. Even i checked the option in kinect manager of allowed turn around. but it doesn't work. Please help me with this.

 

Hi roumenf

I'm doing the OverlayDemo on the vertical screen 900 x 1600 it not fix with the join position. So, please teach me the way. I'm doing the project run on the big vertical screen.

Thanks.

 

I was trying to get a hand joints position relative to the Avatar while evading the inclusion of the smooth factor. So I extended the controller class and tried to fetch the shoulder bone, which worked, but not the hand and it's tips. GetBoneTransform(jointMap2boneIndex[KinectInterop.JointType.HandRight])) doesn't seem to fetch the bone transform.

I have written a short extended Avatar Controller class to quickly test it, which is added as an attachment to this post. Just replacing the controller script on an avatar in the Avatar Demo should be enough for you to recreate this issue.

Any ideas why this is happening, a bug? Or am I just trying to achieve this in a wrong way?

 

One of the limitations of Kinect SDK is that it tracks users correctly, only when they face the camera. The turnaround-option you mention is only experimental and needs more work. Its goal is (at least partially) to overcome the turn-around limitation through some AI algorithms, but this is still to be proved.

 

In overlay demos 1&2, there is a component of BackgroundImage-object, called 'Portrait Background'. Enable it and set the aspect ratio of the game view to be 9:16. You can also modify the code (and aspect ratio) a bit, if it doesn't look 100% good on specific resolution. See this tip for more info: http://rfilkov.com/2015/01/25/kinect-v2-tips-tricks-examples/#t19

 

This way you are going to get the humanoid model transforms used for rotating its joints. The model's hand corresponds to KinectInterop.JointType.WristLeft/Right for instance. The model has no hand tip, but fingers with multiple parts instead. As there is no direct relation between some of the Kinect joints and the humanoid joints, I have commented out some of the entries in boneIndex2MecanimMap-dictionary in AvatarController.cs. You may uncomment or modify them, if you like. But as to me, a better way would be to get the joint positions you need from the KinectManager and subtract the position of the SpineBase, if you need them relative to the user.

 

Good morning and happy holidays,
In the scene Kinect FaceTrackingDemo4 could put the rigid mask superimposed with the image of the camera. I've tried but I can not.
Thank you.

 

Hi @roumenf, i need to get just the Hair of the user.
would there be a way to achieve this?

 

I don't know See the FacetrackingDemo3. Maybe if you look at the upper part of the head, you will get what you need. But, to estimate if the user has hair or not, or to recognize his/her hair style, you would need some computer vision algorithms, like OpenCV for Unity.

 

Thank you and sorry for the delay. See the FacetrackingDemo3 on how to get only the face image from the rgb camera. But I don't know how to apply it on the 3d face. I think there was a DirectX-algorithm for this, but I'm not sure. Please search in Internet, how to apply an image as a texture on a 3d-object.

 

Hello Roumen,
I try to use your FittingRoom1 demo. There is a problem when the user moves to left or right side of the screen, the dress doesn't suite the user, it shifts horizontally. What can we do with the subject? Thanks.

 

You can try to use the 'Adjusted Camera Offset'-setting of OverlayController-component to do a correction of this horizontal shift. May I ask you to send me some screenshots of this erroneous behavior by e-mail, because I'm not sure I understood it fully and correctly.

 

hi RF
I have a problem with your package .
I want to get face properties like eyeclosed , happy , and ...
but i dont find any function to support this in your package . can u help me?

 

See KinectScripts/Samples/GetFacePointsDemo.cs. You need a similar script, but instead of face points, get the face properties you want. Before you start your script, open KinectScripts/Interfaces/Kinect2Interface.cs, find InitFaceTracking()-function and uncomment the face-frame-features you want to get along with the face result. Don't forget to save the file after this modification.

 

I have an interesting problem trying to combine the FaceTracking2 demo and Background removal.

I have a static GUITexture background (of some wood) with it's own camera set to Depth -2 (Depth only)

Next, I'm trying to have a GUI Texture as the video feed from the Kinect, with background removed (Using Background Removal Manager script on the Kinect Controller and Foreground to Image script on the GUI Texture), it has another camera with a Depth of -1 (Depth Only). Then I have my 3D model (The Viking Helmet) with Model Hat Controller on the Main Camera which has a Depth of 0 (Depth only, Mixed Culling, ignoring the layers the background and cam feed cameras are on)

This works pretty well, except the video feed from the Kinect always renders above the Main Camera. Modifying the depth of the Cameras doesn't have any bearing on the ordering.

Basically I'm trying to cut the user out from the video feed, add them on top of a 3D scene or static background, and have them control their Viking helmet (which should appear on top of their head). Any ideas what the problem could be?

 

@MoKargas , i guess the video from the Kinect is drawn in the OnGUI method..this will cause it to be rendered on top of everything.
you will have to probably assign the pixels to your own texture.

Code (CSharp):

1. KinectManager manager = KinectManager.Instance;
2. if(manager && manager.IsInitialized())
3. {
4.     Texture2D depthTexture = manager.GetUsersLblTex();
5.     Texture2D colorTexture = manager.GetUsersClrTex();
6.     // do something with the textures
7. }

http://rfilkov.com/2015/01/25/kinect-v2-tips-tricks-examples/#t19

I could be wrong, but I'm quite certain that this is your problem.

 

I suppose under video feed you mean the background-removed texture, don't you. There is a BackgroundRemovalDemo2-scene, which shows how to overlay 3d objects over the BR-feed. In relation to what vivalavida is saying above, I suppose your issue may be related to the layers rendered by each camera, some setting of the BR-manager or extra script you need to apply the BR-texture to the background. If you cannot resolve the issue alone, please zip the project and send it to me via WeTransfer, so I could take a look.

 

I'm having the same issue than MoKargas, I need background removal and facetracking with 3d models, so yes the example BGR-2 display a 3D object on top of the background-removed texture, but the coordinates of the 3D model are wrong, I'm using the FaceTrackingManager + ModelHatController.
Any idea of what is wrong ?

Thanks a lot !


Edit: Well it was just the camera's coordinates ... My bad.

 

@roumenf
Is there a way to limit the shaking when using the Model Hat controller ? I've been trying to compare the rotation from the last frame and if it's too small or too big then do nothing but it's not working.

Any ideas ?

Thanks a lot!

 

Try to lower the value of the SmoothFactor-setting of ModelHatController-component. This should make rotation changes smoother. But if the shaking is caused by positional changes (this is not smoothed at all in the script), you would need to modify it a bit. Tell me, if you need help for the script.

 

Is there any way to avoid that kinect do tracking for some part of the body? (like hands or foots for example)