Kinect v2 with MS-SDK

I have just bought the plugin and it works fine but I keep getting error messages about it only working with Unity Pro. Is this normal?

I'm also wondering if the plugin supports more than one user? No-one else around so I can't test it.

 

Hi, can you please send me a screenshot or a stack trace of these error messages. The asset should work with Unity free as well, doesn't it? To your 2nd question: Yes, up to 6 users can be tracked simultaneously by the Kinect v2. Feel free to test it.

 

It worked fine with the free version yes but I got a few messages in the Console Window.
I am working with pro now so not sure how I can replicate the messages - would they be saved to a log file somewhere?

 

Unity log file is located in C:\Users\{username}\AppData\Local\Unity\Editor, but there are only two logs, for the latest two editor sessions. Anyway, if the error pops up, please make a screenshot and send it to me.

 

Hi Rumen. I have just bought the asset. Everything works out of the box. However, the latency is noticeable. I use Unity Free on Windows 8, 8GB Ram with a Core I7, GPU is Gigabyte 7850 1GB.

Could you please tell me that it is normal of there must be something wrong with my set up?

Thanks.

 

Hi, what is exactly noticeable - the avatars respond too slow or the FPS is low? Do you have the depth or color camera images turned on (this could decrease the FPS a lot)?
Avatars in the scene have a smooth-factor setting you can increase to get a better responsiveness, or set it to 0 to remove the smoothing altogether. Also, you can remove the FacetrackingManager-component from the MainCamera to gain some speed. It is used mainly to track the head position and neck orientation.

 

I meant the responsiveness. After following your suggestions it is much better now. Thanks

 

Hi,
After checking out / dissecting your cubeman, overlay and kinectmanager script I managed to get something working.
Basically it's a combination of the overlaydemo and cubeman, but I managed to map 3d primitives on top of the human player instead of the line renderers.
Additionally, I altered the ingame Main Camera to actually update to the set sensorheight and sensorangle as given in the Kinectmanager, so it (roughly) matches up the scene with the ingame opbjects.
However, as seen in screenshot part A, my meshman is always tilted backwards slightly, which I can't explain. fully (cam is setup at height 3.45 [2.45+1], angle -25).
I expect this might have something to do with the maybe the ViewportToWorldPoint or KinectTRS (although idk whatever exactly this is).

The sensorheight had to be the actual height + 1m to fix the 0 line ground/foot location (not perfect though)., which I guess is due to the Kinect 0/middle line.

I was hoping you might have any info on where this angle is coming from,
looks like the overlay man is always straight infront of the camera, despite being at an angle??
If I can work this final tilt/angle thing out, I can finally get to actual gameplay coding.


When not updateing the camera position, it does as seen in screenshot part B (it walks from in air down into the floor)

Many thanks in advance for any information / tips whatsoever.

 

I think the scene camera should not be moved or tilted to reflect the sensor's position. The sensor-height and angle values are used to set up the transformation matrix, needed to convert the points, detected in sensor's own space to the world space. The tilted person means to me the sensor angle is not (or not correctly) applied to this transformation. Maybe it will be easier to locate the issue, if you zip your project, share it online and send me a link to it per e-mail.

 

Hi, Appreciate the assistance.
I just emailed you on the address mentioned in the documentation.
please don't hesitate to reply here or to the email if you want to know anything

 

Hello first off bought the package and have loved it so far. I was wondering if you could help me. I have been looking at the Interaction manager script and was wondering how i could add it so you could pick up 2 different objects using 2 separate hands at the same time. So far only line of code i have been able to add is a second hand texture

 

Look at the GrabDropScript.cs. The hand states are tracked independently. So, in your case, you don't need to check for primary hand and track the selected/dragged objects for both hands.

 

Can anyone verify that this package works with Unity 5?

 

When you raise your arm , wrist rotation look unnatural. AvatarsDemo.

 

Look at InteractionManager.cs, it calculates LeftHandScreenPosition with such codes:
leftHandScreenPos.x = Mathf.Clamp01((leftHandPos.x - leftIboxLeftBotBack.x) / (leftIboxRightTopFront.x - leftIboxLeftBotBack.x));
leftHandScreenPos.y = Mathf.Clamp01((leftHandPos.y - leftIboxLeftBotBack.y) / (leftIboxRightTopFront.y - leftIboxLeftBotBack.y));
leftHandScreenPos.z = Mathf.Clamp01((leftIboxLeftBotBack.z - leftHandPos.z) / (leftIboxLeftBotBack.z - leftIboxRightTopFront.z));
but i can not understand, is there anyone tell me why . Now i want to get the ‘HeadScreenPos’ , how to solve this problem, Many thanks in advance for any tips.

 

Do you know how i could lock an objects X position inside of the Interaction demo ? I just want to only be able to move it up and down when i grab and try move left or right it wont move. I have already tried locking it using rigidbody and have tried locking it a few way programmaticaly. Any thing you can suggest >?

 

Hi,

we are using your plugin in a customer project and high quality is a must. Regarding the Background Removal sample: Generating the stenciled user mask of the HD color image results in a texture with the dimensions of the user tex which is 512x424. So I have a low res color image where the rgb info is stenciled out. I guess it is possible to transform the user image coords to color space and stencil it then which would result in a HD stenciled version, am I right? Is that it what you refer to in your release note for the new version: A smoother background removal is in development ?
We would be very interested in getting a stenciled face of the HD image, so combine the facetracking rect with a user stenciled HD version of the RGB image. Can you tell me if you are working on the "smoother background removal" and when to expect it?
best,
Robert

 

has anyone tried applying animation to part of the avatar? apply an animation with a mask so just the arms animate, seems to leave the rest of the body in a bind pose fighting against the kinect????

 

Oh, my! I need to get a leave to respond to all Here some quick answers:

- Yes, the K2-package upgrades nicely to Unity 5. The next release will be fully Unity 5 compatible.
- The InteractionManager-code is like this, because it calculates the cursor position according to hand position within a body-relative space box. You can update the box coordinates, to do something similar for the head.
- To lock in X or Y direction in interaction demo, see the else-part in Update() of the GrabDropScript and comment out the lines you don't need. Experiment a bit.
- I'm working on a smoother background removal demo now. It will be part of the next release, probably at the beginning of April. If you need it sooner and bought the asset already, you may send me an e-mail (please include your invoice #). Then I'll send you back a pre-release version of the package.
- I think we cleared the animator/avatar issue here: http://rfilkov.com/2014/08/01/kinect-v2-with-ms-sdk/comment-page-1/#comment-1222

 

Hello Roumen and thanks again for your work.
I am using Kinect Manager computing User Map and Color Map, to extract contours of the body and I draw them on a plane with this script:

Code (CSharp):

1. public class UserContour : MonoBehaviour {
2.  
3.     Texture2D tex;
4.     public GameObject kinectPlane;
5.     // Use this for initialization
6.     void Start () {
7.  
8.     }
9.  
10.     // Update is called once per frame
11.     void Update () {
12.         KinectManager manager = KinectManager.Instance;
13.         if(manager && manager.IsInitialized())
14.         {
15.             tex    = manager.GetUsersLblTex();
16.             kinectPlane.GetComponent<Renderer>().material.mainTexture = tex;
17.         }
18.     }
19. }
20.  

The closest user should also interact with some objects in the 3D scene. This is the portion of code to click these objects moving a hand:

Code (CSharp):

1.  
2. //inside update()
3.  
4. isHandClick = false;
5. Vector3 screenNormalPos = Vector3.zero;
6. Vector3 screenPixelPos = Vector3.zero;
7. if (manager.IsLeftHandPrimary ()) {
8. //if the left hand is primary, check for left hand grip
9.     if (manager.IsLeftHandClickDetected ()) {      
10.         isHandClick = true;
11.     }
12.     screenNormalPos = manager.GetLeftHandScreenPos ();
13. } else if (manager.IsRightHandPrimary ()) {
14. // if the right hand is primary, check for right hand grip
15.     if (manager.IsRightHandClickDetected ()) {      
16.         isHandClick = true;
17.     }
18.     screenNormalPos = manager.GetRightHandScreenPos ();
19. }
20. if (screenNormalPos != Vector3.zero) {
21. // convert the normalized screen pos to pixel pos
22.     screenPixelPos.x = (int)(screenNormalPos.x * Camera.main.pixelWidth);
23.     screenPixelPos.y = (int)(screenNormalPos.y * Camera.main.pixelHeight);
24.     globalHandPixelPosition = screenPixelPos;
25.     Ray ray = Camera.main.ScreenPointToRay (screenPixelPos);
26.     // check for underlying objects
27.     RaycastHit hit;
28.     if (Physics.Raycast (ray, out hit)) {
29.     //Debug.Log ("HITTING SOMETHING: "+hit.collider.gameObject.name);
30.     foreach (GameObject obj in interactiveObjects) {
31.         if (hit.collider.gameObject == obj) {
32.             //Debug.Log ("HIT OBJECT " + obj.name);
33.             //interact with objects                        
34.         }
35.     }
36. }
37.  
38.  

The problem comes when there are a lot of users (more than 3 or 4)
The interaction lags a lot, and it is really difficult to control the pointer. What can I do to improve performance?
Thank you again

 

Hello roemenf,

I was wondering if you are still working on the hand/wrist rotations? If I hold my hand in front of me it looks like this on the avatar: http://puu.sh/gQcaa/4fa18f86b2.jpg

Hopefully you find the time to update this

aeonixion

 

Yes, I work on wrist rotations almost for each package release, when there is enough dev. time. I think it will be a bit better in the upcoming release. You will see..

 

Hi, at least try to disable (or modify, or simplify) the generation of the user-map texture in this case, to increase the performance. Then use the Unity Profiler to check where the bottleneck is. This might bring some ideas.

 

thank you again for your support.
in the Unity Profiler I can see KinectManager.Update() at about 50%, with peaks at 80% when new user comes. I think I should try to disable the joints updates for the non interacting users, since I don't need them to be fully tracked. Maybe it could be useful to add a "Limit skeleton tracking to closest user" toggle
I'll try commenting some lines..Thank you again!

 

That's good to hear . I want to shoot fireballs from my character's hand. Any idea when the next update will be?

 

Thank you for this suggestion! I added 'Max tracked users'-setting If you set it to 1 and combine it with 'Detect closest user' (enabled by default), you will get a closest-user-only tracking.

 

I submitted it today. It will be published, as soon as the Asset-store staff (hopefully) approves it.

 

that's great, can't wait to try it!
thank you again for your work!

 

Hi,

I want to move my character using rigidbody.addforce. However, if I add a rigidbody to the avatar and move a bit in real life, the character jumps up and spasms up and down. Have you got any tips how to use a rigidbody with the character controller script?

 

Is it possible to use multiple kinects with your plugin?

 

You can enable the 'Is kinematic'-setting of the rigidbody-component. But I don't quite understand, how would you apply forces and move the avatar along with your movements, at the same time.

 

No, this is not possible. Sorry.

 

Hi roumenf!
Excelent work with this asset!!!!
Does this asset work with Unity 5?
When will you release a smoother background removal?

Thanks!

 

Release v2.4 was already posted to the Asset store. If they approve it, it should be out this or next week, I hope.

 

Hey roumenf,

Quick two questions, does this asset support multiple kinects? Also can it detect more than one person?

 

Quick answers: multiple Kinects - no; multiple players - yes, up to 6.

 

Sweet thanks =)

 

Hi,

I was wondering if there's a list of the changes that happened in the last update? I only just updated.

Also, is it possible for you to add all the files into 1 folder and then export a package? I had moved all of your plugin files into a seperate folder and it kinda got messy after updating. If it's not possible, no problem.

Btw, wrist rotations have really improved a lot. Still some issues but much better, so good job on that one.

 

The list of changes is available in the Asset store, when you click on the asset version (currently 2.4): https://www.assetstore.unity3d.com/en/#!/content/18708 They are listed here, as well: http://rfilkov.com/2014/08/01/kinect-v2-with-ms-sdk/ And finally, the full list of changes is the Assets-folder, after you import the package - 'WhatsNew-Kinect2-MsSdk.pdf'.

Ok, I'll move the demo-scenes into a separate folder, not to mix with the redistributable stuff, like KinectScripts, Resources and Standard Assets. Would that be enough?

 

Yep that would be great. Thanks again and keep up the good work!

 

Hi, Roumenf.

Thank you for this great asset, I just bought it and after 10 minutes I am already playing with it. I would like to know if there is any additional setup needed in order to have wrist rotation detection and ankle rotation detection. I can't see any changes when I rotate the wrist or ankle. I have double-checked the avatar and they are both correctly assigned, but still no luck.

Thank you!

 

Hi Alvaro,

There is a setting of KinectManager (a component of MainCamera in the example scenes), called 'Allowed hand rotations'. Select 'All' to see all hand rotations. 'Default' skips the rotations around the bone, which might not always be correct. For the ankles there are no limitations, as far as I remember.

 

Hey Roumenf, is it possible at all to get any support with a non Unity 5 version of this plugin?
I am working in the latest version of 4, and I have been using your plugin from "https://github.com/rfilkov/kinect2-unity-example-with-ms-sdk"
I love the plugin, but there is only one issue Im having (and im struggling furiously to try and resolve it).
The rotations dont seem to apply to the pointman points, so I cant add in arms that rotate / move like what the lines are representing.

Maybe I am missing something as I am quite new to programming with the kinect V2.
Is there some way that I could possibly get some help with this?

Cheers,
Trenalok.

 

Hi Trenalok, you're using an extremely old wrapper, from the early days of Kinect-v2. The updated project is here: https://www.assetstore.unity3d.com/en/#!/content/18708 Please contact me to get its older version, working on Unity 4.6.

 

Hi roumenf,

Your work seems amazing !!

I am a student and working on an immersion project and I would like to ad movement to it, using the xbox one kinect. I would have several questions, if you don't matter :

• In the description of the asset : "be used in Unity Pro and Unity Personal editors", does it mean that we can use it with the free version of Unity ?
  
• Does it works only with the last version of unity ?
  
• Can we detect several avatars on the same scene ? (I think I saw something like 6 ?)

I would be very excited to use it if I can use it for my project !!

Thanks a lot !

Jack

 

To your questions:

• Yes, the K2-asset works with the Pro- and Free-versions of Unity editor. The Free version is now called 'Unity Personal'.
• I have an older release (v.2.3), which was compiled for Unity 4.6. The latest release works with (at least) Unity 5.0.1.
• Kinect v2 can track up to 6 persons. This is also supported by my K2-asset, with a component setting called 'Player Index'. Here, value of 0 means the 1st detected player, 1 - the second, ..., 5 - is the 6th one.

 

@roumenf Your plugin is really great. I had a more general Kinect question for you, and was wondering if you knew if it was possible since you seem to be a Kinect/Unity expert.

I was interested in grabbing the Color source from the Kinect stream and creating an HDRI (.exr/.hdr) texture from it. I was wondering if you think this is possible, and if so, do you know of a way to do this in Unity?

I've seen online tutorials on converting normal images to HDR through taking the same picture at different exposures, and blending the multiple pictures together to get an HDR image. My thoughts were to take the Kinect Color source stream every 3 frames and blending the image somehow, but I'm not sure if that way makes sense, or would be the best way to achieve what I'm trying to attempt.

 

Hi Ben, I'm not expert in HDR, but (maybe) one way to do it would be to enable the HDR-setting of the camera, while rendering the Kinect camera output as background. Look here: http://docs.unity3d.com/Manual/HDR.html

 

Ok, thanks. I was under the impression that for Unity to output proper HDR that we needed to use a HDR texture rather than have it render HDR, but I guess I'll try it out.

 

Well, I said I'm not very familiar with HDR This was just an assumption. Experiment a bit and you will find it out.