Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Kinect v2 with MS-SDK

Discussion in 'Assets and Asset Store' started by roumenf, Aug 1, 2014.

  1. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
  2. Balours

    Balours

    Joined:
    Nov 27, 2013
    Posts:
    59
    @roumenf

    Indeed, lowering the smooth Factor reduces the "jumps" but it gives the impression of latency. Sooo, I think I need some help for the script, please.

    Thanks a lot !
     
  3. eco_bach

    eco_bach

    Joined:
    Jul 8, 2013
    Posts:
    1,601
    Hi, trying to texture the mesh in the face demo with an actual mask or face image. Since the mesh is dynamically created I am at a loss as to how this might be accomplished. Any ideas?
     
  4. eco_bach

    eco_bach

    Joined:
    Jul 8, 2013
    Posts:
    1,601
    I occasionally get the error message 'Kinect Manager is missing or not initialized'. Does anyone know how to get rid of this?
     
  5. adri1992

    adri1992

    Joined:
    Dec 14, 2015
    Posts:
    34
    Last edited: Jan 29, 2016
  6. rizawerx

    rizawerx

    Joined:
    May 30, 2013
    Posts:
    38
    Hi, I just bought this plugin..
    Nice job for making my life easier :)
    Right now I need some help....Every time I run a demo scenes containing avatar like avatars demo, face tracking demo, fitting room demo, Unity always crashed. I try v5.3, v5.2 even v5.0.1 and always crashed..also tried 32 and 64 bit..same result.
    But other demos works well, and Kinect SDK samples also works well. I'm using Kinect for Windows and USB 3.0 pci-e extension....
    Any idea to avoid the crash? Thanks in advance

    Update:
    Sorry, I just read here
    http://rfilkov.com/2014/08/01/kinect-v2-with-ms-sdk/comment-page-1/
    that it's a known bug..
    After I deactivate all facetracking manager, all demos runs fine in unity v5.3.1f1...I also found out kinect sdk itself crashed when I run FaceBasic - WPF sample...
     
    Last edited: Jan 30, 2016
    roumenf likes this.
  7. ralf_b

    ralf_b

    Joined:
    Jul 9, 2013
    Posts:
    48
    @roumenf
    First off thanks for your great Kinect v2 examples they are all working fine.

    Wonder if you have got any idea to realise a holographic effect like this:

    I tried several things tracking my head and moving the scenes camera accordingly but I just cannot get it right. I have been googeling a lot and did not find any clues on how to set something like this up :/

    Any help or tips would be great
     
  8. vivalavida

    vivalavida

    Joined:
    Feb 26, 2014
    Posts:
    85
    I guess an effect like this requires the projection matrix to be modified.
    Changing the camera position and rotation wont give you this effect.
    I'd be interested to see the results when you get this working.
     
    roumenf and ralf_b like this.
  9. ralf_b

    ralf_b

    Joined:
    Jul 9, 2013
    Posts:
    48
    roumenf and vivalavida like this.
  10. eco_bach

    eco_bach

    Joined:
    Jul 8, 2013
    Posts:
    1,601
    I am working with Unity and the Kinect v2 SDK. My goal is to use the data from the Kinect sensor to control facial expressions on an EXISTING face or mask mesh.

    I know that it is possible to create dynamic mesh data from the Kinect as in the demo, but in my case I want to use an existing mesh.

    Can anyone possibly point me to any examples or tutorials that might help?
     
  11. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Hi, sorry for the delayed response. I'm very busy these days at work. What do you mean by 'EXISTING face mesh'? The mesh is generated dynamically at run-time, to match your face. I'm not an expert in texturing meshes, so don't know how to put the real face image over the mesh. On a 2nd thought, maybe the approach should be something similar to the KinectScripts/Samples/PointCloudView.cs. What do you think?
     
  12. Balours

    Balours

    Joined:
    Nov 27, 2013
    Posts:
    59
    Hey @roumenf !

    Any chance to activate the Background Removal Manager at runtime, for example displaying the video feed in full screen, then after a while remove the background ?
     
  13. isterling

    isterling

    Joined:
    Oct 11, 2015
    Posts:
    5
    Hey @roumenf @Ben BearFish and @ralf_b ,

    I am also wanting to integrate the Kinect v2 for a project involving the creation of a holographic effect in Unity. I am using the projection matrix script which was discussed here: http://forum.unity3d.com/threads/using-projection-matrix-to-create-holographic-effect.291123/

    I guess the main challenge I am having right now with the Kinect v2 is inputing the face tracking data from the sensor to control the position of the camera which has the projection matrix component attached. Any assistance would be helpful right now.
     
  14. vivalavida

    vivalavida

    Joined:
    Feb 26, 2014
    Posts:
    85
    Hey @roumenf , @ralf_b , @isterling

    I thought I'd give the problem a shot myself.




    Is this what is expected?
    Let me know if anything needs to be modified.
     
    roumenf, ralf_b and isterling like this.
  15. isterling

    isterling

    Joined:
    Oct 11, 2015
    Posts:
    5
    @vivalavida

    Looks awesome. I'm curious as to how this effect would look with a slow moving particle effect on the screen to see if it would create a certain "pop out" effect as the camera moves in and out of the Z axis and digital objects fall into the frustum. I've yet to have the opportunity to implement the camera script along with the head tracking capabilities of the Kinect v2, so i'm guessing I add the KinectManager and FacetrackingManager components to the Main Camera, which contains the projection matrix algorithm.
     
  16. ralf_b

    ralf_b

    Joined:
    Jul 9, 2013
    Posts:
    48
    @vivalavida it definately looks like what I was trying to achieve

    I have almost done it myself unfortunately I get some weird camera behaviour.

    I used "camera_frustum_display.cs" by Chrarc on the camera for the projection matrix.
    I also put "s02headtrack.cs" on the camera to track the head (it needs the "KinectManager.cs" from the Kinect v2 examples https://www.assetstore.unity3d.com/en/#!/content/18708 which I suppose am not allowed to post here ;) )
     

    Attached Files:

  17. jojoh

    jojoh

    Joined:
    Jan 22, 2013
    Posts:
    15
    Hello hello!

    Do you have any idea why some examples do not work with record/playback function of Microsoft Kinect Studio?

    Afaik, the 3 overlay demos.

    Thank you in advance,
    jochem
     
    Last edited: Feb 8, 2016
  18. eco_bach

    eco_bach

    Joined:
    Jul 8, 2013
    Posts:
    1,601
    Is there any way to improve the quality of the background subtraction so the masking is less "blocky" ?
     
  19. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    This should be possible, but don't forget after you activate the component, to invoke its Start() method in order to initialize it, and before you deactivate it - to invoke its OnDestroy() method. Make these methods of BR-Manager public, too.
     
  20. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    They should work the same way, as with the real sensor. I suppose you forgot to enable the 'Nui Uncompressed Color'-stream when re-playing the saved file, which is a major part of these demos.
     
  21. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
  22. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    @vivalavida Thank you for answering the 'holographic' questions above :) I'm not an expert in the area, but one of the most creative people I know, who was one of the pioneers working on Kinect-based holographic effects, is Davy Loots. Please tell me, if I should ask him say anything on the subject, too.

     
    Last edited: Feb 18, 2016
  23. Balours

    Balours

    Joined:
    Nov 27, 2013
    Posts:
    59
    Indeed, and you need to set the GUITexture to null before activating the Background Removal Manager !
    Thanks.

    I'm also interested in the new background removal so I've sent you a mail :)
     
    roumenf likes this.
  24. jojoh

    jojoh

    Joined:
    Jan 22, 2013
    Posts:
    15
    Thanks man, i actually thought I had it enabled, but they could use an UX designer at microsoft..

    Cheers
     
    Last edited: Feb 11, 2016
    roumenf likes this.
  25. vivalavida

    vivalavida

    Joined:
    Feb 26, 2014
    Posts:
    85
    wow that's some nice work there.
    Anyway if not too much trouble, it would be nice if he's had any experience with human projection-mapping, to point to some sources that implement this with the KinectV2 or even some useful literature if possible.

    Thanks a lot.
     
    roumenf likes this.
  26. isterling

    isterling

    Joined:
    Oct 11, 2015
    Posts:
    5
    Still having some trouble with the headtracking / projection matrix setup. I have the projection matrix script attached to the camera, and it functions beautifully.
    I also have a head tracking script (mentioned here: http://rfilkov.com/2015/01/25/kinect-v2-tips-tricks-examples/ ) attached to the camera.
    I then attached the KinectManager and KinectFaceTracking to the MainCamera as well, yet the camera is unresponsive. Any suggestions?

    Cheers guys.
     
  27. vivalavida

    vivalavida

    Joined:
    Feb 26, 2014
    Posts:
    85
    @isterling,
    1) are you getting the desired effect if the Kinect is not involved? i.e. can you manually move the camera and get the desired perspective?(it seem that you can)
    2)Try out the head tracking script in an empty scene without the projection matrix, try moving the camera.
    3) If it all works i'd suggest attaching the HeadTraking script to an empty and make the parent of the camera.
     
  28. Daloots

    Daloots

    Joined:
    Sep 22, 2012
    Posts:
    4
    Hi guys,
    I've been meaning to post the code for the holographic effect based on Roumenf's awesome plugin to the asset store for a few years now and never got around to it :oops:.
    So I'll just post the part that matter here.

    The script itself is very short and simple and agnostic as to where you get the head position from. So you'll need to update the HeadPosition property somewhere. It should be real simple using Roumenf's plugin. I've used a few different head tracking methods over the years like a reflective dot on the stereoglasses from a 3D TV (really cool!) or the Intel Realsense (not so cool).

    The parameters of the script are:
    • ScreenWidth/ScreenHeight: the (physical) dimensions of the display. This is needed to get the perspective perfect. The dimensions should be given in meter. Unfortunately it also means that you'll have to make those configurable if you want to distribute your application or run on several displays.
    • HeadPosition: this should be updated each frame somewhere to indicate the relative position of the viewer with respect to the middle of the screen (the origin). Take into account the relative offset of the Kinect when you calculate this.
    Even though the effect is really nice on video, in real life you really need stereoscopic viewing for a convincing result. So if you have access to a 3D display, try it! You'll need to modify the script slightly to calculate a projection matrix for each eye, by offsetting x about .03 meters for each eye.

    I'm not sure what you mean with human projection-mapping?

    Cheers!
    Davy
     

    Attached Files:

    MaximumFX, isterling, ralf_b and 2 others like this.
  29. isterling

    isterling

    Joined:
    Oct 11, 2015
    Posts:
    5
    Hey guys, thanks for all the help so far. I am currently having trouble trying to restrict a game object that is being controlled by a player's head to moving along the X axis and rotating along the Y axis. I'm trying to do something like :
    Code (CSharp):
    1.     if (newPosition.x >= MinOffset && newPosition.x <= MaxOffset)
    2.             {
    3.             faceManager.GetHeadPosition.Translate(translateVector);
    4.             }
    5.  
    Can someone point me in the right direction? I've looked through many of the examples and couldn't find anything similar, or maybe I missed it.

    Thanks!
     
  30. vivalavida

    vivalavida

    Joined:
    Feb 26, 2014
    Posts:
    85
    Code (CSharp):
    1. KinectManager manager = KinectManager.Instance;
    2.  
    3. if(manager && manager.IsInitialized())
    4. {
    5.     if(manager.IsUserDetected())
    6.     {
    7.         long userId = manager.GetPrimaryUserID();
    8.  
    9.         if(manager.IsJointTracked(userId, (int)KinectInterop.JointType.Head))
    10.         {
    11.             Vector3 headPosition = manager.GetJointPosition(userId, (int)KinectInterop.JointType.Head);
    12.             Quaternion neckRotation = manager.GetJointOrientation(userId, (int)KinectInterop.JointType.Neck);
    13.             // do something with the head position and neck orientation
    14.         }
    15.     }
    16. }
    and then clamp the xposition after getting the value?

    http://rfilkov.com/2015/01/25/kinect-v2-tips-tricks-examples/#t9
     
  31. isterling

    isterling

    Joined:
    Oct 11, 2015
    Posts:
    5
    @vivalavida

    Hey thanks for pointing that out. So I am still having some trouble implementing a clamp function to the Vector3 headposition. Here's what I have so far:
    Code (CSharp):
    1.                 if (manager.IsJointTracked(userId, (int)KinectInterop.JointType.Head))
    2.                 {
    3.                     Vector3 headPosition = manager.GetJointPosition(userId, (int)KinectInterop.JointType.Head);
    4.                     //Quaternion neckRotation = manager.GetJointOrientation(userId, (int)KinectInterop.JointType.Neck);
    5.  
    6.                     // do something with the head position and neck orientation
    7.  
    8.                     float xMove = headPosition * Time.deltaTime * speed;
    9.                     transform.Translate(headPosition, 0f, 0f);
    10.                     transform.position = new Vector3(Mathf.Clamp(transform.position, xMin, xMax), transform.position.y, transform.position.z);
    However, headPosition does not return float values. I tried calling the GetFacePoint method in a similar fashion, but had some difficulties with that as well. Apologies for the seemingly trivial coding questions, as I have only made 2D sidescrollers in Unity, so this is somewhat of a new batch of challenges. Take care.
     
  32. vivalavida

    vivalavida

    Joined:
    Feb 26, 2014
    Posts:
    85
    Code (CSharp):
    1.   Vector3 headPosition = manager.GetJointPosition(userId, (int)KinectInterop.JointType.Head);
    2.    transform.position = new Vector3(Mathf.Clamp(headPosition.x , xMin, xMax), transform.position.y, transform.position.z);
    headPosition is a Vector3 it wont return float values, but you can get its x,y and z components, which are floats.

    ps. why were you doing Time.deltaTime?
     
    roumenf likes this.
  33. jonathan243132

    jonathan243132

    Joined:
    Feb 26, 2016
    Posts:
    2
    Hi guys,

    Is there a way that I can rotate a character when it is controlling by the avatar controller?
    My character's upper body is now controlled by Kinect with the avator controller classic and I use animator to apply some animation to the lower body, what I want to do is that when I turn my waist to a certain range of angle, it will able to turn the character facing direction. The idea is that I want the character to turn its facing direction and what I need to do is just turn my waist left or right a little and then back to center.

    However, the upper body will not turn at all, it will just base on my body movement and seems that other rotation can't turn it the legs are working fine and the upper body will just remain at the same direction. I tried to use a game object as the parent of the character so that when the game object turn, it should able to turn the whole character but it fail.

    Is it the avatar controller stopping the rotation happen or it will just map my body rotation and I cannot override those rotation?

    Below is my failed implementation, please do not care about the rotation angle because it is just a very brief idea

    Code (CSharp):
    1. public class Rotate : MonoBehaviour {
    2.  
    3.     public GUIText infotext;
    4.     private KinectInterop.BodyFrameData bodyFrame;
    5.     private KinectInterop.BodyData bodyData;
    6.     public Quaternion bodyrotation;
    7.     private KinectManager manager;
    8.     private KinectInterop.SensorData sensorData = null;
    9.  
    10.  
    11.     // Use this for initialization
    12.     void Start () {
    13.      
    14.     }
    15.     void Update()
    16.     {
    17.         if (manager == null)
    18.         {
    19.             manager = KinectManager.Instance;
    20.         }
    21.         if (manager != null && manager.IsInitialized())
    22.         {
    23.             bodyFrame = manager.getBodyFrame();
    24.         }
    25.         long userID = manager.GetPrimaryUserID();
    26.         bodyData = manager.GetUserBodyData(userID);
    27.         if (bodyData.bodyTurnAngle < 315 && bodyData.bodyTurnAngle > 270)
    28.          {        
    29.              Quaternion Left = Quaternion.AngleAxis(-2, Vector3.up);
    30.              transform.rotation = Quaternion.FromToRotation(Vector3.right, new Vector3(-1,0,0));
    31.         }
    32.          if (bodyData.bodyTurnAngle < 90 && bodyData.bodyTurnAngle > 45)
    33.          {
    34.              Quaternion Right = Quaternion.AngleAxis(2, Vector3.up);
    35.              transform.rotation = Quaternion.FromToRotation(Vector3.right, new Vector3(1, 0, 0));
    36.         }
    37.     }
    38. }
     
  34. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    There is a setting of AvatarController, called 'External root motion'. If you enable it, then you could rotate the model and the Kinect-detected rotation will be added to that. The extra is that in this mode you should move the model too, or modify the AvatarController's code a bit.
     
  35. jonathan243132

    jonathan243132

    Joined:
    Feb 26, 2016
    Posts:
    2
    Thanks for your help, I have stuck in this problem for months. However, is this only included in the latest version of the asset? I am using v2.7.0 of this asset and I cannot find the "External root motion" in the Avatar controller. Is there a way that I can add some code to apply this "External root motion" in the Avatar controller if it is not included in v2.7.0?

    I have stuck in this problem for months,
     
    Last edited: Feb 29, 2016
  36. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Sorry for the delay, I'm overloaded with projects at work these days. Yes, I think this setting was included in the latest release. Just download it from the Asset store and try to replace the AvatarController-script (save the original one somewhere first). If compilation or other kinds of errors pop up, open the updated script and see where 'externalRootMotion' is used. Then copy only these parts to your script. I think it was used no more than twice - once for motion and once for rotations.
     
  37. edufurla

    edufurla

    Joined:
    Mar 7, 2014
    Posts:
    19
    Hi.
    I have to say this is an amazing asset.
    I am working in a project that uses background removal and the 3th example I saw in twiter could be a great improve to me.
    Any chance to update in next two or three days or I receive a beta version of this scene?
     
    Last edited: Mar 1, 2016
  38. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    I just answered this on Twitter: https://twitter.com/edudeck16/status/704718042794745857
     
  39. Huy Thach

    Huy Thach

    Joined:
    Mar 27, 2013
    Posts:
    6
    Can you give us a simple version to check how it works.
    Even with logo, watermark or something like that.
     
  40. rickg_la

    rickg_la

    Joined:
    Mar 8, 2016
    Posts:
    2
    Just wondering if things are working for 5.3 yet. I tried it out and I'm getting the error:

    Assets/KinectScripts/Interfaces/Kinect1Interface.cs(569,64): error CS1061: Type `System.IO.FileInfo' does not contain a definition for `Length' and no extension method `Length' of type `System.IO.FileInfo' could be found (are you missing a using directive or an assembly reference?)

    thanks
     
  41. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Well, in order the "things" to work, you'll need to switch from the default (and deprecated) Web-Player to Windows-standalone platform in 'Build settings'. More information is available here: http://rfilkov.com/2014/08/01/kinect-v2-with-ms-sdk/#ki1
     
  42. LennartJohansen

    LennartJohansen

    Joined:
    Dec 1, 2014
    Posts:
    2,394
    Does this asset work with multiple kinect scensors? The Kinect SDK seem to support up to 4 sensors for skeleton tracking.
     
  43. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    No, it doesn't. And Kinect SDK 2.0 works with one sensor only, too.
     
  44. rickg_la

    rickg_la

    Joined:
    Mar 8, 2016
    Posts:
    2
    I was just wondering if anyone has compiled the necessary steps and scripts to get the view-dependent "holographic" camera to work. It would be a good addition to the wiki. I recently jumped over to Unity from UE4 for this project because it just made sense seeing these discussions and info in the forums.

    To distill everything down...
    Is that a node with the KinectManager script and a camera with a modified SimpleHolographicCamera script are all one needs in a scene to make a holographic camera work?

    If so, I'm wondering how to control the "cam.transform.LookAt" properly. I'm getting a camera to move, and it's position appears to be correct, but the orientation is wacky. My thought is that it would point at the position of Kinect sensor (with an offset) rather than using the head position, making the assumption that the target window is stationary. I tried setting the LookAt to (0,0,1000) to have it look down Z and it didn't behave the way I would have expected.

    I attached the modified script that seems to have the position working. If anyone has this virtual window working, I'd love to see an example I could build from, and add the info to the wiki. My hope is to create one that tracks multiple people, has a default position and orientation, and can do a quick lerp from default to a "primary" user, and from user to user when someone new is determined to be "primary" and back to default if no one is in the capture space.

    Many thanks,
    Rick
     

    Attached Files:

    roumenf likes this.
  45. vivalavida

    vivalavida

    Joined:
    Feb 26, 2014
    Posts:
    85
    hey, check out the room alive toolkit.
    Though the Kinect SDK 2.0 works with one sensor, room alive toolkit has a way to synchronize data from multiple kinects over a network.
    I'm not fully sure how this is implemented, but should serve as a possible starting point.
     
    roumenf likes this.
  46. Huy Thach

    Huy Thach

    Joined:
    Mar 27, 2013
    Posts:
    6
    I have bought your package,
    All demos are awesome, but in the KinectFittingRoom2 demo, the height of model is changed depending on the arm movement.
    As shown in the attached file:
    1. If arm is down, the height of model is lower than the height of user,
    2. If arm is horizontal, the heights of both model and user are equal,
    3. If arm is up, the height of model is higher than the height of user.

    Do you know how to fix this problem?
    Thank you.
    1_cut.png 2_cut.png 3_cut.png
     
  47. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Try to disable the 'Continuous scaling' of AvatarScaler-component of the model. You can also adjust the 'Body scale factor', if needed.
     
  48. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Yep, over the network.
     
  49. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    First I must admit I've never tried to do the holographic view (that's why asked the expert - @Daloots - to comment). According to his post, head position means "relative position of the viewer with respect to the middle of the screen (the origin). Take into account the relative offset of the Kinect when you calculate this." So, as far as I understand, the head position should be calculated with respect to the screen center's position. I suppose camera orientation has not much of a meaning, if you're changing its projection matrix. You should substract from the head position the position of the output screen's center (expressed in similar coordinates), in order to get "relative position of the viewer with respect to the middle of the screen". Again, I'm a layman on the subject, but hope this helps a bit :)
     
  50. Huy Thach

    Huy Thach

    Joined:
    Mar 27, 2013
    Posts:
    6
    I have tried to disable the 'Continuous scaling' of AvatarScaler-component, but the problem is not solved.
    Please check it in your computer.