Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Kinect v2 with MS-SDK

Discussion in 'Assets and Asset Store' started by roumenf, Aug 1, 2014.

  1. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Aimed to provide support to "Kinect v2 with MS-SDK"-package users.
    Please post your questions below.
     
  2. youarebritish2

    youarebritish2

    Joined:
    Nov 7, 2013
    Posts:
    29
    You are a life-saver. I had been scratching my head over trying to port the original version of this package to Kinect v2 for weeks now. I'll let you know if I run into any issues.
     
    roumenf likes this.
  3. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Thanks. I hope you're not the one who has rated yesterday my humble package low :)
     
  4. youarebritish2

    youarebritish2

    Joined:
    Nov 7, 2013
    Posts:
    29
    Most certainly not! Actually, the first thing I did after verifying that it worked was gave you 5 stars. :)

    I've been playing with it a little bit and it works great, although on each of the demo scenes, I sporadically receive this error message:

    IndexOutOfRangeException: Array index is out of range.
    KinectManager.UpdateUserHistogramImage () (at Assets/KinectScripts/KinectManager.cs:1405)
    KinectManager.UpdateUserMap () (at Assets/KinectScripts/KinectManager.cs:1298)
    KinectManager.Update () (at Assets/KinectScripts/KinectManager.cs:1175)

    I've not yet been able to manually induce it, but I'll let you know if I can find steps to reproduce.
     
  5. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Thank you for the stars and the valuable feedback!
     
  6. kaz2057

    kaz2057

    Joined:
    Jan 18, 2011
    Posts:
    326
    Why you deleted all the file for kinect V2 from the web?

    I cannot find bitbucket and asset store files ...

    Thanks
     
  7. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Sorry, I hope it is only temporary. It is due to request by a partner of mine. Bitbucket repository is there, but private.
     
  8. pyratej

    pyratej

    Joined:
    Jul 13, 2012
    Posts:
    12
    Doh, i thought i was going crazy. Couldn't find it. o_O Kinda weak mate. Hope its back up soon.
     
    roumenf likes this.
  9. Ecocide

    Ecocide

    Joined:
    Aug 4, 2011
    Posts:
    293
    Hi roumenf,

    first of all: thanks alot for providing this package for free!

    We build the Kinect right into our driving simulator, right now only to detect if the player is moving the head to the left or right while turning. I purchased the Kinect v1 Extras package and used the facetracking manager to achieve this. However, is it possible to track multiple faces?
    A quote from the Microsoft page:
    Is this the only way to do this? I hoped it was as easy as tracking different skeletons, therefore just indexing them correctly :-D

    The reason I posted this in the Kinect v2 thread is: will you provide an Extra package with face tracking abilities for the Kinect v2?

    Thanks again. I will definitely donate some money as soon as everything is working and I am able to use it in our project!
     
  10. Quaker_SDR

    Quaker_SDR

    Joined:
    Jun 21, 2013
    Posts:
    39
    tht package is still exist?
     
  11. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    @.Rattlehead, I'm having holidays now and cannot check all the details, but can't you just use the head rotation around Z for instance to achieve what you want? ShapeUnits are also available in FacetrackingManager. As to the multiple faces - unfortunately the face-tracking wrapper tracks only one face, as long as I remember. If it is important for you and you have cpp coding abilities, send me an email and I can you provide you the source of the wrapper, so you could extend it.
     
  12. Quaker_SDR

    Quaker_SDR

    Joined:
    Jun 21, 2013
    Posts:
    39
    @roumenf how can i get this package??
     
  13. Ecocide

    Ecocide

    Joined:
    Aug 4, 2011
    Posts:
    293
    This is exactly what I did, yes. Using the head rotation is all I need. But unfortunately your Kinect v2 package does not support this. There is a property or field for head rotation but it always returns zero.
    Regarding the fact I only need the head rotation around the z axis, do I have to use the facetracking feature at all?

    Have a nice holiday! It'd be great if you could take a look at it after your holiday time. No need to send the source for the wrapper atm, but thanks!
     
  14. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Send me an e-mail.
     
  15. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    You don't need face-tracking to get the head orientation. I have to look again, but I'm 99% sure the returned head rotation is not 0. Which property/field do you use?
     
  16. Ecocide

    Ecocide

    Joined:
    Aug 4, 2011
    Posts:
    293
    I used the bones array in the PointManController script in the example folder.
    Code (csharp):
    1. Debug.Log (bones [1].transform.localRotation.eulerAngles);
    Code (csharp):
    1.  
    2. //store bones in a list for easier access
    3.      bones = new GameObject[] {
    4.        Hip_Center,
    5.   Spine,
    6.   Neck,
    7.   Head,
    8.   Shoulder_Left,
    9.   Elbow_Left,
    10.   Wrist_Left,
    11.   Hand_Left,
    12.   Shoulder_Right,
    13.   Elbow_Right,
    14.   Wrist_Right,
    15.   Hand_Right,
    16.   Hip_Left,
    17.   Knee_Left,
    18.   Ankle_Left,
    19.   Foot_Left,
    20.   Hip_Right,
    21.   Knee_Right,
    22.   Ankle_Right,
    23.   Foot_Right,
    24.   Spine_Shoulder,
    25.   Hand_Tip_Left,
    26.   Thumb_Left,
    27.   Hand_Tip_Right,
    28.   Thumb_Right
    29.      };
    30.  
    The other bones rotations seem to work :-(
     
  17. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    That's all right. Head is an end-joint and doesn't have its own rotation. The bone is Neck-Head in this case. The Neck-joint gets rotation, as it needs to be oriented to the Head in 3d-space.
     
  18. Ecocide

    Ecocide

    Joined:
    Aug 4, 2011
    Posts:
    293
    Hm I tried using
    Code (csharp):
    1. Debug.Log (bones [2].transform.localRotation.eulerAngles);
    which is supposed to be the neck joint, but the values are pretty random imo. Do I have to take the general rotation of the body related to the Kinect into account or something like this?
     
  19. kirubhaUnity3D

    kirubhaUnity3D

    Joined:
    Sep 4, 2014
    Posts:
    1
    Hi,

    I have mapped kinect 2 with human avataar model in Unity version 4.5 and i am able to see the reflection of me in human avataar and it is really exciting.But only thing is head is not turning left or right side when my head is turning left or right and another thing is when i am standing still, my hands are shaking in avataar because the skeleton joints of hand in kinect is trembling.Can i make this as stable in avataar.

    Please provide your thoughts on it.

    Thanks
    Kirubha
     
  20. FranzGopez

    FranzGopez

    Joined:
    Oct 8, 2014
    Posts:
    1
    @roumenf how can I have a copy of this kinect v2 and MS sdk? its not in the asset store anymore
     
  21. tcboy88

    tcboy88

    Joined:
    Oct 3, 2014
    Posts:
    1
    @roumenf can we get the kinect v2 for MS SDK in private? since it is not in the asset store anymore.
    I want to try it for academic research purpose as a master graduate.
     
  22. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Soon it will be available again in the asset store. If you hurry up, send me an e-mail.
     
  23. jforder

    jforder

    Joined:
    Jun 30, 2012
    Posts:
    22
    Hello all,

    Has anyone got Kinect Face working in Unity yet?

    Cheers!
     
  24. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Not yet :)
     
  25. jforder

    jforder

    Joined:
    Jun 30, 2012
    Posts:
    22
    So just imported the latest Kinect v2 SDK for Unity. I opened up there demo scene and hit play and I'm getting a DLLNotFoundException. Is anyone else getting the same thing?

    The Error is curing at this line:
    Code (CSharp):
    1.  RootSystem.IntPtr objectPointer = Windows_Kinect_KinectSensor_GetDefault();
    This is the link where I got the package from. It's on their website.
    http://go.microsoft.com/fwlink/?LinkID=513177

    Pretty frustrating, because the DLL are there and in the right folders but unity doesn't seem to be able to find them.

    Be good to know if anyone else gets the same issue when importing this package.

    Thanks!
     
  26. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    First of all, which package have you imported and which scene did you run?
    If it is 'Kinect.2.0.1410.19000.unitypackage', make sure you have Unity pro. Otherwise it won't run directly.
     
  27. jforder

    jforder

    Joined:
    Jun 30, 2012
    Posts:
    22
    I'm running Kinect.2.0.1410.19000 with the lastest SDK installed. It's running in pro as well. I've been on the private forums for the Kinect program and not had a problem running previous builds.

    I tried running both scenes that are provided - Kinect View and Green Screen.

    I've tried to move the KinectUnityAddin.dll into different sub-folders of the plugins folder but no luck. I've got my work colleague to test it out on his pc and he doesn't have the same issue it works fine.

    I've tried the Kinect Body Basics on the SDK Browers and that works fine.

    I've changed the Unity project from x86 to x86_64, doesn't have a effect.

    I know its going to be something silly just trying to narrow down what the issue is.

    Here are some of the other things I've tried:
    PC Restart
    New project and reimported

    Here is what I'm getting:

     
  28. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Try to copy 'KinectUnityAddin.dll' from /Assets/Plugins/x86 to the root-folder of your Unity project (the one containing Assets-folder).
     
  29. FiveFingers

    FiveFingers

    Joined:
    Oct 15, 2009
    Posts:
    541
    We have the same exact problem with Kinect Microsoft SDK Uniy packages.
    Even moving the 64 bit .dlls (we are on window7 64 bit) does not help.
    Anybody tried this and know what the problem might be ?
     
  30. FiveFingers

    FiveFingers

    Joined:
    Oct 15, 2009
    Posts:
    541
    Found out you can run the newer 2.0 SDK only on Windows 8 platform.
     
  31. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Yes, exactly. I was quite surprised to read you managed to run it on Win7:
    "Supported Operating System: Embedded 8 Standard, Windows 8, Windows 8.1"
     
    FiveFingers likes this.
  32. FiveFingers

    FiveFingers

    Joined:
    Oct 15, 2009
    Posts:
    541
    We installed 1.7 driver infact and I was surprised to see the Face tracking example in the Microsoft SDK Toolkit work so good on a XBOX360 Kinect.
     
    roumenf likes this.
  33. mgear

    mgear

    Joined:
    Aug 3, 2010
    Posts:
    9,350
    Huh, if someone else is fighting to get kinect v2 starting today,
    it seems the previous sdk had hidden expiration date on 31.10, and full sdk2.0 is out.

    took few hours to find that out,
    there wasnt any notifications, just the device wouldnt start (no lights).
    http://vvvv.org/forum/dx11-kinect-v2-support-broken-as-of-nov.-1st
     
    Last edited: Nov 3, 2014
  34. FiveFingers

    FiveFingers

    Joined:
    Oct 15, 2009
    Posts:
    541
  35. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Kinect v2 can detect up to 6 people at the same time. You don't need to do anything specific to enable it. But how to get the information about the detected people with this package, I don't know. I suppose you need to look deeper into the package's code, or use the available manuals and/or samples.
     
  36. lmbarns

    lmbarns

    Joined:
    Jul 14, 2011
    Posts:
    1,628
    Love your kinect work!
     
    roumenf likes this.
  37. hillang1

    hillang1

    Joined:
    Nov 5, 2014
    Posts:
    2
    Take a look at the BodySourceManager.cs script that comes with the KinectView scene in the package. This shows how to get an array of 6 bodies and keep it updated with valid tracking IDs (which will change as users move into/out of view of the sensor).
     
  38. operator23

    operator23

    Joined:
    May 17, 2013
    Posts:
    14
    Thanks again for all of your work roumenf. This package is awesome and with it I have been able to create a simple, cheap mocap alternative. In previous versions I was unable to get good data from hands and head, so I used alternative motion control solutions to clean them up. I am coming back after being away for a bit, so I need update to the latest and see what has improved. Anyway, my question is your website (http://rfilkov.com/2014/08/01/kinect-v2-with-ms-sdk/) mentions updates in version 2.0, and under #6 it says "Added multi-source reader option (supported by Kinect v2 only)." Does this mean this package works with multiple kinect 2.0? Thanks!
     
  39. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Not really. Multi-source reader is a way to synchronize the data of color, depth and body streams. Multiple sensors are not supported by the Kinect-Unity wrapper for K2, which this package uses internally.
     
  40. darkhus

    darkhus

    Joined:
    Nov 30, 2012
    Posts:
    17
    Hello, I got Windows 8.1 and Kinect SDK installed, project works well on unity editor but problem start when I build application for desktop, I got message "OpenDefaultSensor failed", so I try with previouse verion of your package (that which was free) and got similar error "KinectForUnity cant be loaded. Please check the Kinect SDK installation." What should I do in thi scase? I really hav eno idea. thank you
     
  41. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    In order to have your build working, please select x86-Architecture in Build settings (NOT x86_64). The support for the x64-architecture is part of the next package update, which will be posted to Unity asset store next week.
    Please also mind that the previous package is for Kinect v1 only (i.e. either Kinect-for-Windows or Kinect-for-Xbox 360).
     
  42. darkhus

    darkhus

    Joined:
    Nov 30, 2012
    Posts:
    17
    thank you for reply and your help, now it works fine
     
  43. Gameccino

    Gameccino

    Joined:
    Aug 1, 2012
    Posts:
    40
    Hi, I have some issues with hands and head rotation. Hands looks erratic and innatural, head does not rotate on the y axis. This happens both on your demo scene and on an brand new scene with an humanoid model. Is this an issue you're aware?
     
  44. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Hi there, yes I'm aware. For the head, you can use the head-rotation, reported by the face-tracking manager. I'm working on wrists' rotations now, but it's interesting to me, what exactly do you mean by 'erratic and unnatural'. Please contact me by e-mail and send me a short video or some screenshots, demonstrating the issue, if possible.
     
  45. PranayKamat

    PranayKamat

    Joined:
    May 29, 2013
    Posts:
    6
    Hey Roumen,

    Are Bone constraints specific to the U_Character or is it in Kinect co-ordinates?

    thanks
     
  46. BlueOnE

    BlueOnE

    Joined:
    Jan 24, 2015
    Posts:
    16
    Hi, I was wondering if it's possible to map a character mesh exactly over the user on top of the color stream with this plugin?
    (kinda like what they did in this YouTube example:
    )

    Thanks, looks like a great plugin so far.
     
  47. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Bone constrains are not U_character specific. They are restrictions of the Kinect bone orientations, but are not perfect yet.
     
  48. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Yes, it is pretty much possible (in frontal view), although it is not easy. There is a KinectOverlayDemo-scene in the package. By the way, what you see in this video is not a mesh, but a set of lines, connecting the K2-tracked joints.
     
  49. BlueOnE

    BlueOnE

    Joined:
    Jan 24, 2015
    Posts:
    16
    Yes I know its not a mesh, but it was the closest I could find to a decent example.
    I got ur samples and looked through them, nice work, but I have some leftover questions if that's ok:.

    I'm hoping to align some sort of rigidbody skeleton (if not mesh, then possibly a set of rigidbody lines) over the user and add a matching floorplane under it, although the Overlay example got me in decent direction for the skeleton/joints, I cant seem to align my floorplane correctly with the actual camera's view and foot joints.

    Thanks in advance.
     
  50. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Skeleton and lines should be aligned in different ways, as to me. For the lines, you need to position the line ends at the correct positions, as in the overlay example. In case of skeleton, you need to scale it and orient the bones in such a way that the end-joints reach the correct positions.
    Regarding the floor question: there are two settings of KinectManager - SensorHeight (in meters) and SensorAngle (+/-, in degrees). You need to set them correctly first and then you'll get good positions for feet. Knowing the sensor height above the floor and its angle, you also know where the floor is, don't you.