Kinect with MS-SDK

I've read through 'Howto-Use-Gestures-or-Create-Your-Own-Ones.pdf' and the source code yet I'm still having trouble creating full body gestures such as lean left or lean right. Do you know of any other resources that could possibly be helpful?

 

Here is a resource. I think you can at least fine tune it by yourself and create a similar LeanRight-gesture from it. Should I explain the code?

Code (CSharp):

1.       // check for LeanLeft
2.        case Gestures.LeanLeft:
3.          switch(gestureData.state)
4.          {
5.          case 0:  // gesture detection - phase 1
6.            if(jointsTracked[leftShoulderIndex] && jointsTracked[rightShoulderIndex] && jointsTracked[leftHipIndex] &&
7.               (jointsPos[rightShoulderIndex].z - jointsPos[leftHipIndex].z) < 0f &&
8.               (jointsPos[rightShoulderIndex].z - jointsPos[leftShoulderIndex].z) > -0.15f)
9.            {
10.              SetGestureJoint(ref gestureData, timestamp, rightShoulderIndex, jointsPos[rightShoulderIndex]);
11.              gestureData.progress = 0.5f;
12.            }
13.            break;
14.          
15.          case 1:  // gesture phase 2 = complete
16.            if((timestamp - gestureData.timestamp) < 1.5f)
17.            {
18.              bool isInPose = jointsTracked[leftShoulderIndex] && jointsTracked[rightShoulderIndex] && jointsTracked[leftHipIndex] &&
19.                  (jointsPos[rightShoulderIndex].z - jointsPos[leftShoulderIndex].z) < -0.2f;
20.            
21.              if(isInPose)
22.              {
23.                Vector3 jointPos = jointsPos[gestureData.joint];
24.                CheckPoseComplete(ref gestureData, timestamp, jointPos, isInPose, 0f);
25.              }
26.            }
27.            else
28.            {
29.              // cancel the gesture
30.              SetGestureCancelled(ref gestureData);
31.            }
32.            break;
33.          }
34.          break;
35.  

 

If you wouldn't mind explaining the code, I would very much appreciate it.

 

Each gesture goes through several states. Its start must be detected and then the subsequent states it goes through, too. Leaning left means to me that user's right shoulder must go ahead of the rest of the body, while the left shoulder - not. Our gesture should detect the gesture start in its state 0, i.e. when the right shoulder starts going to the front and the gesture end in its state 1, when the right shoulder is enough ahead. For the position of the body I use the left hip, because it is not likely our right arm to get in front of the left hip and make it invisible to the sensor (like the right hip, for instance).

State 0: checks if our joints are tracked, if the right shoulder is ahead of the body (left hip) and not too much ahead of the left shoulder yet (no more than 0.15f, which means 15cm). If this condition is met, we save the gesture joint and progress. The invoked function SetGestureJoint() increases the state as well, so we get to..
State 1: where we check if within the allowed time (1.5s) our right shoulder is ahead enough to consider the gesture as complete. Ahead enough in this case means at least 20cm. In this case the gesture is considered complete. If the user moves too slow and doesn't lean enough within these 1.5s, the gesture gets canceled and detection gets back to state 0.

You're free to fine tune the constants as to your experience, as well as the code, if you have better ideas. By mirroring this code, you can also develop the respective LeanRight gesture. Hope this explanation helps.

 

Hi there !! How many skeletons can I use with this package ?? I need more than 2 players at same time controlling different skeletons !!

Thanks !

 

The Kinect SDK allows up to two tracked players (up to 6 can be tracked only as player positions). If you need up to six simultaneously tracked players, use Kinect v2.

 

Hello ! Thank you for this great package, it's very useful and well explained. I'm trying to make a game where 2 players have to follow directions to make gestures at the same time. It works perfectly for single player, but when 2 players are moving in the same time, only player 1 gestures are detected. Can you please help with this issue ?

 

Cleared per e-mail. But FYI: In the gesture listener's functions one would need to check if the userId matches Player1ID() or Player2ID().

 

hi roumenf,
The roatation of the character is restricted within +/- 80 degrees. Is it the restriction of the Kinect? I want the charater can rotate 180 degrees or more. Is there any methods to solve this problem?
Thanks.

 

Answered per e-mail. FYI: This limitation is imposed by the Kinect SDK implementation. Beyond +/-80 degrees the SDK gets confused, stops recognizing joints and mixes left and right.

 

hi roumenf,
I find the roatation of the character's knee is wrong,but the roatation of the character's elbow is right, when I work with the KinectAvatarsDemo. It's very hard to bend the character's knee. And It's OK to bend the CubeMan's knee.
Cound you help me and tell me why ?
Thanks !

 

Hi monkeymonkey, it is probably due to the bone-orientation-constraints. Disable the 'Use bone-orientation constraint'-setting of the KinectManager.

 

It works OK.
Thank you,roumenf

 

Hi Roumen,

Thank you for making this package available for free. I am trying to use it with Unity 5.0.1. In the example code I am seeing numerous deprecation notices. Rolling back to Unity 4.6 is not an option for me and I am not quite skilled in C# to safely update the code. Are you planning to have your package updated sometime soon so that it can run cleanly with Unity 5.0?

Thanks

 

Hi evolvingtech, I just tested the package import from Asset store into Unity 5 and, as far as I see, the Kinect-MsSDK-package upgrades nicely to U5. If you see many deprecation errors, maybe you still need to run 'Assets / Run API Updater...' Anyway, I'm going to update this package on the Asset store soon, but there are so many requests regarding the K2-package that there is almost no time left for this one Please have some patience.

 

Roumen,

I've has some mixed success. I am using Unit 5 (the Free version). I have Windows 7 Professional, MS Kinect SDK 1.8 installed. The Kinect I am using is Model 1414. I allow Unity to update the original scripts upon importing your package. No compile errors are reported.

When I try running any of your demo scenes, it will work for about close to a minute. Then it freezes. No error is reported. It just freezes. I stop the Play mode at try rerunning it and then I immediately get the following error:

NuiInitialize Failed - Device is not genuine.
UnityEngine.Debug:LogError(Object)
KinectManager:Awake() (at Assets/KinectScripts/KinectManager.cs:1014)

At this point, the only thing I can do is quit Unity and relaunch. Then I can play any of the sample demo scenes in your page. They will work for about a minute and then freeze, with the same kind of error.

I don't believe it's the hardware, because I have no difficulty running the samples in the Kinect for Windows Developer Toolkit v1.8.0.

I know you are busy with the K-2 package (I will soon be picking up the K-2), but if you can get a chance to look into this, it is greatly appreciated.

 

As I said already, I don't have such an issue. If you tell me how to reproduce this behavior, I can look into it. Otherwise, you need to research it a bit deeper, what exactly stops or goes wrong. It is not a black box. The full source code is there. Look for exceptions or add Log.Debug() messages here and there. If you can stop the play-mode, this means that Unity does not freeze.

 

Hi guys,

I am doing a project for the school and have a question which I cant do already a week.
I need to control main camera position with kinect. watch the video to understand what I am trying to say.

Please help )))

Thank you very much

 

What is the stopper? I would suggest that you use the Neck-joint's position and rotation to control the main camera. Maybe with some tweaks and smoothing (lerp, slerp).

 

Hello, i have a problem with this resource. The tracking works very well, but exactly after 90 seconds it stop track and if i stop and play again i found an error in the console which says "Device is not genuine". I remember not to have this problem in the earlier version.
Do you have any idea? Thank you.

 

Hi Guys, im using this SDK with my Kinect V1 and i want to get the angle between 3 joints. How can i get this?
Using the CubeMan demo i was able to get position and rotation of all bones.
Its possible? This image illustrates what i want.



Thanks!

 

Yes, I know about it. This is a major issue without a definite workaround yet. It is caused by Unity 5. After running Unity 5 you can see that Developer Toolkit 1.8 samples don't work more than 1-2 minutes, too. My best recommendation would be not to use it in Unity 5 environment, when possible. One other user has reported that "Install all windows updates and the newest video card drivers helped me" (thanks Paulo!). Also interesting is that I don't have this issue on my laptop, but only on my office PC, i.e. it still works sometimes in Unity5-environment. I'll definitely post an update, when I (or somebody else) find a workaround.

 

You can calculate the vectors between joints and then the angle between these vectors, like this:
Vector3 vFrom = vPosJoint1 - vPosJoint2;
Vector3 vTo = vPosJoint3 - vPosJoint2;
float fAngle = Vector3.Angle(vFrom, vTo);

 

That is a good news, I though my Kinect was broken. I hope someone can handle this issue soon, i've tried different version of Kinect SDK, from 1.5 to 1.8, but it's not the problem. I'll try what Paulo suggested.

Thank you!

 

First off, thank you for a great tool!

I am using this asset to track the users head position. This is working just fine in editor, and as a standalone build on the same computer. However after running the same build on a different (much more powerfull) computer I experience a jitter where, it would seem, the position of the camera keeps trying to move to (0,0,0) every other frame.

I am doing my tracking in Update(), then updating the camera position in LateUpdate() as I have seem recommended before but the problem still persists. Kinect drivers and video drivers are all up to date as well.

Has anyone else experienced this issue?

 

Hi,

how could i get six people's position,not joint's position ,just people

thanks alot!

 

You can always use Vector3.Lerp() to smooth the head position, but it is a bit surprising that a joint position may go to (0,0,0) each other frame. Can this be somehow related to the LateUpdate()? You can add code like this to your Update()-function, to check if the returned head position was wrong, or something else:
"

Code (CSharp):

1.     KinectManager manager = KinectManager.Instance;
2.  
3.      if(manager && manager.IsInitialized() && manager.IsUserDetected())
4.      {
5.        uint userId = manager.GetPlayer1ID();
6.  
7.        bool headTracked = manager.IsJointTracked(userId, (int)KinectWrapper.NuiSkeletonPositionIndex.Head);
8.        Vector3 headPos = manager.GetJointPosition(userId, (int)KinectWrapper.NuiSkeletonPositionIndex.Head);
9.  
10.        Debug.Log("Head tracked: " + headTracked + ", pos: " + headPos);
11.      }
12.  

"

 

Add this function to the KinectManager.cs. Maybe it will help:
"

Code (CSharp):

1.     public List<Vector3> GetAllUsersPositions()
2.     {
3.         List<Vector3> posBodies = new List<Vector3>();
4.  
5.         for(int i = 0; i < KinectWrapper.Constants.NuiSkeletonCount; i++)
6.         {
7.             KinectWrapper.NuiSkeletonData skeletonData = skeletonFrame.SkeletonData[i];
8.  
9.             if(skeletonData.eTrackingState != KinectWrapper.NuiSkeletonTrackingState.NotTracked)
10.             {
11.                 Vector3 skeletonPos = kinectToWorld.MultiplyPoint3x4(skeletonData.Position);
12.                 posBodies.Add(skeletonPos);
13.             }
14.         }
15.  
16.         return posBodies;
17.     }
18.  
19.  

"

 

Hi roumenf,
The fix turned out to be if(joint.z == 0) then don't do anything. I'm still not sure why it was happening, because the debug window wasn't displaying the behavior.

One other question though. The setup I'm using has a very high camera at about 2.3 meters and -27 degree angle. I'm running into an issue where the y values are becoming skewed as the user approaches the screen. For instnace, in the avitar demo, the avitar is lifted off the ground as the user moves closer. Is there any quick way to fix this? I imagine one could adjust the y values based on depth, but I thought the sensor height vs angle already took care of that.

thanks again!

 

Hi NKlawitter,
Thank you for researching the issue and figuring out the fix!
To your question: Yes, I think the SensorHeight and SensorAngle-settings should take care of this. Maybe such extreme camera position affects the tracking precision and this somehow skews the result. I have to re-check this. Please do a little experiment, to find out if the Y-position error is linear to the Z-distance. If it is, you can add a some linear Y-correction in the MoveAvatar()-function of AvatarController, as a quick workaround.

 

Hi roumenf, i m getting this error. in file from GetKinectColourMapTexture.cs, GetKinectHistogramTexture.cs, to kinect3DTwoHandCursor.cs file. its total 14 error in different files and all are asking about same thing. the error is:
error CS0246: The type or namespace name `FsmStateAction' could not be found. Are you missing a using directive or an assembly reference?
I hope u will help me.

 

You're probably using the 'Kinect and KinectExtras with MS-SDK Playmaker Actions'-asset, which is not mine. Anyway, I think you have not imported the Playmaker-asset before that one (because these are Playmaker actions and demo scenes), and this is causing the syntax errors you get.

 

ok thanks for your reply. that error error is gone. now i want to ask that how to animate characters upper body with kinect and lower body with mecanim animation controller. when i apply animation controller it is not following kinect and only play controllers animation please help it is very important. plz.

 

Look here: http://rfilkov.com/2015/01/25/kinect-v2-tips-tricks-examples/#t11 It is aimed to the K2-asset, but the procedure should be similar for Kinect v1 as well. Tell me, if you don't understand anything.

 

hello roumenf.

I am trying to make a gesture to make the character walk forward when the player moves their legs in a walking movement. Now, I have some troubles fine-tuning the gesture but that isn't the problem.

When I put the character to be moved forward, for some reason it snaps back into the initial position right away. Tried moving the camera instead, the character doesn't move. Tried making both the character and the camera move, the camera goes but the character snaps back to the initial position right away.

Any idea what I have to do to make it work?

 

I suppose your character uses the AvatarController as component. If this is the case, open KinectScripts/AvatarController.cs, find UpdateAvatar()-method and comment out this line: 'MoveAvatar(UserID);'. I hope this will fix your issue.

 

that worked, thank you.

 

Hi roumenf,

At first, I'd like to say Thank You for such a great package for sharing

Now I'm talking about my case: I added my avata into the KinectAvataDemo scene, which has the same bone hierarchy with U_CharacterFront, the AvataController.cs script was also added. When I start the scene, when both of the U_CharacterFront and my avata were detected, but only the bones of U_CharacterFront were tracked very well. The bones of my avata were not tracked! Here is a snap of my workspace:


I hasn't fixed it unitl now, consider as the same configuration, would you please give me some suggestions about this issue? Thanks.

 

Look here: http://rfilkov.com/2015/01/25/kinect-v2-tips-tricks-examples/#t3 The tips are related to the K2-asset, but the steps are similar in the "Kinect with MS-SDK"-asset, as well.

 

Hi roumenf,

Thanks for replying! I've tried and it works very well!!!
Thank you very much!

 

Hi roumenf,

*thank you for your great job with kinect

how can i add an avatar en runtime, i mean i try to instantiate a character avatar as a gameobject then i add to the kinect manager with kinetManager.Player1Avatars.Add(myCharacter.gameObject); but not works is freeze
my character avatar works well when is in the scene before i start, but not when i instantiate, also i try using kinectManager.ResetAvatarControllers(); works but move other avatars to a diferent position
all my avatar has Offset Relative to Sensor active

i want to made something like dressingroom for kinect 1

**Update: i found a solution but i think is bad way, after add my avatar to the kinectManager i put this functions
kinectManager.ClearKinectUsers();
kinectManager.ResetPlayerGestures(kinectM.GetPlayer1ID());
kinectManager.GetUserPosition(kinectM.GetPlayer1ID());
kinectManager.ResetAvatarControllers();

Another question, how can scale the avatar from his center (pelvis) i try many thing

***Update2: i figured and now i can change the scale, about the instantiate an avatar?

thx in advance.

 

Hi superaldo and sorry for the delay.
1. Here is the correct sequence to update the list of avatar controllers:
"

Code (CSharp):

1. kinectManager.Player1Avatars.Clear();
2. kinectManager.Player2Avatars.Clear();
3.  
4. kinectManager.Player1Avatars.Add(myCharacter.gameObject);
5.  
6. kinectManager.ResetAvatarControllers();
7. kinectManager.ClearKinectUsers();

"

2. To change the scale, just change the scale of his Pelvis's transform in Hierarchy.

 

I'm trying to use the Kinect to control a large flying character with wings. The pre-made gestures (handraise and Tpose) seem to be too slow to respond and don't really reflect the movement I'm looking for. Do you have any suggestions for a gesture that would reflect a flapping motion and respond faster? (I was considering using angles, but didn't want to waste days figuring it out if I'm going about it incorrectly.)
Thank you!

 

Both Tpose and HandRaise are poses and use some extra delays, in order to be reported as recognized. I.e. the user needs to stay in the pose for some time, to be recognized. In your case, I would check the Y-positions of the hands and compare them with the Y-positions of the shoulders and with the Y-position of the hip-center. As long as the flapping continues, the Y's of both hands should be either near the Y of the respective shoulder (i.e. Mathf.Abs(Yh - Ys) <= distance) or near to the Y of the hip-center within a predefined time interval. Also, the comparison should change alternatively. I don't think you need to define it as a gesture. Just check the motion in a separate script and provide a boolean property or function that tells your other scripts if the flapping continues or not.

 

Thank you very much for that helpful information and starting me off on the right track

 

Hello, me again...

I tried adding a rigidbody to the avatar so it doesn't float as it walks forward, it seems to be working. However, now when I squat the avatar just raises its legs and floats in midair, even with vertical turned on and with rigidbody. is it a compatibility problem with rigidbody?

 

Hi roumenf,

have some trouble with gestures, I want to do a running gesture so i made two gesture one for legRight and other for legLeft

Code (CSharp):

1.         case Gestures.RunRight:
2.             switch(gestureData.state)
3.             {
4.             case 0:  // gesture detection - phase 1
5.                 if(jointsTracked[rightKneeIndex] && jointsTracked[rightHipIndex] &&
6.                    (jointsPos[rightKneeIndex].y + jointsPos[rightHipIndex].y) > 1.8f)
7.                 {
8.                     SetGestureJoint(ref gestureData, timestamp, rightKneeIndex, jointsPos[rightKneeIndex]);
9.                     gestureData.progress = 0.5f;
10.                 }
11.                 break;
12.              
13.             case 1:  // gesture complete
14.                 if((timestamp - gestureData.timestamp) < 1.5f)
15.                 {
16.                     bool isInPose = jointsTracked[rightKneeIndex] && jointsTracked[rightHipIndex] &&
17.                         Mathf.Abs(jointsPos[rightKneeIndex].y - jointsPos[rightHipIndex].y) < 0.2f;
18.                     if(isInPose)
19.                     {
20.                         Vector3 jointPos = jointsPos[gestureData.joint];
21.                         CheckPoseComplete(ref gestureData, timestamp, jointPos, isInPose, 0f);
22.  
23.                     }
24.                     else
25.                     {
26.                         // cancel the gesture
27.                         SetGestureCancelled(ref gestureData);
28.                     }
29.                     break;
30.                 }
31.                 break;
32.             }
33.             break;

and it works but if I stay with my knee up the gesture dont stop and still counting.
can you help me please? or if you have better solution

edit1// I try with KinectWrapper.Constants.PoseCompleteDuration but is very slow to complete the gesture
Greetings,

 

When you figure out the problem, I'd like to know what it was since I'm also having the opposite, the example squatting is recognizing when I squat but not if I stay squatted

 

Also (I think I may be overusing your help, but...) how do I get the angles at which the head is turned? I plan to make the camera spin around the avatar if the person looks to the side beyond a certain angle (I was thinking 30 or 45 degrees).

 

Disable the 'Use bone orientations contraints'-setting of the KinectManager (component of the MainCamera), if it is enabled, and then try again.