Kinect with MS-SDK

HI roumenf, I have some queries about the gesture capturing. In have add Avatar controller script to my 3d model to track the skelton and it works fine.But i also need some gesture tracking(ex:swipe left ) for moving the 3d model to left or forward.I have added gesture Listener script and kinect manager script on main camera. But it dont work. . It tracks the skelton but dont detect the specific gesture for controlling the movement. What should i do? It will be more fine if u can help me on skype.Will u give me ur skype id? Or add me my skype id is "sajeshpadmanabhan" ......Thanks

 

Have you tried to run the examples that come with Kinect SDK? Don't you have the same problem there?

The Kinect sensor needs to be plugged in to both USB and electrical power outlet to be able to work. And this is from a MS-forum, related to the topic: "Plug the Kinect into another USB-Port, one, that doesn't share the same USB-Controller with another high-bandwith device (like a webcam)."

 

@sajudev already answered by e-mail

 

I try another project I have and the kinect function as normal, but with this one, it is not
I also try create another project with this package only and it still doesn't work
Same error as posted above

Edited: I re-installed my kinect sdk 1.8 and it works fine now

 

Hi again, roumenf.

Your recommendation was right: my character needed to be EXACTLY in t-pose. Previously, the arms were a bit under the horizontal, and that was causing some strange issues.

Now I would like my character to run some Mecanim animations when a gesture is detected. To be more precise, I am working on a basketball game and I want my character to do a throw ball animation when it detects the gesture. Currently, as long as I enable the AnimatorController, even if there is no animation in the current default state, the kinect recognition goes crazy and it only detects correctly the movement in the plane.

Any ideas?

Thank you again!

 

I figured it out. It only happens on maximize on play. I set the resolution to 1024x768 with hard shadows and it works fine.

 

Why don't you disable AvatarController when you are about to enable AnimatorController and vice versa, so they don't disturb each other.

 

Hi. I have tried almost everything, including that. I found that the best working option is using Legacy to play the animation. The problem is, when the animation ends, Kinect seems to continue the operation from the final pose of the animation (last frame), and some strange problems occurs, with some bones in strange positions.

I have seen similar Kinect assets solutions providing a way to mix Mecanim with kinect recognition and I wanted to know if you have faced a similar problem with your asset.

Thank you!

 

Hi Roumen,
First of all, many thanks for the free kinect package, it´s a hell of a job!!!

I´m having trouble executing the "KinectOverlayDemo" scene, it doesn´t seem to work and the HUD says "waiting for users". The other example scenes work just fine out of the box with the avatars and the gestures but this scene in particular doesn´t seem to work at all.

I also tried with the Kinect + Playmaker package that Temper Tantrum has and the same thing happens, none of the example scenes work and the kinect does not detect the user.

I´m using Unity 4.6 Beta, Kinect SDK v1.8 and a Xbox Kinect 1414 and my computer specs are pretty good overall.

What might be the problem here? I appreciate your help.

Thanks a lot,
Juan Maldonado

EDIT: Already found the problem. I changed the USB port connection for the Kinect and worked like a charm.

 

Congratulations!

 

Hi Roumen! Thks for this package been playing with it!

Is it possible to lock the skeletons to players so that some guy that walks in front of doesn't replace any of the original skeletons. If so can u tell me how?

Also how can i make player on the most left side become userIndex 0 and player on the most right be userIndex 1? Edit: i managed to do this sort of using manager.GetRawSkeletonJointPos(userId,0); and seeing whether the vector3 x component is <= or > 0.
thks in advance.

 

Hi idunnuhow,

Well, when somebody walks in front of you, Kinect usually stops tracking your skeleton and "loses" your player. When the person walks away, it founds you and starts tracking you again. Unfortunately the player/user ID changes too. You can provide additional logic and compare (for instance) the position of the lost player with the position of the new. If they don't differ much and the time difference is not big (1-2 seconds), you can consider the player is the same as before.

The 2nd issue you have worked out by yourself. Well done! Only, stick to using GetJointPosition() in your scripts, instead of GetRawSkeletonJointPos(). The difference is that GetJointPosition() takes into account the position and tilt angle of the sensor, while GetRawSkeletonJointPos() doesn't. But in your case there shouldn't be any noticeable difference in the results. Hope this info helps.

 

Hi Roumen!

Thank you for the help. I have another question
How can I disable gesture checking after i have done a gesture? and then enable it later?
Thks in advance again.

 

KinectManager manager = KinectManager.Instance;
if(manager && manager.IsUserDetected())
{
uint userId = manager.GetPlayer1ID();
manager.DeleteGesture(userId, gesture);
}
...
the same, but:
manager.DetectGesture(userId, gesture);

 

Hi again Roumen!

Is it possible for seated mode tracking here?

 

Nope, seated mode is not supported. It changes the skeleton structure.

 

Hi Roumen,

Thanks very much for such an excellent and stable free asset!

I am going to be using your library in a project that will be displayed in a crowded space, so I'd like to implement a z-cutoff to prevent people in the background from messing up the skeleton tracking. I understand in theory how to do this -- just tell the Kinect to ignore every point in the depth map with a z coordinate greater than (cutoff), but I can't figure out where to implement this in the code. Maybe in the PollDepth() method of KinectWrapper?

If you could point me in the right direction I'd really appreciate it!

 

There is no way, as far as I remember, to limit the depth scanning within Kinect-SDK. This also means you can't limit the skeleton tracking in the SDK. But there is a setting of KinectManager-component, called 'Max user distance'. You can set your cutoff-distance there (in meters) to limit the distance to bodies used by the avatars in the scene.

 

You can also limit in X axis (so that only the player in the center's gets tracked, if other person stands 0.5m to the side, he doesnt get skeleton) and can add limit for the user count (if its already 1, then dont add new users)

btw. has anyone had problems with kinect hanging, it just stops tracking and updating the colormap/skeleton or "going to sleep" if nobody is there for some time? (still power saving is off)

 

Well, I had seen MinUserDistance in KinectManager.cs, but there's no corresponding MaxUserDistance. But the fix is simple. MinUserDistance only appears 3 times in KinectManager.cs. Under the first instance

public float MinUserDistance = 1.0f;​

just add another variable

public float MaxUserDistance = 2.5f; //or whatever distance you want to use, it's in meters​

Then in the next two appearances (in the ProcessSkeleton method)

change

if(userId == Player1ID && Mathf.Abs(skeletonPos.z) >= MinUserDistance)​

to

if(userId == Player1ID && Mathf.Abs(skeletonPos.z) >= MinUserDistance && Mathf.Abs(skeletonPos.z) <= MaxUserDistance)​

and do the same for the Player2 instance.

I tested and it works; the only oddity is that when you step out of range the User Map freezes and stops refreshing. But that's not a problem for my purposes.

Thanks, again, Rumen for the help!

 

Hi roumenf!

I have problems using player1 calibration pose; i have twoUsers set to false. It works when there is only 1 player in camera range, but not if there are 2. I'm trying to make it such that whoever does a specific pose becomes Player1 in a single player game.
Also, is it supposed to work like if i have 6 users and one of them does the player 1 calibration pose, that user becomes player 1? or does calibration pose only apply to the 2 closest users?

Thks in advance.

 

Hi roumenf!

I figured out why calibration doesn't work for my purposes and i managed to find a way. Multiple users would trigger multiple calls of method CheckForCalibrationPose. The problem was each call referenced the same gestureData parameter 'player1CalibrationData'.
So I made 6 gesturedata vars and used one for each different userIndex.

 

Hi Roumen,
thanks for this great plugin, I really enjoy using it!
I want to create my character which will be just simple silhouette, with maybe some outline. Something like this:

Does anyone knows how to do something like this? Or maybe there is some option in this plugin for that?
I'm still Unity3d rookie, and don't have any idea how to do this...

Also, another question: does this plugin can detect some other objects, for example chairs etc?

 

ALLAH RAZI OLSUN

 

Hi roumenf,
Is there anyway to set the color of the usermap based on player1/2?

 

Hello,

Great package, I like how you have the gestures in the KinectManager!
However, I have one issue I hope you could help with.....it doesn't seem to want to use the Kinect with Primesense/OpenNI

I've got the package running in Unity 4.5.3f3 (non-Pro) on Windows 7 64 with the Windows Kinect SDK 1.8 installed on my first PC (PC1).

It runs very well in the files where your wrapper seems to use the Windows SDK:
(Avatars Demo, Gestures Demo, and the Overlay Demo).

However, in scenes (Follow Hand, ItemSelector scenes) when it seems to need OpenNI and Primesense, I was getting the following errors:

GeneralException: One or more of the following nodes could not be enumerated:
Device: PrimeSense/SensorKinect/5.1.2.1: The device is not connected!
Device: PrimeSense/SensorV2/5.1.2.1: The device is not connected!

I assumed I was getting that error message because I only had the Windows SDK installed and nothing else on PC1....

On a second PC (PC2) I have been running a Zigfu setup for testing where I had installed the latest (though not updated recently) Zigfu install of OpenNI, NITE and Primesense.

Therefore, to try a basic install of the PrimeSense for testing, I put the Zigfu OpenNI/NITE/Primesense install on my PC1 that uses your RF Solutions Wrapper ----------- No luck, same error as above. The Kinect doesn't seem to want to start under Primesense/OpenNI, but is still fine with Windows SDK.

Does anyone else have this problem? Do you all install OpenNI and Primesense before using this wrapper? Would I need to go to OpenNI 2.0 or 2.2? The only reason I haven't is because the software versions I am targeting are a bit behind, and the Zigfu install seemed to have the right time-frame versions all bundled.

Cheers!

 

hmmmm......Looks like PC2 was defaulting to Windows SDK under Zigfu, the OpenNI/Primesense doesn't work there either. But that's not an issue as I can select the Windows SDK within the Zig script, and I don't seem to lose gestures.

The differences between these two wrappers are very interesting. On the face of it though, I would much prefer to go forward with Rumen, as he's made a nice package and seems to be going forward with support even on the 1.8 version. ( I cannot do 2.0 at this point in time).

Does anyone have advice on what their OpneNI/Primesense setup was when working with this? Or have I got the wrong end of the stick on the problem? Should it work all under Windows SDK?

Ta very much.

 

Ahh.....

Should I be using this?
https://www.assetstore.unity3d.com/en/#!/content/7225

I thought because the OpenNI code was in this wrapper, it would work....

 

Ok, I used the OpenNI2 package:
https://www.assetstore.unity3d.com/en/#!/content/10693

With these OpenNI2.2 and Nite packages:

And that works fine.

I guess I'm confused by the OpenNI scripts and scenes in the Kinect with NS-SDK package.
Should this package be switchable between MS-SDK and OpenNI(2) like Zigfu?
That would be great, as your architecture seems to offer something different from Zigfu in the MS-SDK package, but that is only if the OpenNI works in those gesture scenes.

Since Zigfu fails to work with OpenNI2 on my machines as well, I suspect there is something broken in the whole arrangement of OpenNI2, NITE2, etc.... that is all fixed in the MS-SDK2.

Just wanted to check if OpenNI(2) is supposed to work in this package!

 

Hi roumenf, thanks for shared us the Kinect with MS-SDK, I have a big problem with the mouse control when I work with the new feature canvas GUI buttons, the cursor try to go away from the button every time I try get close to it, how I can fix that?, becomes frustrating at times trying to place the cursor over the button, thanks very much for you support.

 

Hello zaniocz, I'm testing Kinect with MS-SDk, something similar happened, but instead of Gui Button used a GuiTexture, there was the value of the position in the GUI screen is between 0-1 and my button that is an object is between 0 and Screem.width.
In this case you will have to make a convertible values, I used a function that found in the forum Unity.

/********
public float Remap (float value,float from1, float to1, float from2, float to2){
return (value - from1) / (to1 - from1) * (to2 - from2) + from2;
}
******/

Also used the "ScreenToWorldPoint" and then "Raycat" to detect the collision and define an action, such as touch a button or load a new scene.
Sorry, but using Google tranlate, and thank you roumenf.

 

Hi zaniocz, thank you for reporting the issue. May I ask you to contact me by e-mail and, if possible, send me a short video depicting your problem. I suppose you use RightHandCursor and LeftHandCursor to control the cursor on-screen. It would be also great, if you could create a sample project for me, zip it, share it online and send me a link.

 

Yes, there is. Open KinectScripts/KinectManager.cs, find UpdateUserMap()-function, find 'switch(userMap % 4)' near the end of the function and replace the switch with something like this:
"
if(userMap == GetPlayer1Index())
usersMapColors[flipIndex] = Color.xxx * c;
else if(userMap == GetPlayer2Index())
usersMapColors[flipIndex] = Color.yyy * c;
else
usersMapColors[flipIndex] = clrClear;
"

 

Hi homeslice, and sorry for the delayed response. I had some days off. There are several Kinect packages - one for Kinect SDK, one for OpenNI 1.5 and one for OpenNI 2. They have similar functionality, scripts code, but are not directly related or switchable. The latest package "Kinect v2 with MS-SDK" (aimed to Kinect-v2) includes support for Kinect-v1 too. I was thinking to include the OpenNI1.5 and OpenNI2 interfaces into this package as well, but I'm still not sure whether to do it or not. Because OpenNI was literally shut down by Apple. What do you think, is it worth doing this?

 

Thks and Hi again roumenf,

I set the tracking for two users, but when ever a third person is nearer to the kinect than the intended persons, the third person replaces one of the players. For now I am using 'min user distance' in kinectmanager to prevent this sort of. I cannot figure out why the player is lost even though he is not blocked, just that he is no longer one of the two closest user.
How can I prevent the kinect from replacing one of the two calibrated users in this case?

 

I think there was some default option like [x] track closest user?
but you can also add max distance limit to the code (same place with the min.dist)

 

Yes, the option is 'Detect closest user', but that may happen only if someone from previous users gets removed from the list of tracked users. Maybe the newcomer covers the body of a previous user and prevents its tracking.

 

hmm but doesn't detect closest user apply only when all players are not calibrated? my problem is when both players are already calibrated, it detects a player lost when a 3rd person is closer than player 1 or player 2, even though they are standing side by side facing the kinect. Under processSkeleton(), i think it creates a list of users to be removed at the end, and in between it takes out ids from that list who are tracked as player1 or player2. problem is when the 3rd person thing happens, it shows that the same id was taken out twice, therefore the other calibrated id is removed. so i am confused at this part.

 

Hm, are you sure they have equal IDs? Can you please send me a short video demonstrating this behavior (with console window turned on, if possible)?

Yes, in ProcessSkeleton() the logic is to find out which of the tracked skeletons is no more tracked, and remove it from the list. As you probably know, the SDK for Kinect1 can track only two users (in detail) out of six, the other are tracked only positionally. May it be possible that the new user gets tracked in detail (because it is closer) and the other one switches his skeleton state from tracked to position-only? You can check this by adding some Debug.Log()-statements in the function, to debug the skeleton status of the users (skip the non-tracked skeletons).

 

Thanks for your hard work and sharing this great package with us!

Maybe this is too obvious, but I can't find a solution to get the raw depth image only (without colored player, without matte). I tried the playmaker action "get depth map", too (allows to get depth-and-user map), but it returns me the user-map (colored image with alpha). I would like to use the depth image as a heightmap for a shader....

 

You can get the raw depth image by using GetRawDepthMap()-function of the KinectManager, like this:
"
KinectManager manager = KinectManager.Instance;
if(manager && manager.IsInitialized())
{
ushort[] pixels = manager.GetRawDepthMap();
// do something with the raw depth map here
}
"

 

Hi,
I was reading your Gesture Listener script and happened to notice there were only a few hand gestures to choose from. How should one go about creating new gestures for the whole body or even just the legs?

Thanks

 

Hi,
I was also having trouble with 2 compiler errors; " Assets/OverlayDemo/Scripts/KinectOverlayer.cs(62,108): error CS0117: `KinectWrapper.Constants' does not contain a definition for `ColorImageWidth'" and "Assets/OverlayDemo/Scripts/KinectOverlayer.cs(63,115): error CS0117: `KinectWrapper.Constants' does not contain a definition for `ColorImageHeight'". If you've seen these before or know what in the code I should change, please let me know.

Thank you.

 

Hi, regarding the gestures: the gesture recognition code for the new gestures should be added to KinectGestures.cs-script. The full source code is there. There is also a short manual 'Howto-Use-Gestures-or-Create-Your-Own-Ones.pdf', located in the Assets-folder of the package.

 

Look LadyGuinevere, there is no way to submit an asset with compilation errors to the Unity asset store Obviously the KinectWrapper or KinectOverlayer scripts were modified afterwards. The simplest workaround might be to create a new Unity project, download the asset once again from the Asset store, try the scenes and then copy the KinectScripts-folder (or just the missing parts of the scripts) to your project.

 

Hi hi. Great asset.

Do anyone know how can I load my own gestures database?

Thanks.

 

I did some debug logs and that is the case, if so how do i prevent a new user from getting tracked in detail just because he is standing closer?

 

Hi Roumenf,

Great asset, but there is always question .
In Interaction Demo how can we push or pull (z+, z-) objects.
Cheers,

Max

 

Well, I think you can't. This is something Kinect SDK decides internally. So far I couldn't find a suitable setting or method to override this decision.

 

Hi nmax, do you ask about InteractionDemo in KinectExtras or Kinect v2-asset?