Kinect with MS-SDK

Aimed to provide support to "Kinect with MS-SDK"-package users - https://www.assetstore.unity3d.com/#/content/7747
Please post your questions below.

 

Hi Rumen,

A very big thank for your job and give us for free.

I'm making tests with your package, and all is ok for me.

Do you think, that you can add easly infrared support ?

I have add these lines in KinectInterop.cs
ColorInfrared, // lolo USHORT
ColorRawBayer // lolo 8-bit Bayer
in public enum NuiImageType and try to modify DisplayColor.cs but I am a bad developper ;-)

Do you have plane to add Facetrack possibility ?

Thank you again for your help. I can made tests if you need.

Regards,

Neoxeo

 

Hello, Roumen.

We are currently working on a Kinect minigame with the first version of Kinect.

I have just downloaded this package and it works like a charm. Working out-of-the-box with no additional set-up needed. Thank you very much!

Anyway, I have discovered a strange issue:

While jumping and crouching, it seems that the "cube guy" is doing it almost fine, but the movement is not correctly applied to the models. It seems that it try the "pelvis" to always remain in the same place, no matter what the "cube guy" is doing.

Any ideas on what's the cause of this? Any thoughts on how to solve it?

EDIT: Found it: The vertical movement was disabled on the AvatarController script. This package has a lot of great options. I really think you should put a bit more effort to documentation. Anyway, great work!!!

 

Hi neoxeo.

Sorry for the late reply. I have forgot to subscribe to my own thread
I have to think about the infrared support. Probably in the next version...

Best,
Rumen

 

Hi Alvaro,

Sorry (again) for the late reply.
To your question: There is a parameter of AvatarController-component, attached to the model, called 'Vertical Movement'. Check this checkbox to allow movements in vertical direction.

Hope this helps.

Best,
Rumen

 

Hello Roumenf,

A big thanks for releasing this for free, so far this is the easiest way to integrate Kinect to Unity that I've found.

I just started working on a Kinect related project and I've stumbled into a problem:

- About the gesture detection... I've tried to use the IsGestureCompleted function in the KinectManager.cs, but somehow it can't detect the completed gesture (the Gesture.complete always returns false) even though the gesture info guitext display it as completed.

 

Hi monochr0m3.

To your question: The reason for not detecting the completed gesture is, as to me, that GestureComplete()-handler in AvatarController detects and resets the gesture state, before you call IsGestureCompleted(). To work around this, I'd suggest that you modify GestureComplete() and GestureCancelled()-functions of AvatarController to return false (this means: don't reset data) instead of true. Then, in your code, to handle resetting data by yourself. The last parameter true/false signals to reset or don't reset the corresponding gesture state:

if(IsGestureComplete(UserID, gesture, true))
{
// gesture is complete
}
else (if(IsGestureCancelled(UserID, gesture, true))
{
// gesture is cancelled
}

Hope this helps.

Best,
Rumen

 

Hello,

I am trying your Kinect with MS-SDK, and it works very well. I have got a question: i have two Kinect connected to my pc, I want to stream data from another application to Unity using a Kinect and your application to capture skeleton inside Unity with the other Kinect. Unfrotunatly both applications try to access the same Kinect, even if it is on another USB controller.

Is there a way to tell Kinect with MS-SDK to take a specific Kinect? Or any other workaround?

Thank you

 

Hello megisto,

There is no easy way to do this with 'Kinect with MS-SDK' package. It initializes and uses the first available Kinect sensor. Can't you select the 2nd Kinect for the other app? Another option is to use two PCs. Or, if you like coding and testing, you can utilize INuiSensor interface and select the sensor programmatically

Greetings!

 

Well, i like coding, but i just can't that stuff is way far from my knowledge...

it is strange, if hit play the first application running takes control of the Kinect, while the other application fails. But if i start Kinect with MS-SDK first and then i start the other one it can run the other Kinect, but at that point the initialization of the streaming data to Unity fails, because it initialize at startup.

 

Hello and thanks for this asset!

I am developing a game with the kinect and was wondering if it is possible to detect hand and head interaction (is it possible to figure out if the player is touching certain parts of his/her head) - for example if there is snow falling on the player's head could I detect if the player brushes off the snow from a certain part of his/her head?

Also is it possible to display the raw camera input?

Sorry for the newbie questions - I am just getting started with Kinect and Unity3d

Thanks

 

Hi roumenf,,
Always thanks for your Kinect with MS-SDK..

I Have a quastion,,


How does the rotation of the characters?
Standing in the front of KINECT, I turn around 180 degrees, I turn to the other side
Character does not rotate.

 

Hi mattteoh,

Sorry for the late reply. I don't know why, but I don't get emails when someone posts questions to this thread. Please, if I don't answer within a 2-3 days, just contact me by email.

To your questions:
I don't think such a hand-head interaction might be possible.

The raw color camera input can be displayed by ticking ComputeColorMap and DisplayColorMap-parameters of KinectManager. In the example scene KinectManager is a component of MainCamera.

Greetings!

 

Hi luxertlee,

Please stick to body rotations within +/-80 degrees

Greetings!

 

Hello!
I really love your work but i stumbled across this issue:
When i use KinectManager attached to main camera on the first scene everything is ok, but when i load new one(with script attached to camera as well) everything slows down and gets unable to use.
When i load secod scene directly from unity there is no problem

Greetings

 

Hi!

When there are multiple scenes in your game, MainCamera is not a good choice to attach KinectManager to. Please read 'Howto-Use-KinectManager-Across-Multiple-Scenes.pdf' , located into the Assets-folder of the example package. The file-name says what it is all about

Greetings!

 

Worked like a charm.
Thanks!

 

Hiya Rumen,

Great package. Just wondering if there was any plans to support the Kinect2? Looking around cant find any Kinect2 Unity code yet, so wondering if will have to write myself or if someone will get something on the asset store or online somewhere first.

Cheers,

Chris

 

Hello,

I am trying your Kinect with MS-SDK, and it has been working very well.

I have got a question: I have two Kinect but they are connected to two PCs. I want to be able to communicate between these two PC and get the information of the Kinects to complete the information of the other one. They will be detecting the same Users (I want to have two users) but I want the Kinects to complete the visibility of each other. How can I do that?

Also I have another question, When the kinect is not detecting you well and we have the joints that are pretty messed up, How can we make Unity to understand that he has lost the track of the user, so I will be able to complete it with the other Kinect? Is there a technique on unity that can identify whether the skeleton detected by the kinect is a human or not? Or that we have just the track of person.

Thank you for your help in advance,

 

Hi WorkingsUK,

There is more than plans about K2-support. Here are my first Kinect2-experiments, if you're interested: https://github.com/rfilkov/kinect2-unity-example-with-ms-sdk

When the package gets more complete, I'll publish it on Unity asset store.

Greetings!

 

Hi Yasi,

Unfortunately this package can only help you partially. So, you will detect the user(s) and their body parts (joints), relative to the one sensor and relative to the other (on the 2nd pc). The exact user's and joints' positions can be taken from the public functions of the KinectManager / KinectManager manager = KinectManager.Instance; /. Just use GetJointPosition() and GetJointPosition()-functions. PointManController-script is a good and fairly simple example for using these functions. But after that, knowing the positions of the two sensors and detected positions of the user and joints from the 2 perspectives, you need to convert all the joints' data to world positions, process the data and do the interference resolution by yourself. This is up to your own algorithm.

To your second question: Kinect is quite good at recognizing humans, so don't care about this at first. Apart of positions, there are also flags for each joint, which is returned by IsJointTracked()-function. This would be enough at first for a fair joint recognition and issue resolution. Hope this info helps.

Greetings!

 

Hi roumenf,

Thank you very much for your answer. I am working on it!

But I had an idea, I know that users identification by the Kinects is based on their position and their entrance (correct me if I’m wrong) therefore for example user1 in Kinect1 is not the same user in Kinect2 (possibly)! But what if I give them my own identification beside their own, for example I say the first user that comes into the scene, is UserA which would be on the right side of the scene, and I transfer this information to the other Kinect so it can identify my user as UserA. And then I can define their roles and…. Is it possible? And How can I do it?

About the second question, well I’ve been working with your Plug-in for a while and I really cannot thank you enough for it! But actually my problem now is that the kinect is finding a humanoid form on the chair that is in his vision, and when the user is on it the joints get mixed up! I don’t know what to do about it! The chair was just a test I did but it is possible that sometimes I may have such thing in the scene!

Maybe it would be clearer if I send you the description of my project?! Can I contact you by email?

Thank you very much for your help.

Greetings as well!

 

Hi yasi,

Sure, you can contact me by e-mail. It is published at the Unity asset store. Just keep in mind I can't always reply right away.

Yes, you're right. KinectManager detects usually the closest user or the first user that enters the FOV of the sensor. I think you've said earlier that there will be one user only. But even in case of two users, the logic might be changed, as the full source code is there. You can change it as to your needs.

Greetings!

 

Hello again!

Yes I've contacted you before by email and you kindly have responded to me and helped me! I just sent you the description right now!

Thank you

 

Hi roumenf,

So I'm fighting with Kinect and Unity and have some problem.

As you described I have StartUpScene and FirstScene because I would like to use KinectManager across mulitple scenes. In the startupscene I created a KinectObject, but please tell me how to access it in the FirstScene, now I'm going to implement some gestures for example LeftHandCursor and RightHandCursor and I don't know how to do it? I also implemented a GestureListener but I do not have a reference to KinectManager in FirstScene.

Regards,
Tomek

 

Hi Tomek,

Actually, it's quite simple. You're not interested in the KinectObject from StartupScene, but rather in the KinectManager alone. The KinectManager has a singleton Instance-property that may be used to return the needed reference:

KinectManager manager = KinectManager.Instance;
if(manager manager.IsInitialized())
{
// do whatever you want with the manager, for instance:
manager.GestureListeners.Add(myGestureListener);
}

Greetings!

 

OK, please tell me if I understand you well in this case.

I have to create script GestureListener which will implement GestureListenerInterface with field KinectManager in the FirstScene.
But to which component shall I add it? To the MainCamera from the FirstScene? I also added GestureInfo but it's not working.
and how to use then HandCursor in this scenerio?

Best,
Tomek

 

Do you need different gestures in different scenes or the same gestures (including HandCursor, etc.) should be available throughout the whole game (i.e. in all scenes)?

R.

 

I need HandCursor in whole game and specific kind of gestures in each scene.

 

Here is an example scene: https://www.dropbox.com/s/mdq22gacb8gdmva/Kinect with MS SDK-Multiple-Scenes.unitypackage?dl=1

I added Start()-methods to LoadMainLevel.cs (to prevent the cursor object from destroying) and to SimpleGestureListener.cs (to register the listener with KinectManager from _StartupScene). You need to start Assets/MultipleScenesDemo/_StartupScene. The hand cursor-gestures and the Click-gesture are added to Player1Gestures-parameter of the KinectManager. The rest are dynamically added by SimpleGestureListener.

Hope it will help you do more by yourself now.

Greetings!

 

Big Thanks Rumen, works great, still have a lot of questions but this give nice background
Best

 

Hi Rumen,

first of all congratulation for very good wrapper. It works nice.

I have two questions and one request:
1) How can I apply a rotation offset to the character? Since my character start with -90 offset degree from my scene, I try to rotate the PointMan but this create issue with X axis movement.

2) can I use two (three or four) kinect at the same time on a unique machine to track joint position? MS doc specify SdK support until 4 Kinect ... And your wrapper?

3) Please next update put all the kinect asset in a root folder as: "KinectMS". Adding your asset in a giga project is not so good

Thanks

 

Hi Rumen,
It's me once again. Please tell me how to properly add SpeechManager to this scenerio, I did it as you described in pdf but when speech recognition is working all of the others functions like gestures and stop working. I am adding speech manager to KinectObject in the startupscene.

//EDITED

I solve this problem by uncommenting #define USE_KINECT_MANAGER in SpeechWrapper.cs and #define USE_SPEECH_RECOGNITION in KinectWrapper.cs. Imho it should be written in one of these PDF.

Best,
Tomasz

 

Hallo Rumen,

first of all thanks a lot for sharing the "Kinect with MS-SDK" Scripts. It is really awesome..

However, I do struggle a little bit with the use calibration poses in a two player game. It seems that the presence of two persons at the same time prevents the "CheckForCalibrationPose(...)" method from returning true for Player 2. Have you ever experienced something like this?
Is there anything special to consider when using calibration poses for two players?
Any help is appreciated..

All the best,
Felix

 

UP

 

Hi kaz2057,

Sorry, I missed your post before To your questions:

1. Parent your character to an empty game object (Ctrl-objects in the example) and rotate it as to your requirements. The character should be in T-pose and with (0,0,0)-rotation. Then attach the appropriate script (AvatarController or PointManController) to your character's object and set the appropriate script's parameters. For an example, look at KinectAvatarsDemo-scene.
2. No, you can't directly use them. The wrapper uses only the first (default) sensor. You'd need to modify the KinectWrapper and KinectManager and then sync the data from different sensors.
2. I suppose you mean to put all folders under KinectMS. Good suggestion, thanks! Actually, you only need to copy the KinectScripts-folder to your giga-project. All the rest is examples.

Greetings!

 

Hello felix_muc,

For player 2 there is a separate calibration pose ('Player 2 Calibration Pose'-parameter). If this doesn't work, this might be a bug. Try to look up in Mono where the this parameter is used in KinectManager. If the issue persists and you can't find the reason, please tell me. I'm a bit too busy with other things at the moment, but will do my best to provide you a fix script a.s.a.p.

Greetings!

 

Hi roumenf,

I'm trying to use Facetracking I need only data from head especially rotation. I can't get to work with this. In the provided sample you have a HeadTransform variable but where are you setting it? Here is my code, but HeadTransform is always null, why?

public Transform HeadTransform;

private FacetrackingManager FaceManager;

private Vector3 HeadInitialPosition;
private Quaternion HeadInitialRotation;

void Awake()
{
FaceManager = GameObject.FindGameObjectWithTag("KinectSecondObject").GetComponent<FacetrackingManager>();
}

void Start()
{
if(FaceManager != null)
{
if(HeadTransform != null)
{
HeadInitialPosition = HeadTransform.localPosition;
//HeadInitialPosition.z = 0;
HeadInitialRotation = HeadTransform.localRotation;
}
}
}

 

HeadTransform is a public parameter of ModelFaceController-script. It needs to be set in the Unity Inspector and its goal is to allow the user to specify which bone/transform of the model represents the head. In the FT-example it is FaceRigged (the root). In the code copied above the initial position and rotation of the head-transform is saved, so that applying the real head translations and rotations to the head-transform to be based on its initial state. Look at the start of the Update()-method for example how to get/update the head position and rotation.

To your question: To get the rotation of the head, as detected by Kinect, all you would need is something like this:

FacetrackingManager manager = FacetrackingManager.Instance;
if(manager manager.IsTracking())
{
Quaternion headRotation = manager.GetHeadRotation();
// do something with the head rotation
}

It is that simple

Greetings!

 

Congratulations for you wrapper.

Too bad Web Player restrictions don't allow it to run on the web.

 

Hey I'm starting to work with your asset and it looks great, but I have a question, I want to replace the avatar that is in the asset with another that I have, but the skeleton its quite different, so I wonder if there's a way to restructure the skeleton of the example and put the new model.
Hope you can help me!

Thanks!

 

Hey, I'm new to use kinect and i have question please: is this package support multi kinect working at the same time ?

 

Of course, this is the idea Your avatar needs to be in T-pose. Put it in the scene, attach AvatarController-script as a component of the avatar and assign the avatar's joints the corresponding parameters of AvatarController. For a reference, look at the example scene. Take also a look at the Readme-file in the Assets-folder. There is some important info about the MirroredMovement-flag and the needed control-object.

 

No, this package uses the default Kinect-sensor only.

 

Hi, Roumen.

I have been trying to integrate the new Unity's simple assets' third person character into your package, but it seems that it have any problem working out of the box. I set the bones exactly like in you example but it is moving in a weird way. Does you plugin require a specific bone hierarchy? I think it might be interesting to update the package to work with this official unity asset.

Thank you very much in advance.

Aernarion.

 

It should work, as long as the avatar is in T-pose, parented to an empty control object (meaning that its initial position and rotation are set to 0,0,0) and MirroredMovement-flag/control object rotation set as to your preference. Take a look at the 'Why there are two avatars'-section in the Readme-file, located in the Assets-folder of the package.

Greetings!

 

Is anyone else having issues with blinking shadows with the Kinect v2 for Windows?

 

Why do you mean by "blinking shadows"?

 

The shadow is there for a second and then gone for a second. Like a strobe light.

 

Hi
I have Nuilnitialize Failed - Device is not power
I am sure my kinect is pluged and power
Any one has this problem ? Please show me how to solve it

Thanks