Kinect v2 with MS-SDK

Hi, I tried to used the unity kinect wrapper for kinect version 2. I tried according to the instruction about using KinectAvatarsDemo.unity. After importing all the packages, I double clicked this file. However, there is no avatar in the scene and when I run it, my Kinect was not turned on. Not sure if I did anything wrong. Could you please have a look. Let me know if you need more information. Thank you,

 

I don't really understand what you mean by "importing all the packages". You need to import only one package. If the hierarchy in the scene is empty after you open it, you probably have Unity version prior to 5.0.1. In this case just upgrade Unity. If you don't want to upgrade, e-mail me and I'll send you an older version of the package, suitable for Unity 4.6.

 

Thank you Runmen, I solved the problem by updating the Unity. The example works good. Thanks a lot.

 

Hi,

Can you use 2 kinects with this plugin 'kinect v2 with MS-SDK'?

If not can anyone recommend one?

Many thanks
Seb

 

You cannot because those are Microsoft's own current restrictions, which are you can only run one Kinect v2 per computer. If you need to do a multi-Kinect v2 setup, you'll need to have multiple machines communicate over networking. Microsoft has said they will support this at some point, but I believe it's related to the USB 3.0 performance.

EDIT:
It looks like this plugin allows for multi-Kinects v2. You might be able to integrate it into the "'Kinect v2 with MS-SDK" plugin.
https://github.com/stimulant/MS-Cube-SDK

What do you think @roumenf ?

 

Yes, the KinectTransport looks interesting. Thank you for the link! I added it to my To-look-at list

 

roumenf,
I am trying your package, and I have a questions to you.
Is it possible to run captured sound from the kinect, together with animation.

Thanks

 

Hi roumenf,

Thank you for your great work! I tried to run demo-scene named "KinectBackgroundRemoval" and "DepthColliderDemo", but nothing was shown at foreground.

Another demo-scenes like "KinectFittingRoom" , "KinectOverlayDemo1" and "KinectOverlayDemo2" are OK.

- Windows 8.1
- Unity 5.1.0f3
- Kinect for Windows SDK v2.0 v2.0.1410.19000 ( KinectSDK-v2.0_1409-Setup.exe )
- Kinect v2 with MS-SDK 2.5.1

Does this demo-scene works with Unity 5.1.0f3?

Thanks.

 

Got the same Problem with the Background Removal-Demo. Can't quite figure out why it doesn't work yet.
The rest is pretty amazing though.

 

Hello roumenf,

First thank you for the great plugin.

Is it possible to use both kinects (v1 and v2) in the same game? What I want to do is make a game where I can use both kinects (not at the same time), the game must be able to know which version is plugged in and work accordingly.
I ask this because I've tested your plugin with the first Kinect (Windows 7 SDK 1.8), and it works fine. I will get access to a Windows 8 machine in a couple of weeks and I'm sure it will work great with Kinect v2 and SDK v2.

I'm just not sure if with just SDK 2.0 Kinect v1 will work, or if I can install both SDKs (1.8 and 2.0) in Windows 8 without any conflicts.

Best regards

 

I suppose, yes. Kinect is just another microphone to the system.

 

Oh, sorry for the late reply, I saw your message just now. Unity 5.1 introduced this new issue, which cause some of the asset shaders not to work correctly any more. I'm researching it for more than a week now, but it looks like I won't be able to resolve it without some support from the Unity staff. In the meantime, you can find a quick workaround here: http://rfilkov.com/2014/08/01/kinect-v2-with-ms-sdk/#ki

 

Sorry again for the late reply. Please see the previous post.

 

You need to install SDK 1.8 for Kinect v1 and SDK 2.0 for Kinect v2. They can work simultaneously on the same machine, without any conflicts. Please also open KinectScrips/Interfaces/Kinect2Interface.cs, find 'public static bool sensorAlwaysAvailable = true;' and change it to '= false;'. This will allow Kinect v2/v1 detection and the KinectManager will use the currently plugged in sensor. If the flag is true, Kinect v2-interface is used whenever possible, even if no sensor is plugged in. This allows utilization of the Kinect Studio emulation without physical sensor though.

 

Hi,

First of all, the package is great and thanks a lot.

I am currently researching for a way to use a projector to simulate an AR world. Thus, I need the Kinect to face the same direction as the projector (opposite to the screen). I need the Kinect to track me from behind.
Is there a way to change the setting so such that Kinect is in back view?

Thanks

 

I'm digging this plugin, but does anyone know the protocol for mapping the face/expression tracking to a custom face mesh/rig?

Thanks!!

 

I think I answered this by e-mail, but here it is once again: The limitation of facial-tracking-only comes from Kinect SDK, not the K2-asset. I can't check this right now, but I suggest the body position is being tracked always (which would be enough for the RoomAlive-setup).

 

I suppose you want to apply the face animation units (available from the FacetrackingManager) to your custom face model. In this regard, see the first face-tracking demo and ModelFaceController-script. The source is there, so you can modify or tweak it in the way your rigged face model requires.

 

I've been using this package for a few days and so far it's quite nice and works great. In order to learn better, I've been trying to get a model of a foot ball helmet to align to a users face/head. I am using the Face Tracking Manager and getting the position and rotation from the manager like: manager.GetHeadPosition, manager.GetHeadRotation. I grabbed some code from the face tracking demo... I get the helmet model rotating properly but cannot get any position transforms to work at all. I can see the purple face tracking rectangle but when I move my face side to side on the screen, the helmet stays in center. Seems some kind of scale issue, but I've not been able to figure it out. Any hints?

 

I can't be sure about the matter of the issue without seeing your code, but if you e-mail me, I can send you an intermediate version of the K2-asset that contains a similar demo-scene (please mention your invoice #, too). Then you can compare it with your code and find out what exactly is wrong.

 

Hi roumenf,

The package is great and thanks a lot!!
I tried to run demo-scene named "KinectAvatarsDemo1.unity", but it crashed.

Another demo-scenes are OK...
plz, help me.

Thanks.

 

Well, it is really strange that one scene crashes, while the others run OK. They use the same components.
What crashed exactly? Any error messages or screenshot?

 

Thanks for maintaining the package. Showing off initial fruits that are part of your labor:

 

Hey man,

I really love your plugin!, but I was wondering is it possible track a specific color or IR reflecting object? I have a small kinect project, and I don't have much experience with it.

 

Thank you! I think you can track any object with some help from image processing libraries, like OpenCV. You can look for object with specific color in the rgb image returned by the sensor, or for IR-reflecting object in the IR-image, or for any object (you could recognize) in the depth-image.

 

Hello ^^

Using "Kinect v2 with MS-SDK" -package try to develop online games

I try to use the Photon

If I change the Kinect v2 with MS-SDK "-package Will you be developing online games?

 

Hi, I don't really understand what you mean. Could you be more specific?

 

Hi

I've been developing a game using your package. Everything works fine in the editor. However, when I build the game and run, Kinect won't run in the game. I've tried to build and run a sample scene from Kinect Unity Pro Package Add-on and Kinect worked. Do you know what might be the problem and how to fix it?

 

Make sure you copy the Resources-folder along, as described here: http://rfilkov.com/2015/01/25/kinect-v2-tips-tricks-examples/#t2 Also, look for error-messages in the log-file in the xxxx_Data-folder. Tell me, if you need further help.

 

does the Avatar Controller support for hand states? (open, close, lasso). Thanks in advance

 

Hii roumenf, great plugin,I have bought your new SDK, I have worked on it, right now i want to do virtual fitting room like this can you please tell me how can i achieve like that, I have exact map depth issue cloth doesn't look like fitt on player , rotation issue (side view-when i rotate myself 90 degree) model doesn't rotate as i thought, and last but not least how can I view my hand above my model. I have made much more efforts on it and stuck on these things, hope you understand my problem and give me some suggestion, please see video you can understand what i want. Thanks in advance

 

Hi roumenf,

What I'm trying to do is get just the foreground data of only the body (just like the background removal example, but without the head).

Post that, I would like to know if there is a way to differentiate between skin and clothing (eg, get which region is the shirt/pant and which is the visible portion of the hand).

Have a great day.

 

Sorry for the delayed response. The AvatarController does not use the hand states, but the InteractionManager does. You can also get the current hand states by using GetLeftHandState() and GetRightHandState()-functions of KinectManager.Instance.

 

Already answered here: http://rfilkov.com/2015/01/25/kinect-v2-tips-tricks-examples/comment-page-1/#comment-1711

 

I know what you mean, but still thinking of the best way to achieve it. Please contact me in a week or two to ask again.

 

Hi roumenf,

Is it possible to display the player, using the background removal manager, inside the game rather than on top of it? I would like the user to be at a Z position of 0, while the camera is at -10. However when I try this it will show the player on top of objects that are below a Z position of 0 which confuses the player a lot. Is there any way I would be able to fix this you know of?

The product works great btw .

Edit: A render texture seems to do the trick just fine.

 

I see you've found a solution already Anyway, if you clear the 'Foreground camera'-setting of the BackgroundRemovalManager-component, the foreground texture will not be rendered on the GUI, but you can still get the texture and apply it on a 3d plane, or the same way as the background image, with one more intermediate camera:

Code (CSharp):

1. Texture fgTex = BackgroundRemovalManager.Instance.GetForegroundTex();

 

Allright, next time I will use that method as it sounds like the best method.

Besides that I have been able to create two games with the Kinect already and they are great to play with two players, however I won't be able to share them with you during the development process. Once I'm allowed to share them I will make sure to post a video on your website (in the comments section) .

 

Hi roumenf, any progress with this?

 

Hi roumenf, i’ve purchased kinectv2 for unity3d, every demo scene runs fine. But in Background removal scene, i can’t see any thing in foreground, it just a background image when i run the scene. I installed kinectv2 SDK. Could you help me . Tks.

 

Sorry vivalavida, I was away for some time and have delays in all directions, including this one. I think it should be done with the help of the camera's depth/stencil buffer, but couldn't research and experiment enough yet. Please research and experiment as well, if you can. Or ask me again in some time, so I don't forget.

 

I think I answered this question once. Just in case: Make sure you're not using Unity 5.1.0 or 5.1.1. The workaround is to upgrade to a newer Unity version, as described here: http://rfilkov.com/2014/08/01/kinect-v2-with-ms-sdk/#ki

 

Can this have two kinect's connected to Unity at once?

 

No, not really. Kinect SDK 2.0 does not allow more than one K2-sensor to be connected to a single machine.

 

Hi roumenf, first of all nice plug in, the Kinect SDK 2.0 is amazing. I have a question, I'm trying to make a game like fruit ninja and I use the overlay example 1, but the green ball it seems like the Kinect lose track of the joint when I move my hand....But when I use the avatar example, it doesn't lose track of any joint....Any idea of why this his happening. Thanks for the help.
PS. I'm using the contact V1 for this project.

 

Hi @roumenf ,
Is there a way to get a solid colour mask for each tracked user?

 

Hi DonTito, I just tested the 1st overlay example with Kinect V1 and can't reproduce what you say, i.e. it doesn't lose track of my hand. Could you please tell me what should I do, in order to reproduce it.

 

Open Resources/DepthShader.shader and replace 'float hist = 1 - (_HistBuffer[depth] / _TotalPoints);' with 'float hist = 1;'.

 

@roumenf , Thanks a lot.
I have one more question (have been excitedly playing with your asset a lot lately, hope it's not much of an inconvenience).

Is it possible to get a gray-scale depth image of everything that is perceived by the depth sensor? showing objects far away darker and close by objects brighter; without any human tracking.

 

[QUOTE="vivalavida, post: 2346692, member: 564879Is it possible to get a gray-scale depth image of everything that is perceived by the depth sensor? showing objects far away darker and close by objects brighter; without any human tracking.[/QUOTE]

In the same shader, comment out the: 'if (player != 0) { .... }' block, and add this code below it:

Code (CSharp):

1.                     int dx = (int)(i.uv.x * _TexResX);
2.                     int dy = (int)(i.uv.y * _TexResY);
3.                     int di = (int)(dx + dy * _TexResX);
4.                    
5.                     float depth = _DepthBuffer[di];
6.                     if(depth == 0) depth = 5000;
7.                     float hist = 1 - depth / 5000;
8.                    
9.                     return float4(hist, hist, hist, 1); // gray
10.