Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

help: moving kinect camera and depth data into a Texture2D

Discussion in 'Scripting' started by amir, Feb 9, 2011.

  1. amir

    amir

    Joined:
    Oct 5, 2010
    Posts:
    75
    Hi!

    I'm working on Kinect/OpenNI bindings for Unity.

    I want to get a Texture2D from the IR, Camera, Depth and User Labels provided by OpenNI. I can do it in vanilla unity and I only get like 14-18 fps.

    These maps have a generic structure (eg: MapData<RGBPixel24> ImageMap and MapData<ushort> DepthMap) From these maps I can generate a Color[], using a look-up-table to map the 11-bit depth values to Colors. These for loops over the map data don't appear to suck up time.

    But the Texture2D SetPixels and Apply calls for these 640x480 Color[] every update makes my framerate drop to about 14-17 fps on my MacBook Pro.

    I should be able to get 30 fps updated the camera and depth data at 640x480.

    I have searched the forums and all signs point to me using GetNativeTextureID(); to get the Native ID of the target Texture2D and pass it to a plugin that uses glBindTexture(GL_TEXTURE_2D, nativeTextureID); and glTexSubImage2D to render a 640x480 window. I've built these plugins, even built a simpler ones that just color a small square of pixels

    I can't seem to get a working solution to modify a texture from a plugin. The Texture plugin provided by Unity doesn't have the plugin code and runs slow anyway -- it seems to use setpixels and apply.

    I understand that I'll need to do some tricks to make openGL plugins work on windows too, or write another one for DirectX. I really didn't expect this would be so complicated. Can someone enlighten me?
     
  2. amir

    amir

    Joined:
    Oct 5, 2010
    Posts:
    75
    I want to report that I have successfully created a very simple plugin that can draw the 640x480 camera data at a high framerate:

    Code (csharp):
    1. void camTexture( int nTexId , int width, int height, int * dataPtr)
    2. {
    3.     glBindTexture(GL_TEXTURE_2D, nTexId);
    4.     glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, width, height,
    5.                     GL_RGB, GL_UNSIGNED_BYTE, (GLvoid *)dataPtr);
    6.    
    7. }
    which gets the OpenNI camera data called from Update like this:
    Code (csharp):
    1.    
    2.  
    3. [DllImport ("xcodeproj")]
    4. private static extern void camTexture (int nTexId , int width, int height, IntPtr p);
    5.  
    6.         private readonly string SAMPLE_XML_FILE = @".//OpenNI.xml";
    7.         private Context context;
    8.         private ImageGenerator image;
    9.  
    10. ....
    11.    
    12.    
    13.     // Use this for initialization
    14.     void Start () {
    15.        
    16.         this.context = new Context(SAMPLE_XML_FILE); // (see the
    17.         this.image = context.FindExistingNode(NodeType.Image) as ImageGenerator;
    18.         ...
    19.  
    20. void Update () {
    21.     this.context.WaitNoneUpdateAll();
    22.     IntPtr p = this.image.GetImageMapPtr();
    23.     camTexture( tex.GetNativeTextureID(), 640, 480, p  );
    24.     }

    I am now going to generate some textures / normal map from the depth data. Here, context is an OpenNI context and image is an image generator. my repo here has a lot of stuff, but It's all very messy now: https://github.com/tinkerer/UnityWrapper

    I also uploaded a handgen.cs example recently.
    Sorry features first, documentation later!
     
    Last edited: Feb 14, 2011
  3. albatross

    albatross

    Joined:
    Sep 16, 2010
    Posts:
    15
    Awesome. I'd love to see your c++ source files (even if they are a work-in-progress)
     
  4. amir

    amir

    Joined:
    Oct 5, 2010
    Posts:
    75
    well that was my C++ source... that and a corresponding .h file which externs it. Add OpenGL framework build a bundle and zoom.

    extern "C" {

    void camTexture( int , int , int , int * );
    }
     
  5. albatross

    albatross

    Joined:
    Sep 16, 2010
    Posts:
    15
    I am looking for Context and ImageGenerator classes. Are these available?
     
  6. sayezz

    sayezz

    Joined:
    Mar 9, 2011
    Posts:
    16
    Hey,
    is there a way to get the Videostream from ImageGenerator into a Texture2D ? I dont get it :(
     
  7. middleman

    middleman

    Joined:
    Mar 10, 2011
    Posts:
    4
    Hello All-

    I am having problems with finding the existing Image node. I have Unity set up to search for the existing node with:

    this.image = context.FindExistingNode(NodeType.Image) as ImageGenerator;

    but I get an error at this line claiming:

    "XnStatusException: No match found
    xn.WrapperUtils.CheckStatus (UInt32 status)
    xn.Context.FindExistingNode (NodeType type)
    NiteUserTracker.Start () (at Assets/Standard Assets/OpenNI/NiteUserTracker.cs:49)"

    The "NiteUserTracker.cs" file referenced in this error is almost an exact clone of the "Nite3" script that comes with tinkerer's example project, simply with the bone tracking capabilities re-routed to my own javascript file.

    I can make a new ImageGenerator, and have done so. The code throws no errors but also does not feed any video. Does anyone have any ideas?

    -MiddleMan