Unity Community


Page 1 of 9 123 ... LastLast
Results 1 to 20 of 170

  1. Location
    Wiesbaden, Germany
    Posts
    24

    UnityAR - ARToolkit Interface

    UnityAr provides an interface to the ARToolkit. Local connected webcams are used as input source for the pattern recognition. This interface allows to detect multiple ARToolkit marker from the videostream and provides the position and rotation of each marker in the 3D scene.

    ARToolKit is a computer vision tracking library that allows for the creation of augmented reality applications that overlay virtual imagery on the real world. To do this, it uses video tracking capabilities in order to calculate the real camera position and orientation relative to square physical markers in real time. Once the real camera position is known a virtual camera can be positioned at the same point and 3D computer graphics models drawn exactly overlaid on the real marker. So ARToolKit solves two of the key problems in Augmented Reality; viewpoint tracking and virtual object interaction. (From Wikipedia)
    A free beta version is available on our website: http://produktion.weltenbauer.com on the "entwicklungen." section.

    ARToolkit Details: http://www.hitl.washington.edu/artoolkit

    CodeVis VidCap Details (Video stream input): http://www.codevis.com/vidcapture
    Attached Images  
    Christian Rathemacher
    Lead Programmer

    ------------------------------------------------------------------------------------------------

    weltenbauer. Software Entwicklung GmbH

    Humboldtstr. 7
    65189 Wiesbaden

    0611 / 97 14 314
    produktion.weltenbauer.com
    produkion@weltenbauer.com


  2. Posts
    26
    Thx for sharing the beta. Looks quite promising
    I got a question though. On your webpage you have a curtain visualization that seems to behave like a cloth thing.
    Would you mind sharing how you achieved this effect ?


  3. Location
    Austin, TX
    Posts
    604
    wow, that looks cool. is a mac version possible?


  4. Location
    Melbourne, Australia
    Posts
    64

    My server version

    Here is a screenshot of my AR server streaming full frame video @ 25 fps, into Unity Indie.

    Will provide download links in due course. Windows 32 bit only.

    Carl
    Attached Images  


  5. Posts
    313
    nice work,keep up


  6. Posts
    1,362
    How much information is required, ie, focal lengths/height etc? Or do you derive all that information from statistical analysis from parralax etc?

    Essentially, is this a realtime equivilent of Bojou/PixelFarm/Matchmover/Syntheyez?
    Jamie T. Bentley
    Technical Director


  7. Location
    Melbourne, Australia
    Posts
    64
    Bojou/PixelFarm/Matchmover/Syntheyez/Icarus/Etc are markerless systems so they have a lot more work to do - but yes, the idea is the same.

    The ARToolkit (on top of which I've built my server version), tracks a known rectangular marker in the live video image.

    See here for more information:

    http://www.hitl.washington.edu/artoolkit/

    Intrinsic camera parameters such as focal length are pre-programmed (or auto-derived from prior analysis of a grid via the camera).

    Carl


  8. Location
    Melbourne, Australia
    Posts
    64
    Hi Christian,

    you are asking about live video in Unity. The following code is my render loop and I've had it running up to about 35 fps. My video data runs raw over a socket connection as a byte stream, three bytes per pixel.

    The following code has been tested with PAL video, 720 x 576 pixels, rendering at 25 frames per second.

    I'm using a dual processor 2.8 GHz which probably helps. I don't know where the cutoff point is in terms of CPU power however the code is optimal as far as I can tell (short of writing a DLL).

    C# optimises inner loops and I've written the pixel data iteration as a single dimensional loop.

    It would go a lot faster if I didn't have to do the byte to float conversion (multiplication by m). But that would require sending floats over the socket ( 4 times more data!!!)

    Note the use of the function: setPixels (plural).

    Code:  
    1. // per frame render
    2.  
    3. int i=0;
    4. int k;
    5. float m = 1.0f/255.0f;
    6. float r,g,b;
    7. int miplevel = 0;
    8. int numPixels = frameWidth * frameHeight;
    9. int numPixelBytes = numPixels * 3;
    10.        
    11. // even though this is a per/pixel operation
    12. // it works quite fast in C#
    13. k = frameOffset;
    14.        
    15. for(i=0; i<numPixels; i++)
    16. {
    17.   r = dataBuffer[k++] * m;
    18.   g = dataBuffer[k++] * m;
    19.   b = dataBuffer[k++] * m;
    20.   cols[i] = new Color(r, g, b);
    21. }
    22.        
    23. int xpos;
    24. int ypos;
    25.        
    26. xpos = (1024 - frameWidth)/2;
    27. ypos = (1024- frameHeight)/2;
    28.        
    29. texture.SetPixels (xpos, ypos, frameWidth, frameHeight, cols, miplevel);
    30.    
    31. // copy texture to the GPU
    32. texture.Apply(false);


  9. Location
    Melbourne, Australia
    Posts
    64
    Hi Christian,

    just looking at your demo client code:

    Code:  
    1. Color[] val = new Color[width * height];
    2. float[] color = new float[3];
    3. for(int y=0; y<height; y++){
    4.  for(int x=0; x<width; x++){
    5.   getPixel(x, y, color);
    6.   val[y*width + x] = new Color(color[0], color[1], color[2]);
    7.  }
    8. }
    9.        
    10. tex.SetPixels(val);
    11. tex.Apply();

    This is what I do in my Start function (ie. just once). Note class level variables accessible by update

    Code:  
    1. // class level globals
    2.  
    3. Texture2D texture;
    4. Color [] cols;
    5. Color colour;
    6.  
    7.     void Start ()
    8.     {
    9.    
    10.         // create a new texture for the video frame data
    11.         texture = new Texture2D (1024, 1024, TextureFormat.RGB24, false);
    12.        
    13.         // do we need expensive texture filtering? Use the least expensive:
    14.         texture.filterMode = FilterMode.Point;
    15.        
    16.         // assign our new texture to the object's texture
    17.         renderer.material.mainTexture = texture;
    18.    
    19.         // apply the object's texture to the object (ie. copy texture to GPU)
    20.         texture.Apply(false);
    21.        
    22.         int i=0;
    23.         int miplevel = 0;
    24.         int numPixels = frameWidth * frameHeight;
    25.    
    26.         int xpos;
    27.         int ypos;
    28.        
    29.         xpos = (1024 - frameWidth)/2;
    30.         ypos = (1024 - frameHeight)/2;
    31.        
    32.         // init cols array by getting pixels from the object ??
    33.         //cols = texture.GetPixels (xpos, ypos, frameWidth, frameHeight, miplevel);
    34.        
    35.         // Can't we just do it explicitly - yes we can ...
    36.         cols = new Color[numPixels];
    37.        
    38.         // TEST - set all pixels white
    39.         for(i=0; i<numPixels; i++)
    40.         {
    41.             cols[i] = new Color(1.0f, 1.0f, 1.0f);
    42.         }
    43.        
    44.         // copy pixels back to texture
    45.         texture.SetPixels (xpos, ypos, frameWidth, frameHeight, cols, miplevel);
    46.    
    47.         // copy texture to the GPU
    48.         texture.Apply(false);

    The update function's only responsibility is then to populate the cols array, setPixels to cols and apply to object (send to GPU).

    Carl


  10. Location
    Dubai, UAE
    Posts
    74
    Hi,

    I downloaded UnityAR and so far have tried it on a bootcamped MacBook Pro with iSight as well as a PC with a Philips webcam, but unfortunately non of them work.

    The demo can get as far as detecting modes, but it fails to set them (I have tried different ones, by changing modeNr in script).

    Any ideas why this is? Are there any other requirements to download perhaps?

    Thanks,
    attila


  11. Location
    Wiesbaden, Germany
    Posts
    24
    Quote Originally Posted by indy138
    Hi,

    I downloaded UnityAR and so far have tried it on a bootcamped MacBook Pro with iSight as well as a PC with a Philips webcam, but unfortunately non of them work.

    The demo can get as far as detecting modes, but it fails to set them (I have tried different ones, by changing modeNr in script).

    Any ideas why this is? Are there any other requirements to download perhaps?

    Thanks,
    attila
    Hm? That's is strange. Are there any error reports or exceptions that are coming up in the console?
    Christian Rathemacher
    Lead Programmer

    ------------------------------------------------------------------------------------------------

    weltenbauer. Software Entwicklung GmbH

    Humboldtstr. 7
    65189 Wiesbaden

    0611 / 97 14 314
    produktion.weltenbauer.com
    produkion@weltenbauer.com


  12. Location
    Wiesbaden, Germany
    Posts
    24
    Quote Originally Posted by carllooper
    Hi Christian,

    you are asking about live video in Unity. The following code is my render loop and I've had it running up to about 35 fps. My video data runs raw over a socket connection as a byte stream, three bytes per pixel.

    The following code has been tested with PAL video, 720 x 576 pixels, rendering at 25 frames per second.

    I'm using a dual processor 2.8 GHz which probably helps. I don't know where the cutoff point is in terms of CPU power however the code is optimal as far as I can tell (short of writing a DLL).

    C# optimises inner loops and I've written the pixel data iteration as a single dimensional loop.

    It would go a lot faster if I didn't have to do the byte to float conversion (multiplication by m). But that would require sending floats over the socket ( 4 times more data!!!)
    Thanks for your advice. I have nearly the same FPS on my test system with pal resolution. But do you realy think that using SetPixels() instead of SetPixel() for every pixel brings that speed-up. I think what SetPixel() is doing internaly, is setting a Color struct to a two dimensional Color array that is located in cpu memory. The function which slows down the whole process is tex.Apply() that copys this Color array to the gpu mem.

    The main question for me is, if there is a better way to apply all pixels from the webcam image in the gpu mem every frame. Maybe streaming the pixel data direct to gpu mem or something like that.

    But I think currently is SetPixel() and SetPixels() the only way to modify textures in gpu mem. Correct me if I'm wrong.
    Christian Rathemacher
    Lead Programmer

    ------------------------------------------------------------------------------------------------

    weltenbauer. Software Entwicklung GmbH

    Humboldtstr. 7
    65189 Wiesbaden

    0611 / 97 14 314
    produktion.weltenbauer.com
    produkion@weltenbauer.com


  13. Location
    Melbourne, Australia
    Posts
    64
    Hi Christian,

    when I compare the rate difference between setPixel and setPixels there is a significant difference. One should expect this since a loop around setPixel involves the overhead (per pixel) of a function call that, apart from anything else, must calculate an address from the supplied x and y, ie. something equivalent to:

    i = y * width + x

    An extra mutiiplication and addition per pixel! Plus the loop iterators for generating x and y in the first place:

    x++
    y++

    And the bounds check on both:

    x<width
    y<height

    And when you go back to how your original data was obtained in the first place ... I'm just using ARToolkit's builtin camera capture (referenced in SimpleTest by "dataPtr" - a single dimensional array of 4 bytes per pixel ... well, you can start to see what I mean.

    With respect to getting data from the CPU to the GPU - I don't know any other way within the framework of Unity. However I didn't find that was the bottleneck.

    If I prerender a 1024 x 1024 array of Colors in Startup, I can send this to the GPU, on every update, at about 75 fps. But if I try to populate every element of the Colors array during every update call then the fps drops right down to around 20 fps.

    So the bottleneck is in how to populate the array of Colors.

    Now if your data was already an array of Color structs one could (maybe) pass that directly into setPixels ... but I haven't tried that. You'd have to reverse engineer the internal structure of the Color array - or make some eduacted assumptions and test such - and I'm not sure C# would be happy about it anyway. In C/C++ you can always ensure how every bit of every byte is structured ...

    I'm still learning how C# works.

    So I guess that's an experiment to try. Casting a C prepared byte array - to a Colors array! Is that even possible?

    Carl


  14. Location
    China
    Posts
    1,009
    How to print "paper"?

    My Blog:Http://Www.1Vr.Cn
    Unity Tencent QQ Group:2453819,34643309,7838310,7838228
    Unity Forum: http://www.iu3d.com
    Development Environment:Unity Pro & iPhone Adv & MacBook Pro & iPhone & iPod & iPad & Zpad.


  15. Location
    Melbourne, Australia
    Posts
    64
    Ok, there is more optimisation one can do. C# provides a means by which to use what is called "unsafe" code. You qualify such code with the keyword "unsafe". That's an actual C# keyword. This tells the compiler not do any bounds checking on your code allowing you to access memory directly, ie. as you would in C/C++.

    The demo code I've got (from 'C# in A Nutshell' by Albahari and Albahari) is:

    Code:  
    1. unsafe void RedFilter(int[,] bitmap)
    2. {
    3.   int length = bitmap.length;
    4.   fixed (int* b = bitmap)
    5.   {  
    6.      int * p = b;
    7.      for(int i=0; i<length; i++)
    8.        *p++ &= 0xFF;
    9.   }
    10. }

    The above is C# code! I didn't realise until I read this that one could use pointers in C#. In other words one can do what is normally the province of C/C++. And according to Albahari and Albahari, it can be faster than calling an external C function because there's no overhead in leaving the managed environment.

    --

    Regarding nested loop iteration. Suppose we need to copy data from src to dest via a transport data structure. The simplest approach is:

    Code:  
    1. int src[100];
    2. int transport[100];
    3. int dest[100];
    4.  
    5. // source domain
    6. for(i=0; i<100; i++) transport[i] = src[i];
    7.  
    8. // destination domain
    9. for(i=0; i<100; i++) dest[i] = transport[i];

    Compare the above to this:

    Code:  
    1. int src[100];
    2. int transport[10,10];
    3. int dest[100];
    4.  
    5. // source domain
    6.  
    7. i=0;
    8. for(y=0; y<10; y++)
    9. {
    10.   for(x=0; x<10; x++)
    11.   {
    12.     transport[y,x] = src[i++];
    13.  
    14.   }
    15. }
    16.  
    17. // destination domain
    18.  
    19. for(y=0; y<10; y++)
    20. {
    21.   for(x=0; x<10; x++)
    22.   {
    23.       // do what setPixel must do
    24.       i = y * 10 + x;
    25.       dest[i] = transport[y,x];
    26.   }
    27. }

    As can be seen the first approach is going to be faster than the second approach, because there is simply much less for the compiled code to do.

    Carl


  16. Location
    Melbourne, Australia
    Posts
    64
    The next question is how to get Unity's complier to accept "unsafe" code.

    Carl


  17. Location
    Melbourne, Australia
    Posts
    64
    The next question is how to get Unity's complier to accept "unsafe" code.

    ...

    Ok, found it here:

    http://forum.unity3d.com/viewtopic.p...ht=unsafe+code

    On windows the default filepath is:

    C:\Program Files\Unity\Editor\Data\MonoCompiler.framework\com pile_any.pl

    And the code to change:

    From:

    my @base_args=($mono_path, $mcs_path, '-debug', '-target:library', '-nowarn:0169', '-out:'.$output, );

    To:

    my @base_args=($mono_path, $mcs_path, '-debug', '-unsafe', '-target:library', '-nowarn:0169', '-out:'.$output, );


    ----

    BUT - having done the above, Unity is still asking me to set the 'unsafe' option. Although I closed down Unity and started it up again, it may be that the compiler is temporarily cached (sitting in memory for a little while). I'll try leaving Unity shut down for a while.

    ----

    No joy. Not sure what's happening there.

    ----

    Oh great. Found this advice:

    http://forum.unity3d.com/viewtopic.p...=unsafe+option

    Unity no longer supports the use of unsafe. The only way to use unsafe is compile your own dlls - which defeats what I'm doing since I'm working with Unity Indie.

    ----

    Some other possibilitys here:

    http://forum.unity3d.com/viewtopic.p...=unsafe+option




    Carl


  18. Location
    Melbourne, Australia
    Posts
    64
    This works.

    Demonstrates writing unmanaged data into a Color.

    Code:  
    1. using System;
    2. using System.Runtime.InteropServices;
    3.  
    4. public struct MyColor
    5. {
    6.    public float r;
    7.    public float g;
    8.    public float b;
    9.    public float a;
    10.        
    11. }
    12.  
    13. void Startup()
    14. {
    15.  
    16. MyColor myColor;
    17.        
    18. myColor.r = 0.42f;
    19. myColor.g= 0.0f;
    20. myColor.b= 0.0f;
    21. myColor.a= 0.0f;
    22.        
    23. Color unityColor = new Color(0.0f, 0.0f, 0.0f);
    24.        
    25. info = "unityColor = " + unityColor.ToString() + "\n";
    26.        
    27. // Initialize unmanged memory to hold myColor
    28. IntPtr pnt = Marshal.AllocHGlobal(Marshal.SizeOf(myColor));
    29.        
    30. info += "Size of myColor = " + Marshal.SizeOf(myColor).ToString() + " bytes\n";
    31. info += "Size of unityColor = " + Marshal.SizeOf(unityColor).ToString() + " bytes\n";
    32.  
    33. // Copy myColor to unmanaged memory.
    34. Marshal.StructureToPtr(myColor, pnt, false);
    35.        
    36. // display the address of unmanaged memory
    37. info += "pnt = " + pnt.ToString() + "\n";
    38.  
    39. // This is the import one
    40. // Copy the data at address pnt, from unmanaged memory, into Color
    41.  
    42. unityColor = (Color) Marshal.PtrToStructure( pnt, typeof(Color) );
    43.        
    44. info += "unityColor = " + unityColor.ToString() + "\n";
    45.    
    46. // Free the unmanaged memory.
    47. Marshal.FreeHGlobal(pnt);
    48.  
    49. }


  19. Location
    Melbourne, Australia
    Posts
    64
    Now I'm really quite perplexed, amazed and somewhat relieved.

    I've just discovered that I can access DLLs from within Unity Indie.

    I thought that wasn't possible. The comparison chart between Indie and Pro and says that Indie doesn't support the following Pro feature:

    C/C++/Objective-C Plugins Support
    Have a custom native library that you absolutely must use? Use a C, C++ or Objective-C plugin to directly call into it.


    So why does the following succeed in invoking the Windows OS message box in the Indie version of Unity3D?

    Code:  
    1. using UnityEngine;
    2. using System.Collections;
    3.  
    4. using System;
    5. using System.Runtime.InteropServices;
    6. using System.Text;
    7. using System.IO;
    8.  
    9. class MsgBoxTest
    10. {
    11.     [DllImport("user32.dll")]
    12.     static extern int MessageBox (IntPtr hWnd, string text, string caption, int type);
    13.     public static void Main()
    14.     {
    15.         MessageBox (IntPtr.Zero, "Unity3D Indie version supports access to DLLs", "Attention", 0);
    16.     }
    17. }
    18.  
    19. public class DLLclientTest : MonoBehaviour
    20. {
    21.     void Start ()
    22.     {
    23.         MsgBoxTest.Main();
    24.     }
    25. }

    The entire time I've been working on getting ARToolkit data into Unity3D via a socket connection I could have just written a DLL instead.

    ?????

    Oh well - it was worth the effort anyway - I now know how to do sockets in C#.


  20. Location
    Melbourne, Australia
    Posts
    64
    So here we go:

    C++:

    Code:  
    1. #ifdef __cplusplus
    2.    extern "C" {  
    3. #endif
    4.    __declspec( dllexport ) int secretOfTheUniverse();
    5. #ifdef __cplusplus
    6. }
    7. #endif
    8.  
    9. __declspec( dllexport ) int secretOfTheUniverse()
    10. {
    11.     return 42;
    12. }

    C#:

    Code:  
    1. using UnityEngine;
    2. using System.Collections;
    3.  
    4. using System;
    5. using System.Runtime.InteropServices;
    6.  
    7. class TestDLL
    8. {      
    9.    [DllImport("MyDLL.dll")]
    10.    static extern int secretOfTheUniverse();
    11.    
    12.    public static int Secret()
    13.    {
    14.       return secretOfTheUniverse();
    15.    }   
    16. }
    17.  
    18. public class DLLTest : MonoBehaviour
    19. {
    20.    static String info = "Testing DLL\n";
    21.    
    22.    void Start ()
    23.    {
    24.       int i;
    25.       i = TestDLL.Secret();
    26.       info += "The secret of the Universe is " + i.ToString() + "\n";
    27.        
    28.    }
    29.    
    30.    void OnGUI ()
    31.    {
    32.       GUI.Label (new Rect (0, 0, 200, 200), info);
    33.    }
    34. }

    The result in Unity:

    Testing DLL
    The secret of the Universe is 42


    So in relation to performance of an ARToolkit port I'm assuming the following will be pretty fast:

    On Startup:

    1. Allocate enough unmanaged memory (using Marshalling) to represent an array of Colors, getting back a pointer to such.

    2. Pass the pointer to a DLL, which holds it for future reference.

    On Update:

    1. Call a DLL function (which has the unmanaged memory pointer) to populate the unmanaged memory with four floats per pixel (which will be very fast).

    2. Copy the memory back into managed memory (using Marshalling) where it is now referenced by a Color reference.

    3. Pass the Color reference to texture.SetPixels.

    4. Call apply on the texture to copy the Color data to the GPU.

    On ApplicationQuit:

    1. Free the unmanaged memory.

    ---

    Now I found the only thing stopping my Indie version of Unity3D accessing a DLL was if I put the DLL where the docs (for the Pro version) say to put it, ie in:

    Assets > Plugins.

    Doesn't work.

    The Indie version (not surprisingly) doesn't look there. But if I put the DLL in the System search path (a normal location for DLLs), eg.

    C:\Windows\System32

    No problem.

    Or if I put it in the other place where DLLs typically become available to their hosts:

    The same folder as a published standalone executable.

    No problem.

    Or I put it in the same folder as the Unity editor executable:

    C:\Program Files\Unity\Editor

    No problem.

    So I guess the only thing the Pro version offers in this respect is just the convenience of it being in a single project specific location where both the editor can find it and from where it will be published. Fair enough. That's a value. But it's still bit of a stretch to sell such convenience as "plugins are a Pro only feature". Should I laugh or cry?

    Carl

Page 1 of 9 123 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •