Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

ARToolkit extension

Discussion in 'Scripting' started by wimeck, Jun 5, 2008.

  1. wimeck

    wimeck

    Joined:
    May 26, 2008
    Posts:
    50
    Hi There!

    I’m quite interested in ARToolkit ( http://www.hitl.washington.edu/artoolkit/ ), a software library for building Augmented Reality (AR) applications. These are applications that involve the overlay of virtual imagery on the real world.
    GameBlender and Virtools already have plugins to communicate with ARToolkit. Check http://www.ash.webstranka.info/?p=36 for an example movie.
    My question is, how difficult will it be to combine ARToolkit with Unity? And could it be possible to make a connection without owning Unity pro? I am not a programmer by any means. I actually just started trying Unity to learn programming in a more playful way.
    If anyone is interested in making a connection, ARToolkit plus ( http://studierstube.icg.tu-graz.ac.at/handheld_ar/artoolkitplus.php ) would be an even better base to build ontop.

    Links:

    GameBlender plugin: http://www.ash.webstranka.info/?p=45

    Virtools plugin: http://mirevicast.medien.fh-duesseldorf.de/projects/virtools-extensions

    As an example what kind of games you could create, check this amazing project by Julian Oliver: http://julianoliver.com/levelhead

    Greetings,

    Wim
     
  2. Marc

    Marc

    Joined:
    Oct 4, 2007
    Posts:
    499
    From an initial scan of the docs it seems like its written in C and i couldn't find any API in C# so think it will have to be done in a plugin but it should be possible. It might be hairy though to actually combine the rendering pipeline into unity, but can't say for sure as i havent done it :D

    Is a cool technology.

    Regards,
    Marc
     
  3. milkytreat

    milkytreat

    Joined:
    Jan 15, 2007
    Posts:
    267
  4. solmyr-fr

    solmyr-fr

    Joined:
    Jun 18, 2009
    Posts:
    24
    Hi,

    I'm also doing a plugin with ARToolKit.

    A few things I managed to do :

    - The pattern recognition works.

    - I've integrated the image grabbed by the webcam on a plane (Texture2D) which is far away from the Unity camera.

    - Thanks to openGL I have access to 2 matrices [16] of 16 elements that I convert in 2 matrices [3][4] so i can extract easily a position vector and a quaternion with an ARToolkit function "arUtilMat2QuatPos". The first one is the projection matrix for the Unity Camera (the real webcam is not in the position (0,0,0) and rotation (0,0,0) in Unity world, this is due to camera calibration). --> "glGetFloatv(GL_PROJECTION_MATRIX,MatrixProjection);"
    The other one is the model-view matrix for the Unity object which will appear on the pattern. -->"glGetFloatv(GL_MODELVIEW_MATRIX,MatrixModelView);"

    -The object (in Unity -> a cube) I want to be on the pattern follow the pattern when I move the pattern.




    Nevertheless, I have some projection (or position) problems : the object is not on the middle of the pattern but is shifted.

    I think that the problem is that one or two of my gameObject (the camera, the cube and the plane) is not is the right position. Because when the pattern is on the left extremety of the screen, the cube has a better position than if the pattern was on the right extremity of the screen.

    The distance plane-camera is a constant and is set to an arbitrary value ("4150" so that the camera look only at the plane in the Unity world)
    The plane move only one time with the camera in the "start" function (due to the projection matrix).
    The cube move alone (all the time) (due to the model-view matrix) in the "update" function.

    Does someone (who knows well Unity/OpenGL or ARToolkit) thinks that I've missed a step to determine the positions of my gameObjects.

    Or maybe the problem is somewhere else :s .
     

    Attached Files:

  5. burnumd

    burnumd

    Joined:
    May 27, 2008
    Posts:
    367
    I've likewise implemented an ARTk plugin and got exactly to where solmyr is. Something gets mucked up in the translation from the plugin coordinates to Unity. I tried asking on the ARTk forums and emailed the maintainer about it and all I ever got was a rather brusque reply that I must be reversing the projection matrix (ie, calculating it in camera space rather than world space, which can be, and was, tested by adding a single function call to the plugin). There really isn't much support for ARTk anymore outside of the rather lackluster forums, unfortunately, and the maintainers didn't seem interested in looking into the problem at all.
     
  6. solmyr-fr

    solmyr-fr

    Joined:
    Jun 18, 2009
    Posts:
    24
    yeah... it's really strange, it's working perfectly when the drawing are done in OpenGL (in ARToolKIt code samples).
    I've followed every function that is necessary in their code to get the 2 matrices. Then the adaptation to unity should have be easy :?
     
  7. zeroZshadow

    zeroZshadow

    Joined:
    Feb 11, 2009
    Posts:
    30
    looks interesting, is this useable for indie too ?
    if so, could you make an guide or example project ?
    i'd like to try it out
     
  8. burnumd

    burnumd

    Joined:
    May 27, 2008
    Posts:
    367
    It's Pro only. When last I checked the ARTk library is C and C++ only (with wrappers for some other languages), which means it has to be done via plugin.
     
  9. horsman

    horsman

    Joined:
    Jul 19, 2008
    Posts:
    156
    This is a long shot, but:

    as far as I can tell, Unity uses a left handed co-ordinate system. OpenGL uses a right handed co-ordinate system.

    If you are stacking your matrixes in Unity, do them in the reverse order. Let me know if this works.
     
  10. burnumd

    burnumd

    Joined:
    May 27, 2008
    Posts:
    367
    I can't speak for solmyr, but I wasn't mucking with the rendering pipeline at all (aside from using Graphics.DrawTexture() to overlay the incoming video image) since doing so would require getting the camera modelview matrix post-render, sending it to the plugin and drawing the AR objects there and then setting it again (which defeats the purpose of using a 3D engine). I was simply using the transformation values ARTk produced to update the position of Unity prefabs. There is a utility function in ARTk to get a right-handed transformation matrix, but just from looking at my notes (the project won't open in 2.5 without some conversion I can't really do right now), we were actually getting different transform values in the trace from the plugin versus what Unity was actually receiving.
     
  11. solmyr-fr

    solmyr-fr

    Joined:
    Jun 18, 2009
    Posts:
    24
    Yes I'm doing it in the same way (don't work either)

    I don't need it, i'm just switch Y and Z axes. Apparently it works because the cube is following the pattern in all directions (up/down/left/right/front/back)

    I'd rather think that the problem comes from an openGL transformation we don't understand (with the 2 matrices projection/model-view)

    Or perhaps it comes from another coordinates problem:
    -the projection matrix has to be inversed (because we need camera coordinates in the real world)
    -the modelview matrix musn't be inversed (because it's the pattern coordinates in the camera coordinates)

    3 possibilities , 3 failures ^^
     

    Attached Files:

  12. solmyr-fr

    solmyr-fr

    Joined:
    Jun 18, 2009
    Posts:
    24
    Another strange thing (burnumd have the same problem I suppose)

    With ARToolKit code sample, the openGL modelview matrix is :
    -0.210773 | 0.411159 | -0.886861 | 0
    0.969434 | -0.0286372 | -0.243674 | 0
    -0.125586 | -0.911114 | -0.392556 | 0
    51.7304 | -38.3579 | 384.759 | 1


    With my code in Unity3d, the openGL modelview matrix is :
    0.02105 | 0.377307 | 0.925849 | 0
    0.99863 | 0.0364404 | -0.0375552 | 0
    -0.0479081 | 0.925371 | -0.376023 | 0
    51.9647 | 21.9308 | 391.948 | 1

    The pattern is exactly in the same position.

    The more important thing is the position of the pattern ( i'll deal with the rotation problem after). The position XYZ is the last line:
    51.7304 | -38.3579 | 384.759 in the first case
    51.9647 | 21.9308 | 391.948 in the other case.

    :eek:
     
  13. burnumd

    burnumd

    Joined:
    May 27, 2008
    Posts:
    367
    That's pretty much exactly the problem we were having. We tried to do some corrections to the values, but it didn't seem to correspond linearly and I, at least, couldn't see a pattern to the deviation. I didn't use the modelview matrix directly (I used the utility function to convert the matrix to two double arrays. Looking at your data, I wonder now if it's a bug in the plugin system. Maybe an overflow or type conversion issue since ARTk gives the matricies as doubles rather than floats? From looking at my plugin, it doesn't seem like it would be the latter, since my plugin print statements display the converted float value and I saw the same issue. It didn't occur to me at the time to report it because I assumed it to be either an ARTk problem or something I was just doing wrong, but if you've got a working(-ish) project, you might consider submitting a ticket to Unity QA with your project (along with the reams of fiducial patterns so they can actually test it).
     
  14. zeroZshadow

    zeroZshadow

    Joined:
    Feb 11, 2009
    Posts:
    30
    just well.. did you try mirroring the camera texture >_> ?
     
  15. solmyr-fr

    solmyr-fr

    Joined:
    Jun 18, 2009
    Posts:
    24
    First, thanks for your help :).

    Can't be that, I send a floats matrix to Unity.

    The main steps of my program are (in red C# script , in blue C++ plugin , green comments):

    -Start()
    -->initARToolKit(); // init the webcam and the projection matrix
    -->GetQuatAndPos(quaternionCam, positionCam, MODE_CAM,INVERSE_CAM); // get position and rotation for the cam using arUtilMat2QuatPos(newMatrix, q, p);
    // "quaternionCam" and" positionCam" are float matrices whereas "q" and "p" are double matrices

    --> UpdateCamera(...) // update positions and rotations of the Unity camera


    -Update()
    --> mainLoop(); // grab the image,detect the marker, calculate the "patt_trans" matrix
    -->UpdateTexture1(...) //Update the texture on the plane like in the texture plugin
    --> if(IsPatternDetected()
    ){
    ---->GetQuatAndPos(quaternionOrient, positionPatt,MODE_CUBE,NULL); // get the position and rotation of the cube in float matrices
    ---->UpdateCube(...) // update positions and rotations of the cube
    }

    Here the code for "GetQuatAndPos(quaternionOrient, positionPatt,MODE_CUBE,NULL)" . You can see that there is no problem of double or float conversion through the unity plugin
    Code (csharp):
    1.  
    2. {
    3. argConvGlpara(patt_trans, gl_para);
    4. glMatrixMode(GL_MODELVIEW);
    5. glLoadMatrixd( gl_para );
    6. glTranslatef( 0.0, 0.0, 25.0 );
    7. glGetFloatv(GL_MODELVIEW_MATRIX,matriceModelView);
    8. for(int ii = 0; ii < 3; ii++ ){
    9.     for( int jj = 0; jj < 4; jj++){
    10.             newMatrix[ii][jj] = matriceModelView[jj*4+ii];
    11.     }
    12. }  
    13.                     arUtilMat2QuatPos(newMatrix, q, p);
    14.  
    15. for( int ii = 0; ii < 4; ii++ ){quat[ii] = (float) q[ii];}
    16. for( int ii = 0; ii < 3; ii++ ){pos[ii] = (float) p[ii];}
    17. argSwapBuffers();
    18. }

    Already did because the camera grab a mirrored image (and when I move my pattern in the left side in reality, it does the same thing in the texture).
    Moreover the problem isn't here zeroZshadow.

    There is nothing that could explain the difference between the values in the modelview matrices in the 2 cases (ArTk sample of code and my unity plugin) with a pattern in a same position.
    -I follow every step they do
    -I don't modify the intermediate matrices
    -I compare the openGL modelview matrix that is used in the display function to my openGL modelview matrix that is used just before sending the coordinates to Unity.
     
  16. spacecat

    spacecat

    Joined:
    Jul 17, 2008
    Posts:
    16
    Hi, I didn't read this very closely, but I am incorporating StbES into Unity and have change both the position and quaternions to Unity's left handed system. For the position of a marker (z up) I found I had to invert the Y value. For the quaternion, I had to invert the X and Z quat values. We are not putting the projection matrix delivered by StbES directly into the camera because it seems to interfere with picking. So we are just setting the transform and FOV from calibration data.
     
  17. Christian

    Christian

    Joined:
    Dec 2, 2008
    Posts:
    24
    Hey Guys!

    This is my way to calculate the position and rotation of an object using ArToolkit. It works fine if the camera position is (0,0,0) and looks to +z axis, the image plane is at position (0,0,960) with a scale of (128,1,96):
    Code (csharp):
    1.  
    2. // position
    3. double[] trans = new double[3];
    4.  
    5. // rotation vector as quaternion
    6. double[] rot = new double[4];
    7.  
    8. // Get the translation and the rotation with
    9. // the AR function arUtilMat2QuatPos
    10. getDetectedTranslationRotation(trans, rot);
    11.  
    12. // Set position
    13. transform.position = new Vector3(-(float)trans[0] * 1.5f, -(float)trans[1] * 1.5f, (float)trans[2]);
    14.  
    15. // Set rotation
    16. transform.rotation = new Quaternion((float)rot[0], (float)rot[1], -(float)rot[2], (float)rot[3]);
    17.  
    18.  
    The getDetectedTranslationRotation is a DLL function where patt_trans is the dedected marker matrix:

    Code (csharp):
    1.  
    2. EXPORT void getDetectedTranslationRotation(double* translation, double* rotation){
    3.     arUtilMat2QuatPos(patt_trans, rotation, translation);
    4. }
    5.  
    6.  
    This implementation works fine in my case, even if the marker are in the corner of the webcam image.
     

    Attached Files:

  18. 3dever

    3dever

    Joined:
    Mar 2, 2009
    Posts:
    71
    Can someone publish sources? :)
     
  19. Narlix

    Narlix

    Joined:
    Jul 3, 2009
    Posts:
    111
    wow, the sources would be awesome: :eek:
     
  20. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    Hi,

    I have experience in the AR side of things - I can code in C/C++ (amongst other things) and understand camera matrices.

    But I'm new to Unity3D. What do I need to do to get webcam video streaming into a Unity 3D authored demo?

    Will I need the pro version of Unity3D? Or will the indie version do it?

    Carl
     
  21. half_voxel

    half_voxel

    Joined:
    Oct 20, 2007
    Posts:
    978
    Cool!!

    Is this pro only?
     
  22. Dreamora

    Dreamora

    Joined:
    Apr 5, 2008
    Posts:
    26,601
    Plugins are always Pro only
     
  23. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    dreamora said: "Plugins are always Pro only"

    Ah yes - I understand plugins require the pro version - but I'm not proposing a plugin per se. Although that may be what I'll end up writing.

    In the meantime I'm hoping there is some way to pull in a webcam stream without writing a plugin, ie. using a script.

    But if not I'll just have to purchase the pro version and write a plugin.

    Carl
     
  24. jorisw

    jorisw

    Joined:
    Aug 13, 2009
    Posts:
    5
    Hi everybody!
    Our team has developed a so-called Performance engine which is a modular system that can dynamically connect many in- and outputs (being sensors and media) within a nice GUI (mostly build with MaxMSP). We're using Unity as our 3D engine. Soon we want to intergrate mixed reality ideas in our performance.

    Anybody (Christian, Carllooper?) made any progress in implementing ARToolkit in Unity, AND is willing to share the scripts or plugin?

    Our performance engine software will soon be available to anybody under GPL when we have a stable version.
    Have a look at http://www.talltreelabs.org/wordpress/
     
  25. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    I now understand that the indie version of unity can do normal socket communication using C#, so my project is progressing.

    An ARToolkit application is being tested as a server independant of a unity application. The idea is that any application (eg. a unity app) can connect to the ARToolkit server through a socket. It then gets each frame of video, plus calibration data for that frame, through the socket.

    It is up to the unity application (or any other app) what it actually does with the data but typically the calibration data (in the form of a camera matrix), would be used by the application to render objects in the correct perspective. And the video frame would be applied as a dynamic texture to a polygon set up as the background.

    I haven't yet tested this pipeline in Unity. The server is currently being use-tested with Flash/swf clients, rather than unity clients. But I hope to have a unity client up and running in the coming weeks.

    This server based approach is suited to the indie version of unity, ie. no plugins required - just C# scripts.

    Carl
     
  26. jorisw

    jorisw

    Joined:
    Aug 13, 2009
    Posts:
    5
    Hi carllooper,
    any progress inthe last two weeks?
    We will start a project in oktober and I'm very interested in using ARToolkit with Unity 3D. Either embedded or as you ar now doing through a server...

    Thnx!
    Joris.
     
  27. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    Hi Joris,

    I haven't had a chance to start on it! Have been sidetracked by a couple of other jobs. But I'm now booked in to do it next week - so that's great. I'll be posting source-code in due course - and win32 binarys.

    The project we're doing doesn't actually require Unity as the graphics engine - and indeed, I'll probably be building a custom graphics engine for the particular project on which I'm booked - but the design is modular ie. so that any graphics engine with socket support (such as Unity) can use the server.

    Carl
     
  28. jorisw

    jorisw

    Joined:
    Aug 13, 2009
    Posts:
    5
    Hi Carllooper,
    sounds great, look foreward to the results.

    Do you have any indicators how (through what protocol) the data from that server can be interpreted. Are you going to use UDP, OSC? And what exactly are you going to send: marker ID, xyz?

    Thing is we have very limited skills in writing C# scripts for the client side of things. So some information and advice would be welcome...

    Good luck with the project!
    J.
     
  29. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    I haven't fully defined the protocol but the following information will be available on a per frame basis:

    camera parameters
    video image frame (ie. rgb pixels)

    I'll be using the "CAHV" model for the camera parameters:

    http://www.digilab.uni-hannover.de/docs/manual.html#cahvmodel

    It will be up to the graphics engine client to transform the camera parameters into a form more convenient for their particular purpose, but I'll post example usage.

    The CAHV model described above is specified in terms of a "convenient" composite matrix (that includes the perspective transform, pixel aspect, etc) AND one without such (which is actually much more convenient) - an invertible matrix allowing transforms between coordinate systems (camera to world and back again) before/after which one can factor in/out the perspective/pixel-aspect/offsets etc.

    But I'll post tutorials on this - if only to remind myself how to use such.

    Carl
     
  30. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    The following code is in Javascript.

    It demonstrates transforming between camera and world coordinates using the CAHV camera model. This was prepared for use in Flash where the graphics engine is custom coded.

    Code (csharp):
    1. /*
    2.  
    3. The following demonstrates usage of the data provided by the Voodoo camera tracking software.
    4.  
    5. The data uses the "CAHV" camera model. See: [url]http://www.digilab.uni-hannover.de/docs/manual.html#cahvmodel[/url]
    6.  
    7. The following demo code uses the same variable names as that used in Voodoo:
    8.  
    9.         Cx Cy Cz
    10.         Ax Ay Az
    11.         Hx Hy Hz
    12.         Vx Vy Vz
    13.         K3 K5
    14.         sx sy
    15.         Width Height
    16.         ppx ppy
    17.         f
    18.         fov
    19.         H0x H0y H0z
    20.         V0x V0y V0z
    21.  
    22. */
    23.  
    24. //=========================
    25. // EXAMPLE CAHV CAMERA DATA
    26. //=========================
    27.  
    28. var cahv:Object = new Object();
    29.  
    30. // Camera position
    31.  
    32. cahv.Cx = 29.0416229219
    33. cahv.Cy = 8.3053284856 
    34. cahv.Cz = -18.0094353929   
    35.  
    36. // rotation matrix - this is a composite matrix where H and V factor in sx, sy, f
    37.  
    38. cahv.Ax = 0.4532135977 
    39. cahv.Ay = 0.0047848041 
    40. cahv.Az = 0.8913891072 
    41. cahv.Hx = 1228.2586202365  
    42. cahv.Hy = 44.2655269119
    43. cahv.Hz = -624.7275242183  
    44. cahv.Vx = -45.2768078343   
    45. cahv.Vy = 1469.8574547117  
    46. cahv.Vz = 15.1304125920
    47.  
    48. // radial distortion
    49. cahv.K3 = 0.0000000000 
    50. cahv.K5 = 0.0000000000
    51.  
    52. // aspect (in this case a standard definition DV camera)
    53. cahv.sx = 0.0106666667 
    54. cahv.sy = 0.0100000000 
    55.  
    56. // image size (SD camera)
    57. cahv.Width = 720   
    58. cahv.Height = 576  
    59.  
    60. // principle point (we have a well engineered camera)
    61. cahv.ppx = 0.0000000000
    62. cahv.ppy = 0.0000000000
    63.  
    64. // focal length
    65. cahv.f = 14.7063246798
    66.  
    67. // field of view
    68. cahv.fov = 29.2677729859   
    69.  
    70. // the following provides H and V independant of sx, sy, and f
    71. // so when combined with A, one has an invertible rotation matrix (of unit vectors)
    72.  
    73. cahv.H0x = 0.8908701234
    74. cahv.H0y = 0.0321062965
    75. cahv.H0z = -0.4531220685   
    76. cahv.V0x = -0.0307873033   
    77. cahv.V0y = 0.9994730068
    78. cahv.V0z = 0.0102883711
    79.  
    80. //=================================
    81. // TEST THE DATA
    82. //=================================
    83.  
    84. trace("Given an arbitrary point in world coordinates:");
    85.  
    86. var w:Object = new Object();
    87. var c:Object;
    88.  
    89. w.x = 400;
    90. w.y = 300;
    91. w.z = 200;
    92.  
    93. trace("");
    94. trace("w.x = " + w.x);
    95. trace("w.y = " + w.y);
    96. trace("w.z = " + w.z);
    97.  
    98. trace("");
    99. trace("Transform the point into camera coordinates ...");
    100.  
    101. trace("");
    102. trace("transform: camera->world:");
    103.  
    104. c = world2camera(cahv, w);
    105.  
    106. trace("");
    107. trace("c.x = " + c.x);
    108. trace("c.y = " + c.y);
    109. trace("c.z = " + c.z);
    110.  
    111. trace("");
    112. trace("OR using the ortho matrix transform instead we get the same result ...");
    113.  
    114. c = world2cameraOrtho(cahv, w);
    115.  
    116. trace("");
    117. trace("c.x = " + c.x);
    118. trace("c.y = " + c.y);
    119. trace("c.z = " + c.z);
    120.  
    121. trace("");
    122. trace("Now we can transform the above into world coordinates ...")
    123.  
    124. trace("");
    125. trace("transform: camera -> world:");
    126.  
    127. w = camera2world(cahv, c);
    128.  
    129. trace("");
    130. trace("w.x = " + w.x);
    131. trace("w.y = " + w.y);
    132. trace("w.z = " + w.z);
    133.  
    134. trace("");
    135. trace("But this point is not the same as our original world point because")
    136. trace("our input point wasn't in the image plane - but since this one is in the image plane")
    137. trace("we can do the tests again starting with the above world point (in the image plane)");
    138.  
    139. trace("");
    140. trace("w.x = " + w.x);
    141. trace("w.y = " + w.y);
    142. trace("w.z = " + w.z);
    143. trace("");
    144. trace("transform: world -> camera:");
    145. c = world2camera(cahv, w);
    146. trace("");
    147. trace("c.x = " + c.x);
    148. trace("c.y = " + c.y);
    149. trace("c.z = " + c.z);
    150. trace("");
    151. trace("transform: camera->world:");
    152. w = camera2world(cahv, c);
    153. trace("");
    154. trace("w.x = " + w.x);
    155. trace("w.y = " + w.y);
    156. trace("w.z = " + w.z);
    157.  
    158.  
    159. //-------------------------
    160. // WORLD > CAMERA TRANSFORM
    161. //-------------------------
    162.  
    163. function world2camera(cahv:Object, w:Object):Object
    164. {
    165.     // 1. translate by inverse of camera position
    166.    
    167.     var x1 = w.x - cahv.Cx;
    168.     var y1 = w.y - cahv.Cy;
    169.     var z1 = w.z - cahv.Cz;
    170.    
    171.     // 2. transform using the *composite* matrix
    172.    
    173.     var x2 = cahv.Hx*(x1) + cahv.Hy*(y1) + cahv.Hz*(z1)
    174.     var y2 = cahv.Vx*(x1) + cahv.Vy*(y1) + cahv.Vz*(z1)
    175.     var z2 = cahv.Ax*(x1) + cahv.Ay*(y1) + cahv.Az*(z1)
    176.    
    177.     // 3. project onto image plane (perspective projection)
    178.    
    179.     var x3 = (x2/z2) * 0.01;
    180.     var y3 = (y2/z2) * 0.01;
    181.     var z3 = (z2/z2) * 0.01;
    182.    
    183.     // answer
    184.     var c:Object = new Object();
    185.    
    186.     c.x = x3;
    187.     c.y = y3;
    188.     c.z = z3;
    189.    
    190.     return c;
    191.    
    192. }
    193.  
    194.  
    195. //----------------------------------------------------
    196. // WORLD > CAMERA TRANSFORM
    197.  
    198. // Gives same result as above but uses the ortho
    199. // matrix instead. Using the ortho matrix requires an
    200. // additional step (step 3) to accomodate pixel aspect
    201. // and focal length
    202. //----------------------------------------------------
    203.  
    204. function world2cameraOrtho(cahv, w)
    205. {
    206.     // 1. translate by inverse of camera position
    207.    
    208.     var x1 = w.x - cahv.Cx;
    209.     var y1 = w.y - cahv.Cy;
    210.     var z1 = w.z - cahv.Cz;
    211.    
    212.     // 2. transform using the *orthogonal* matrix (affine rotation)
    213.    
    214.     var x2 = cahv.H0x*(x1) + cahv.H0y*(y1) + cahv.H0z*(z1);
    215.     var y2 = cahv.V0x*(x1) + cahv.V0y*(y1) + cahv.V0z*(z1);
    216.     var z2 = cahv.Ax *(x1) + cahv.Ay *(y1) + cahv.Az *(z1);
    217.    
    218.     // 3. ADDITIONAL STEP to account for pixel-aspect and focal length
    219.    
    220.     var x3 = x2 * (1.0/cahv.sx);
    221.     var y3 = y2 * (1.0/cahv.sy);
    222.     var z3 = z2 * (1.0/cahv.f);
    223.    
    224.     // 4. project onto image plane (perspective projection)
    225.    
    226.     var x4 = (x3/z3) * 0.01;
    227.     var y4 = (y3/z3) * 0.01;
    228.     var z4 = (z3/z3) * 0.01;
    229.    
    230.     // answer
    231.     var c:Object = new Object();
    232.    
    233.     c.x = x4;
    234.     c.y = y4;
    235.     c.z = z4;
    236.    
    237.     return c;
    238. }
    239.  
    240. //---------------------------------------------
    241. // CAMERA > WORLD TRANSFORM
    242. //
    243. // This can only be done using the ortho matrix
    244. //---------------------------------------------
    245.  
    246. function camera2world(cahv, c):Object
    247. {
    248.     // 1. transform point by pixel-aspect factors and multiply by a 100
    249.  
    250.     var x1 = c.x * cahv.sx * 100;
    251.     var y1 = c.y * cahv.sy * 100;
    252.     var z1 = c.z * cahv.f  * 100;
    253.    
    254.     // 2. transform pixel by the inverse of the rotation matrix
    255.    
    256.     var x2 = cahv.H0x*(x1) + cahv.V0x*(y1) + cahv.Ax*(z1);
    257.     var y2 = cahv.H0y*(x1) + cahv.V0y*(y1) + cahv.Ay*(z1);
    258.     var z2 = cahv.H0z*(x1) + cahv.V0z*(y1) + cahv.Az*(z1);
    259.    
    260.     // 3. translate by camera position
    261.    
    262.     var x3 = x2 + cahv.Cx;
    263.     var y3 = y2 + cahv.Cy;
    264.     var z3 = z2 + cahv.Cz;
    265.    
    266.     // answer
    267.     var w:Object = new Object();
    268.     w.x = x3;
    269.     w.y = y3;
    270.     w.z = z3;
    271.    
    272.     return w;
    273.  
    274. }
    275.  
    276.  
     
  31. jorisw

    jorisw

    Joined:
    Aug 13, 2009
    Posts:
    5
    Hi carl!
    Thanks for this info, I will show this to my group hoping they can actually work with this. I was hoping for having a few scripts for inside Unity either in c# or indeed javascript (although it is called Unity-script and really no javascript alltogether) that I can attach to the camera and/or objects.

    Do you think you or one of your collegues are going to look into that? I will try to find people too. Maybe somebody reading this??

    Thanks in advance for sharing!
    Joris.
     
  32. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    Hi Joris,

    I'll be writing up unity specific C# code in due course. No problem. The code I posted was just an example of the sort of thing one would be doing on the client side. But I'll be writing up usable client specific code - at least for the following rendering contexts:

    DarkGDK
    Qt OpenGL
    Unity3D
    FlashPlayer10

    Carl
     
  33. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    In low level code, such as OpenGL, the camera is always stationary and everything else is transformed from one frame to the next (moving) relative to the stationary camera.

    But in higher-level coding eg. in frameworks such as Unity 3D, one typically specifys both camera and object transformations (motions). Nevertheless - under the hood - such specifications just get re-encoded in terms of a stationary camera (and moving objects) because the graphics hardware only needs to know the appropriate transforms to apply to objects.

    So it all depends at which level one is working - low level coding (in which the camera doesn't move) or higher level coding (in which both camera and objects are regarded as moving).

    For the client side coding I'll be describing the video in terms of a moving camera - and the marker as stationary.

    But a single (and simple) matrix inversion provides for the opposite scenario - where one treats the camera as stationary and the marker as moving ...

    ... the sort of thing the previously posted javascript code demonstrates.

    Carl
     
    jethrogillgren likes this.
  34. jorisw

    jorisw

    Joined:
    Aug 13, 2009
    Posts:
    5
    Hey Carl,
    wow, that's great. Didn't dare to hope for so much.
    I know what you mean with the moving or stationary cam. In fact I've used the Virtools ARToolkit plug and noticed that indeed tha camera would stay at 0,0,0 and the rest would move. Great if you want to animate something...

    Which brings me to the point that Unity has some great physics engine! But if the world would constantly change its angle in relation to a stationary camera we will see interesting effects. Of course something like gravity has a direction, -Y for example. If the world turns, objects will slide of the ground plane... Or am I completely wrong here?

    Anyway. Again thank you. Do you have any indication in terms of a schedule so I know when to expect some samples?

    Cheers,
    joris.
     
  35. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    Hi Joris,

    yes - a physics component is on my list of client-side ideas as well. I'm thinking of an object attached by a spring to the marker so that as you move the marker around the object springs about in a life-like way relative to the marker: a Jack-In-The-Box toy. In this case the camera (in real-life) would be stationary.

    There are some unsolvable problems with a free moving camera eg. the relationship between camera and gravity - or the camera and light source, can't be defined (would become invalid). One could define gravity and light relative to a marker. ie. one treats the marker as staionary (with respect to gravity and light source). Completely unsolvable is if both are moving relative to gravity and light source. The only solution would be to use ambient lighting and simply ignore gravity - as if one were in orbit/free fall).


    Anyway - back to work ...

    cheers
    Carl
     
  36. Christian

    Christian

    Joined:
    Dec 2, 2008
    Posts:
    24
  37. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    The provided weblink to UnityAR crashes my browser.

    I've finished my server version, but have yet to finish writing up the client side. Operates at the speed of whatever camera is being used, eg. 720 x 576 @ 25 fps, in Unity Indie.

    Uses ARToolkit as well.

    Carl
     
  38. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    That's me in there. Ugly aren't I? Most screenshots have the subject looking offscreen. For better effect I looked at the camera during the screenshot.

    Will post a download link later.

    cheers
    Carl
     

    Attached Files:

  39. unseenthings

    unseenthings

    Joined:
    Nov 12, 2009
    Posts:
    32
    Sorry to resurrect a couple-months-old thread, but I'm doing research for a commercial AR project. These look like good solutions, but if they're all based on ARToolKit, that means that you have to use their commercial license and so you're looking at $5,000+ just to start working with it for anything commercial, is that correct? Are people just using this for non-commercial testing and such right now?

    Is there a non-ARToolKit option??

    thanks.
     
  40. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    A commercial license allows you to protect any contribution you might like to make to the ARToolkit, eg. keeping your contribution private and confidential.

    If you are making your contribution open-source then you don't really need to purchase a commercial license. But you will need to make your source code open-source.

    There are other benefits to a commercial license - access to more documentation I believe, and more recent updates.

    Note that you can use ARToolkit (and your own extensions) in commercial projects, irregardless of whichever license you adopt. The commercial license is simply there as an option for those who need to protect their contribution.

    Disclaimer: this is a big generalisation - you'll need to read the ARToolkit licence for more particular information.


    Carl
     
  41. unseenthings

    unseenthings

    Joined:
    Nov 12, 2009
    Posts:
    32
    But even if I just use the ARtoolkit "as is" and don't make any changes (or "contributions") to the code, I still have to open-source everything that I do with it (e.g. the Unity Code) since it's GPL, if I understand it all correctly. I was just wondering if there was some other option besides that, since the budget isn't just huge and paying upwards of $5k just to start using the toolkit really eats into the profit of the project.
     
  42. unseenthings

    unseenthings

    Joined:
    Nov 12, 2009
    Posts:
    32
    Okay, I've been talking to the ARToolKit people and am getting the licensing stuff worked out -- here's the next biggie -- can any of this be used from the browser, or is this all standalone only?

    thanks,
    Greg
     
  43. bruceb

    bruceb

    Joined:
    Nov 4, 2009
    Posts:
    9
    This sure feels like a right/left handed coordinate system issue. As you know, the sense of what is a positive rotation changes between right and left, so it is more than just switching axes.
     
  44. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    Consider making your software open-source. It need not impinge on your capacity to sell your work. Most users of your software won't have the ability, patience or even remote interest in compiling your source-code. And those that do are probably those you want to talk to anyway (collaborate etc).

    An alternative is this. And it can become necessary where start-up funding is short and stakeholders are demanding. There is a loophole one can exploit. If one really needs to do so.

    There is nothing (that I know or could argue) stopping open-source software communicating with closed-source software over a network.

    For example, I can write a server under an open-source license, and release that to the public, and run it on a box somewhere.

    If someone else (including myself by the way) writes a client, that communicates with that open-source server, do I have any chance of successfully suing for access of the source-code of the client?

    Or vice-versa, and this is the conceptual clincher - if I write an open-source client (that connects to a closed-source server), do I have any chance of successfully suing for access to the source-code of the said server?

    Note that there is no real difference in which side you define as the server and which you define as the client. They are simply the opposite endpoints of a communication channel/protocol.

    There are basically three forms of communication, the endpoints of which GPL may or may not be able to integrate:

    1. Static linking
    2. Dynamic linking (DLLs)
    3. Network (socket)

    The first is generally understood to be integrated by GPL (ie. the whole becomes GPL). The second is debatable (see links below) and the third does not appear to be addressed anywhere I could find (but I'd be interested in any where it is)

    http://en.wikipedia.org/wiki/GNU_General_Public_License#Linking_and_derived_works

    http://www.gnu.org/licenses/gpl-faq.html#GPLIncompatibleLibs

    Of course, all of this only matters if you are suing or being sued over otherwise closed-source code. Making your source code open-source in the first place alleviates the problem, but see:

    http://en.wikipedia.org/wiki/GNU_General_Public_License#Criticism

    for situations where this may not completely alleviate concerns - eg. where a contributor clandestinely contributes closed-source to an open-source project, the managers of the open-source project become liable.

    Also, keep in mind that it is only the software (and it's redistribution) which is integrated by GPL. The output of GPL software is not itself GPL (unless, of course, the output includes any or all of the GPL source-code itself!).

    *Disclaimer. As previously mentioned this discussion is a generalisation and the best advice is to research the topic fully and throw it past your lawyer.

    Also I'm an advocate of open-source, so I don't in any way suggest you should actually try exploiting any of the loopholes I've mentioned. I'm making this information available as a matter of principle - that all information (for better or worse) be free.

    What you do with it is up to you.

    Carl
     
  45. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    ARToolkit uses matrices while Unity3D uses quaternions. The conversion of a matrix to a quaternion is required. One can work it out from first principles or otherwise google the appropriate code.

    You'll also need to ensure you include the field of view in your conversion. And that you properly account for any changes of coordinate systems - eg. transposing the rotation component may be required (flipping rows and columns in the 3x3 sub-matrix).

    Whatever you do, don't use Euler angles as an intermediary. You'll be trying every possible arrangement under the sun before you realise that heading and pitch need to be transposed and that you've gimbal-locked half your results.

    Carl
     
  46. carllooper

    carllooper

    Joined:
    Aug 8, 2009
    Posts:
    64
    The Unity web browser plugin can't run any unity solutions that involve a DLL (in the unity build). Such use is excluded on security grounds as much as the technical difficultys it would otherwise involve.

    There is nothing stopping one from developing and distributing another plugin (which uses DLLs or their equivalent) which then communicates with the unity plugin. Of course there are the considerable technical difficultys involved (satisfying multiple operating systems and browsers).

    Another solution (which I've adopted in the past) is to run an executable outside the browser altogether - which then communicates with unity (running inside the browser) but the executable is PC only and if it was ever distributed to end-users would require them (of course) to download and install such to their operating system, making it somewhat pointless running the output in a web browser!

    But it's fun anyway. I always learn something new.

    Carl
     
  47. andresp

    andresp

    Joined:
    Aug 12, 2011
    Posts:
    38
    Are you sure about this? Can you give us a legal clarification from ARToolkit supporting your point?

    Because on their first page they are very bold:

    @unseenthings
    Can your share more information about your licensing process with ARToolkit?