[Code sample] Off Center Projection Code for VR, CAVE, or just for fun

Hi,
Not sure where to stick this but hopefully here will do.
I'm demonstrating my working implementation of Off Center Projection code I wrote for my masters research project.
I've look around the forums and it seems a few people have been struggling with the various methods of Off Center Projection (OCP) for VR or other applications (I know I was when I started out) so I figured I would share my working code.

This post assumes you are already familiar with the concepts of off center projection and projection matrixes in general. I won't be going into detail about the maths behind OCP, however if you want to really understand what's going on and how this particular implementation of OCP works then a decent understand of Matrix and Vector maths is a must, the paper linked below gives a decent guide on how this method works.

I implemented a version of Robert Kooima's Generalized Perspective Projection (2008) in Unity in C#. Before going further you should read his short paper on it: csc.lsu.edu/~kooima/pdfs/gen-perspective.pdf
In short this method allows the developer to create correct skewed projection of a 3D scene for any given head position and given screen position or orientation. It is particularly useful for multi-screen display environments, such as the virtual reality installations know as a CAVE (or CAVE alikes). But it's also a very good method to use for one screen display setups, such as shown in Johnny Lee's video on head tracking with a WiiRemote (http://youtu.be/Jd3-eiid-Uw).

I adapted Robert Kooima's C code into C#. All credit must go to him for this code, it's pretty much a straight copy.
Here it is:

Code (csharp):

1.  
2. public static Matrix4x4 GeneralizedPerspectiveProjection(Vector3 pa, Vector3 pb, Vector3 pc, Vector3 pe, float near, float far)
3.     {
4.         Vector3 va, vb, vc;
5.         Vector3 vr, vu, vn;
6.        
7.         float left, right, bottom, top, eyedistance;
8.        
9.         Matrix4x4 transformMatrix;
10.         Matrix4x4 projectionM;
11.         Matrix4x4 eyeTranslateM;
12.         Matrix4x4 finalProjection;
13.        
14.         ///Calculate the orthonormal for the screen (the screen coordinate system
15.         vr = pb - pa;
16.         vr.Normalize();
17.         vu = pc - pa;
18.         vu.Normalize();
19.         vn = Vector3.Cross(vr, vu);
20.         vn.Normalize();
21.        
22.         //Calculate the vector from eye (pe) to screen corners (pa, pb, pc)
23.         va = pa-pe;
24.         vb = pb-pe;
25.         vc = pc-pe;
26.        
27.         //Get the distance;; from the eye to the screen plane
28.         eyedistance = -(Vector3.Dot(va, vn));
29.        
30.         //Get the varaibles for the off center projection
31.         left = (Vector3.Dot(vr, va)*near)/eyedistance;
32.         right  = (Vector3.Dot(vr, vb)*near)/eyedistance;
33.         bottom  = (Vector3.Dot(vu, va)*near)/eyedistance;
34.         top = (Vector3.Dot(vu, vc)*near)/eyedistance;
35.        
36.         //Get this projection
37.         projectionM = PerspectiveOffCenter(left, right, bottom, top, near, far);
38.        
39.         //Fill in the transform matrix
40.         transformMatrix = new Matrix4x4();
41.         transformMatrix[0, 0] = vr.x;
42.         transformMatrix[0, 1] = vr.y;
43.         transformMatrix[0, 2] = vr.z;
44.         transformMatrix[0, 3] = 0;
45.         transformMatrix[1, 0] = vu.x;
46.         transformMatrix[1, 1] = vu.y;
47.         transformMatrix[1, 2] = vu.z;
48.         transformMatrix[1, 3] = 0;
49.         transformMatrix[2, 0] = vn.x;
50.         transformMatrix[2, 1] = vn.y;
51.         transformMatrix[2, 2] = vn.z;
52.         transformMatrix[2, 3] = 0;
53.         transformMatrix[3, 0] = 0;
54.         transformMatrix[3, 1] = 0;
55.         transformMatrix[3, 2] = 0;
56.         transformMatrix[3, 3] = 1;
57.        
58.         //Now for the eye transform
59.         eyeTranslateM = new Matrix4x4();
60.         eyeTranslateM[0, 0] = 1;
61.         eyeTranslateM[0, 1] = 0;
62.         eyeTranslateM[0, 2] = 0;
63.         eyeTranslateM[0, 3] = -pe.x;
64.         eyeTranslateM[1, 0] = 0;
65.         eyeTranslateM[1, 1] = 1;
66.         eyeTranslateM[1, 2] = 0;
67.         eyeTranslateM[1, 3] = -pe.y;
68.         eyeTranslateM[2, 0] = 0;
69.         eyeTranslateM[2, 1] = 0;
70.         eyeTranslateM[2, 2] = 1;
71.         eyeTranslateM[2, 3] = -pe.z;
72.         eyeTranslateM[3, 0] = 0;
73.         eyeTranslateM[3, 1] = 0;
74.         eyeTranslateM[3, 2] = 0;
75.         eyeTranslateM[3, 3] = 1f;
76.        
77.         //Multiply all together
78.         finalProjection = new Matrix4x4();
79.         finalProjection = Matrix4x4.identity * projectionM*transformMatrix*eyeTranslateM;
80.        
81.         //finally return
82.         return finalProjection;
83.     }
84.  

This code may look a little complicated but ultimately the end result is very simple.
The function inputs are as follows:
Vector3 pa is the vector position of the Bottom Left Corner of the screen
Vector3 pb is the vector position of the Bottom Right Corner of the screen
Vector3 pc is the vector position of the Top Left Corner of the screen
Vector3 pe is the vector position of the head tracker location (or eye position)
All these are vector positions in 'real' space, you have to model the screen as well as the players head position in the real world for this method to work, hence allowing you to create an image for any given screen position or orientation which is what makes this method a little different (and in my opinion more complete) than of OCP methods.
Float's near and far are the near and far clipping planes of the camera.
This method then compiles all this data into a projection matrix. This then simply overwrites the Unity camera's existing one.
Something like this:

Code (csharp):

1.  
2. // Update is called once per frame
3. public void FixedUpdate () {
4.     Camera cam = camera;
5.     //calculate projection
6.     Matrix4x4 genProjection = GeneralizedPerspectiveProjection(
7.                 BottomLeftCorner, BottomRightCorner,
8.                 TopLeftCorner, trackerPosition,
9.                 cam.nearClipPlane, cam.farClipPlane);
10.     cam.projectionMatrix = genProjection;  
11. }
12.  

In this case the script has been attached to a Unity Camera object (hence it can reference the 'camera'). You then just need to feed the function the screen positions coordination's and the head tracker position (from a wiimote, kinect or other tracking device, all handled in other scripts) and theory says that you'll get a lovely OCP display.
These are just code snippets from a large script, however since the full script has a lot of fluffy and project specific code (including setting up split screen and stereoscopic images) I won't include it here, since it will probably just add confusion. I hope this is enough for y'all to get started.
If you have any questions or queries on the implementation or want some help feel free to post them up, I'll try and reply to them in a timely manner.

Bit about my project:
So I did a research masters in trying to get video game engines to work on a CAVE-like virtual reality setup. Unity was the final engine I selected to implement on the CAVE (after considering UDK, Panda3D, Irrlicht and XNA). I finished it back in December 2011. I've got a video demonstration of some of the features of the final demo I made for the CAVE. You can watch it here: http://youtu.be/Pud_RyQonI0?hd=1
Unfortunately I don't have any footage of it running the CAVE, but I have been assured by my testing volunteers (read housemates and friends) that it was 'very cool'.
Thanks for reading, hope you find it useful!

 

Skiddings,

I just want to say thanks a bunch for sharing!! Your video was awesome, I would love to see it in action.

So, I'm in the middle of implementing this OCP... I've read through all the docs you linked and have a good grasp on it, but I'm having just a few issues.. It's SO close to working for me! Maybe you can steer a poor man with a headache in the right direction

Firstly, what do you suggest for Unity's rig location/setup??


This is one of many ways I tried but I keep getting lots of reversed directions, camera being upside down, X movements being inverted, Camera rotating around eyeMarker etc etc.

I haven't altered the code, although I've been using the PerspectiveOffCenter() function your code calls from Unity's Script Reference.

Beyond the setup within Unity, I've been playing with getting an FPS char to move... but it seems the second I move any of the OCP parts it all goes to hell, and moving them all at once does absolutely nothing?

 

Hello,

thanks for sharing as well. This really looks very interesting.

The script mentioned above calls a method at line 36: PerspectiveOffCenter(left, right, bottom, top, near, far);
Unfortunately this method seems to be missing. Can anyone maybe help me solve this issue?

Thanks in advance.

 

Nevermind, I guess I've found an answer to my question: http://docs.unity3d.com/Documentation/ScriptReference/Camera-projectionMatrix.html

 

Ok, I guess I'm on the verge of getting something working up and running. I'll post a demo project if I succeed

 

Hello. Anybody has any success with this? Something is definately wrong with math - I just can't understand what exactly.

 

Hey,
I am having minor successes. But I am still not content with my results. They are way too static.

I think the math is correct. What confuses me is setting up things in Unity.
Especially the notional display.

Are you using the Kinect Skeleton wrapper as well?
Is anybody up for working together?

 

Hi,
Sorry to dig up this topic, but it matches exactly with my issue.
I have difficulty making this script works smoothly, and I can't have a flexible camera setup working in all camera, screen and tracker positions/directions.
Has somebody here successfully made it work ?

My colleague created a topic some time ago with an example project showing some light issues relative to this offcenter center projection method (https://forum.unity3d.com/threads/c...inetuning-my-script-and-light-problem.425328/), maybe we got this issue because there is something wrong in our setup.
If somebody has an idea and could take a look on this sample project and make it work (and thus providing the community a fully working example ), it would be really appreciated !

Thank you in advance