Reflection Rendering Wrong in OpenVR + hTC VIVE

5.4 beta 15 forward linear rendering. Single pass stereo rendering is disabled.

the material of those glass is standard(specular) shader set to transparent mode,
it renders reflection from a baked reflection probe.

 

I've got a similar issue, that is likely caused by the same underlying problem. After upgrading from 5.3.4 to 5.4b15 yesterday, I noticed that the planar mirrors in our VR game (based on the code found here) are terribly broken. The mirror code assumes that it can simply place a camera behind the mirror, render the scene into a texture as it would be seen from that perspective, and then sample the resulting texture in a shader applied to the surface of the mirror. In 5.3.4 this worked because each eye was effectively its own camera, and thus had its own reflection camera. Now in 5.4, the two eyes are handled by a single camera, which means there's a single reflection camera rendering to a single texture that's then used by both the left and right eye. This clearly produces the wrong effect.

What I believe I need is:
1) A way to obtain the local position/rotation of each eye relative to the camera object so that I can setup two cameras and produce two reflection textures.
2) A way in my pixel shader to identify which eye is being drawn and choose the correct texture.

Can someone confirm that my understanding of the problem and solution is accurate? And assuming I'm on the right path, is there a way to do this with the APIs currently available?

 

Yeah I've already reported a bug for this issue, the issue id of this bug is:
790752

 

Hi Dan-

I have been able to get MirrorReflection to work by doing the following:
1. Create 1 main camera for normal rendering of everything but Water (the layer you put the mirror on). Set this camera to "Both Eyes" and Layer 0. That way your main rendering gets all the advantages both eyes being rendered at same time.
2. Create 2 additional cameras. These will be the reflection cameras. Set them to only render "Water". Set 1 to Left and the other to Right. Important part: Set near clipping on both of these reflections cameras really low, to 0.01 or lower or you will have objects near the mirror from layer 0 camera that will seem to be behind mirror.
3. Don't use "Single-pass stereo rendering". It doesn't agree with this layered camera arrangement.

You could also just go with 2 total cameras, 1 for left and 1 for right, but you will lose some of the performance advantage of the both eye camera for the bulk of your scene rendering.

ccs

 

Thanks for the reply ccs. I should have replied to this thread a while ago with an update on how I've resolved the issue. Your solution sounds reasonable enough, but it does require modifying the main camera / render passes which isn't ideal. My solution modifies the MirrorReflection code and enables it to work without requiring any changes to the main camera (ie: a single main camera rendering all layers to both eyes). It also has the added benefit that it works just fine for single-pass stereo rendering. If we could get support for creating our own stereo cameras that render to split textures, then this could be improved even further.

Basically, I modified the MirrorReflection.shader so that it has two reflection textures, one for each eye, and it samples the appropriate texture based on which eye is being rendered (you can determine this by examining unity_StereoEyeIndex when using single pass stereo rendering, or by checking the sign of unity_CameraProjection[0][2] when using traditional multi-pass stereo rendering).

The MirrorReflection.cs is then responsible for generating the appropriate texture(s) for the camera that is being mirrored. Normal monocular cameras only require a single reflection render pass. Stereo cameras require two reflection render passes (this is where having engine support for creating our own stereo cameras that render to a single side-by-side split render texture would be super useful). Unfortunately, Unity does not currently expose the per-eye details that we need from a stereo camera (ie: position/rotation of each individual eye and the projection matrix used by each eye). I had to rely on some SteamVR APIs to access this information, and haven't yet looked into how (or if) one could do this for other VR platforms.

I've attached our modified MirrorReflection scripts in case anyone is interested.

 

Cool solution Dan. Thanks for sharing it. That should improve performance of this a bit for me as it only requires a single camera setup. I also like the reduced complexity. I was also having trouble with image effects that use depth textures with the layered approach (1 both + 2 left/right), so this might solve that as well.

 

Hi Dan- You solution works great, but I found a bug that causes odd clipping in the mirror depending on how the mirror is placed. I suggest this fix:

Yours:
Vector3 normal = -transform.up;

Fixed:
Vector3 normal = transform.up;

 

Hi,
I'm using the AdvancedMirrorReflection shader/script from the asset store and also have this problem - unfortunately the author appears to have dissapeared. I'm keen to retain the functionality of the shader (rather than use just the base shader as fixed by Dan) but don't know how to fix the lack of stereo in reflections (I cannot script sadly). Would anyone be willing/able to show me? The shader and script is attached. Many thanks in advance.
Felix

 

I am having problems with this on all the water prefabs. Is this just a generic reflection problem? I downloaded what dan kroyman's scripts, but couldn't figure out how to get them to work (i'm a shader newb). I attempted to do what ccsanders did, but also couldn't follow exactly how to do that. I'm totally stuck, since I need water working for the Vive.

I went to the unity3d debug area, the main one referenced is closed duplicate, which didn't make any sense to me after I read the referenced debug that it was suppose to be closed to. I'd hate to bug someone, but is there a prefab that actually works with reflections and vive, that way I can figure out the exact steps on how to do it?

 

If you have clipping problems, try removing these two lines.

Code (CSharp):

1. Vector4 cameraSpaceClipPlane = CameraSpacePlane(reflectionCamera, pos + offset, normal);
2. reflectionCamera.projectionMatrix = reflectionCamera.CalculateObliqueMatrix(cameraSpaceClipPlane);

 

First, thank you for posting this!

Second, this worked GREAT for me in the 5.4b11, but now Oculus fails to run through Open VR, and it looks like the Steam VR camera fails to work with the Oculus VR SDK.

Anyone have any idea on achieving the same results in a VR agnostic fashion? (ie offsetting the two views for two renders in the reflection code without depending on the Steam VR camera, and then differentiating the eyes in shader in a way that works on the Oculus SDK?).

Thanks!

 

The VRpanorama asset that runs in batch, does it correctly outside the whole "vr" process. This is for it's 360 stereoscopic mode. They kind of explain how they are doing it in their support thread : http://forum.unity3d.com/threads/released-vr-panorama-render-360-stereo-videos.336086/

This general bug drives me crazy btw, there was a library change in b24 that I hope people will start using so this won't be a complex problem. I also btw dislike not having the parallax support in the standard shaders, so throwing away anything with a bump/normal map drives me crazy as I port things to vr.

 

Alright - turns out that if you have a SteamVR_Camera as a child of another SteamVR_Camera (which worked great before for basic camera layering), SteamVR stack overflows and locks up Unity. This issue removed, from what I can SteamVR / OpenVR handles support for the Oculus, and I can at least run my game.

Three problems remained since the upgrade to 5.4.0f1:

a) I'm using a derivative of the water reflection above - it worked GREAT before on 5.4.b11.... but now the reflections are offset and warped. I can confirm that two are still being generated, and that the eye differentiation in the shader is functioning correctly. I'm not sure what changed. Has anyone been able to make anything like this Mirror Reflection work in Unity 5.4.0f1?

b) My camera was not recentering / was floating awkwardly in the air. Turns out Open VR does not properly respect InputTracking.Recenter();

What does work though:
Valve.VR.OpenVR.System.ResetSeatedZeroPose();
Valve.VR.OpenVR.Compositor.SetTrackingSpace(Valve.VR.ETrackingUniverseOrigin.TrackingUniverseSeated);

c) My layered camera doesn't work at all any more. It used to work fine (seperate stereo camera, child of the head, let me render my other layers ontop in exactly they way I want). Now if I have the SteamVR_Camera component, we crash. If I remove the component, its view is horribly warped and offset. Has anyone had any success layering two cameras in SteamVR?

 

Yes. The mirror solution that I provided earlier in this thread is still working great for us as of 5.4.0f2 (NOTE: I haven't tried this on an Oculus in quite a while so there's a possibility that it doesn't work anymore). I've updated it a bit to keep pace with improvements Unity has implemented in their shader libraries, but none of those changes were necessary. Here's a simple unitypackage containing a stripped down version of our latest mirror script+shader (I had to remove a bunch of dependencies on other features of our game). This should be usable as a simple starting point for anyone that wants to create their own custom mirror functionality.

 

There seems to still be issues with VR water reflection in 5.5. (Open bug report about it here.)

I tried the Mirror unitypackage above and the reflection do work correct for me, except that occlusion culling is not working correctly. So in the reflections, objects flicker in and out of existence if occlusion culling is enabled.

Did anyone ever manage to fix up the water assets from Standard Assets (Water script / FXWaterPro shader) to work with single pass VR rendering? And with occlusion culling support too?

(I've asked about this internally in Unity too, but there's no resources for this right now, so I wanted to hear if anyone from the community have managed to fix this. I've searched around a lot and it's hard to get clarity on this issue...)

 

Mike Tucker says here he was able to fix it
https://github.com/ValveSoftware/openvr/issues/251

UPDATE:
I spoke with Mike and he said he was able to fix the reflections, but never got to the refractions bit. It's been broken for quite awhile... surely someone at Unity can tackle this, surely?

 

dan-kroymann's solution was working great (thanks!), but the reflections are now very incorrect for me on 5.6.1f1 (was fine on 5.6.0f3). Anyone else seeing that? Possibly a 5.6.1f1 bug?

 

Bug reported. In 5.6.1f1, cameras with a rendertarget are ignoring their viewport rect.

 

You just found this occlusion bug? This is been around since the reflective water exists and has as far as I know never been fixed. I thought someone reported it about 4 years ago.

 

It HAS been fixed for standard assets water. But standard assets water doesn't work correctly with single pass stereoscopic rendering. And certain third party water shaders still exhibit the occlusion culling issue due to not setting the right matrices or something like that.

Currently I don't know if it's possible to write a water shader that works for both occlusion culling and single pass stereoscopic rendering, or that's mutually exclusive the way Unity is currently built. I have seen shaders that work for one or the other, but not both.

 

I did some digging on this issue over the past few days and wanted to share what I've found. dan-kroymann's example works as expected for reflections prior to 5.6.1f1 but as DylanF notes, in 5.6.1f1 cameras with a render target ignore the viewport rect. I needed to work around this for a 5.6.1 project so I was able to modify the standard water script/shader combined with the Mirror example to use two RenderTextures so that I didn't need to change the viewport rect. It's pretty hacky but I'll attach it in case anyone is interested.

I haven't looked into refraction yet but I'm hesitant to work on this further since the viewport issue is clearly a Unity bug and using two textures for this is not ideal. I also still need to attempt occlusion culling, it may be as simple as just setting the culling matrix correctly.

 

Does this fix work for the the "Water4" (advanced) asset? Just to clarify, both the script and shader are simply applied directly to the water prefab?

 

Hi, I tried your mirror in unity 5.6.2 with HTC Vive, it worked well with a little fix. Thank you.
I have a problem, I need two mirrors reflect each other, it's important in my game. I realized it in 5.3 with help of others, you can find how it realized in the file. But I don't know how to do it based on your mirror. Could you help me with this? I will pay for this. Thank you.

 

In theory it should be fairly simple to do this. Right now, the MirrorReflection.cs code intentionally prevents recursive rendering of mirrors for performance reasons. You could fairly easily modify this to allow recursion down to a specific depth. Just change the s_InsideRendering bool to be an integer that counts the recursion depth and prevents recursion past whatever depth you care about.

 

I'm interested into getting *any* kind of mirror effect to work on the Vive with Unity 5.6.2.

The script you attached doesn't include @dan-kroymann's modifications to support two eye textures, so it won't work with the shader he provided (which expects a _LeftReflectionTex and a _RightReflectionTex). Could it be you mixed up the script or did you modify the shader, too (possible to single-pass stereo?)

So far, the only mirror I could get to work even partially was from the Vive stereo rendering toolkit (after modifying a bunch of calls to Valve-specific APIs that were apparently present in Unity 5.4), but the reflection either had the eyes switched or not separated.

 

I tried what you said, but one of the two mirrors did not rend recursively.

 

Yes, the script I attached is that I use in 5.3 with only one eye mode. In that script, the two mirror can reflect each other recursively, so I want to follow dan-droymann's MirrorReflection to modify it. But after several days trying, I still didn't get correct result. I did used dan-droymann's shader also.

 

I just tried water for the first time in VR. I am glad to see that you folks have been working on it.

Just to be clear, can I use one of the fixed mirror reflection shaders in this thread with the standard WaterProNighttime asset? The thread seems to have moved on to mirrors, not just water. What is the current fix using 5.6.2 or later, and Steam VR? Using the Mirror.unitypackage near the top of this thread?

 

Answering my own question - the solution by kaelb above, the water.cs and FXWaterPro.shader, worked well for me for reflecting objects in the scene. There's still an odd doubling of skybox reflections but that is not an issue in my case.

Thanks again!

 

Building on kaelb's work, I've updated the script and shader to work with refraction using his methodology. Not a fast solution, but it works for my small project. Thanks kaelb!

Note: This is for WaterProDaytime/WaterProNighttime. I'm using Unity 2017.2.0f3.

 

How to Use reflection in single pass Render Mode With VR Game

 

How to make the two mirror can reflect each other recursively,

 

Reflection rendering is different for each eye and noticeably different, did you ever notice that?