1. Help us improve the editor usability and artist workflows. Join our discussion to provide your feedback.
    Dismiss Notice
  2. We're looking for feedback on Unity Starter Kits! Let us know what you’d like.
    Dismiss Notice
  3. We’re giving 2017.1 beta testers a chance to win t-shirts and a Nintendo Switch. Read more on the blog.
    Dismiss Notice
  4. We want to know how you learned Unity! Help us by taking this quick survey and have a chance at a $25 gift card
    Dismiss Notice
  5. Are you an artist or level designer going to Unite Europe? Join our roundtables there to discuss artist features.
    Dismiss Notice
  6. Unity 5.6 is now released.
    Dismiss Notice
  7. Check out all the fixes for 5.6 on the patch releases page.
    Dismiss Notice

Reflection Rendering Wrong in OpenVR + hTC VIVE

Discussion in 'Virtual Reality' started by betalulu, Apr 20, 2016.

  1. betalulu

    betalulu

    Joined:
    Apr 9, 2014
    Posts:
    9
    unity53correct.png
    unity54b15bug.png

    5.4 beta 15 forward linear rendering. Single pass stereo rendering is disabled.

    the material of those glass is standard(specular) shader set to transparent mode,
    it renders reflection from a baked reflection probe.
     
    Last edited: Apr 20, 2016
  2. dan-kroymann

    dan-kroymann

    Joined:
    Feb 5, 2013
    Posts:
    16
    I've got a similar issue, that is likely caused by the same underlying problem. After upgrading from 5.3.4 to 5.4b15 yesterday, I noticed that the planar mirrors in our VR game (based on the code found here) are terribly broken. The mirror code assumes that it can simply place a camera behind the mirror, render the scene into a texture as it would be seen from that perspective, and then sample the resulting texture in a shader applied to the surface of the mirror. In 5.3.4 this worked because each eye was effectively its own camera, and thus had its own reflection camera. Now in 5.4, the two eyes are handled by a single camera, which means there's a single reflection camera rendering to a single texture that's then used by both the left and right eye. This clearly produces the wrong effect.

    What I believe I need is:
    1) A way to obtain the local position/rotation of each eye relative to the camera object so that I can setup two cameras and produce two reflection textures.
    2) A way in my pixel shader to identify which eye is being drawn and choose the correct texture.

    Can someone confirm that my understanding of the problem and solution is accurate? And assuming I'm on the right path, is there a way to do this with the APIs currently available?
     
  3. CarlosRincon

    CarlosRincon

    Unity Technologies

    Joined:
    Jan 28, 2016
    Posts:
    73
    betalulu: thanks for reporting this - it has been raised with the team.

    dan.kryomann: looking to get someone from the VR team to advise.
     
  4. CarlosRincon

    CarlosRincon

    Unity Technologies

    Joined:
    Jan 28, 2016
    Posts:
    73
    betalulu: actually can you raise a bug yourself and post the number here so that we can follow up internally? The team is aware of the issue as mentioned above, but it's better if you raise the bug so that you are kept in the loop via the bug ticket.

    Bug reporting recommendations: https://unity3d.com/unity/qa/bug-reporting
     
  5. betalulu

    betalulu

    Joined:
    Apr 9, 2014
    Posts:
    9
    Yeah I've already reported a bug for this issue, the issue id of this bug is:
    790752

     
  6. CarlosRincon

    CarlosRincon

    Unity Technologies

    Joined:
    Jan 28, 2016
    Posts:
    73
    Great, many thanks.
     
  7. ccsander

    ccsander

    Joined:
    May 17, 2013
    Posts:
    34
    Hi Dan-

    I have been able to get MirrorReflection to work by doing the following:
    1. Create 1 main camera for normal rendering of everything but Water (the layer you put the mirror on). Set this camera to "Both Eyes" and Layer 0. That way your main rendering gets all the advantages both eyes being rendered at same time.
    2. Create 2 additional cameras. These will be the reflection cameras. Set them to only render "Water". Set 1 to Left and the other to Right. Important part: Set near clipping on both of these reflections cameras really low, to 0.01 or lower or you will have objects near the mirror from layer 0 camera that will seem to be behind mirror.
    3. Don't use "Single-pass stereo rendering". It doesn't agree with this layered camera arrangement.

    You could also just go with 2 total cameras, 1 for left and 1 for right, but you will lose some of the performance advantage of the both eye camera for the bulk of your scene rendering.

    ccs
     
  8. dan-kroymann

    dan-kroymann

    Joined:
    Feb 5, 2013
    Posts:
    16
    Thanks for the reply ccs. I should have replied to this thread a while ago with an update on how I've resolved the issue. Your solution sounds reasonable enough, but it does require modifying the main camera / render passes which isn't ideal. My solution modifies the MirrorReflection code and enables it to work without requiring any changes to the main camera (ie: a single main camera rendering all layers to both eyes). It also has the added benefit that it works just fine for single-pass stereo rendering. If we could get support for creating our own stereo cameras that render to split textures, then this could be improved even further.

    Basically, I modified the MirrorReflection.shader so that it has two reflection textures, one for each eye, and it samples the appropriate texture based on which eye is being rendered (you can determine this by examining unity_StereoEyeIndex when using single pass stereo rendering, or by checking the sign of unity_CameraProjection[0][2] when using traditional multi-pass stereo rendering).

    The MirrorReflection.cs is then responsible for generating the appropriate texture(s) for the camera that is being mirrored. Normal monocular cameras only require a single reflection render pass. Stereo cameras require two reflection render passes (this is where having engine support for creating our own stereo cameras that render to a single side-by-side split render texture would be super useful). Unfortunately, Unity does not currently expose the per-eye details that we need from a stereo camera (ie: position/rotation of each individual eye and the projection matrix used by each eye). I had to rely on some SteamVR APIs to access this information, and haven't yet looked into how (or if) one could do this for other VR platforms.

    I've attached our modified MirrorReflection scripts in case anyone is interested.
     

    Attached Files:

    Last edited: May 5, 2016
    jhuynh_isobar and ccsander like this.
  9. ccsander

    ccsander

    Joined:
    May 17, 2013
    Posts:
    34
    Cool solution Dan. Thanks for sharing it. That should improve performance of this a bit for me as it only requires a single camera setup. I also like the reduced complexity. I was also having trouble with image effects that use depth textures with the layered approach (1 both + 2 left/right), so this might solve that as well.
     
  10. ccsander

    ccsander

    Joined:
    May 17, 2013
    Posts:
    34
    Hi Dan- You solution works great, but I found a bug that causes odd clipping in the mirror depending on how the mirror is placed. I suggest this fix:

    Yours:
    Vector3 normal = -transform.up;

    Fixed:
    Vector3 normal = transform.up;
     
    jhuynh_isobar likes this.
  11. deflix73

    deflix73

    Joined:
    Aug 1, 2015
    Posts:
    4
    Hi,
    I'm using the AdvancedMirrorReflection shader/script from the asset store and also have this problem - unfortunately the author appears to have dissapeared. I'm keen to retain the functionality of the shader (rather than use just the base shader as fixed by Dan) but don't know how to fix the lack of stereo in reflections (I cannot script sadly). Would anyone be willing/able to show me? The shader and script is attached. Many thanks in advance.
    Felix
     

    Attached Files:

  12. PatrickFaith

    PatrickFaith

    Joined:
    Jun 28, 2016
    Posts:
    7
    I am having problems with this on all the water prefabs. Is this just a generic reflection problem? I downloaded what dan kroyman's scripts, but couldn't figure out how to get them to work (i'm a shader newb). I attempted to do what ccsanders did, but also couldn't follow exactly how to do that. I'm totally stuck, since I need water working for the Vive.

    I went to the unity3d debug area, the main one referenced is closed duplicate, which didn't make any sense to me after I read the referenced debug that it was suppose to be closed to. I'd hate to bug someone, but is there a prefab that actually works with reflections and vive, that way I can figure out the exact steps on how to do it?
     
  13. svarlingo

    svarlingo

    Joined:
    Oct 19, 2015
    Posts:
    2
    If you have clipping problems, try removing these two lines.
    Code (CSharp):
    1. Vector4 cameraSpaceClipPlane = CameraSpacePlane(reflectionCamera, pos + offset, normal);
    2. reflectionCamera.projectionMatrix = reflectionCamera.CalculateObliqueMatrix(cameraSpaceClipPlane);
     
  14. SeriousACZ

    SeriousACZ

    Joined:
    Jul 23, 2013
    Posts:
    19
    First, thank you for posting this!

    Second, this worked GREAT for me in the 5.4b11, but now Oculus fails to run through Open VR, and it looks like the Steam VR camera fails to work with the Oculus VR SDK.

    Anyone have any idea on achieving the same results in a VR agnostic fashion? (ie offsetting the two views for two renders in the reflection code without depending on the Steam VR camera, and then differentiating the eyes in shader in a way that works on the Oculus SDK?).

    Thanks!
     
  15. PatrickFaith

    PatrickFaith

    Joined:
    Jun 28, 2016
    Posts:
    7
    The VRpanorama asset that runs in batch, does it correctly outside the whole "vr" process. This is for it's 360 stereoscopic mode. They kind of explain how they are doing it in their support thread : http://forum.unity3d.com/threads/released-vr-panorama-render-360-stereo-videos.336086/

    This general bug drives me crazy btw, there was a library change in b24 that I hope people will start using so this won't be a complex problem. I also btw dislike not having the parallax support in the standard shaders, so throwing away anything with a bump/normal map drives me crazy as I port things to vr.
     
  16. SeriousACZ

    SeriousACZ

    Joined:
    Jul 23, 2013
    Posts:
    19
    Alright - turns out that if you have a SteamVR_Camera as a child of another SteamVR_Camera (which worked great before for basic camera layering), SteamVR stack overflows and locks up Unity. This issue removed, from what I can SteamVR / OpenVR handles support for the Oculus, and I can at least run my game.

    Three problems remained since the upgrade to 5.4.0f1:

    a) I'm using a derivative of the water reflection above - it worked GREAT before on 5.4.b11.... but now the reflections are offset and warped. I can confirm that two are still being generated, and that the eye differentiation in the shader is functioning correctly. I'm not sure what changed. Has anyone been able to make anything like this Mirror Reflection work in Unity 5.4.0f1?

    b) My camera was not recentering / was floating awkwardly in the air. Turns out Open VR does not properly respect InputTracking.Recenter();

    What does work though:
    Valve.VR.OpenVR.System.ResetSeatedZeroPose();
    Valve.VR.OpenVR.Compositor.SetTrackingSpace(Valve.VR.ETrackingUniverseOrigin.TrackingUniverseSeated);

    c) My layered camera doesn't work at all any more. It used to work fine (seperate stereo camera, child of the head, let me render my other layers ontop in exactly they way I want). Now if I have the SteamVR_Camera component, we crash. If I remove the component, its view is horribly warped and offset. Has anyone had any success layering two cameras in SteamVR?
     
  17. dan-kroymann

    dan-kroymann

    Joined:
    Feb 5, 2013
    Posts:
    16
    Yes. The mirror solution that I provided earlier in this thread is still working great for us as of 5.4.0f2 (NOTE: I haven't tried this on an Oculus in quite a while so there's a possibility that it doesn't work anymore). I've updated it a bit to keep pace with improvements Unity has implemented in their shader libraries, but none of those changes were necessary. Here's a simple unitypackage containing a stripped down version of our latest mirror script+shader (I had to remove a bunch of dependencies on other features of our game). This should be usable as a simple starting point for anyone that wants to create their own custom mirror functionality.
     

    Attached Files:

    waffle_maker, Rog, deflix73 and 3 others like this.
  18. runevision

    runevision

    Unity Technologies

    Joined:
    Nov 28, 2007
    Posts:
    1,178
    There seems to still be issues with VR water reflection in 5.5. (Open bug report about it here.)

    I tried the Mirror unitypackage above and the reflection do work correct for me, except that occlusion culling is not working correctly. So in the reflections, objects flicker in and out of existence if occlusion culling is enabled.

    Did anyone ever manage to fix up the water assets from Standard Assets (Water script / FXWaterPro shader) to work with single pass VR rendering? And with occlusion culling support too?

    (I've asked about this internally in Unity too, but there's no resources for this right now, so I wanted to hear if anyone from the community have managed to fix this. I've searched around a lot and it's hard to get clarity on this issue...)
     
  19. SuppleTeets

    SuppleTeets

    Joined:
    Aug 22, 2014
    Posts:
    7
    Mike Tucker says here he was able to fix it :)
    https://github.com/ValveSoftware/openvr/issues/251

    UPDATE:
    I spoke with Mike and he said he was able to fix the reflections, but never got to the refractions bit. It's been broken for quite awhile... surely someone at Unity can tackle this, surely?
     
    Last edited: Apr 10, 2017
  20. DylanF

    DylanF

    Joined:
    Jun 25, 2013
    Posts:
    9
    dan-kroymann's solution was working great (thanks!), but the reflections are now very incorrect for me on 5.6.1f1 (was fine on 5.6.0f3). Anyone else seeing that? Possibly a 5.6.1f1 bug?
     
  21. DylanF

    DylanF

    Joined:
    Jun 25, 2013
    Posts:
    9
    Bug reported. In 5.6.1f1, cameras with a rendertarget are ignoring their viewport rect.
     
  22. JDMulti

    JDMulti

    Joined:
    Jan 4, 2012
    Posts:
    334
    You just found this occlusion bug? This is been around since the reflective water exists and has as far as I know never been fixed. I thought someone reported it about 4 years ago.
     
  23. runevision

    runevision

    Unity Technologies

    Joined:
    Nov 28, 2007
    Posts:
    1,178
    It HAS been fixed for standard assets water. But standard assets water doesn't work correctly with single pass stereoscopic rendering. And certain third party water shaders still exhibit the occlusion culling issue due to not setting the right matrices or something like that.

    Currently I don't know if it's possible to write a water shader that works for both occlusion culling and single pass stereoscopic rendering, or that's mutually exclusive the way Unity is currently built. I have seen shaders that work for one or the other, but not both.
     
  24. kaelb

    kaelb

    Joined:
    Dec 4, 2014
    Posts:
    1
    I did some digging on this issue over the past few days and wanted to share what I've found. dan-kroymann's example works as expected for reflections prior to 5.6.1f1 but as DylanF notes, in 5.6.1f1 cameras with a render target ignore the viewport rect. I needed to work around this for a 5.6.1 project so I was able to modify the standard water script/shader combined with the Mirror example to use two RenderTextures so that I didn't need to change the viewport rect. It's pretty hacky but I'll attach it in case anyone is interested.

    I haven't looked into refraction yet but I'm hesitant to work on this further since the viewport issue is clearly a Unity bug and using two textures for this is not ideal. I also still need to attempt occlusion culling, it may be as simple as just setting the culling matrix correctly.
     

    Attached Files:

    antoripa likes this.
  25. b4l8ter

    b4l8ter

    Joined:
    May 28, 2015
    Posts:
    1
    Does this fix work for the the "Water4" (advanced) asset? Just to clarify, both the script and shader are simply applied directly to the water prefab?