Only render parts of objects intersecting with a special invisible object

Greetings forum, thanks in advance for any assistance y'all can provide. And sorry for any idiocy on my part, this is my first post.

I'm having a heck of a hard time achieving a special effect. I want to transition from one scene to the next using a growing spherical portal. Sort of the 3d equivalent of a circle-out wipe transition such as you might see in a movie (here's an example from Star Wars, where a circular window into Endor opens from the center of the screen outwards, replacing the previous shot of space:



). But in this case, parts of the next scene inside a sphere, not a circle, should replace the previous scene.

My first impulse was to use the stencil buffer, in its most basic implementation. That is, the portal sphere has ColorMask 0 and ZWrite off, but always passes the stencil test and replaces the buffer with 1 (arbitrary number, chosen for simplicity); then all the objects in the next scene use materials which are culled unless they pass the stencil test equal to 1.

This basically works except for one problem: some objects in the next scene which are outside the portal sphere are visible anyway - namely, those objects which are between the sphere and the camera! Again, only objects from the next scene that are inside the sphere are supposed to be revealed. Here's a screenshot:



As you can probably tell, the closest corner of the house is outside the sphere, and therefore it shouldn't be visible yet. As the sphere grows outward from a point, the house should be revealed in a series of spherical slices. I want to see crazy curved cross-sections of the walls and chairs and stairs, etc. as the sphere grows. The way this is working now, it’s ultimately no different than the 2D circle-out effect from the movies: the sphere just cuts a circular hole in the screen through which the next scene is seen.

I believe the fundamental problem is with the way the stencil buffer functions not with respect to depth in 3d space but with respect to ultimate pixels on the 2d screen. In other words, the renderer considers the screen, and any pixel where the sphere exists, regardless of depth or occlusion, becomes "stencil 1". Then when it comes time to render the actual physical objects in the scene, they have no idea whether they are in front of, inside of, or behind the portal sphere, and if they're stencil 1 objects they just get rendered irrespective to that important factor.

My understanding is that in order to get the stencil to apply in space, I need to turn ZWrite back on in my shader. Otherwise the stencil information the sphere writes has no depth info associated with it. The problem with that is that when Zwrite is on, objects behind the sphere get occluded. Well, that won’t work, because I need to see the objects inside the sphere and behind it. Even when I set the ColorMask to 0 making it invisible, it still occludes things, essentially cutting a hole in the world (aka a DepthMask, like so: http://answers.unity3d.com/questions/404637/subtractive-rendering.html).

So really what I need to figure out how to do is attribute depth information to my stencil buffer entries without actually writing to the depth buffer, or go the standard route and write to the depth buffer but somehow override the default behavior of occluding more distant objects. I'm not sure which is better, but probably either way would require some scripting.

For example, if I could use a variable as the Ref of the stencil buffer I could convert the stencil buffer into a sort of custom depth buffer serving my needs (This guy CaptainScience seems to have found custom depth buffer functionality in Unreal, at least: https://forums.unrealengine.com/archive/index.php/t-739.html). Or I might be able to script the sorting of the objects behind the sphere, using a technique similar to here: https://chrismflynn.wordpress.com/2012/09/06/fun-with-shaders-and-the-depth-buffer/ using texcoord and ComputeScreenPos.

I learned a lot about stencil buffers from exploring the demo project posted here: http://forum.unity3d.com/threads/unity-4-2-stencils-for-portal-rendering.191890/ Unfortunately as this project’s pdf readme admits, this solution is limited, and only works in this special case. The stencil mask is drawn before geometry. As soon as you pull an object in front of the glass the effect is ruined, because the part of it outside is still seen; it becomes clear that the portal simply draws on the surface of the screen and defines where on the screen the object can be seen, not where in the game world the object can be seen. (Here's another example of what I mean with "in front" being a problem, in the very last comment, the shot of the boat protruding through the wall: http://answers.unity3d.com/questions/590800/how-to-cullrender-to-through-a-window.html) As expected, the project writer ultimately suggests scripting solutions for sorting in depth if I need to move the portal into rendering simultaneous with the geometry, like I do need.

Depth + Stencil


Now I have done my homework enough to know that depth buffer and stencil buffer can be worked with together. For example, "Carmack's reverse trick" for volumetric shadows, where the buffer gets inverted/toggled each time it crosses the border of a shadow. My solution would need up to three states instead of two: when you are still outside the sphere as it grows, current scene between you and the sphere, next scene inside the sphere, and current scene on the other side of the sphere where there are no objects inside the sphere occluding them (only two when you’re already inside the sphere: next scene, and current scene [although at the point you’re inside the sphere it may make more sense to begin referring to the “next” scene as the current and the “current” as the previous; not sure what better time there’d be for making that terminological distinction]). So if I could figure out how to realize my sphere as the same type of object these volumetric shadows get realized as, that might work.

This might have helped: http://forum.unity3d.com/threads/rendering-inside-a-fishtank.271955/ I like the idea of flipping the stencil buffer on when I go in, and back off again when I go out of the sphere, but in practice I can't get this to work.

This might have helped if it went anywhere: http://forum.unity3d.com/threads/po...ers-at-intersections-with-another-mesh.93430/

The section in the documentation on "Transparent shader with depth writes" (http://docs.unity3d.com/Manual/SL-CullAndDepth.html) seems like it could be useful if I could parse it. It certainly seems possible to see through objects that write to the depth buffer, which is really all I need! It's just that the docs seem really sparse and vague on all of this (or assume a much greater familiarity with the basic material)

CSG


People have spoken of Stencil Buffers used to create CSG in graphics in general (https://www.opengl.org/discussion_boards/showthread.php/159762-advanced-csg-using-stencil-buffer) , so I'm sure it's somehow possible in Unity. Erik Rufelt's post here (http://www.gamedev.net/topic/660220...umes-and-carmacks-reverse-for-hole-rendering/) seems promising, for example, and this guy got something started: http://forum.unity3d.com/threads/semi-constructive-solid-geometry-in-the-stencil-buffer.263600/

Then I looked into Realtime Constructive Solid Geometry (runtime CSG). Has to be runtime, Boolean Ops (https://www.assetstore.unity3d.com/en/#!/content/18544) doesn't work. BooleanRT, GameDraw, MegaFiers... also: http://forum.unity3d.com/threads/boolean-mesh-project.227196/ and: http://forum.unity3d.com/threads/constructive-solid-geometry-csg-in-unity.125861/ This guy tried a couple more I didn't find and said they were all buggy and broke down when attempting more complex geometry (which my scene definitely has): http://www.reddit.com/r/Unity3D/comments/28va97/any_working_bugfree_realtime_boolean_operation/


Fidding with Z-Testing


A friend suggested inverting the depth test, so that objects behind the rim of the sphere wouldn't get occluded. The problem is then that there are multiple layers of them into the infinite distance, and now they're ALL inverted in depth and appearing on top of each other in funky ways.

I attemped a solution where the FIRST object in the next world seen through the sphere ALWAYS passes the z-test and then flips the stencil up to 2, where objects can pass the stencil test at 2 but with default z- behavior; unfortunately all objects in the next world would need BOTH of these materials for the effect to work (because in some situations they could be the very first object past the sphere lip, in others not) and Unity doesn't support multiple shaders.

I played around a bit with the ZFail param of the Stencil buffer. I thought, "if it doesn't fail the z-test, then it can't be inside the sphere yet!" But if there were ever more than one object from the next world in front of the sphere, this would only cull the first one.


Other Failed Rendering Strategies


Perhaps it would be possible to achieve this solution using a single shader with multiple passes (http://forum.unity3d.com/threads/achieving-a-multi-pass-effect-with-a-surface-shader.96393/) and/or replacement shaders (seems like the "queue" tag uniquely can enable ordering of passes? http://forum.unity3d.com/threads/simple-multi-pass-rendering.130705/), but the documentation doesn't provide any examples of multiple passes that I can find, nor can I find any examples online, so that possibility remains a mystery.

I looked into Layer Masks, but it doesn't matter how many cameras you use - it still doesn't solve the problem of cropping out the stuff from the next scene that is between the portal sphere and a camera.

I looked into Alpha blending but don't think it's right for this job.

And render textures seem more for projecting views into parts of the world onto flat surfaces, wouldn't really apply here.

I understand that transparent objects use the classical "painter's algorithm" style of rendering, back to front, which also might come in handy, even if none of the objects in my scene actually are transparent, I could try labelling them as such and put up with whatever possible weird artifacts from intersection / equal distance may occur...

There may be some way to adapt the work done here: http://forum.unity3d.com/threads/simple-cross-section-shader.34508/

This guy's three part post seemed promising: http://willychyr.com/2013/11/unity-shaders-depth-and-normal-textures/
Since the effect from the game Quantum Conundrum he was inspired by turned out to be exactly what I'm going for! Unfortunately he never actually gets there in his three-part tutorial, it just ends up with a depth texture. I guess I'm not surprised that a project designed by one of the folks behind Portal would feature this effect, but it does make the surprising dauntingness of achieving it easier to bear.

FYI, I just started learning about graphics and rendering this weekend, so I apologize I'm being an idiot noob here. Any guidance would be deeply appreciated.

 

Dang, well, it certainly seems like this is a non-trivial effect. Or I did a bang-up job articulating my issue. Plenty of views but no bites. Would still love to get this working!

 

If you're still around trying this, it would help if you re-upload your images. (They're gone! )