Interaction between Volume Rendered 3D Texture and Mesh Objects

I'm currently working on Volume Rendering using Unity. For that purpose, I took some inspiration from Brian Su who developed a Volume Renderer based on raymarching (github repo).

Basically what is being done by Mr Su is the following:

1 - Create a cube that will be used as a bounding box for the 3DTexture renderization
2 - Raymarch through the cube and calculate the 3D texture contribution
3 - Blend the result of the previous step with the scene.

For the completion of each step, there are 3 very important shaders: the material of the cube, the raymarching shader and the blending shader. The shaders that I'm using are extremely similar to what Mr Su developed.

Everything is working fine for rendering a simple 3D Texture. The problem appears when a mesh object is inserted in scene. First, the following happened:



Despite having the cube behind it, this always appears in front of it. My first thought was that this should be related to the cube's shader and that Z writing was off. So I checked the shader and removed the line "ZWrite Off". This worked... more or less:



As you can see, "colorless" parts of the cube are blocking, in this case, the mesh object. This should only happen for the parts of the cube that contain colored pixels and with opacity > 0. I've tried different methods to solve this problem: set the alpha to 0 for the parts of the cube with no color; make the cube transparent (in the cube shader, "Queue" = "Transparent") but this leads to the previous problem (check first image of this thread); mess with the blending part in the blending shader; and other methods.

Nothing works and that's why I'm asking you guys for your help. Any guidance for solving this problem is appreciated and if there's anymore information that you need to solve this issue just ask me.

Thank you

 

Edit: Forgot to mention that I've changed the frag function of the blending shader to the following:

Code (CSharp):

1. half4 frag(v2f i) : COLOR
2.     {      
3.         half4 src = tex2D(_MainTex, i.uv[1]);
4.         half4 dst = tex2D(_BlendTex, i.uv[0]);
5.         return src * src.a + (1 - src.a) * dst * dst.a;
6.     }

to get the result displayed in the second image.

Sorry for the inconvenience

 

I am not sure, but i have some ideas:
- it does not matter if the alpha is 0, if the queue is not on transparent, the pixel will be treated as filled / solid by the zbuffer, and your object (screw?) will be obscured even by the 'black' parts (if it is behind)
- if the queue is transparent on the other hand, you can't write to the z-buffer
It is a bit of a either-or situation - either it works if the object is behind, or if it is in front.
Now the internal Z-sorting of unity should handle that (Queue Transparent), so i wonder why it would lead again to the problem in the fist image

Much bigger problems lie ahead if you what to have solid geometry inside the 3d Volume.

Anyway, i had similar problems, and this was my workaround:
Google 'Stochastic Transparency' - it is a quite simple approach that is fully order independent. The only downside is speed. But with an accumulation buffer, you get a very clean image in less than half a second, only while you move the camera/geo it degrades to pixel-noise.

The Stochastic Transparency i mean refers to Gpu rendering, the same term is used in ray-marching optimisations, but thats not it.

Edit: for clarification - the fundamental problem are not the colors or blending functions, but the Depth buffer, which does not know anything about transparencies, so you need to address that.

 

First of all, thank you for your reply. Indeed I want to have solid geometry inside the 3D Volume.

Can this be solved by using stochastic transparency, or more generally an order independent transparency technique?
And how? It feels like the raymarching shader needs to be changed to check for other objects depths to check for the other color contributions. A problem that arises is to send the "screw"'s information to the shader, since it is a custom mesh, meaning, triangles. How can I avoid doing this? Or it has nothing to do with this?

Maybe I'm getting ahead of myself.

 

It's actually non of the above. You can do both alpha blending and writing to the z-buffer no matter what the queue value is.

I wonder why you need a separate blending shader/pass. It seems to me that you can blend the raymarched result directly into the backbuffer. If you write to the z-buffer, it will not be 100% correct, since the depth will of the surrounding box. Even though you could correct this, it should not even be needed in this case as far as I can see.

 

actually jvo3dc is quite right, you can do a lot of things with the z-buffer. My remarks are not correct, as i was solely thinking of forward rendering (where i had similar problems).
But even then - maybe you can render the solid geometry depth buffer to a rendertexture, and use this in your raymarching shader to handle the screws..

 

Script wise what I'm doing is the following:
- Create a render texture where the raymarching step renders to.
- Then this render texture is sent to the blending shader to blend the result of the previous step;

This is being done in the onRenderImage function:

Code (CSharp):

1. private void OnRenderImage(RenderTexture source, RenderTexture destination)
2.     {
3.        
4.         var width = source.width;
5.         var height = source.height;
6.  
7.         if(_ppCamera == null)
8.         {
9.             var go = new GameObject("PPCamera");
10.             _ppCamera = go.AddComponent<Camera>();
11.             _ppCamera.enabled = false;
12.             _ppCamera.transparencySortMode = TransparencySortMode.Orthographic;
13.         }
14.  
15.         _ppCamera.CopyFrom(GetComponent<Camera>());
16.         _ppCamera.clearFlags = CameraClearFlags.SolidColor;
17.         _ppCamera.backgroundColor = Color.white;
18.         _ppCamera.cullingMask = volumeLayer;
19.  
20.         var frontDepth = RenderTexture.GetTemporary(width, height, 0, RenderTextureFormat.ARGBFloat);
21.         var backDepth = RenderTexture.GetTemporary(width, height, 0, RenderTextureFormat.ARGBFloat);
22.  
23.         var volumeTarget = RenderTexture.GetTemporary(width, height, 0);
24.  
25.         // Render depths
26.         _ppCamera.targetTexture = frontDepth;
27.         _ppCamera.RenderWithShader(renderFrontDepthShader, "RenderType");
28.         _ppCamera.targetTexture = backDepth;
29.         _ppCamera.RenderWithShader(renderBackDepthShader, "RenderType");
30.  
31.         // Render volume
32.         _rayMarchMaterial.SetTexture("_VolumeTex", _volumeBuffer);
33.         _rayMarchMaterial.SetTexture("_FrontTex", frontDepth);
34.         _rayMarchMaterial.SetTexture("_BackTex", backDepth);
35.  
36.         _rayMarchMaterial.SetInt ("_Iterations", iterations);
37.         _rayMarchMaterial.SetFloat("_Opacity", opacity); // Blending strength
38.         _rayMarchMaterial.SetFloat ("_Brt", bright);
39.  
40.         Graphics.Blit(null, volumeTarget, _rayMarchMaterial);
41.  
42.         //Composite
43.         _compositeMaterial.SetTexture("_BlendTex", volumeTarget);
44.         Graphics.Blit(source, destination, _compositeMaterial, 0);
45.  
46.         RenderTexture.ReleaseTemporary(volumeTarget);
47.         RenderTexture.ReleaseTemporary(frontDepth);
48.         RenderTexture.ReleaseTemporary(backDepth);
49.     }

Considering this how can I do what you guys are suggesting - Atair an jvo3dc?

 

Well, that would take some diving into. Apparently the main volume shader uses the front and back depth that is rendered to separate targets beforehand. Surprisingly both are rendered to a ARGBFloat target. It seems to me that RFloat should offer enough space for this.

 

Basically I want the cube's depth information (so it can be noticeable that the 3DTexture is behind or front of our "screw"), but I don't want it to be drawn on the scene. This is my problem right now.

 

Edit: jvo3dc, the front and back textures are being used to render in a quicker fashion the 3DTexture.

This technique is perfectly explained in this awesome blog post.