Retro Shader problems

I am trying to recreate a retro shader like RetroAA found on the asset store. I tried Shader Forge and Amplify Shader Editor. It works well in general, except that there is unwanted noise at the edge of the texture pixels. After several tests, I found interesting elements, but I have not found a solution. Here are the details. I made the shader available if you want to test.

Explanation of the algorithm:
The algorithm calculates the UV coordinates in the middle of the source texture pixel. Most of the time, that returns the correct color of the source texture. Because the texture uses "bilinear filtering", the parts around the center of the texture pixels change colors linearly with the neighbor texture pixels (I also included a screenshot to illustrate the values of the default bilinear texture).

Explanation of the problem:
I tend to believe that there is an error margin calculation when getting the pixels with a steeper angle (accentuated on the sides of the screen). I cannot explain why it happens that way.


I also included another test "TestRetro2" where the noise completely disappears. The only difference in the shader is that I use a "MIP Level" property to plug into the shader. When I set the level to 0, the final result is the same as having a texture with "Point Filter". I am wondering why "TestRetro2" works without noise for the part of the texture that uses a MIP Level 0. The algorithm is the same when dealing with the calculation of the UV coordinates. I cannot explain where the noise comes from exactly because of this test. This test light could be a clue to investigate.

I tried an orthographic camera with perpendicular angles and the noise disappeared. However, if I modify the angle to have variable angles with the texture, the noise is visible again.

I also tested "TestShader1" with similar nodes in the "Amplify Shader Editor". The result was exactly the same. After this test, I do not tend to believe this is a bug related to ShaderForge, but I am not a 100% sure.

Another note about RetroAA. I believe this asset also has the same problem. I did a test with this shader. There is noise, but only a few pixels. It is not really apparent. I believe it this aspect is far less visible because RetroAA uses a "smoothstep" function to create anti-aliasing. That removes a lot of noise.

- I use a 128x128 texture with bilinear filtering (the shader works only with square textures at the moment).
- I do not use Anti-Aliasing on the camera in the test to avoid changing the end result.
- I highlighted a few areas on the image with the shader "TestRetro1". NOTE: to see the noise, view the image in full resolution or just zoom in. The noisy lines are about 1 pixel in width.
- I also used green and magenta markers on the test texture to highlight the problem. Because the rest of the texture is mostly gray, the noise tends to blend. With the colors and the white lines, we can clearly see the problem.
- The test shader includes a functionality to switch from bilinear to point filter to debug.

Does anyone have clues about the real nature of the noise in the shader?

To download the sources, here is a temporary link:
https://www.dropbox.com/s/n4ctvloyebavtie/RetroShaderBug.zip?dl=0

 

The two test shaders can be downloaded in attachment.

 

For your shader this is an issue with how GPUs determine the LOD for a texture. LOD is determined by how much a texture UV changes in 2x2 pixel areas, and in this case because you're forcing a hard edge the change is quite a lot, so it's dropping the LOD level for that 2x2 pixel area probably 1 mip lower than you want. This is why when you override the LOD the issue goes away.

A solution to this is passing uv derivatives into the tex2D function (tex2Dgrad). I don't know if Shader Forge has this, only thing I found was someone making a reference to disabling OpenGL on the shader exposed the option on the shader node. Take the original, unmodified UV and pass it through a DDX and DDY node and plug that into the relevant Texture2D node inputs, assuming there are inputs for that. I don't have ShaderForge, nor have I ever used it, so I don't know for sure.

The other way to handle this, if using tex2Dgrad is not an option, is by calculating the LODs yourself and passing that in using a function like this:

Code (CSharp):

1. float mip_map_level(in float2 texel_coords) // uv * texture size
2. {
3.     float2 x = ddx(texel_coords);
4.     float2 y = ddy(texel_coords);
5.     float delta_max_sqr = max(dot(x, x), dot(y, y));
6.     float lod = 0.5 * log2(delta_max_sqr);
7.     return max( 0.0, lod );
8. }

That's the original UVs multiplied by the texture size, not the "retro" modified UVs. If you use the modified UVs you'll see the exact same artifacts.

Now as for the artifacts with RetroAA, I don't have the shader myself to know for sure how they've implemented it, but my own implementation of this shader using the tex2Dgrad solution doesn't have the artifacts you see in your shader. It does however start to have issues if anisotropic filtering is enabled on the texture, but only on stuff in the distance at extreme angles instead of in the foreground. My assumption would be that this is the same issue RetroAA is having as the some of the screenshots of RetroAA look like the texture is using anisotropic filtering and they make mention of using "aniso level > 0" for best quality, but manually setting the texture LOD actually disables anisotropic filtering from being used where tex2Dgrad does not. It's a very small artifact though, and I only can really see it when using a black and white grid texture.

This too would be solvable. My solution would be to have one texture sample using tex2Dgrad or tex2Dlod and doing the "retro aa" modifications to the UV, and a second texture sample just using the default unmodified UVs straight, then fade between them using the calculated lod. A more elegant solution might be to calculate the mip map level using the method used for anisotropic filtering and applying the UV modifications separately per axis, but none of the GPU vendors actually implement anisotropic filtering using the reference implementation so it would difficult to match. It would also potentially mean a completely different shader for if anisotropic filtering is enabled or not.

As an aside, it's kind of ugly that you have to use a material parameter to define the texture size. Unity passes that information to the shader automatically. Someone should bug Joachim about adding a node for that.

 

Thanks bgolus. I understand that the problem, according to you, is related to the mipmaps. That would make sense. I had this thought during the process, but I discarded this possibility at a certain moment. I tried the "Mipmap Visualization" asset from the asset store to verify that possibility. It did not show isolated pixels with different colors. I expected to see plain colors with transitions and different colors where the noise was. That was not the case. Therefore, I discarded that scenario. I have my doubts about the method used to show the mipmaps information with this asset.

The mipmaps scenario would make sense with the 2x2 pixel areas. The noise is sometimes 1x1 or 2x2. Furthermore, that problem occurs when the source pixels have more distortion as it happens when moving away from the center.

On the other hand, I am surprised to notice that the noise is always at the junction of the source pixels. I cannot fully explain why the mipmaps appear at this specific area. I did not expect to have a mipmap level of 1 when being so close. I thought that the level changed when the number of pixels could not be fit within the drawn area on the screen. At this point, the graphic card used a lower resolution. According to your theory and related to my case, there are MIP Level 1 pixels on the edges the MIP Level 0 pixels. It is not necessarily a smooth transition based on the distance and the angle of the surface (which affects the rendered area).

I took a screenshot of the rendered image and I applied a high contrast image effect in Photoshop. I can clearly see the same noise at the pixel junctions. It was just less apparent because of the similar gray pixels.

I also tried the Mipmaps view in the Unity Editor, but there's only a blue shading. Not useful. It works with the Standard and default shaders, but not with all custom shaders. I am wondering if there is a switch.

For the solution, I will have to be creative. I understand there are limitations and achieving this effect is not simple. I experimented with DDX and DDY. I am not quite familiar yet, but I got some interesting data.

Concerning your code, I was not able to get a better result with the MIP Level. It just gave the same result as the point filter.

Concerning the ugliness of the texture size you mentioned, I had already searched solutions and I found a few that could explain the situation. Otherwise, the shaders seem to have this integrated. I should try writing native code in Shader Forge to get the size.

https://shaderforge.userecho.com/to...d-how-about-a-recttransform-component-height/

http://answers.unity3d.com/questions/678193/is-it-possible-to-access-the-dimensions-of-a-textu.html

Another approach would be to use point filter textures instead of linear and try to simulate the mipmaps when we are farther or with steep angles.

I also thought about having a point filter texture and bilinear filter texture. When closer, the point filter texture would be used progressively. The problem is creating two identical textures in Unity with just the filter parameters as the difference is not a good approach to me. It is twice the memory. Using mipmap Level 0 as parameters and changing it manually in the shader could be a better solution as saw in my tests.

 

Okay, try this, specifically for the shader you're using:
Use a fixed mip 0 for your modified UV texture sample.
Use that mip calculation to replace or modify the "point filter amount". You'll need to clamp it to 0.0 ~ 1.0, and either swap the inputs into your current lerp or do a 1.0 - clamped_mip.

As for the texture size thing, in the first link Joachim is I believe specifically referencing the issue with getting information about the rect transform, and is perhaps still under the impression the texture size is not available to shaders ... which is no longer true. That second link is a great example as the 2014 answer (the one accepted as the "best answer") correctly for the time states it is not available, while the next answer down from 2015 links to Unity documentation referencing the _TextureName_TexelSize parameter which includes the texture size in the zw components. I use the _TexelSize parameter quite frequently in my own shaders, including my version of the RetroAA style shader, so it absolutely works.

 

Here's an update.

I've succeeded in getting your function mip_map_level() to work, but not perfectly. The main big problem is that the texture near the camera is clear and very quickly, it changes to the maximum MIP level. I would have expected a more progressive MIP level.

After further investigation, I translated the RetroAA into Shader Forge. The result is matching. I believe only the tone differs. RetroAA doesn't use the MIP entry. I don't fully understand the algorithm yet. I found that the fwidth() is available as the DDXY() in SF. Here's some detail.

The width and the height of the texture don't match. I used a texture of 128x64. If I use the height of 64 with the original algorithm, the rendered mapped texture is stretched. So I use the width to replace the height, even though the size is 128x64. I don't get this one!
UPDATE: Never mind. The texture was not resized correctly. This is fixed.
UDPATE2: It's getting late and I might have posted too soon. There's still a stretch in the texture, but I found another fix. The ratio seems good, but it's still not the original algorithm.

In my test shader, I also used the test color to visualize the DDXY().

That could give a good idea of the algorithm. As I mentioned, the result is pretty much the same. There's noise, but very few and there's an anti-aliasing already output in the pixels. It looks polished and pixelated while keeping a good Mipmap effect when fading into the horizon. Probably with a better AA script, the remaining noise could disappear. That could be the final result. I hope this might be useful. Let me know what you think if that inspires you too for your retro shader.

 

Yep, that shader is nearly identical to mine, apart from me not using smoothstep (which I should, it makes it look better) and me using tex2Dgrad (which they should if they're not).

You might try using the mip level function with that shader to set the texture LOD. If it's working properly then it shouldn't look any different apart from removing the artifacts (assuming you don't have anisotropic filtering enabled). If it gets fuzzier and the texture's aniso level is zero, then something isn't being calculated correctly.

Can you post the test texture you're using? I know it's simple, but the specific placement of your lines in the texture is part of the reason it's showing up do obviously. I wanted to post some examples of what using tex2Dlod and tex2Dgrad produce.

 

Here's the test texture in attachment. Be sure to set the mipmaps on, compression to none and use bilinear filter. Furthermore, the project uses "Anisotropic Textures: Forced On" in the Quality settings. Without this last setting, the texture is kind of blurred when being shrunk in the distance.

 

This is my version of the shader. It's a little less elegant than the RetroAA shader, and slightly slower, but also has slightly reduced artifacts just by being lucky with how I'm manipulating the texture UVs.

Code (CSharp):

1. Shader "Unlit/Pixel Art AA"
2. {
3.     Properties
4.     {
5.         _MainTex ("Texture", 2D) = "white" {}
6.     }
7.     SubShader
8.     {
9.         Tags { "RenderType"="Opaque" }
10.         LOD 100
11.  
12.         Pass
13.         {
14.             CGPROGRAM
15.             #pragma vertex vert
16.             #pragma fragment frag
17.        
18.             #include "UnityCG.cginc"
19.  
20.             struct appdata
21.             {
22.                 float4 vertex : POSITION;
23.                 float2 uv : TEXCOORD0;
24.             };
25.  
26.             struct v2f
27.             {
28.                 float2 uv : TEXCOORD0;
29.                 float4 vertex : SV_POSITION;
30.             };
31.  
32.             sampler2D _MainTex;
33.             float4 _MainTex_ST;
34.             float4 _MainTex_TexelSize;
35.        
36.             v2f vert (appdata v)
37.             {
38.                 v2f o;
39.                 o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
40.                 o.uv = TRANSFORM_TEX(v.uv, _MainTex);
41.                 return o;
42.             }
43.        
44.             fixed4 frag (v2f i) : SV_Target
45.             {
46.                 float2 uvPixels = i.uv * _MainTex_TexelSize.zw;
47.  
48.                 float2 uvPixels_fwidth = min(float2(1.0, 1.0), fwidth(uvPixels));
49.                 float2 uvPixels_floor = floor(uvPixels);
50.                 float2 uvPixels_frac = frac(uvPixels);
51.                 // magic ...
52.                 float2 uvPixels_aa = (1.0 - saturate((1.0 - abs(uvPixels_frac * 2.0 - 1.0)) / uvPixels_fwidth)) * sign(uvPixels_frac - 0.5) * 0.5 + 0.5;
53.                 float2 uv = (floor(uvPixels) + uvPixels_aa) * _MainTex_TexelSize.xy;
54.  
55.                 fixed4 col = tex2Dgrad(_MainTex, uv, ddx(i.uv), ddy(i.uv));
56.                 return col;
57.             }
58.             ENDCG
59.         }
60.     }
61. }
62.  

And here's an example of your re-implementation of Retro AA in ShaderForge, vs my "Pixel AA" (as it would look if implemented similarly in ShaderForge), and then the above shader. This shows off what the benefit of using tex2Dgrad is, note the Retro AA implementation looks nearly identical to mine when using tex2Dgrad.


This is an intentionally worse case texture with single pixel white lines that are spaced out in a way to cause the maximal error possible.

Note that setting Anisotropic Textures: Forced On and having the Aniso Level on the texture 1 or greater ignores the "Bilinear / Trilinear" setting. Anisotropic is the "next" filtering type after that, even though Unity's UI makes it seem like it's something you can use with the filtering type in the drop down. Forcing on clamps the aniso level for all textures not set to level 0 to between 9 and 16.

 

I've been able to translate your solution into Shader Forge. Thanks a lot. It's certainly the best adaptation I've seen so far. If I can give a big thumbs up somewhere, let me know.

tex2Dgrad() seems the key component in the solution. Unfortunately, Shader Forge didn't have a node for it, but there's a function to paste code. It just wrote tex2Dgrad() as code.

The result seems exactly the same as the standard shader when far and it keeps an interesting amount of point filtering when close.

I like your idea of testing with a black and white high-contrast texture. I did it too to test.

I also found that the line
"float2 uvPixels_floor = floor(uvPixels); "
is useless because the variable is not used. You already calculate the floor() below only once. Removing it will make your script faster.

Furthermore, I was wondering if you can remove the AA. The AA is of a great quality. The problem is that I'll need to apply an AA script to the whole image and that probably makes a little too much of AA at the end.

 

I would probably just replace the place where I do floor(uvPixels) again with that variable. That's probably from me playing with the code to see where things where breaking down. However removing it won't actually change the compiled code as shader compilers are pretty smart about removing unused code.

 

Wow, this shader is amazing! But I've been wondering if there is anyway to get lighting or shadow casting support working? With these features the shader would fix quite a few issues I have with my current game project.

I've got the transparency working on my own and already tried to implement lighting/shadow casting myself, but I basically have no shader coding experience/knowledge, so sadly my results were far from successful.

Also, under which license is the shader released?

 

I'm going to say it's under the WTF license, or unlicense, or MIT. Which ever you want. I don't need atribution.

If you want one that works with lighting you can write your own custom surface shader version of this, or buy the RetroAA asset.

 

Dammit. I was actually afraid of this.

I had already a email conversation with the RetroAA people and they told me that thier solution have issues with
screen space shadows in forward rendering because they're using MSAA and this "mixes quite badly" with it (creates artifacts around transparent sprites).

I'm actually tempted to buy it to add shadows on my own just to see if it works on my side but since they told me it's not working properly on their side I'll probably wait for a sale.

Anyway, thanks for your quick reply!

 

Oh, yes, of course it will. This has nothing to do with RetroAA though, that's a problem with how Unity's screen space shadows work on desktop and console. This or any asset will have problems there.

Specifically they're talking this issue:
https://forum.unity.com/threads/fixing-screen-space-directional-shadows-and-anti-aliasing.379902/

 

Bumping this to ask for advice. I haven't read how the technique works yet, but do you think you can factor this out to generically use it for any kind of 'pixel art' shader? In order to easily use it when you make custom shader effects (using shader graph, or just manually written shader) but still want this kind of anti-aliasing. E.g. as a shader graph node.

 

That is kind of exactly what this technique is for. It takes any texture intended to be used as pixel art and produces an anti-aliased version of that texture for very little additional shader cost. While the code example above won't work with Shader Graph, even in a Custom node, the technique is totally applicable. It could either be done with the built in nodes, or a Custom node with the code updated to be in Direct3D 11 HLSL (the above is in Direct3D 9 HLSL, which doesn't always work with Shader Graph).

 

It works great, thank you for sharing! I have a problem though when I am trying to sample normal map using this technique. There are artifacts in some places (but not everywhere!) which are not there if I use point filtering and standard sampling. They look like these: (lod0, normal pass with lower the resolution and high bump scale for them to be more visible)


Do you possibly know what could cause them? Thanks in advance

 

There’s not actually anything wrong. What you’re seeing is the expected result of sharpened normal maps. This technique doesn’t really work when applied to anything other than the color texture. Normals, specular, smoothness, using this technique on any of those will not produce the desired result.

The reason why is it’s abusing the texture sampling hardware to interpolate between the texture’s values along the texel edges, and you want to be interpolating between the colors that will be displayed to the viewer. When you apply it to the base color / albedo, the interpolated value is the displayed color. When you apply it to the normal map you’re creating a sharp 1 pixel wide crease in the normal that can alias when lighting is applied, which is what you’re seeing.

The “correct” approach for normal maps would be to sample the normal map at each of the 4 closest texels, calculate the lighting for each texel / normal separately, and then manually interpolate between those. That’s obviously more expensive, but that’s the only way to get the correct anti-aliasing.

Alternatively just use point filtering and fix it with post processing anti-aliasing like SMAA or TAA.

But both of those options makes it a little harder to transition between the bilinear mip mapped far away surfaces and the near ones.