Cutout Shader - Help

I have been working on and off for a week trying to make a shader for my 2D game and I have had no luck finding one, or in making one. Here is an image explaining what the desired shader is supposed to do:


The shader is supposed to be only applied to that one image because the other two images could be any image from the game. Any help would be really awesome! Thanks!

 

I'm not sure if i follow, confirm me, those 3 images are actually separate objects? and you want the middle one to serve as a mask? and you want a shader for that middle object alone, is this correct?

Could you be more specific on where you want this in your game? There are ways to create masks but you are asking for such a specific way of doing it that it might be easier with another solution, it could help knowing for what it is for

 

Sure. So I have multiple tiled backgrounds (from sprite sheets) for the game with water/lava in the middle. Because the images are tiled there is no smooth transition between them (e.g. grass suddenly stops and runs into the water, see the first picture below). So I want a shader that I can apply to a sprite sheet that will act as a mask to allow all the possible transitions. The black dot in my previous post's image would be one of the images in the mask sprite sheet (also see bottom image).

The way I was originally going to make this was by manually making all possible transitions be separate images, then have the game detect which transition occurred and then have it place the images. This method would be pretty easy for me to make, but I didn't want to do that because I am afraid that it might make the game lag because this is a mobile game (and I wanted to try a more elegant solution by using shaders).

Oh and everything in this game is randomly generated so I can't manually place anything.

Hopefully that explanation was sufficient.

(Original game with no transitions)

(Example smooth transitions, photoshopped onto it)

(Example of a single lava piece in the middle of grass)

 

Now i get, but then, how are you going to create the middle mask? you'll end up creating a custom map every time you want to mask something. I'm not sure if the performance saving you are trying to get is even a valid one, tilemap technique exists for ages now and unless the tiles are really really small you wont get much of a performance hit in today's mobile phones

2D Toolkit as it's own tilemap solution and i'm sure there are other assets to sport one for the unity 2D sprites. I've shipping a mobile game with tilemaps myself so i can testify it works.

What i'm trying to say is, for your situation i think you should create the different sprite combinations and use them in a tilemap. BUT, if you really want to know how to do what you want here it is: http://docs.unity3d.com/Manual/SL-Stencil.html

The stencil buffer lets you create a mask using shaders but you would still have to mark the shaders below and above the mask for it to work, so you can't simple create just one shader and call it a day

here's a nice example http://answers.unity3d.com/questions/590800/how-to-cullrender-to-through-a-window.html

Also, just my advice, take it as you will, you should focus on creating the game first and optimizing it if needed later. Sure there are things that should be thought before doing them but right now you are trying to solve a problem that you don't even know if you ever going to have

 

I wasn’t very clear on that - my game is actually completely playable right now; it already has some sound effects, music, GUI, and temporary sprites. That top picture in my previous post is a cropped screenshot – in the second picture I just photoshopped that area to show how I wanted it to look.

Ok cool, I have never made a game for a phone - I have been testing my game on a Nexus 7 tablet and it currently runs smoothly without me having taken much time to optimize the game. There are about 13x20 tiles on the screen at a time, not including visible player and enemy sprites.

There really aren’t that many custom masks required, I think only ~4 different ones needed in order to do everything necessary. A separate one is needed for: a lone block, end cap, corner, side… that’s all I can think of off the top of my head.
(All the possible mask shapes that are needed to make everything... I think)

Bummer. I really don't want to try to mess with how my game currently deals with shaders for spritesheets. It will be much faster to just make a few separate mask images from the grass/other sprites (shaped like what you see in the picture above) and place them on top of the water/lava and see if that lags the game. If it does lag I'll come back and try to do this with shaders.

Thanks a lot for all the help!

 

If you are using SpriteRenderer, you can use the Easy Sprite Pattern shader from your asset Easy Sprite
https://www.assetstore.unity3d.com/en/#!/content/17675

Here is a web demo, the SpriteRenderer is the Mask, the Pattern, is your texture.
http://www.vetasoft.com/unity/easysprite.html

It's should work like your lava exemple + you can auto animated the lava inside the mask.