Masking in Terrain base

[Priyank.Jain2]

In our application the .ter file generated is of square shape, but actual terrain boundaries are different than that.

Is there any way we can ignore rendering of all the values that are outside the terrain bounds?

Just like we do masking in Layers to apply texture on a particular region can we do same for the terrain_base?

[Priyank.Jain2]

@silent can you please provide some pointers to this?

[silent]

Hi Priyank,

Sorry, can't understand what you want to achieve. Could you please provide more details (maybe with screenshots)?

Thanks!

[Priyank.Jain2]

This is how the terrain file looks like currently.

And this is what we are expecting it to be

We basically want to mask out all the the fragments out side the terrain boundaries.

I am currently playing with Terrain holes but no luck. Are there any sample for hole? Also what are the different ways we can achieve this?

[silent]

There is no such functionality available out of the box.

Maybe specially created terrain occluder textures can do the trick? 

[Greg.Mildenhall]

We're still trying to do this. The obvious way (to me) is to use a slightly modified version of the terrain shader that discards the unwanted fragments based on alpha. Unfortunately, this doesn't affect the depth pre-pass so it still prevents the backdrop from rendering. I've disabled the pre-pass for terrain (I think that should be OK because it's very simple terrain?) and am trying to write terrain depth in the deferred pass instead. The depth buffer appears to be bound for writing*, but I'm having no success actually writing to it. I imagine I'm just not getting the shader code right - even OUT_DEPTH = 0.5; at the end of the MAIN_BEGIN_DEFERRED section doesn't seem to have any effect.

*EDIT: Oops sorry, I think I got confused about that. It's _not_ write-enabled (and the test is set to EQUAL) so now I'm wondering if there's a way to change that in the material definition for terrain? Otherwise I'll start looking for a way to plumb the texture holding the alpha values into the depth pre-pass, or else somehow insert an extra pass (Is that what an Auxiliary pass is for?) that will write the depth.

[Greg.Mildenhall]

None of the approaches I mentioned above have borne any fruit. :( Is there any documentation about which aspects of the material file will actually have an effect on terrain and which will not? It doesn't appear to apply all the configuration features of a normal material file.

Recapping: We tried discard()ing the terrain fragments with zero alpha, but found the depth was still getting written in the pre-pass, preventing the backdrop from rendering. So what we need is a way to make the terrain render its depth in the main gbuffer-filling pass OR a way to pass in our terrain texture to the pre-pass shader so we can use the alpha information there. Is there a way to do either of those, or is the terrain material file just not flexible enough?

As for your suggestion to use occluder textures (or should we use occluder terrain?), is that a per-fragment effect, or is it per-mesh? I'm worried that it only occludes per-mesh, only blanking out squares of terrain as a whole, and that which squares get blanked could change as the LOD level changes. Is that true?