Stencil Buffer usage while rendering to texture.

[sebastian.vesenmayer]

Hello,

is it possible to do a stencil test while rendering to a 2d texture.

This is my example code:

	camera->setModelview(Unigine::Math::inverse(m_player->getWorldTransform()));

	// saving current render state and clearing it 
	Unigine::RenderState::saveState();
	Unigine::RenderState::clearStates();
	Unigine::RenderState::clearBuffer(Unigine::RenderState::BUFFER_STENCIL, Unigine::Math::vec4(), 0.f, 1);
	Unigine::RenderState::setStencilFunc(Unigine::RenderState::STENCIL_EQUAL, Unigine::RenderState::STENCIL_KEEP, 2);
	Unigine::RenderState::flushStates();

	// rendering an image from the camera to the texture
	viewportwide->renderTexture2D(camera, texture);

	// restoring back render state
	Unigine::RenderState::restoreState();

The stencil function I use should skip filling the texture, but it still works.

What am I doing wrong?

[andrey-kozlov]

Hello Sebastian,

RenderState is a low level thin wrapper over graphics api. The rendering pipeline changing it a lot during renderTexture2D call. So stencil state which is set before the call is overridden. And also the rendering pipeline uses its own depth-stencil buffer

What you probably need is to render viewport to a separate texture and then apply that texture with render_copy_2d material using your depth-stencil buffer and stencil state. Something like this:

    const auto rt = Renderer::getPostRenderTarget();
    rt->enable();
    rt->bindColorTexture(0, target);
    rt->bindDepthTexture(depth_stencil);

    const auto mat = Materials::findMaterial("render_copy_2d");
    const auto color_id = mat->findTexture("color");

    mat->setProceduralTexture(color_id, viewport_texture);

    RenderState::saveState();
    RenderState::clearStates();
    RenderState::setStencilFunc(RenderState::STENCIL_EQUAL);
    RenderState::setStencilPass(RenderState::STENCIL_KEEP);
    RenderState::setStencilRef(2);

    mat->renderScreen(Render::PASS_POST);
    mat->setProceduralTexture(color_id, nullptr);

    RenderState::restoreState();

    rt->disable();

 

[sebastian.vesenmayer]

Ok, I guess it is not that simple then.

I want to mask pixels with the stencil buffer to reduce fragment shading time while rendering to a big texture where lots of information is not used.

Do I understand it right, that I can do stencil masking only on the already shaded texture in a post process?

 

[andrey-kozlov]

You can mark unneeded pixels with depth=1.0. Next stages of pipeline will check for that value and skip calculations if possible. Also gbuffer fillrate is saved by depth test. We use that approach to optimize out invisible pixels for VR headsets. It can be achieved with following code:

Render::addCallback(Render::CALLBACK_BEGIN_OPACITY_GBUFFER, MakeCallback(callback_body));

RenderTargetPtr rt;

void callback_body()
{
	const auto mat = Materials::findMaterial("mark_hidden_pixels");
	const auto shader = mat->fetchShader(Render::PASS_POST);
	// mat->renderSсreen doesn't write depth therefore manual setup here
	Renderer::setMaterial(Render::PASS_POST, mat);
	Renderer::setShaderParameters(Render::PASS_POST, mat, 1);
	shader->setParameterArrayFloat4("s_transform", (const Math::vec4*)Math::mat4_identity.mat, 3);

	if (!rt)
		rt = RenderTarget::create();

	RenderState::setBufferMask(0, RenderState::BUFFER_DEPTH_STENCIL);
	RenderState::setPolygonCull(RenderState::CULL_NONE);
	RenderState::setDepthFunc(RenderState::DEPTH_ALWAYS);
	RenderState::flushStates();

	rt->unbindAll();
	rt->enable();
	rt->bindDepthTexture(Renderer::getTextureCurrentDepth());

	Ffp::renderScreen();

	rt->disable();
	rt->unbindAll();
}

BaseMaterial mark_hidden_pixels <var_prefix=var texture_prefix=tex node=none hidden=1>
{
	Pass post <node=none>
	{
		Fragment=
		#{
			#include <core/shaders/common/fragment.h>
			#define DISABLE_OUT_RT

			MAIN_BEGIN(FRAGMENT_OUT, FRAGMENT_IN)
				if(length((IN_UV - float2_half) * 2.0f) < 0.95f)
					discard; // don't write depth for visible pixels
			MAIN_END
		#}
	}
}