[SOLVED] RenderImage2D and callback for capture screen

[fabre.cedric]

Hello,
I have one problem I use a renderImage2D with a post process to compute a depthMap but when I capture my video frame by frame (engine limited to 5 fps) from :
Unigine::Render::addCallback(Unigine::Render::CALLBACK_BEGIN_VISUALIZER, Unigine::MakeCallback(this, &EngineVideoManager::captureBufferCallback));

for renderImage2D :
 

    Unigine::ImagePtr imageToRender = Unigine::Image::create();
    imageToRender->create2D(size, size, Unigine::Image::FORMAT_R32F);

    Unigine::RenderState::saveState();
    Unigine::RenderState::clearStates();
    Unigine::Render::setPostMaterials("post_depth");
    Unigine::Render::renderImage2D(camera, imageToRender, Unigine::Viewport::SKIP_VELOCITY_BUFFER | Unigine::Viewport::SKIP_VISUALIZER);
    Unigine::RenderState::restoreState();

My video is flickering with my depthmap and my final image.
How this is possible to render properly from callback and renderImage2D without having this merge please ?
Thanks.

[silent]

Hi Fabre,

Can you send us a minimal test scene were you have your provided code and capturing code as well? So we can compile it and debug on our side?

Thanks!

[fabre.cedric]

GBufferRead.cpp
Sure, try to replace the GBufferRead.cpp file from the sample with this and you can view a similar problem I think.
EDIT : "I have modified the depth model to the left but the bug appear on other quad"
Thanks Silent

Edited December 16, 2021 by fabre.cedric

[sweetluna]

Hi Fabre,

GBufferRead sample uses depth buffer to cutout background for targets.
By changing this line:

RenderState::setTexture(RenderState::BIND_FRAGMENT, 0, renderer->getTextureOpacityDepth());

to:

RenderState::setTexture(RenderState::BIND_FRAGMENT, 0, renderer->getTextureColor());

you effectively replaced depth buffer texture required to remove background with screen color texture :)


Those lines in the sample are responsible for getting rid of repeated background:

float native_depth = TEXTURE_BIAS_ZERO(TEX_DEPTH, uv).r;

float clear = sign(abs(native_depth));

OUT_MRT(OUT_LDEPTH) *= clear;
// ...

This effect caused by that fact that we don't clear most of the textures each frame and only depth buffers and some other textures are cleared, since all those unnecessary clears accumulate to excess time per frame that can be done doing something useful.

As another possible solution, you can account for that fact that in Screen Color texture alpha channel is responsible for sky distinction. Sky always has an alpha of 0 (might need to test with transparency).

[fabre.cedric]

I need to compute final image for video rendering and I need depthBuffer texture from a specific point of view camera.
So, what's the right way to get it to work ? I don't understand your answer sorry :-/.

Edited December 16, 2021 by fabre.cedric

[sweetluna]

1. For retrieving depth buffer just add callback for Render::CALLBACK_END_SCREEN. And copy it to a different texture.

viewport->addCallback(Render::CALLBACK_END_SCREEN, MakeCallback(this, &render_finished_callback));

// ..
void render_finished_callback(Unigine::Renderer *renderer)
{
    depth->copy(Renderer::getTextureCurrentDepth());
}

2. Remove setPostMaterial as it no longer needed.
3. Modify your composition material (when you combine video with something or something similar) so that you account for depth/sky alpha, as described in my answer above, if needed.

[fabre.cedric]

No I need the depth texture from a specific (camera) point of view, near and far value, with this method the depth buffer is captured from the player point of view.

[sweetluna]

Create Viewport, set Skip Flags, add Callback and use renderImage2D to render straight to the CPU image.

[fabre.cedric]

Okay are you sure with the renderImage2D from viewport this problem doesn't appear with my other callback to grab the video from getTextureColor() ?

[fabre.cedric]

I have tested :
First viewport with first camera and a callback-GrabFinalImage to grab the final render from getTextureColor();
Second viewport with second camera (set post material to the camera) and a callback-getDepthTexture with viewport2->renderimage2D and exactly the same problem...
 

[sweetluna]

If you attach callbacks to individual viewports it should work just fine

[fabre.cedric]

Give me a minimal test code please because I have tested and that doesn't work, maybe I made something wrong.

[sweetluna]

Hi,

Something like that should work just fine. The problem with your approach was because you added global render callback with Render::addCallback, instead of Viewport::addCallback. Render callbacks will be executed for all viewports, while viewport callbacks will be executed only for assigned viewports. Also, you can check for nullptr in your global callback.
Don't forget to add PlayerDummy into your test scene. To check out if tex_depth is correct you can use RenderDoc on app compiled in debug mode.

ViewportPtr vp_special;
PlayerPtr player_special;
TexturePtr tex_depth;
ImagePtr image;

void begin_visualizer_callback()
{
	if (Renderer::getViewport() == vp_special)
		return;


	// Do something
	Log::message("test\n");
}

void end_screen_callback()
{
	tex_depth->copy(Renderer::getTextureCurrentDepth());
}

int AppWorldLogic::init()
{
	// Write here code to be called on world initialization: initialize resources for your world scene during the world start.
	vp_special = Viewport::create();
	player_special = checked_ptr_cast<Player>(World::getNodeByName("PlayerDummy"));
	tex_depth = Texture::create();


	int flags = Texture::DEFAULT_FLAGS | Texture::USAGE_RENDER;
	tex_depth->create2D(RenderState::getWidth(), RenderState::getHeight(), Texture::FORMAT_D32FS8, flags);

	// Add callback for retrieving depth texture
	vp_special->addCallback(Render::CALLBACK_END_SCREEN, MakeCallback(end_screen_callback));
	image = Image::create();

	// Add callback for main viewport
	Render::addCallback(Render::CALLBACK_BEGIN_VISUALIZER, MakeCallback(begin_visualizer_callback));

	return 1;
}

////////////////////////////////////////////////////////////////////////////////
// start of the main loop
////////////////////////////////////////////////////////////////////////////////

int AppWorldLogic::update()
{
	// Write here code to be called before updating each render frame: specify all graphics-related functions you want to be called every frame while your application executes.
	vp_special->renderImage2D(player_special->getCamera(), image);

	return 1;
}

This way you can have separate viewports for Main player's camera and for different camera, and get different textures/buffers for each of them.
Hope this will help you!

[fabre.cedric]

Well ok I understand, you update the viewport by renderImage2D in the update function and a specific camera.
I'll try that thanks !

[fabre.cedric]

I think you can add this exemple in the sdk samples because this is not really clear to understand how to achieve this.
That work like a charm now !
Many many many thanks for your patience and your times !!!!

[fabre.cedric]

The getTextureCurrentDepth return to me an alpha texture totaly strange but I have the right texture with my depth post process and color texture.
So that work perfectly now !
I would like to share my result with you.
I have developed with the help of a colleague and your team,  my proper particles engine billboard based with many particles and collisions test from depth buffer directly.
This is really much faster than the integrated collisions particles system for complex mesh and I can control all particles individually for life, speed and other.
Why don't integrate a similar collision system for particles in your engine ?
Thanks to the Unigine team for his support ;).

[silent]

Nice snow in the middle of the summer :)

[fabre.cedric]

36 minutes ago, silent said:

Nice snow in the middle of the summer :)

Yes, this is my test scene :D