Custom rendering middleware integration

[adrian.licuriceanu]

Hello,

Sorry for the lengthy post...

 

We have a very big rendering middleware integrated into Unigine, which basically render its elements as outputs into Unigine gbuffers (for deferred shading). This is implemented as a plugin with main rendering done by inheriting from an ObjectExternBase, with a very big bounding box (the middleware has its own internal visibility) and two surfaces (one for opaque elements and one for transparent ones). Each have a simple material so I have ambient, deferred and shadow passes being called on the extern object.

The implementation of the middleware is DirectX11 based and it cannot use Unigine objects, textures, nodes. It has its own internal objects.

 

So the only solution we've come up with it, required some heavy modifications to Unigine source code so our custom functionality is being called in various places (most of the modifications are in RenderRenderer.cpp, but some are in RenderLights.cpp too).

These callbacks are implemented in a somewhat generic way, but it is still a pain for us to apply these source code modifications with each Unigine engine upgrade. So I guess the main question here is if there is some other way to do this, that doesn't require a heavy engine source code modification? We've started this way back, maybe now we have better solutions with latest engine versions.

And if there is not, can you consider, in the future Unigine versions, to add something similar so extensibility with external middlewares is easier? For example, some form of generic callbacks for various engine rendering events (I can send the entire list we need :) ? Or maybe more virtual functions to ObjectExternBase interface to cover all these events?

 

Some details:

To give you a better context here are more details how these callbacks are being implemented:

• in EnginePlugins class we have a customCallback(enum id, void *custom_data) function receiving an id and custom data (depending on ID). This function will call a similar customCallback function (also added by us) on every Unigine::Plugin interface, which by default does nothing.

• id can be:

enum PLUGIN_CALLBACK
{
	BEGIN_VIDEORESTART_CALLBACK=0,
	END_VIDEORESTART_CALLBACK,
	BEGINFRAME_CALLBACK,
	POSTBEGINFRAME_CALLBACK,
	ENDFRAME_CALLBACK,
	VISIBILITY_CALLBACK,
	RENDER_CALLBACK,
	ENGINE_CALLBACKS_NO
};

So basically all the plugins (that override customCallback function) are announced by engine when a new when video mode is reset, frame starts and ends, when visibility is to be computed, when rendering is to be done in various cases etc.

• For example the rendering callback (RENDER_CALLBACK) it is called in different parts of RenderRenderer (what kind of render uniquely identified by more data passed to callback). A case would be:

  • before render_deferred_surfaces, so our middleware also gets a hold of gbuffers to render to (it expects all D3D11Texture* to be sent in a structure in the callback custom data). So this callback custom data looks for example something like:

struct RenderCallbackData
{
	// Pass this rendering is referring to.
	RENDER_CALLBACK_PASS pass;
	// Custom date depending on the pass.
	void *pass_data;
	// Unigine viewport id.
	int viewport_id;
};

and for example for render_deferred_surfaces we have pass member valued as another enum constant (RENDER_PASS_BEFORE_DEFERRED_TEXS) and pass_data points to:

struct RenderCallbackData_BeforeDeferredTexs
{
	// Deferred buffer for depths.
	void *depth_texture;
	// Deferred buffer holding diffuse color.
	void *color_texture;
	// Deferred buffer holding the view space normals.
	void *normal_texture;
	// Deferred shading information.
	void *shading_texture;
	// Velocity texture.
	void *velocity_texture;
	// Lightmap texture.
	void *light_map_texture;
};

 

• .

• visibility callback is also called in different contexts (for reflection, for shadows, for normal rendering etc) so the middleware does different visibility checks and uses different settings. It uses its own internal data structures.

• frame start and end does some custom processing needed per frame by the middleware.

• and so on.

I hope this gives you some context of the task. If needed, I can send more details about code modifications.

 

Kind Regards,

Adrian L.

[silent]

Hi Adrian,

Did you have time to check C++ Samples -> Rendering -> GBufferRead / GBufferWrite? There are a lot of callback while frame is being constructed and you can replace any buffer as you want without heavy engine modifications.

Thanks!

[adrian.licuriceanu]

Hi, will check, thank you! I knew I had to ask, maybe there are other better options. I will let you know if this doesn't cover all our cases.

Regards,

Adrian

[adrian.licuriceanu]

Hi,

I've checked the samples and I think this approach may help us. But we are still missing some callbacks on the engine side to pull this off without engine source code modifications:

- When visibility is performed. We may be able to apply visibility right before rendering, but are there any cases when multiple render calls are to be made on the same visibility? Again we may be able to compare frustum to not repeat visibility.

- At shadowmaps rendering: I need a call per each cascade in render_light_world_shadow.

- At ambient pass: Seems that there is no support for a callback to know when render_scene_transparent is being called.

- At begin/end frame and at begin/end video restart - needed by some housekeeping for the middleware.

Are there any workarounds for these? Or can you consider these for additions in the future engine versions?

Regards,

Adrian L.

[andrey-kozlov]

Hello Adrian,

You may take a look at ObjectExtern (C++ API -> Nodes -> ObjectExtern sample). It may help with shadowmap and ambient passes and maybe with visibility

[silent]

Adding callbacks for each shadowmap sounds very reasonable and should be in theory possible to achieve with relatively small changes. Adding callback for each and every possible pass will simply slow down the rendering, so we need to choose only a subset from the all available options.

 

 

Quote

- At begin/end frame and at begin/end video restart - needed by some housekeeping for the middleware.

We are moving away from video_restart concept when you need to destroy all the resource to recreate the window (or change the windowed mode to fullscreen). It's now already possible with 2.11 update to seamlessly switch between fullscreen / windowed mode. As for Begin / End frame - these callbacks already available in API (RENDER_CALLBACK_BEGIN = 0 / RENDER_CALLBACK_END = 61).

 

 

Quote

So the only solution we've come up with it, required some heavy modifications to Unigine source code so our custom functionality is being called in various places (most of the modifications are in RenderRenderer.cpp, but some are in RenderLights.cpp too).

Is it possible to check these modifications? Especially if they covering following cases:

- When visibility is performed. We may be able to apply visibility right before rendering, but are there any cases when multiple render calls are to be made on the same visibility? Again we may be able to compare frustum to not repeat visibility.
- At ambient pass: Seems that there is no support for a callback to know when render_scene_transparent is being called.

because right now we can't find out the way to achieve it.

Thanks!

[adrian.licuriceanu]

Hello again,

Ok, so some progress was done on this. Some of the callbacks were removed.

Some issues though:

- Am I correct to just ignore video restart? I mean in the future (2.11?) this will not be called at all?

- I need the actual shaders oblique frustum plane parameter (the one from cbuffer_camera.oblique_frustum_plane). Seems that API returns the oblique_frustum_plane, which is like a rendering API independent version of it. I can see it being transformed to cbuffer format with:

		mat4 oblique_projection;
		vec4 v = vec4(oblique_frustum_plane * imodelview) * inverse(projection);
		float z = Math::abs(v.z);
		if (z > 1.0f)
			v /= z;
		v.w -= 1.0f;
		mat4 transform = mat4::IDENTITY;
		transform.setRow(2, v);
		mul(oblique_projection, transform, projection);
		oblique_projection = hardwareProjection(oblique_projection);
		cbuffer_camera.oblique_frustum_plane = oblique_projection.getRow(2);

But I also need the above cbuffer_camera.oblique_frustum_plane to be exposed via API, otherwise I just need to duplicate that on my end and hope it will not change in the future.

- For stereo rendering I need RenderRenderer combined_modelview to be exposed in the API. 

- You said to use replace my begin/end frame callbacks with RENDER_CALLBACK_BEGIN/END, but am I correct that these are called multiple times per frame? Like, for example, one time for normal rendering, one time for reflection, another time for another viewport? If so, I actually need this one time per entire tick (update, frame or how you want to call it). A solution would be to get some sort of unique frame index (some id that engine is using to discern between a frame and the next) and apply my code only if this differs (that is if I go to another frame). Is there a way to get a frame id?

Kind Regards,

Adrian L.

[morbid]

Hi Adrian,

Sorry for the delay with the answer.

On 6/5/2020 at 11:08 PM, adrian.licuriceanu said:

- Am I correct to just ignore video restart? I mean in the future (2.11?) this will not be called at all?

In 2.11 video_restart doesn't clean resources from VRAM, that's the major change we've made for the LandscapeTerrain. If you want to change resolution or anything related to video settings you still need to call video_restart.

I've talked to the engine team and we can extend API for your case. As for the frame index we already have getFrame.

Thanks.

[adrian.licuriceanu]

Hi,

Thanks for all the info.

Quote

In 2.11 video_restart doesn't clean resources from VRAM, that's the major change we've made for the LandscapeTerrain. If you want to change resolution or anything related to video settings you still need to call video_restart.

Good that video_restart doesn't clean resources, this was basically for what I was previously using it (to make sure I recreated my VRAM objects). As for resolution changes, I do it on the fly in my code (detect if changed) so all is good.

Quote

I've talked to the engine team and we can extend API for your case.

Excellent.

Quote

As for the frame index we already have getFrame.

Good, I remember I read something about it. Yes, that did the trick :)

One last question: does the texture from Unigine::Renderer::get()->getTextureGBufferAlbedo() always have the same size as the one from Unigine::RenderState::get()->getScreenColorTexture (gbuffers sizes == screen sizes)? Asking since, during some of my code, I need to check the size of the gbuffers, but this is not during a callback and seems that getTextureGBufferAlbedo returns null if not inside a callback. Btw, any reason for this:

 

void Render::runCallbacks(int callback, RenderScreen *screen)
{
    renderer->setRenderScreen(screen);
    signals[callback].invoke(renderer);
    renderer->setRenderScreen(nullptr);
}

This is why it returns null, the screen is null outside callbacks calls.

 

Regards,

Adrian L.

Edited June 11, 2020 by adrian.licuriceanu

[andrey-kozlov]

Hello Adrian,

Size of getTextureGBufferAlbedo should be equal to getScreenColorTexture()'s size. Although it may differ from window size due stuff like supersampling or render_border.

I believe screen is null outside of callbacks to avoid returning dangling pointers or outdated textures. I would recommend to move screen buffers referring into callback bodies if possible.

[adrian.licuriceanu]

Hi Andrey,

Quote

Size of getTextureGBufferAlbedo should be equal to getScreenColorTexture()'s size

Good, that is what I am looking for (I create another temporary buffer I use for my own rendering which needs to have the same size as gbuffer, not really interested in window size). So, also in the end, I can just use screen color texture and avoid heavily refactor that specific code to be moved inside a callback.

Thanks,

Adrian L.

[morbid]

Hello Adrian,

We'd like to get more info on the requested functionalities.

1. Where and when exactly do you expect shadows callbacks? Do you need access to the shadows buffer? I would appreciate a direction to the engine's source code.
2. Do you have any vision of new API methods for oblique frustum plane and combined_modelview? If you need a new method (a getter I suppose?) what should it return?

Thank you.

 

[morbid]

In 2.12 you'll find new callbacks for shadows:

CALLBACK_BEGIN_SHADOWS,

	CALLBACK_BEGIN_WORLD_SHADOW,
	CALLBACK_END_WORLD_SHADOW,

	CALLBACK_BEGIN_PROJ_SHADOW,
	CALLBACK_END_PROJ_SHADOW,

	CALLBACK_BEGIN_OMNI_SHADOW,
	CALLBACK_END_OMNI_SHADOW,

CALLBACK_END_SHADOWS,

And new methods

Renderer::get CBufferCamera ShaderCBufferCamera
Renderer::get CBufferScattering ShaderCBufferScattering

combined_modelview will be available with CALLBACK_BEGIN Renderer::getModelview().

Will be waiting for your feedback.

[adrian.licuriceanu]

Thank you for the support. We are still in the middle of upgrading to 2.11 and fixing all the issues from that. After that we will upgrade to 2.12 and provide the feedback for the new API support.

Kind Regards,

Adrian L.

[adrian.licuriceanu]

Hello, resurrecting this thread a little...

I know you've added getRenderShadowCascadeProjection and getRenderShadowCascadeModelview, which are great, but they need the current cascade as parameter. As I am rendering cascades inside renderShadow for object extern there is no API to get the current cascade I am rendering. Am I correct to assume that always cascades are rendered in ascending order and I can just have a static cascade index incremented (and looped) with each call of renderShadow?

Kind Regards,
Adrian L.

[silent]

Hi Adrian,

That's correct, shadow cascade are being rendered in the ascending order.

[adrian.licuriceanu]

Ok, thanks. And the followup question: how to get the current world light the cascade is being rendered for? I mean I need all these during an object extern renderShadow, so I need to call there light->getRenderShadowCascadeProjection for example. I was hoping that at least RENDER_CALLBACK_BEGIN_WORLD_SHADOW callback will have the light as parameter.

[adrian.licuriceanu]

Sorry, nevermind (Renderer::getCurrentLight() was the answer).