Jump to content

[SOLVED] Lidar - segmentation labelling and reflectivity


photo

Recommended Posts

Hello,

I am using a custom version of the lidar sample to generate point clouds.

I have several questions regarding the generation of lidar data:

- Lidar point clouds usually come with a reflectivity value. I currently grab the albedo value and average it over R, G and B. Is there a way to assess the reflectivity more realistically?

- Is there a way to know which object every hit belongs to? I would like to generate a label for each point (X, Y, Z, I, Label).

- If this is too complicated, I would at least need to separate the ground from other objects placed on the ground (i.e. add a boolean label for each generated points, ground or not ground). Would this be possible?

Thanks!

Link to comment

Hi Coppel.Damien,

Is there a way to assess the reflectivity more realistically?
Please look at the "GBufferRead" sample (SDK->Samples->C++ API->Render->GBufferRead). You can get metallness and roughness values from each pixel on a render texture (void gbuffers_ready_callback(Renderer *renderer) and Render::CALLBACK_END_OPACITY_GBUFFER).
image.png
image.png

Best regards,
Alexander

Link to comment

- Is there a way to know which object every hit belongs to? I would like to generate a label for each point (X, Y, Z, I, Label).

- If this is too complicated, I would at least need to separate the ground from other objects placed on the ground (i.e. add a boolean label for each generated points, ground or not ground). Would this be possible?

You can try to use Auxiliary buffer for this:
image.png
image.png
Then for each type of object, set an individual color. For example: ground is red, everything else is green. Using the previous approach, you can get an auxiliary color from each pixel and interpret the result the way you want.

Link to comment

Thanks for your answers!

Regarding reflectivity, I already have the code to get metallness and roughness. I will check if there is a formula to assess reflectivity better using these values.

Regarding labelling I thought using of auxiliary buffer since I was already using that to label object in images with pixel masks.

I was wondering if there was a better way because sometimes the same material is used for different objects so it can cause trouble for the labelling.

Link to comment

I am struggling to get auxiliary value the same way I get albedo, depth, roughness etc...

I attached the .basemat file I modified to try to get auxiliary value but it doesn't seem to work. It works for albedo, depth and roughness but 'gbuffer.auxiliary' doesn't exist so I don't really know how to proceed.

I didn't find any helpful example in the sdk. Do you have a solution or should I do it with a Render::CALLBACK_END_AUXILIARY_BUFFER ?

fetch_buffers_post.basemat

Link to comment

Hi Damien,

The material that you've sent reads GBuffer's textures and does just a bit of trickery to clear alpha if there is nothing behind the object.
If you want to read auxiliary buffer you have to provide it to this shader.

RenderState::setTexture(RenderState::BIND_FRAGMENT, 4, renderer->getTextureAuxiliary());


Regarding this bit of code:

#ifdef STATE_AUXILIARY
	INIT_MRT(float4, OUT_AUXILIARY)
#endif

STATE_AUXILIARY is undefined for this material. I would suggest reading this useful documentation about states in ULON materials.

Also s_auxiliary_color is sent only for Water Global. If I understood this correctly you should read texture instead.
 

Link to comment
Posted (edited)

Hello,

For my tests I use a simple scene where the ground is covered with one simple texture with an auxiliary color set.

image.thumb.png.8d27b2b2ae833874e4fe23d77ab5ca04.png

I did add the suggested line to my code. But I still can't get the auxiliary buffer.

RenderState::setTexture(RenderState::BIND_FRAGMENT, 4, renderer->getTextureAuxiliary());

In order to grab the gbuffer values I need these lines:

viewport->setSkipFlags(~0);
viewport->setRenderMode(Viewport::RENDER_DEPTH_GBUFFER);
viewport->addCallback(Render::CALLBACK_END_SCREEN, MakeCallback(this, &Lidar::render_finished));

I tested to change the callback to:

viewport->addCallback(Render::CALLBACK_END_AUXILIARY_BUFFER, MakeCallback(this, &Lidar::render_finished));

But this callback is not called in RENDER_DEPTH_GBUFFER renderMode which explains why I can't get the auxiliary buffer.

Since I want to get the auxiliary texture which is not in the gbuffer I suppose I need to set RENDER_DEPTH_GBUFFER_FINAL  instead.
In this renderMode the END_AUXILIARY_BUFFER callback is called but nothing works anymore.

In RENDER_DEPTH_GBUFFER mode with CALLBACK_END_SCREEN everything else works fine (i.e. getting albedo, normal, roughness, metalness...) except getting the auxiliary buffer.

I attached the c++ file and the basemat files I use for the lidar part of my project. My code is heavily inspired by the GBufferRead sample and the lidar sample.

Do you have any idea on how to keep getting gbuffer values and grab auxiliary buffer at the same time ?

fetch_gbuffers.basemat Lidar.cpp

Edited by Coppel.Damien
Link to comment

Hi Damien,

I took a fast glance at your code.
You can add VIEWPORT_SKIP_POSTEFFECTS flag for skipping post effects and keep VIEWPORT_RENDER_DEPTH_GBUFFER_FINAL.
Also adding more skip flags like VIEWPORT_SKIP_SHADOWS, VIEWPORT_SKIP_VISUALIZER, VIEWPORT_SKIP_DYNAMIC_REFLECTIONS, VIEWPORT_SKIP_SRGB might help too.

Also i'd suggest moving this part:
 

    RenderState::clearStates();
    RenderState::setViewport(0, 0, render_size, render_size);
    render_target->bindColorTexture(0, depth_texture);
    render_target->bindColorTexture(1, albedo_texture);
    render_target->bindColorTexture(2, shading_texture);
    render_target->bindColorTexture(3, auxiliary_texture);
    render_target->enable();

and this one:

    render_target->disable();
    render_target->unbindColorTexture(0);
    render_target->unbindColorTexture(1);
    render_target->unbindColorTexture(2);
    render_target->unbindColorTexture(3);


Down into the render_finished callback for more redability and possibly fixing your issue.

Something along the lines:

{
	RenderTargetPtr render_target = Render::getTemporaryRenderTarget();

	RenderState::saveState();
	RenderState::clearStates();
	RenderState::setViewport(0, 0, render_size, render_size);
	render_target->bindColorTexture(0, depth_texture);
	render_target->bindColorTexture(1, albedo_texture);
	render_target->bindColorTexture(2, shading_texture);
	render_target->bindColorTexture(3, auxiliary_texture);
	render_target->enable();
	{
		RenderState::setTexture(RenderState::BIND_FRAGMENT, 0, Renderer::getTextureCurrentDepth());
		RenderState::setTexture(RenderState::BIND_FRAGMENT, 1, Renderer::getTextureGBufferAlbedo());
		RenderState::setTexture(RenderState::BIND_FRAGMENT, 2, Renderer::getTextureGBufferShading());
		RenderState::setTexture(RenderState::BIND_FRAGMENT, 3, Renderer::getTextureAuxiliary());
		RenderState::setTexture(RenderState::BIND_FRAGMENT, 4, Renderer::getTextureGBufferNormal());

		Render::renderScreenMaterial("fetch_gbuffers");
	}
	render_target->disable();
	render_target->unbindColorTexture(0);
	render_target->unbindColorTexture(1);
	render_target->unbindColorTexture(2);
	render_target->unbindColorTexture(3);

	RenderState::restoreState();
}

 

Link to comment
Posted (edited)

Well, I am not sure what happened but everything works! Doing what you suggested in the render callback seemed to fix it.

As for the skip flags they where all already enabled thanks to

viewport->setSkipFlags(~0);

But it seems they are no longer necessary for my code to work although I suppose it speeds up the render.

Thank you!

Edited by Coppel.Damien
  • Like 1
Link to comment
  • silent changed the title to [SOLVED] Lidar - segmentation labelling and reflectivity
×
×
  • Create New...