Sign in to follow this  
photo

2D Mask of objects in Screenshot

Recommended Posts

Posted (edited)

Hello,

I would like to have a 2D mask for each object in my screenshots (a mask being a binary 2d matrix with the same size as the image, with '1' values in pixels of the object and '0' elsewhere).

To do so I need to have the coordinates of all surface points of the objects in the scene when I take a screenshot. My objects are loaded from imported fbx models.

NodePtr testNode = World::loadNode(
"/home/ganoufa/unigine_projects/dev_project/data/models/car.fbx");
testNode->setEnabled(true);
testNode->setWorldPosition(Vec3(0., 0., 0.));

How can I get the coordinated of these surface points ? I saw the function "getVertex" that returns the coordinates of a vertex for a given surface of a mesh. My idea was to loop over all vertices of every surfaces of the mesh of my object and then convert these coordinates into 2D screen coordinates. But I can't find how to get the mesh associated with my node.

Is it a good way to do it ?

How can I get the Mesh associated with my node ?

Edited by Coppel.Damien

Share this post


Link to post

Hi Damien,

It looks like you can simply use auxiliary pass for the mesh_base material and fill it with whatever color you need:
auxiliary_sm.jpg
 

After that you need to grab the Aux buffer contents and that's it.

How to get access to the rendering buffers you can find in C++ samples in SDK Browser (Render -> GBufferRead).

Thanks!

Share this post


Link to post

Thank you for your answer.

I didn't fully understand the "for the mesh_base material and fill it with whatever color you need" part of your answer. From what I understand you are suggesting to define a unique color for each object (instead of white like in your image) in order to retrieve a unique mask for each object ?

If so: how do I define the color of an object in the auxiliary pass ?

 

Share this post


Link to post

It can be defined by texture or via simple vec4 color in parameters tab (please, make sure that Auxiliary state is enabled):

image.png

Share this post


Link to post

Dear @Coppel.Damien,

@silent has already explained the technical side of the mask generation using auxiliary buffer.

Quote

I would like to have a 2D mask for each object in my screenshots

Important Question: Your question seems that functionally you require every objects mask separately, doesn't it? If yes, let me know. I have implemented something like this recently with callbacks.

Here is a simple code how you will implement the G-Buffer  callbacks.

Render::addCallback(Render::CALLBACK_END_AUXILIARY_BUFFER, MakeCallback(this, &YourClass::render_custom_auxillary_mask));

.
.
.
.

void YourClass::render_custom_auxillary_mask(Renderer* _renderer)
{
	TexturePtr texture = _renderer->getTextureAuxiliary();

	//Write post filter here to do what you want to do with the mask.
}

Now, If answer to my question is No. Then you may get it to work with above approach.

  • Suppose there are 4 objects in your scene.
  • Setup material for all objects separately and give unique different color.
  • Use callback like above
  • In your implementation you can write post filter to separate out the objects based on color comparison.

If answer to my question is Yes then we will discuss the tricky part with the callbacks and you will receive separate masks for each object only. This will eat up little performance but best for dynamic implementation using component system. This will scale for sure. 

Rohit

Share this post


Link to post

Dear @silent, @rohit.gonsalves,

I have implemented a code with callbacks to get the auxiliary buffer.

I don't need every object to have a separate mask but only to know which pixels belong to which object in the image. Therefore the approach mentioned by Silent, that you detailed Rohit should work well. My interest in these masks is to train machine learning algorithm for object detection. Therefore having a precise mask of the labeled object instead of a bounding box to feed the algorithm as an input is very valuable.

Thank you for your help !

Guillaume Anoufa

 

 

Share this post


Link to post

Hi, why don't you try to use Material Mask? You can easily configure a bit for your '2d matrix' and is a lightweight version of auxiliary approach as uses Int instead of Float. 

Share this post


Link to post
Posted (edited)

Hello Javier,

Could you explain your idea with material masks please ? I don't understand how they could be used for my purpose.

Edited by Coppel.Damien

Share this post


Link to post
Sign in to follow this