photo

Questions regarding YUV textures

Recommended Posts

Hi !

I am currently tring to create a custom material with a dynamic emission texture. In real time, three YUV textures will come from an external stream and shall be converted in code to one RGB texture which will then be applied to the material's emission texture.

I tried creating a custom material that references custom deferred and ambient shaders but the emission texture is never taken into account.

Am I doing it right ? As rendering is a process that I never really worked on I don't know if I'm going the right way or if there is an easier path to take that I missed.

Should I inherit my material from mesh_base ?

In order to be more precise for what I need :

  • A way to convert three YUV textures data to one RGB texture data in real time (custom shader?)
  • Apply the given RGB texture to the material's emission texture
  • All of this dynamically in C++ and UUSL for the conversion.

Thanks for your time !

Antoine

Share this post


Link to post
Posted (edited)

Dear @Antoine.YVAN,

I may help with all three of your needs. 

First question?.

You want to take three yuv textures and make one RGB texture, don't you?

I did not understand this. For me, every yuv texture should give you one RGB texture. What is the purpose of three yuv textures which are incoming? 

If this is known. I can guide you to write custom material, yuv to RGB with gamma correction, uusl shader.

And you can set it easily to emmision texture. 

Rohit

Edited by rohit.gonsalves

Share this post


Link to post

Hello @rohit.gonsalves !

Thanks for your answer. 

Let's just skip this three YUV textures thing.

Quote

 I can guide you to write custom material, yuv to RGB with gamma correction, uusl shader.

It would be much appreciated ! If, as you say, it is easy to set it to the emission texture I would absolutely love that.

Thank you for your time.

Antoine

Share this post


Link to post
1 hour ago, Antoine.YVAN said:

Let's just skip this three YUV textures thing.

It would be much appreciated ! If, as you say, it is easy to set it to the emission texture I would absolutely love that.

Dear @Antoine.YVAN,

YUV to RGB shader code...

float3 yuv2rgb(float3 YUV)
{
	float3 RGB;
	
	float2 ChromaClamped = YUV.gb - 128.0 / 255.0;
	float3 YCbCr = float3(YUV.r, ChromaClamped.g, ChromaClamped.b);
	RGB.r = saturate(dot(YCbCr, float3( 1.164,  0.000,  1.596)));
	RGB.g = saturate(dot(YCbCr, float3( 1.164, -0.392, -0.813)));
	RGB.b = saturate(dot(YCbCr, float3( 1.164,  2.017,  0.000)));

	return RGB; 	
}

Look at the weights. This is the most important factor while converting from YUV to RGB. YUV is a color difference model to RGB. And this color difference also depends on the physical property of the display you are using. It changes two things, Gamma correction for our eyes and Weights. 

For monitors it is sRGB. For TV displays with different frame rates and resolutions the weights differs with the standard BT.601, BT.709, BT.2020 and BT.2100 standards.

The above weights are good to for your tests.  We don't have any gamma correction in it till now. So we will do it later.

Can you write a custom material and UUSL to embed this code with incoming YUV textures?

Rohit

  • Like 1

Share this post


Link to post
Posted (edited)

Okay thanks for the code and explanations.

Quote

Can you write a custom material and UUSL to embed this code with incoming YUV textures?

I created the custom material and pasted the code into a .shader file.

Here are the content of my files

<?xml version="1.0" encoding="utf-8"?>
<base_material version="2.0" name="custom_post" editable="0">

	<shader pass="emission"
		defines="BASE_EMISSION"
		fragment="shaders/yuv.shader"/>
	
	<!-- Textures -->
	<texture name="albedo" pass="deferred" unit="0"/>
	<texture name="emission" pass="emission" unit="1"/>

	<supported node="object_mesh_static"/>
	
</base_material>
float3 yuv2rgb(float3 YUV)
{
	float3 RGB;

	float2 ChromaClamped = YUV.gb - 128.0 / 255.0;
	float3 YCbCr = float3(YUV.r, ChromaClamped.g, ChromaClamped.b);
	RGB.r = saturate(dot(YCbCr, float3( 1.164, 0.000, 1.596)));
	RGB.g = saturate(dot(YCbCr, float3( 1.164, -0.392, -0.813)));
	RGB.b = saturate(dot(YCbCr, float3( 1.164, 2.017, 0.000)));

	return RGB;
}

INIT_TEXTURE(1,TEX_EMISSION)


MAIN_BEGIN(FRAGMENT_OUT, FRAGMENT_IN)
	rgb = yuv2rgb(TEX_EMISSION.rgb);

	OUT_COLOR = rgb;

MAIN_END

Are they both correct ? I am not comfortable with shadering, especially on Unigine. 

Antoine

Edited by Antoine.YVAN
  • Like 1

Share this post


Link to post

Dear @Antoine.YVAN,

I rather thought it differently, like my product workflow, Where I do this conversion and save the texture in video memory and assign it to required surfaces using API. This approach is bad for editor work. As it will not work. 

If you want to put this thing to emission texture and with converted YUV texture, You need to keep the big flow of rendering sequence with materials, if you need it to work with other things.

As a first time creator of such material and shader I would have done following.

  1. Make a copy of data\core\materials\default\mesh\mesh_base.basemat and rename it like mesh_base_yuv_emmision.basemat
  2. Change fragment="core/shaders/mesh/emission/emission.shader" to some other shader like emissionYUV.shader
  3. Make a copy of \data\core\shaders\mesh\emission\emission.shader and rename it like emissionYUV.shader
  4. In emissionYUV.shader file locate  OUT_COLOR.rgb = TEXTURE_EMISSION(TEX_EMISSION).rgb * m_emission_color.rgb;
  5. Change this OUT_COLOR.rgb = TEXTURE_EMISSION(TEX_EMISSION).rgb * m_emission_color.rgb; to

rgb = yuv2rgb(TEXTURE_EMISSION(TEX_EMISSION).rgb);
OUT_COLOR.rgb = rgb  * m_emission_color.rgb;

And put the yuv2rgb method above.

I think this should work in editor. When you apply your material, Select emission state, and choose the YUV texture, The material will show RGB on the object.

Try this out and let us know. 

Beware, You need to think of sizing of pixels. YUV are half the size than RGB textures as from one YUV pixel we calculate two RGB pixels. Need to take care of that. But let it be, First see if you can achieve what I have described. 

Regards,

Rohit

  • Like 1

Share this post


Link to post

Good morning @rohit.gonsalves !

Thanks for your help,

I managed to follow the steps you advised me and it does work: I can see the emission texture updating with the new rgb !

I just have a few questions : will this be relevant for YUV420 with a tri plannar UV mapping ? Or is it only for YUV444 (or any other format)? I am a true rookie with the video world.

Regards,

Antoine

Share this post


Link to post

Dear @Antoine.YVAN,

The shader for conversion is good enough for all your needs. 420, 422 and 444.

The only thing would change is how you handle the pixel sizing. Check this https://en.wikipedia.org/wiki/Chroma_subsampling

YUV 420 is 1/4th bandwidth of RGB texture. 

YUV 422 is 1/2nd bandwidth of RGB texture

YUV 444 is exactly same bandwidth of RGB texture.

So how you manage your Tex coords to get correct value out of YUV texture sampling is important.

Hope this answers your question.

Regards,

Rohit

  • Like 1

Share this post


Link to post