Questions about 2.8 data streaming.

[sebastian.vesenmayer]

Hi,

we have migrated to 2.8 version of the engine and I have some questions about some changes.

Our start up time increased noticeable. Is there a connection with the data streaming setup and cache?

Is it possible to disable unloading of streamed textures which where already loaded?

Streaming from HDD is really slow, is a SSD mandatory for your engine now ?

Is it possible to force resource upload into VRAM? We are observing many spikes when moving the camera arround in the microprofiler when textures are not loaded and view direction moves again to highpoly/material count area.

Thanks in advance for your help.

Regards

Sebastian

[morbid]

Hello Sebastian,

This microprofile capture from the OpenGL, right? If so, there's additional settings that should be tweaked for GL.

I suggest increasing Async Buffer and Async Buffer Indices size and checking how it will affect scene performance.

All settings are described here: https://developer.unigine.com/en/docs/2.8/editor2/settings/render_settings/streaming/ You also can change them in the console: https://developer.unigine.com/en/docs/2.8/code/console/#render_resources

12 hours ago, sebastian.vesenmayer said:

Is it possible to disable unloading of streamed textures which where already loaded?

You can control time for resources cleanup, in frames.

13 hours ago, sebastian.vesenmayer said:

Streaming from HDD is really slow, is a SSD mandatory for your engine now ?

SSD is preferable, but it should be fine with HDD. I suggest configuring streaming for your build.

13 hours ago, sebastian.vesenmayer said:

Is it possible to force resource upload into VRAM? We are observing many spikes when moving the camera arround in the microprofiler when textures are not loaded and view direction moves again to highpoly/material count area.

There's force mode for streaming which loads every resource required for the current frame, check the settings I mentioned above.

We can help you to configure streaming if you'll send us a scene.

Thank you.

[sebastian.vesenmayer]

Yes morbid, it is OpenGL.

4 hours ago, morbid said:

You can control time for resources cleanup, in frames.

Can I change it to unlimited frames? The limit in the manual is 300 now.

4 hours ago, morbid said:

There's force mode for streaming which loads every resource required for the current frame, check the settings I mentioned above.

I was already using this option for the screenshots. Is it possible to force upload when a node is loaded or created and not at the first time of usage?

I have narrowed the problems with the spikes from the second picture and it seems that decal meshes and the world light shadow are the problem, when all geometry and texture data is loaded into VRAM.

When I am moving high into the air and the shadow distance has no more geometry to render the "Buffer Shadows" usage changes to 0 MB
When moving down, it takes some time till the Buffers Shadow is created again. Do you have a setting for this?
Turning of shadows results in no spikes.

When Decal meshes go out of sight and move into the Viewport again, there is also a high spike.
Disabling decals results in no spikes.
We are using arround 35 decal meshes, which are projected on an 2000 x 10000 units area.

Thanks

 

[morbid]

1 hour ago, sebastian.vesenmayer said:

Can I change it to unlimited frames? The limit in the manual is 300 now.

Have you tried increasing memory limit? These settings are on the same page.

When you see a texture/mesh its being loaded to the VRAM. Memory will be filled until the specified limit and only after this point resources will be unloaded. So if you want to keep as many resources as you want - increase the limits.

1 hour ago, sebastian.vesenmayer said:

I was already using this option for the screenshots. Is it possible to force upload when a node is loaded or created and not at the first time of usage?

I'm afraid there's no such option. All data supposed to be smoothly streamed.

1 hour ago, sebastian.vesenmayer said:

I have narrowed the problems with the spikes from the second picture and it seems that decal meshes and the world light shadow are the problem, when all geometry and texture data is loaded into VRAM.

It sounds like you've facing problems with video memory consumption. OpenGL reacts really badly when you run out of memory. Can you provide additional info:

1. GPU model and driver version
2. Full microprofile html dump with spikes
3. VRAM consumption. You can use our profiler (show_profiler 2) or other monitoring software

Thank you.

[sebastian.vesenmayer]

Hi morbid,

I was able to force loading of a nodes texture by creating a temporary Image object out of the textures of a material.

But I could not get rid of the spikes which are produced when decals appear again int the viewport or moving into shadow map distance again.

I am using a gtx 1070 with 8 GB vram with driver version 430.39

GPUZ shows 6791 MB vram usage.

Added the microprofile dump for shadows.

Reducing the vram usage did not help to prevent unloading of shadow texture.

Thanks

 

MicroProfile Capture.html

Edited September 19, 2019 by sebastian.vesenmayer

[silent]

Hi Sebastian,

Sorry for the late reply.

Is it possible to get an actual test scene for reproduction? I can see the spike on microprofile, but we need to attach to a running application and debug on our side to get more info.

Maybe as a temporary solution you can simply try to force update the nodes, so they will be never unloaded and will keep textures in memory?

In general OpenGL + multithreading streaming not working as good as we expect (unlike DX11 version which works great) due to multiple API limitations and drivers issues. Also GL driver starts to work very bad when you are getting close to VRAM limits (unlike DX11). Checking the test scene will give us a chance to see what else we can do here.

Btw, switching to DX11 API is not planned for your application in the nearest future? It would give you additional performance and overall stability will increase significantly.

Thanks!

[sebastian.vesenmayer]

Hi silent,

I can setup a project,

Nodes are fine now, the problem is the deletion of renderbuffers and rendertextures on the engine side. Every time decals/shadows/water get out of viewpoint the resources get released, and it takes a lot of time to allocate them again for 4K. Also this can fragmentate the vram, which can lead to other problems.

We are not using multithreading streaming now, beacause it is to slow.

Thanks

Edited September 24, 2019 by sebastian.vesenmayer

[silent]

It would be great to take a look at your typical scene setup, if possible.

[sebastian.vesenmayer]

This little Testscene with opengl and a ObjectWaterMesh shows the problem of releasing renderbuffers.

This will show the problem with Shadow map release, shadowmap distance is 10 units in this example.

And this is the problem with the decal mesh renderbuffer, with only one decal mesh projected.

I hope this will help you

 

Edit: For shadow maps it can be observed also with directx but not as heavy as with opengl, it scales with memory size needed.

Edited September 24, 2019 by sebastian.vesenmayer

[sebastian.vesenmayer]

DirectX is no option for us now. We have chosen the engine because of OpenGL compatibility.

We need a fix for this, this is definitely a show stopper for us.

Edited September 25, 2019 by sebastian.vesenmayer

[sebastian.vesenmayer]

Hi,

any information what your plans are to get rid of this problem?

Thanks

[silent]

Hi Sebastian,

These spikes are for sure should not be existed in the application.Our dev team needs some additional time for research the possible ways of fixing them.

We will have more spare time after 2.10 SDK Release.

Spikes are caused mostly by the fact that multi-threaded GL context exists (even if render_streaming_mode is set to 1).

So right now the possible solutions are:

1. Use 2.7.3 SDK version
2. Switch to DX11 render inside GL application (maybe we can help you in that direction)

Possible solutions for 2.10+ SDKs:

1. Rewrite streaming pools so we can explicitly say to the specific object to never unload texture (will more likely cause crashes after a long period of time)
2. Spend unpredictable amount of time debugging vendor's GL drivers until some workaround will be found (that, of course, will stop working with a new driver update)

We can't remove multi-threading GL context due to the nature of new developed ObjectTerrain that simply will not work without it.

No matter what we will contact GPU vendors and provide them a basic test scene so they can find out the root cause of this issue. Unfortunately, vendors doesn't pay much attention these days to the OpenGL bugs fixing because there are plenty new APIs (like Vulkan or DX12 / DX11 which are being used by the majority of customers).

Thanks!