Jump to content

Best way to organise 30GB of content


photo

Recommended Posts

Hi,

Over the years we have been using Unigine, out data folder has grown to over 30GB in size. In previous versions of Unigine this worked fine, but starting with the update to I believe 2.9 and now 2.11 it seems out extensive content database causes a significantly higher memory usage. This seems to be at least partially due to Unigine scanning files and pre-loading materials, there are over 100,000 files in total and over 8,000 materials. Even though nothing is actually loaded, just the size of the C++ VirtualFile and Material structs multiplied by this many files takes up quite a bit of memory.

Now in our case this content will never all be used at the same now. More than half of the files are environments (over 80, although most of them are very basic and do not have a lot of files) and vehicles (about 400) and these numbers are only expected to rise further in the coming years. Users can only pick one environment to use though and typically only a dozen of different vehicles are used in the simulation.

Can we utilise mount points for this? Mount points have to be dynamically created if we want to avoid scanning the unused content, but when making a build they should still be included as we cannot know at build time which environment the user will pick. And when the user switches environment, previous environment should be unmounted and materials should be unloaded. And how would we deal with content that is shared between two environments?

Link to comment

Hello Bemined,

It's a good question, indeed.

We recently encountered similar issues with big projects and by now haven't come to a generic solution. However, there are possible options.

  1. Dynamic folder mount/unmount. This will save some loading time, disk space, and RAM but this makes caching impossible. This approach might cause FPS drops when a newly created object appears in the scene (mostly because of shader cache compilation).
  2. In 2.12 we're going to present revamped engine configuration that allows you to choose between 3 different cache modes:
    1. No caching. Here you'll get the fastest loading time, less RAM consumption and worse performance. By "worse" I mean possible hiccups with real-time shader cache generation
    2. Partial caching. This is the compromise between quality and resource consumption
    3. Full caching. This mode is the best for stable performance but is hungry for RAM.

Unfortunately, we can not provide a solution for partial asset library loading because of shader cache algorithm generation. Valid cache requires all materials to be known on the engine initialization stage.

For better frame rate all materials should be always loaded. Yes, this will cause bigger RAM consumption, bot this is the cost of the smooth performance. Other assets (meshes, nodes) can be loaded as mount points on demand.

Thanks.

How to submit a good bug report
---
FTP server for test scenes and user uploads:

Link to comment

Is the new caching the reason the memory usage is a lot more in 2.11? In 2.9 we had just over 2GB of memory usage in an empty world, in 2.11 it's already 4GB. And that is with our custom build for the engine, where we lowered some of the microprofiler buffer size as some older developer pcs could not handle it otherwise. When I tried the original development binaries from Unigine (without any of our custom plugins, and without any custom scripts) it went as far as 7GB for an empty world due to our massive content folder.

For environments, these are always selected when the simulation is paused so no smooth performance is needed here. Vehicles typically are added to the scene before starting the simulation as well, so in our case I think we can get away with a drop in FPS when mounting new content. Does caching take place as soon as you mount, or will there be a hiccup on first appearance in that case?

As for shader compilation, if we can provide end-users with a cache file, should that solve this? We can ensure the build computer has enough memory to mount all content and create a cache file when making a build, if this cache is then compatible with a build that only partially mounted content.

Link to comment
On 6/9/2020 at 9:03 PM, Bemined said:

Is the new caching the reason the memory usage is a lot more in 2.11?

Yes, that's the main reason we implemented new load options in 2.12. At the moment in the internal branch RAM consumption is almost equal to 2.10.

On 6/9/2020 at 9:03 PM, Bemined said:

And that is with our custom build for the engine, where we lowered some of the microprofiler buffer size as some older developer pcs could not handle it otherwise.

That's interesting. Our engine team would be glad to see how you optimized microprofiler, if you have time to share this, of course. Microprofilers RAM consumption depends on the threads count because for each core or thread it dumps required information. On modern 8+ cores CPUs the minimal RAM requirement for development build is quite big.

On 6/9/2020 at 9:03 PM, Bemined said:

Does caching take place as soon as you mount, or will there be a hiccup on first appearance in that case?

Mount excludes caching since caching can be performed only on the engine start. When you'll see a node from a mount it might cause a spike / hiccup.

On 6/9/2020 at 9:03 PM, Bemined said:

As for shader compilation, if we can provide end-users with a cache file, should that solve this?

Yes, shader cache file should improve this a lot.

Thanks!

How to submit a good bug report
---
FTP server for test scenes and user uploads:

Link to comment

Ok, thanks, I guess we'll check again after 2.12 comes out. Our upcoming release will be done using the previous Unigine version anyway.

  

55 minutes ago, morbid said:

That's interesting. Our engine team would be glad to see how you optimized microprofiler, if you have time to share this, of course.

It's not so much an optimisation, we just reduced the memory it's allowed to use by changing one of the defines. As a consequence we can see fewer frames into the past for heavy scenes, so it would be nicer if this can be changed on the fly. Now developers who do have a decent pc will also be stuck with fewer frames, upgrading all developer pcs takes time so we will probably have people working on computers with less ideal specs for quite some time.

Link to comment

We also experienced some issues with development build due to the higher memory consumption. Unfortunately, our devs didn't find a way to keep all data required for profiling with lesser resource drain.

I'd like to add that general trend we're following is "more RAM for caching — better performance in runtime". In the future SDK versions Unigine-based apps will be more and more hungry for RAM.

Thanks.

How to submit a good bug report
---
FTP server for test scenes and user uploads:

Link to comment
×
×
  • Create New...