NodeSector and ObjectTerrain

[sergey.shandar]

We are trying to load ObjectTerrain from a world file in background using NodeSector but it looks like it loads all terrain data instantly.

[ulf.schroeter]

ObjectTerrain has its own texture streaming logic. Not really sure about height field data streaming, but according to 2011-11-11 terrain loading speed should have increased significantly.

[Guest shane.ploenges]

We are trying to load 14 or so tiles of 4097 x 4097.. we do not want them all to load textures and Heightmaps at once but to be streamed in according to camera location. Currently when moving the camera around there is a significant pause when new tiles come in to view.

 

The video card memory usage also seems to suggest that we are hitting the limit, so it seems there is no streaming going on, but all textures and heightmaps loaded at once causing paging from disk to occur.

 

Are we doing something wrong to get node sectors to load and unload terrain tiles and their maps? Or is this not what node sector is designed for (The documentation and example is very limited unfortunarely)

 

If Node sector is not the correct method, are there any other suggestions for having many terrain tiles in a world loading and unloading to keep memory usage to a decent level?

[ulf.schroeter]

Shane, first of all there is no "interconnection" between NodeSector and ObjectTerrain data streaming.

 

NodeSector only streams

• ObjectMesh mesh resources
  
• Object material textures
  
• ObjectGrass and WorldClutter mask images
  

of child nodes. At the moment there is no streaming for WorldOccluderMesh mesh resources, nor WorldOccluderTerrain mask images. It should be quite easy for Unigine to include streaming for these objects, which might be helpful for really large worlds.

 

ObjectTerrain on the other hand implements its own data streaming. As far as I understand the source code coarse height and texture data of ObjectTerrain instance is always loaded on startup and kept in memory. For terrain surfaces textures will be streamed asyounchronously, but surface height data still will be read dynamically from disk by a blocking read (so no real background streaming at the moment). That might be the reason for encountered render stalls on camera movement.

 

Secondly, what do you exactly mean when talking about terrain tiles (as Unigine uses terrain coarse, surface and patch terminology) ? Are you using multiple ObjectTerrain instances in your world ?

[Guest shane.ploenges]

Hi Ulf, thanks for the info.

 

Yes we are using multiple terrains in our world - our current test world contains 14 4097 x 4097 terrains, these were generated in World machine and cover roughly a 30km area. This is a small area for one of our simulators, and is for testing purposes only - when we start building actual worlds, we will be making terrains of hundreds or even thousands of of Kilometers in size.

 

So for our purposes it seems The only way we will get Unigine to work with our simulators for large worlds will be to have some sort of background streaming of terrain tiles, so our current task is to try implement this. We thought node sector would do the job, however you have pointed out it does not, so i am really unsure what we can do now.

 

The current performance of using this system is not good at all, so We may need to speak to unigine team and request some better terrain streaming features, or we cannot do what we thought with building large worlds based on multiple terrain tiles.

 

 

 

Currently our terrain tiles contain the following:

4097 x 4097 height map

2048 x 2048 Diffuse map

2048 x 2048 normal map

1024 x 1024 detail map mask

3 x 2048 x 2048 detail textures

 

We do not even have any grass or world cluster added yet, so as you can see multiplying all these maps by 14 tiles in our world means a huge memory consumption. Our plan was to have a master node sector for each tile and when deactivated according to camera position all related children assets including terrain would by unloaded from memory.

 

This is going to be a real problem for us and i hope we can reach a solution quickly.

[ulf.schroeter]

Actually you're facing the same problems with large database support as we did in the field of simulation. I would expect that you will run into more troubles with your current multi-instance ObjectTerrain approach

 

Besides discussed streaming/memory issues ObjectTerrain is not designed to provide crack-free stiching between multiple ObjectTerrain instances. Actually ObjectTerrain does not even guaranty crack-free transition between patch-surface-coarse transitions of a single instance (see this post, where we encourtered these problems while trying to optimize performance for large view distances).

 

Unless Unigine is wiling to re-design ObjectTerrain to fully support single-instance, full-fledged large-world streaming usage of mesh-based TIN terrain tile representation with mesh_terrain_base material might be an alternative. While losing automatic distance-based mesh LOD tesselation at least this would allow you to use NodeSector mesh and texture streaming for your individual world tiles.

 

As already discussed in another thread I would expect that smaller tile texture sizes (e.g 1x1k) / higher total tile count for your world would be advantageous for more fine grained resource allocation and to avoid render stalls due to massive GPU texture uploads in single frames.

 

Nevertheless - as you are already anticipating - there are more problems with large worlds ahead of you. With your targeted database sizes the huge amount of required Nodes for non-terrain object representation for dense worlds will quickly become the next bottleneck. That's why we have asked Unigine for a real tiled world background node file streaming capabilty for a very long time.

[ulf.schroeter]

Here are some more details on our large world streaming scenarios which we have discussed with Unigine (I am quite sure that you will most probably have quite similar requirements). Some basic features (double-precision support, basic NodeSector streaming) have found their way into the engine, but still there are quite some more challenges.

 

 

1. Use cases, database sizes and structure

 

1.1 ground warfare simulations

 

Typical database sizes are between 50x50km and 100x100 km. Databases are typically organized as 1x1km to 3x3km regular tiles. Details are spread uniformly in the database. Typical viewing distance 8km.

 

1.2 naval simulations

 

Typical database sizes are between 80x80km and 300x300km. Databases are typically organized as 5x5km to 10x10km low resolution base terrain mesh tiles with high detail ground structures (buildings, constructions, trees) organized as 1x1 to 3x3km overlay tiles along navigable coastlines, rivers or habour areas. Typical viewing distance 30km for low resolution terrain meshes and 5-8km for high detail ground structures.

 

1.3 flight simulations

 

Typical database sizes are between 200x200 and 1000x1000km. Databases are typically organized as 10x10km regulare tiles with high-detail ground insets (e.g. airports, training areas e.g shooting ranges). Terrain meshes with multiple LOD surfaces/textures. Typical viewing distances 30km.

 

 

In general for all use cases there is some kind of regular gridded world base tile structure covering the complete database and multiple localized, more fine-grained gridded or irregular placed world detail tiles overlay structures.

 

 

2. Scene content

 

Scene content mainly consists of

 

2.1 ground surface meshes

 

Individual ObjectMesh instances per zone/tile each with individual diffuse, detail material mask and additional ground type mask (e.g for type specific dust particles and step sound selection) per zone/tile

 

2.2 vast forest areas

 

Individual ObjectForest instances per zone/tile but shared tree resources for whole database. See https://developer.un...__fromsearch__1 for details

 

2.3 ground clutter

 

Individual ObjectGrass/WorldClutter instances for multiple grass, corn fields, rock layers each with individual mask images per zone/tile

 

2.4 roads, streets

 

Individual ObjectMesh instances per zone/tile as overlay on base ground surface mesh. Generic materials (100+) can be shared for whole database

 

2.5 urban building and constructions

 

Individual NodeReferences per zone/tile to a pool of 200-300 building templates for whole database.

 

Each building is modelled as ObjectMesh in undestroyed, partially and full destroyed state with 2-3 LOD levels for each destruction state. Referenced building node files can also contain additial ObjectLights (e.g. street latern), ObjectParticles (e.g. smoke pipe), ObjectCloth (e.g. flags) and WorldExpressions (e.g. procedural animation of movable parts (e.g wind mill, flap bridges, etc)

 

 

3. General functional requirements

 

3.1 no render stalls caused by background zone streaming

 

3.2 radial streaming around world streaming reference point

 

To avoid popping artifacts on fast user direction changes radial pre-caching of tiles/zones based on user-definable zone cache/purge distances around streaming reference point should be implemented. In general cache-in distance should be larger than render distance. Purge-out distance should be larger than cach-in distance to avoid resource unloading/loading-ping-pong.

 

Most of the time streaming reference point will be player position, but for some application-specific streaming scenarios (see 3.3) explicit control about streaming reference point(s) might be required.

 

3.3 support for multiple world streaming reference points

 

Required for multiple independant viewports of a single client or server-side resource streaming for multiple connected clients.

 

3.4 customizable background processing on zone load/unload

 

This might be a critical requirement. In nearly all cases additional game worls processing has to be done on world zone loading/unloading (e.g. initialisation/destruction of corresponding game objects). This processing might require traversal of newly loaded node tree e.g. for game object properties retrieval, creation of additional game data structures etc.

 

As this processing might take sigificant time most probably it has also to be done within background resource loading thread to avoid render stalls. Most natural solution might be background invocation of user-defined WorldZone callback functions e.g. WorldZone::setLoad-/UnloadCallback().

 

 

4. Design issues

 

4.1 streaming organization

 

Most probably a new node type WorldZone controlling streaming/purging of attached child nodes might be useful. WorldZone node data will be stored in separate node file like WorldLayer. WorldZone definable cache-in/purge-out distances on top of aggregated child-node bounding box sizes and render distances can be used controlling streaming behavior. WorldZone states unloaded, loading, loaded, unloading. Automatic callback invocation of onLoad/UnloadCallback() for customized game data preparation.

 

Usage of both WorldZone and WorldLayer would give quite flexible spatial and logical organization possibilities for large worlds.

 

4.2 streaming order and priorities

 

There should be some priority strategy for loading resources. Resources in view should be loaded with highest priority based on distance from streaming reference point. Resources outside the view frustum should be loaded with lower priority, but still there should be a prioritization based on viewer distance and resource position relative to view frustum orientation. Resources nearer to view frustum forward direction should be loaded earlier than resources in back of the viewer.

 

4.3 streaming data format

 

Usage of standard XML or binary format for better streaming/processing efficiency ?

 

4.4 per-frame time-budget management for resource preparation/destruction

 

How to limit per-frame resource (de-)allocation within render thread to avoid render stalls ?

 

4.5 editor handling of streamable worlds

 

How can streamable worlds be handled within editor (as only a certain part of the world will fit into memory) ? Maybe best approach might be to load WorldZones based on editor player position distance and dynamically stream-in/purge WorldZones on editor player movement. But how should situation best be handled when WorldZone child structure has been modified, editor player moves an modified, but unsaved WorldZone would be paged out without prior saving ? Maybe some more explicit user-initiated WorldZone editing approach comparable to NodeReference editing would be more suitable.

 

 

Based on our experiences it will be quite hard to realize interactive large worlds without these features (and be sure there are even some more requirements not covered here).

[Guest shane.ploenges]

Ulf, once again thanks for the detailed reply - Unigine should be paying you since you seem to be the only one doing any support around these forums anymore, i cant remember the last time there was an official and detailed response from anyone in Unigine support to one of my questions.

 

this indeed seems like very bad news for us, considering we purchased unigine on the understanding that it would have large world support, it seems this is not the case and we will not be able to do what we initially thoiught possible with Unigine.

 

We cannot move to a mesh based terrain as we need the ability to edit Height maps and terrain in the editor. We have come from a mesh based terrain in our previous engine and it proved too unweildy for our world sizes and editing.

 

Wwe will do some testing using smaller tile sizes, but i think this will not help as we will still be loading the same amount of height map data.

 

Thanks again for your help - hopefully we can get some answers or suggestions from unigine team themselves, but i think our best solution would be to have ObjectTerrain working as part of Node Sectors, with loading and unloading of its associated heightmaps textures and Mask.

[ulf.schroeter]

Unigine should be paying you since you seem to be the only one doing any support around these forums anymore, i cant remember the last time there was an official and detailed response from anyone in Unigine support to one of my questions.

 

Great idea ! I have to admit that my personal enthusiasm of spending my free time here is more and more fading...its a pity that Unigine does not really realize the importance and power of the forum for their own sake

[manguste]

We are really, really sorry for that. There was a severe crunch caused by OilRush release: actually, most of our team worked without any weekends or holidays (3 days to celebrate the New Year and that's it) for two months. It just ate up all of our human resources :( We are trying to make up for it right now.

 

Ulf, stay with us, your help is really invaluable :)

[binstream]

Shane, we are in contact with Dave Lannan from Sydac on your project, at the moment we are still waiting for your assets being uploaded (Dave said they will be ready by tomorrow).

As we discussed with him a week ago over email, we will work on engine optimizations for your large terrain scenario once the assets will be delivered.

 

Having your real-life assets on hands speed up the process significantly, we did it several times before with other customers.

[Rob]

Shane, we are in contact with Dave Lannan from Sydac on your project, at the moment we are still waiting for your assets being uploaded (Dave said they will be ready by tomorrow).

As we discussed with him a week ago over email, we will work on engine optimizations for your large terrain scenario once the assets will be delivered.

 

Having your real-life assets on hands speed up the process significantly, we did it several times before with other customers.

 

Will these engine optimizations be released for everyone?

[binstream]

Ulf, we are really sorry about being quiet here lately.

 

As Nadezhda mentioned, we were in the end of 2.5 years long project (Oil Rush), which consumed all of our resources. The good thing is that it allowed us to understand and optimize a lot in terms of performance and game systems. Another good thing is that now the project is released and we have much more resources for the engine improvements.

 

There is no way to make improvements in the technology without working on the project of some type or having access to assets of some specific project. That's why we ask full content from our customers to make helpful improvements; otherwise they are way too synthetic (like recently implemented zone-based loading) and useless. Unfortunately, a lot of customers are afraid of providing access to their projects (even having mutual NDA incorporated into Unigine licensing agreement), so it makes the process very hard.

 

Anyway, at the moment we are optimizing the engine for flight simulators because we have such content on hands. Later this week Sydac will provide their detailed requirements with the large terrain content, so we'll be able to proceed with it as well.

[binstream]

Will these engine optimizations be released for everyone?

 

Yes, they will.

[ulf.schroeter]

Denis, first of all congratulations to the whole team for successful release your OilRush game, hope it will be a big commercial success ! 100 % with you that such real-world project work is by far the best way to improve the engine functionality and performance. Also great news that the Sydac guys will provide some huge dataset for improving large world support. Still I am absolutely sure that this will be one of the best unique selling points for Unigine compared to other engines, even the big-players.

[binstream]

Ulf, thank you for your kind words and your really invaluable help!

We'll do our best to deserve such trust with upcoming SDK updates this year.

[manguste]

As mentioned in another topic, NodeSector and terrain do not have anothing to do with each other. They have different streaming logic: terrain streaming is automated (and will be improved to support background streaming around the camera), while NodeSector is for supporting custom streaming logic of nodes (ObjectMeshes, material textures, grass and clutter textures).

 

The general recommendations for terrains are as follows:

• Create a large terrain - small terrains will have problems with seams at the border.
  
• The larger the Step the better! If the step is too small and the terrain is large, performance penalty is inevitable. For example, Valley that is yet to be released uses 8193x8193 terrain with step 1 (1 step is aprox. equal to 1 m). Relief details can be added via a normal map.
  
• Do not set Lods distance (the distance to switch on the first LOD) too high - the smaller it is, the higher the performance would be (especially in case of mostly flat or moderate relief - there's no visual difference while surfaces loading becomes much, much faster).
  

[ulf.schroeter]

Do not set Lods distance (the distance to switch on the first LOD) too high - the smaller it is, the higher the performance would be (especially in case of mostly flat or moderate relief - there's no visual difference while surfaces loading becomes much, much faster).

 

Unfortunately setting Lods distance to smaller values for improving performance might also cause cracks as described in this post.

[Rob]

As mentioned in another topic, NodeSector and terrain do not have anothing to do with each other. They have different streaming logic: terrain streaming is automated (and will be improved to support background streaming around the camera), while NodeSector is for supporting custom streaming logic of nodes (ObjectMeshes, material textures, grass and clutter textures).

 

The general recommendations for terrains are as follows:

• Create a large terrain - small terrains will have problems with seams at the border.
  
• The larger the Step the better! If the step is too small and the terrain is large, performance penalty is inevitable. For example, Valley that is yet to be released uses 8193x8193 terrain with step 1 (1 step is aprox. equal to 1 m). Relief details can be added via a normal map.
  
• Do not set Lods distance (the distance to switch on the first LOD) too high - the smaller it is, the higher the performance would be (especially in case of mostly flat or moderate relief - there's no visual difference while surfaces loading becomes much, much faster).
  

 

How would you add relief details with a normal map? Are we talking about the detail materials? Or an extra normal map?

[manguste]

It was about the basic normal map generated from a height map. It's just a possible way in case necessary. It can be done only manually (basically, by creating the terrain twice and then manually replacing lower resolution normal maps with higher resolution ones).

[Rob]

It was about the basic normal map generated from a height map. It's just a possible way in case necessary. It can be done only manually (basically, by creating the terrain twice and then manually replacing lower resolution normal maps with higher resolution ones).

 

Wouldnt it be possible to add an option for a custom normalmap? One that isnt generated by the height map

So it would work the same as the diffuse map and regenerate all the 256x256 textures

[manguste]

Yep, we've already added this one. Now a normal map can be imported just like any other texture (and it works with WM normal maps without any problems as far as we checked).

[v.sergeev]

Here are some more details on our large world streaming scenarios which we have discussed with Unigine (I am quite sure that you will most probably have quite similar requirements). Some basic features (double-precision support, basic NodeSector streaming) have found their way into the engine, but still there are quite some more challenges.

 

 

1. Use cases, database sizes and structure

 

1.1 ground warfare simulations

 

Typical database sizes are between 50x50km and 100x100 km. Databases are typically organized as 1x1km to 3x3km regular tiles. Details are spread uniformly in the database. Typical viewing distance 8km.

 

1.2 naval simulations

 

Typical database sizes are between 80x80km and 300x300km. Databases are typically organized as 5x5km to 10x10km low resolution base terrain mesh tiles with high detail ground structures (buildings, constructions, trees) organized as 1x1 to 3x3km overlay tiles along navigable coastlines, rivers or habour areas. Typical viewing distance 30km for low resolution terrain meshes and 5-8km for high detail ground structures.

 

1.3 flight simulations

 

Typical database sizes are between 200x200 and 1000x1000km. Databases are typically organized as 10x10km regulare tiles with high-detail ground insets (e.g. airports, training areas e.g shooting ranges). Terrain meshes with multiple LOD surfaces/textures. Typical viewing distances 30km.

 

 

In general for all use cases there is some kind of regular gridded world base tile structure covering the complete database and multiple localized, more fine-grained gridded or irregular placed world detail tiles overlay structures.

 

 

2. Scene content

 

Scene content mainly consists of

 

2.1 ground surface meshes

 

Individual ObjectMesh instances per zone/tile each with individual diffuse, detail material mask and additional ground type mask (e.g for type specific dust particles and step sound selection) per zone/tile

 

2.2 vast forest areas

 

Individual ObjectForest instances per zone/tile but shared tree resources for whole database. See https://developer.un...__fromsearch__1 for details

 

2.3 ground clutter

 

Individual ObjectGrass/WorldClutter instances for multiple grass, corn fields, rock layers each with individual mask images per zone/tile

 

2.4 roads, streets

 

Individual ObjectMesh instances per zone/tile as overlay on base ground surface mesh. Generic materials (100+) can be shared for whole database

 

2.5 urban building and constructions

 

Individual NodeReferences per zone/tile to a pool of 200-300 building templates for whole database.

 

Each building is modelled as ObjectMesh in undestroyed, partially and full destroyed state with 2-3 LOD levels for each destruction state. Referenced building node files can also contain additial ObjectLights (e.g. street latern), ObjectParticles (e.g. smoke pipe), ObjectCloth (e.g. flags) and WorldExpressions (e.g. procedural animation of movable parts (e.g wind mill, flap bridges, etc)

 

 

3. General functional requirements

 

3.1 no render stalls caused by background zone streaming

 

3.2 radial streaming around world streaming reference point

 

To avoid popping artifacts on fast user direction changes radial pre-caching of tiles/zones based on user-definable zone cache/purge distances around streaming reference point should be implemented. In general cache-in distance should be larger than render distance. Purge-out distance should be larger than cach-in distance to avoid resource unloading/loading-ping-pong.

 

Most of the time streaming reference point will be player position, but for some application-specific streaming scenarios (see 3.3) explicit control about streaming reference point(s) might be required.

 

3.3 support for multiple world streaming reference points

 

Required for multiple independant viewports of a single client or server-side resource streaming for multiple connected clients.

 

3.4 customizable background processing on zone load/unload

 

This might be a critical requirement. In nearly all cases additional game worls processing has to be done on world zone loading/unloading (e.g. initialisation/destruction of corresponding game objects). This processing might require traversal of newly loaded node tree e.g. for game object properties retrieval, creation of additional game data structures etc.

 

As this processing might take sigificant time most probably it has also to be done within background resource loading thread to avoid render stalls. Most natural solution might be background invocation of user-defined WorldZone callback functions e.g. WorldZone::setLoad-/UnloadCallback().

 

 

4. Design issues

 

4.1 streaming organization

 

Most probably a new node type WorldZone controlling streaming/purging of attached child nodes might be useful. WorldZone node data will be stored in separate node file like WorldLayer. WorldZone definable cache-in/purge-out distances on top of aggregated child-node bounding box sizes and render distances can be used controlling streaming behavior. WorldZone states unloaded, loading, loaded, unloading. Automatic callback invocation of onLoad/UnloadCallback() for customized game data preparation.

 

Usage of both WorldZone and WorldLayer would give quite flexible spatial and logical organization possibilities for large worlds.

 

4.2 streaming order and priorities

 

There should be some priority strategy for loading resources. Resources in view should be loaded with highest priority based on distance from streaming reference point. Resources outside the view frustum should be loaded with lower priority, but still there should be a prioritization based on viewer distance and resource position relative to view frustum orientation. Resources nearer to view frustum forward direction should be loaded earlier than resources in back of the viewer.

 

4.3 streaming data format

 

Usage of standard XML or binary format for better streaming/processing efficiency ?

 

4.4 per-frame time-budget management for resource preparation/destruction

 

How to limit per-frame resource (de-)allocation within render thread to avoid render stalls ?

 

4.5 editor handling of streamable worlds

 

How can streamable worlds be handled within editor (as only a certain part of the world will fit into memory) ? Maybe best approach might be to load WorldZones based on editor player position distance and dynamically stream-in/purge WorldZones on editor player movement. But how should situation best be handled when WorldZone child structure has been modified, editor player moves an modified, but unsaved WorldZone would be paged out without prior saving ? Maybe some more explicit user-initiated WorldZone editing approach comparable to NodeReference editing would be more suitable.

 

 

Based on our experiences it will be quite hard to realize interactive large worlds without these features (and be sure there are even some more requirements not covered here).

Ulf, you are absolutely right!

 

On the first iteration we used ObjectTerrain, afterwards refused it. We implement our custom rendering algorithm based on ObjectMesh (there are many problems with good texturing GIS database)

 

Next step - streaming.

And we rested against a blank wall. It is necessary to make a lot number of changes in an engine code

[binstream]

Streaming improvements, a plugin for multi-terrain databases handling and per-frame generation budget have been added in the recent SDK update.