[SOLVED] Loading time of terrain tiles

[Guest shane.ploenges]

Hi, after much hard work over the last few months, we have finally managed to be able to setup a very large tiled world in unigine. We are currently loading 60 tiles of 8193 x 8193 with Diffuse, Normal and mask maps of 2048 x 2048. (we switched to a 64 bit build to achieve this)

 

Once loaded it runs relatively smoothly when driving along the terrain at 350kmph, except for the occasional frame rate spike/drop - we will try tune this to improve it and remove these spikes as textures are currently uncompressed.

 

however our world loading time has gone up to 4-5 minutes now. Is there any way to speed this up? Is the only fix for us to create our own streaming system for terrain tiles? which are currently considering.

 

Thanks

 

Shane

[frustum]

Can you share several tiles of your terrain system? We need access to reproduction sample to remove the loading bottlenecks.

[Guest shane.ploenges]

How much would you need? currently our world contains 60 tiles and around 50gb of data - our netwrok cannot handle this. What would be the minmum amount of tiles you would like supplied?

[chao.zhang]

Hi shane,

 

would you please share the experiences about how to organize the 60 tiles in Unigine editor? Are these tiles managed by Node Sectors?

What would be the bottlenecks if to expand the numbers of terrain tiles to several hundreds?

[Guest shane.ploenges]

Hi Chao. We have a script that automatically imports our tileset that has been generated in World machine (a terrain generation software), we just click an import button - and the script does the rest.

 

We do not use node sectors as this would offer no benefit, Node sectors only manage mesh and texture data from clutter and grass, they do not have anything to do with terrain.

 

We have had to write several tools to manage our worlds with multi tiles, such as multi tile editing of detail paramenters. lods and flatness, as well as occluder parameters (these are also generated in the import process)

 

500km is the largest we have tried so far, however i am pretty sure this should scale up quite well, one concern we have is texture memory as we eventually we plan to go for a 1500km terrain, and potentailly more if a certain contract is won by us. and i am confident we can achieve this.

 

The main problem as stated in the original post is that we get intermettent pausing and frame drops when turning the camera, we are yet to try compressing our terrain data, so i will post an update once that is done.

 

Cheers

[Guest shane.ploenges]

Just to update this, we have fixed our framerate spikes and updates have also dropped considerabley - we did this by getting rid of coarse and normal textures per terrain surface, and only use one coarse normal and diffuse map per terrain tile (these are 2048 x 2048 for each 8193 x 8193 terrain tile.. we have 60 of these in our world)

 

The other problem as mentioned was loading - originally the loading bar would go very slow the whole way through and it took 5 minutes to load our world.

 

After we made this change the loading bar gets to the end in about a second - then waits for about 2 minutes , we found after removing or terrain occluders, this went away and the world now loads in about 10 seconds .. any idea why this would be the case?

 

i think we may ditch world occluders altogether if they have this hit on loading time.

 

its great to finally get our large terrain world working after loads of messing about.

[ulf.schroeter]

just a quick guess:  probably world occluders still load their mask on startup. I already proposed a different approach in a post on world streaming a year ago or two...if I remember right it was related to the fact that during world xml load there will be an internal call to setMaskFilename() causing a forced mask load even if mask will be immediately be paged out for far away tiles...but I might be wrong here, just dumping my brain asst the end of a tough week...

[Guest shane.ploenges]

End of a tough week, i feel you pain Ulf, we have been battling this large world terrain stuff for nearly a year now.. but at last i feel like we are getting closer.. i really wish we could show off some of the very cool tools our guys have made here at Sydac, we have pushed the Unigine terrain stuff well beyond its intended use and tools that have been made are really great, i would love to share them with the Unigine community, , however our company would never allow it :D

 

Thanks as always for your suggestion and input

[ulf.schroeter]

Thanks as always for your suggestion and input

 

You are welcome, and it seems that my brain still isn't too bad: this was the post referenced above. Have a look at "Additional dynamic CPU-side image resource management"...all problems e.g. with large loading times already mentioned nearly 2 years ago...In the meantime image and mask deferred background loading has been implemented, but I think it hasn't been done for WorldOccluderTerrain...it's a shame that UNIGINE improvements of streaming takes sooo much time..but I already spend far too much time arguing about it. Too bad that you cannot share t least some of your large world examples, would be very interesting

[binstream]

Shane, if you could share test content (please send a link to engine-support@unigine.com), it will help a lot. Without it there will be little improvements in base Unigine terrain code.

[ulf.schroeter]

Denis, it should be quite easy to produce some synthetic test scene via scripting because the actual texture/mask data doesn't really matter, it's only the size of the data ( 60 ObjectTerrain x ( 8192x8192 textures + 2048x2048 normal/specular) + 60 x WorldOccluderTerrain ) causing load delays. We did the same based on only one set of template textures/masks and simply copied that data to differently named file instances to simulate unique data instances.

[frustum]

Your problem is not related to the terrain streaming issue. You have a huge stall each time when you assign a height map image to the new WorldOccluderTerrain node. WorldOccluderTerrain uses cone step tracing internally. WorldOccluderTerrain performs calculation of cone step ratios for specified height map image when you assign a height map image in the non special formed RGB8 format. This step is very slow and should be performed only once at content generation time. After calculation step you can get RGB8 image from WorldOccluderTerrain and save it.

[ulf.schroeter]

@Manguste: I think it would be a good idea to add some clearifying comments on proper usage of WorldOccluderTerrain to the documentation to avoid this kind of problems (Also the documentation of setHeightsImageName still states older RG8 format). The implicit - time-intensive - data conversion from R8 heightmap to RGB8 cone-step data map is more or less undocument.

1. provisioning of R8 heightmap image with name x via WorldOccluderTerrain::setHeightsImageName()
   
2. internal WorldOccluderTerrain conversion to RGB8 cone-step data map
   
3. get this RGB8 image via WorldOccluderTerrain::getHeightsImage()
   
4. save this RGB8 image on disk with same name x to avoid time-consuming cone-step recalculation on next world load
   

[manguste]

Ulf,

Sure, thank you. Added into docs.