[SOLVED] Terrain streaming

[Rob]

We are working on a flight.race game and we are wondering how streaming works, in another thread it is mentioned that it is possible to have node sector streaming and tiled import for terrains but how does this exactly work? We will be using WM2 Pro so we have access to tiles and multiple extents.

 

So what it the best way to create a huge terrain and divide it into multiple objects and then stream them in Unigine to get the best performance possible?

I would like to know what other people have achieved in Unigine and maybe this thread can be used as a knowledge base about streaming

[Guest shane.ploenges]

A Possible Solution (and this is only from an Artist point of view, so dont ask me how it works technically)

-Generate tiles in WM (including Height, Diffuse Mask and Normal Maps)

-In Unigine you will need to script something to generate Node sectors and under these sectors generate Terrain tiles and apply all the relevent Maps and settings. You will need to calculate an offset for your tile as Unigine terrain treats the 0,0 point as the bottom left corner (incidentaly so does WM)

-Keep in mind you need to tick the Flip y tick box in WM and make sure also that you ad +1 to your tile size in the "custom" button for the WM tile size settings.

 

I hope this gives you a starting point - i cannot comment on how the loading/unloading of these node sectors works, but i believe it is a process you need to setup yourself.. in other words Unigine does not do it automatically.

[Guest shane.ploenges]

If people would find it usefell i am happy to do a tutorial on generating a network in WM and outputting tiles for different maps from it - this is something i have been doing a lot over the last few days.

[steve3d]

Please post this tutorial, I really want to know how to do this. Thanks

[Rob]

If people would find it usefell i am happy to do a tutorial on generating a network in WM and outputting tiles for different maps from it - this is something i have been doing a lot over the last few days.

 

I would love to see a tutorial, I already have a network in WM and it exports tiles but I just started working with it so im sure I could learn a lot from this

 

Today ive been testing tiled output from WM in Unigine and I noticed when using a 513 terrain with a 513px heightmap I see very noticable seams on where 1 terrain edge meets the next terrain edge

When using a heightmap of 1025px the seam becomes less noticable but it is still there, does anyone know how to get rid of these seams?

[Guest shane.ploenges]

What level of tile blending are you using in WM? and also are you using the weld verts option in WM?

 

These 2 things can add a ton of time to the render time, but give better results in the end.

 

This brings me to another point about tiled worlds, and i wonder if anyone has ideas on this :

 

When editing heights or detail textures across tiles, does anyone have an idea how this might be done? for example, i am painting heights across a tile join region - obviously it will only effect the tile i am editing so i will never easily be able to rejoin the edges of the tile that joins it,

 

It seems to me that for large tiled worlds we need a system that treats all tiles as a single terrain for the purposes of editing.

 

Unigine certainly has a long way to go in regards to actually supporting editing of tiled large worlds..i think it is not what we expected when we purchased it, and the support lately from unigine team has been less than encouraging.

 

Hopefully they have these things in mind for future updates, however this does not help us meet a very tight deadline for large worlds for simulators.

 

I am not bagging Unigine, as i should also add that we tested several engines before we decided on unigine, including cryengine 3 and we found that unigine was the only one that came close to supporting what we needed at the time.

 

As has been said before by others on this forum, unigine sits in a difficult place at the moment, compared to other engines such as unity and UDK, but they really have the opportunity to differentiate themselves by concentrating on simulation and large tiled world support.

 

rant over <_<

[Rob]

Yes I use the weld vert option and im using a blend of 12%, what settings do you advise?

I was also thinking about the same problems you just stated, thats why im not going to edit the heights in Unigine and will just be using the detail materials and hope they blend well along the edges of the terrain tiles

 

1 way to solve the problem is to output masks for the detail textures so the textures always meet up at the edges, but the seam problem is not the terrain tiles or the heightmap/diffuse as I lined them up in Photoshop and there was no seam to be found I think its the way Unigine calculates the lighting, when I used the higher resolution heightmap the edges and verts were lined up and from a distance it looked like there were no seams but up close you would notice a seam almost as if the normal map was flipped but not as bad ofcourse

maybe it starts a new normalmap calculation on every new terrain?

[cor]

Maybe a Unigine representative could help us here. How are we supposed to make large worlds without visible artifacts.

[ulf.schroeter]

I think we are mixing too much aspects (terrain streaming, world streaming, terrain editing, terrain creation with WM2, etc) in a single thread. Each aspect on its own is quite complex.

 

Nevertheless we where able to create this Unigine terrain without artifacts using single 2k x 2k heightmap image and 8k x 8k diffuse image as inputs. Result was a Unigine ObjectTerrain with 8x8 surfaces.

 

I might be wrong but my understanding is that ObjectTerrain always requires and stores internally a single heightmap data texture. On terrain creation you also have to specify one large diffuse texture and then terrain import generates single smaller sized coarse textures (diffuse/normal) and splits input diffuse texture into smaller texture tiles which will be streamed automatically during run-time (while coarse textures will always be kept in memory).

 

Based on this assumption usage of WM2 pre-tiled heightmap/diffuse/mask data is not straight forward and most probably requires some specialized Unigine script code for proper Unigine terrain data preparation.

 

Have a look into <sdk_root>\data\core\editor\editor_objects_terrain.h to get the idea how texture surface splitting is handled (e.g look into import_diffuse_clicked()) There is also a trick for removal of seams between adjacent texture tiles which might be helpful for preparation of your WM2 exported texture tiles.

[Guest shane.ploenges]

There is also a trick for removal of seams between adjacent texture tiles which might be helpful for preparation of your WM2 exported texture tiles.

 

What is this trick you speak of Ulf?

[ulf.schroeter]

What is this trick you speak of Ulf?

 

Have a look into import_diffuse_clicked(). On large diffuse texture (e.g. 8k x 8k) splitting into smaller independent tiles (e.g. 256x256) they first copy a 1 pixel larger texture tile region (e.g. 257x257) and then rescale it down to target tile size (e.g. 256x256). By doing this texture border colors of adjacent texture tiles are the same which avoids visible seams (at least for diffuse texture).

 

Still I don't understand your seam problem for the heightmap and effects of heightmap resolution and mentioned usage of heightmap tiles (we only imported 1 large heightmap). Could you post some screenshots of your setup and artifacts for clarification ?

[manguste]

Good news everyone! Terrain streaming logic is going to be improved. The idea is to add background streaming of terrain surface textures around the player, which should eliminate the stalls. This feature is going to be available in the next update (not the pending one, which is most probably due out at the end of next week).

 

Streaming of terrain and streaming via NodeSectors are two separate things. Do not mix them together.Terrain has its own streaming logic that is implemented under the hood and NodeSector allows programmers to write custom streaming logic for objects.

 

I might be wrong but my understanding is that ObjectTerrain always requires and stores internally a single heightmap data texture. On terrain creation you also have to specify one large diffuse texture and then terrain import generates single smaller sized coarse textures (diffuse/normal) and splits input diffuse texture into smaller texture tiles which will be streamed automatically during run-time (while coarse textures will always be kept in the memory).

Yes, you are right, except that a heightmap is also split per surface and streamed.

[ulf.schroeter]

Hello manguste, as quite a lot of users seem to have requirements for really HUGE terrains, just a question: does ObjectTerrain support double precision e.g. in case of 100km x 100km ?

 

Of course the base transformation of ObjectTerrain node will do so like all other nodes in double-precision-builds, but I was wondering about all the internal transformation calculations for patches/surfaces where I only see float precion used, where I would also expect float precision issues for patches/surfaces far away from from ObjectTerrain origin.

[manguste]

As Frustum said, because terrain is only a grid, internal transformations with float precision seem to be sufficient (at least it worked for now).

[ulf.schroeter]

one of the best articles on large world handling I have found so far

 

http://www.google.de/url?q=http://scottbilas.com/files/2003/gdc_san_jose/continuous_world_paper.pdf&sa=U&ei=-Kc-T5vANJGHhQfzzI3hBQ&ved=0CBEQFjAA&usg=AFQjCNFJRMkI28jFyqXgjDTAKMeOMvtmjA

[manguste]

Thank you, Ulf, for sharing link to interesting articles!

[ulf.schroeter]

Terrain System used in Battlefield 3

 

- Unlimited Streaming

- Virtual Texturing

- Multi-Layer Decorations and Decals

- Procedural Texture Splattering

- Destruction

- Tesselation

- ...

 

wow

[ulf.schroeter]

New Continous Level-Of-Detail Terrain Heightmap Rendering approach with a clever mesh morphing between different LOD levels avoiding any form of stiching. Full source code and demos available. Check out provided animations showing very smooth LOD morphing.

[frustum]

This is a great idea for the LODs transitions. Unfortunately we can't use it because of non-uniform terrain tessellation which reduces heap of unnecessary polygons.

[ulf.schroeter]

Does non-uniform mesh tesselation still pays off nowadays with incredible triangle throughput (especially with DX11 HW tesselation) ? Also this has some other implications as far as I understand it (e.g. TerrainRelax requirements)

 

BF3 terrain implementation (which I would see as the market reference at the moment) also just uses uniform mesh tessellation with screen-space constant trinagle size and there seems to be no need for additional normal textures

 

[manguste]

Indeed, for DX11 it can be a great option. (But man shall not live by DX11 alone :) Frustum promised to think about it when the time comes to rewrite the rendering pipeline.

[Guest shane.ploenges]

Hi Manguste, back in your post #14 you mentioned terrain streaming would be improved in an update - i was wondering did this happen yet? i i dont recall seeing anything in any change logs around the time.

 

thanks

[manguste]

Shane,

 

All terrain textures are streamed in the background already, as for background streaming of terrain geometry (*.ter files), it is going to be implemented, it's waiting for its turn according to the development plan.

[Guest shane.ploenges]

Thanks for the update.

 

i also have another question terrain related, as we are now getting some reasonable performance from our terrain tiles (500km worth of tiles), we did this by removing diffuse and normal coarse textures per surface, and now use one coarse diffuse and normal per terrain - and only generate surface textures for masks.

 

my question is regarding compressing terrain textures , i am unsure how this actually works.

 

If we compress all our textures, i can see the video card memory usage drop a bit, however it does not seem to write any new .dds files anywhere that i can find? in the coarse diffuse and normal slot for each terrain, it still refers to our .png files.

 

i thought when you did a compress it would swap your uncompressed file for a .dds file, however this does not seem to be the case , visible to me anyway. Would it be a fair usage to pre-compress all our diffuse and normals for terrain prior to importing?

Thanks

[ulf.schroeter]

If I remember old terrain documentation right there was a statement that at least normal compression is not recommended due to compression artifacts...might be the reason why UNIGINE does not compress normal maps