[SOLVED] TilesetFile: header is corrupted

[demostenes]

We have this message in log:

13:51:50 TilesetFile::load(): can't load tileset esqworld/terrain_global/heights/lod0/masks: header is corrupted

13:51:50 TilesetFile::load(): can't load tileset esqworld/terrain_global/heights/lod1/masks: header is corrupted

But terrain is visible (seems ok on the first sight). What does it mean? How this can happen and how we can fix it?

Thanks.

Edited August 14, 2018 by demostenes

[morbid]

Hi,

I also paid attention to this error and did a little research in your repo.

As I see, in revision 376 only one header file was damaged - for LOD 1. I had no time for careful investigation.

I also observed that the most detailed height LOD in esqworld\esqworld.world (current revision) is rendered with artifacts, some tiles are missing. I can't say how exactly this happened, the most adequate version is that something went wrong while someone was committing modified files to the repo.

What can you do:

• find a revision where terrain header's integrity is unaffected
• replace height data from this revision

I can't estimate how many changes you did to the height LODs, but you also can regenerate heights in Landscape tool using landscape asset.

If you can give us more feedback on this - it'll be appreciated.

Thank you!

[demostenes]

Ok, thanks for your help! I will try regeneration of heights.

[demostenes]

Regeneration probably helps, but we loose all changes since migration. We were able to create height/mask exporter, but it is not working on this corrupted terrain (in esqworld.world), exported height map is not OK, tiles does not fit together (on any new terrain it works OK). Probably because of this header error, just to be sure, is this really connected to height map? /lod0/masks: header is corrupted

Btw we didnt notice any tiles missing, terrain is editable, everything is working as it should, except that header error. So it is wierd, that our exporter is not able to read height map properly.

Anyway to not loose this changes, our idea is to manually fix header. Can you tell us, where is this header located and how long it is? And what contains? So we are able to replace it with header from version, which was not damaged (or regenerated version).

Thanks.

Edited October 11, 2018 by demostenes

[morbid]

Hi @demostenes,

we're investigating this issue deeply. I'll keep you posted.

Thanks!

[morbid]

Okay, we made some progress on this.

The corrupted mask header has nothing to do with it. Technically, masks for heights are generated only for insets, but for your terrain these masks were made due to a "holes" in ObjectTerrain.

To avoid errors in log you can simply remove mask files from heigh lods. This means you'll also lose holes created for dungeons or caves. But visual quality of terrain global holes is really bad, that's why we made decal-based holes. I mean that after this you'll have to switch to decals.

The real problem is with second height lod (5.860 mppx). It was broken right after migration from ObjectTerrain to GlobalTerrain (rev 280) and I can't say why. I've tried to migrate it once more and all data was fine. You have two options:

• delete broken lod and keep going
• write a script that will resize 1st lod (the correct one) according to the density of the 2nd lod.

But at first, please, test your tool without 5.860 lod. Will it work?

If you want to recreate 2nd lod, here some thoughts from our developer.

You'll need Unigine::TileSetFile Class and Unigine::Image Class.

• Use load method to get correct data from the first lod
• then check a total number of tiles and their coordinates via int getNumTiles() and int getTilePos(int num, int & x, int & y)
• after that int getTile(int x, int y, unsigned char * data) will allow you put data to the image. Here you can store the data using Unigine::Blob Class

For instance:

int tile_x,tile_y;
tileset->getTilePos(0, tile_x,tile_y); //retrieving tile position
const auto blob = Unigine::Blob::create(); // this string will allow you to extract the data from tile
unsigned char * data = new unsigned char[tileset->getTileSize()];
const auto img = Unigine::Image::create(); // making an image
tileset->getTile(tile_x, tile_y, data);

blob->setData(data, tileset->getTileSize()); // here we're putting the data into blob
img->load(blob->getStream()); // here we've go an image
img->decompress();
blob->setData(nullptr, 0);
delete []data;

This cycle will allow you to fetch images from the good lod. Afterward, you'll need to resize them and build into a new lod.

BlobPtr blob; // create a blob
blob = Blob::create();
blob->reserve((image)->getSize());
(image)->save(blob->getStream()); // saving an image to a blob
if (!output) // this condition will allow you to put images into one tileset file .
{
    output = TilesetFile::create(); // create a tileset
    output->createFile(put here the tileset name for broken lod, blob->getSize());
}
output->setTile(x, y, blob->getData(), 0); // put an image according to calculated position into a tileset file

When you're finished with images and tiles:

output->flushHeader(); // saving everything into utsh file
output->close(); // closing tileset file

Thank you.

[demostenes]

Perfect, we will definitelly use decal holes. Thanks!

Edited October 16, 2018 by demostenes

[morbid]

Okay, but what about your export tool? Does it work without broken lod?

Thanks.

[demostenes]

3 hours ago, morbid said:

Okay, but what about your export tool? Does it work without broken lod?

Thanks.

We are able to get LOD0 as dds texture, is seems OK on the first sight. 

[roboch.matej]

Hi! We have strange problem... When we export heights and regenerate the terrain with the exact same parameters, the terrain height is not exactly same, there is small difference.

The first is the original terrain. Second and surprisingly the worst is export in R32F, third is exported in R16F:

Edited October 22, 2018 by roboch.matej

[silent]

Can you compare exported heights with original in some 3rd party graphics editor? The files itself can be different and that's why you are getting different results at the end. 

[demostenes]

8 hours ago, silent said:

Can you compare exported heights with original in some 3rd party graphics editor? The files itself can be different and that's why you are getting different results at the end. 

Well, they are different, so we get slightly different results. But because there is very small diffence (cca 20 - 30 cm in height on some places), it does not look like principal issue in our tool, but probably some decimals are lost (or rounded up) during saving/conversion. Our tool takes  terrain height tiles from memory and saves them as one .dds. Interesting is, that if we save it as .png, height tiles does not fit together well. Also wierd is, that R32F is less precise than R16F.

Edited October 23, 2018 by demostenes

[roboch.matej]

These are the source files we are using (please don't mind the cz comments, they don't contain anything important)

TilesetExport.h
TilesetExport.cpp

[demostenes]

Any idea, what export has some small height differences?

[morbid]

Hi,

sorry, had no time for this last week. The code seems fine, can you send us R16 and R32 files so we can check what's happening?

Thank you.

[roboch.matej]

Hi!

HeightMap.zip

It also contains .png exported files, which are completely messed up (we don't know why).

[morbid]

It looks like the attached R32F image was resized (or modified in some other way?), take a look at the screenshot. 

We did some test and changed your code, now the data is grabbed directly from tileset file. After that, we get the same images regardless of color depth.

Please check the attached code and images.

tileset_export.zip

[demostenes]

Hello, no, we didnt modify r32f in any way. Now it seems there is almost no difference in compare to original heightmap. Thanks!

Edited November 12, 2018 by demostenes

[morbid]

Good!
Do you need further assistance with this?

[demostenes]

Not now. We will continue testing this tool and if anything goes wrong, we will let you know :) Thanks!

PS: You can provide this script to anyone who have some use for it, of course no guarantee from our side :)

 

Edited November 12, 2018 by demostenes