Jump to content

meta file regeneration does not generate stable guid


photo

Recommended Posts

Hi

After deleting the .meta file for a mesh, a new one is generated with a different guid. This is a problem because the references contain the original guid. How to solve this problem? I assumed that the guid is deterministically generated. An example is attached.

Between versions of .meta file <asset version = "2.70"> and <asset version = "2.7.2.2"> did not change the referenced mesh file, but the hash is different.

Edit: What is the algorithm for generating guid? Is it derived from a hash of file? If so, how do you solve the loss of .meta file? Especially if references already exist.

guids.png

Edited by aargh
Link to comment

Hello aargh,

On 8/12/2018 at 4:02 AM, aargh said:

After deleting the .meta file for a mesh, a new one is generated with a different guid

This is the correct behavior, we generate guids non-deterministically. Each time .meta is lost it will be regenerated with different value. To keep all nodes you need the original meta. If node lost its material\mesh or anything else you'll need to fix it manually.

On 8/12/2018 at 4:02 AM, aargh said:

Between versions of .meta file <asset version = "2.70"> and <asset version = "2.7.2.2"> did not change the referenced mesh file, but the hash is different.

It's also a valid case because version change usually comes with some modifications, so we need to reimport everything once after migration. 

On 8/12/2018 at 4:02 AM, aargh said:

Edit: What is the algorithm for generating guid? Is it derived from a hash of file?

Well, it's random and doesn't depend on hash sum.

Thanks!

 

How to submit a good bug report
---
FTP server for test scenes and user uploads:

Link to comment

Hi

if I delete a meta file, I will always get the same guid when creating a new one. The change only occurs between SDKs. That's why I asked about the generation algorithm

Link to comment

Sorry, I missed one point in the previous message.

  1. If you delete .meta file when the Editor is running - the original values for asset will be restored (not generated).
  2. If you delete .meta file when the Editor is not loaded, just in file manager, Editor will treat this file without meta as a new one and completely new GUID will be generated.

let's go back to your case. What exactly happened? You wrote that .meta was deleted, if it was, then the generation of a new GUID is correct behavior. If this happened after migration - it could be a bug that needs a way to be reproduced.

Thanks!

How to submit a good bug report
---
FTP server for test scenes and user uploads:

Link to comment
6 minutes ago, morbid said:

Sorry, I missed one point in the previous message.

  1. If you delete .meta file when the Editor is running - the original values for asset will be restored (not generated).
  2. If you delete .meta file when the Editor is not loaded, just in file manager, Editor will treat this file without meta as a new one and completely new GUID will be generated.

let's go back to your case. What exactly happened? You wrote that .meta was deleted, if it was, then the generation of a new GUID is correct behavior. If this happened after migration - it could be a bug that needs a way to be reproduced.

Thanks!

Hi

Ad 2 the editor generates the same guid for the file. This has led me to question whether the guid is deterministic.
in the appendix is original meta (first) and meta after new creation by the editor (second)  files are for
data\asset_library\characters\UMA\meshes\UMA_male.mesh

UMA_male.mesh.meta_first

UMA_male.mesh.meta_second

Link to comment
40 minutes ago, morbid said:

let's go back to your case. What exactly happened? You wrote that .meta was deleted, if it was, then the generation of a new GUID is correct behavior. If this happened after migration - it could be a bug that needs a way to be reproduced.

Thanks!

1) It seems, that GUID generation algorithm was changed recently, correct?

2) We are facing crashes and duplicate guid issues: https://developer.unigine.com/forum/topic/4988-duplicate-guids-in-log-and-crashes/ We did all node duplications via editor, so it seems like some bug. Because editor is crashing/hanging quite often during various duplicating operations https://developer.unigine.com/forum/topic/4972-editor-sometimes-hangs-when-duplicating-node/ it leads us to suspiction, that this can be reason, why duplicated guids appeared. 

3) it would be very helpful to extend logging, so we cant find, which file is problematic. Such error message is not much descriptive: crash.jpg

Thanks!

 

 

 

 

Link to comment

And back to my case...

After several crashes of the editor when copying nodes, I found this in the log

14:13:49 FileSystemAssets::load_runtime(): guid "7c1ce7913f7ea73c09945bb57383cd4ef96c8310" is already in use
14:13:49 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:49 FileSystemAssets::load_runtime(): guid "6713e5b9c2afcf3370262a5f6dbd4d1cb7599720" is already in use
14:13:50 FileSystemAssets::load_runtime(): guid "9ed63fe1805b90815509d71dd6040a5ce3d48b30" is already in use
14:13:50 FileSystemAssets::load_runtime(): guid "c05b8da7ca91d078f5fdae08f69fc8e6a827b2c0" is already in use
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): guid "9b7330ea1fcb4b362a8f12741a92b18341ab4f51" is already in use
14:13:50 FileSystemAssets::load_runtime(): unknown "4cb9dd49d33dcce01c002c56ed790273b7d838c2" asset guid
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): guid "31f78f7e9324be5428d312d272cf7b2dd709c6b1" is already in use
14:13:50 FileSystemAssets::load_runtime(): guid "d2850c6ce3bf7c897c905997cf4e7e6bcf42bdc1" is already in use
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): unknown "4cb9dd49d33dcce01c002c56ed790273b7d838c2" asset guid
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): guid "2fb6ac833535108ff3516d217529086757da2603" is already in use
14:13:50 FileSystemAssets::load_runtime(): guid "57cd839eef6f88dda51983672c9d0793c9daa333" is already in use
14:13:50 FileSystemAssets::load_runtime(): guid "65dcf2e0b3a458e574d7a3ed96c1c43c2ea45943" is already in use
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): guid "bca209fee6f3fee3653381579810cbec6d3aaa93" is already in use
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): guid "e861fd0e434b1c2be83cca2070d999ef12f6da34" is already in use
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): guid "902d8ab6f9914a6cdf18950fea13bb63d2c7e984" is already in use
14:13:50 FileSystemAssets::load_runtime(): guid "c7a2500d1fea65dd337407aa6de620b6414cf8a4" is already in use
14:13:50 FileSystemAssets::load_runtime(): guid "ffdf790627c0df05c4b04d91f34eb64b2bdd29a4" is already in use
14:13:50 FileSystemAssets::load_runtime(): guid "33204f6bd16d43c013f6bd4356c827393e19a205" is already in use
14:13:50 FileSystemAssets::load_runtime(): guid "8be192237710bae03bd98405eb27e609de8cc7f5" is already in use
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): guid "6a695da42c887a43f9b37cd8f22e3740115cc9c6" is already in use
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:50 FileSystemAssets::load_runtime(): guid "7fecdf313054e3ea88567dd1d1d605f8c618d817" is already in use
14:13:50 FileSystemAssets::load_runtime(): guid "dbaf863f2666844318942abe55fa7dd4f36c74c7" is already in use
14:13:50 FileSystemAssets::load_runtime(): guid "f037d9580e68fef900bf5c1d77ad808a52428bf7" is already in use
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): guid "ecc9287e6eac45e88984467088d234a0b4f68368" is already in use
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): guid "03d37f3270451c15e2d07ed9b37a4ec1030acda8" is already in use
14:13:51 FileSystemAssets::load_runtime(): guid "78ba8f215d98a91029bfab3f6fc464fc2f52d3e8" is already in use
14:13:51 FileSystemAssets::load_runtime(): guid "af8dc01daa4603bbeb8236b4d30decc6b3d22f19" is already in use
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): guid "6d82a7580a021d86848325f7546db8c8227217d9" is already in use
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): guid "c2f532af108894b16139d051a42bddac98fe136b" is already in use
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): guid "16255eb07203d4b19d5d20c6e4e4227ee77e1bdb" is already in use
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): guid "6163269213cbebea750079b089612ade934d4dbc" is already in use
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): guid "ffc02c10477ae6a3022b28e1c23a5346356b0d0d" is already in use
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): guid "15036ca7e1452c4bf6a26ac91004cf635070eb6d" is already in use
14:13:51 FileSystemAssets::load_runtime(): guid "fdc2617595584a52a05fa8b44ecdb22e070a0e8d" is already in use
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): guid "f0d451fd380480bceaacefca29c5c5287a67cd5e" is already in use
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:51 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:52 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid
14:13:52 FileSystemAssets::load_runtime(): guid "ddbc3d08139ea7c36354776d305387e93ebd31af" is already in use
14:13:52 FileSystemAssets::load_runtime(): unknown "f920836fab836cd224b426e9e1eb3eb959646aef" asset guid

I started looking for the cause and discovered that some files are not up to date on the latest version. I reported this elsewhere.
I made a change to upgrade the script to take place for all old files.

Based on my assumption of guid stability, I deleted the meta files and let them re-create them. Which was a mistake.
- restore the meta files from the backup
- you need to find out why the file editor creates errors in files (duplicates or invalid xml)
- Upgrade script only checks the first found file for the version, others skip
- write a precise description of the metafile mechanics in the documentation. This would help a lot

 

Link to comment

Small addition

I found this error on log file


14:15:21 "AssetManager::init_core_assets(): FileSystem error. GUID = e038d46e26ad35b1e7775c22e5377971790cb075, path = core/textures/sensor/lut_green.dds"
14:15:21 "AssetManager::init_core_assets(): FileSystem error. GUID = 2a3d6d41edb8ac91336265af489c891e9d772658, path = core/textures/sensor/lut_heat.dds"
14:15:21 "AssetManager::init_core_assets(): FileSystem error. GUID = 434ab133edf4b06c916e3d36bd646d04746da384, path = core/textures/sensor/lut_red.dds"
14:15:21 "AssetManager::init_core_assets(): FileSystem error. GUID = e237a1333473ddcc4729b8cfa3bee53e75991b77, path = core/textures/sensor/lut_white.dds"
14:15:21 "AssetManager::init_core_assets(): FileSystem error. GUID = a6d3c23f9fb69fffd5fe6aaa55c6a64559ba1f06, path = core/materials/default/sensor/post_sensor.basemat"
14:15:21 "AssetManager::init_core_assets(): FileSystem error. GUID = 19a181cc831bedbdefe97d193d1aa76fbac66260, path = core/materials/default/sensor/post_sensor_green.mat"
14:15:21 "AssetManager::init_core_assets(): FileSystem error. GUID = 153e6c29c36ec1a0be3922f12928e8c75d53e58b, path = core/materials/default/sensor/post_sensor_heat.mat"
14:15:21 "AssetManager::init_core_assets(): FileSystem error. GUID = 668515a810feac4b6532b0d6f2cf53947b33ca32, path = core/materials/default/sensor/post_sensor_red.mat"
14:15:21 "AssetManager::init_core_assets(): FileSystem error. GUID = 6c9028091619e0c167f3e8e32b7d067784cbfcb0, path = core/materials/default/sensor/post_sensor_white.mat"

This is situation on version before my changes

 

Please extend logging, this is cryptic output for error checking on big project

14:30:57 ----Properties----
14:30:57 Xml::getBoolData(): unknown bool value "" in "parameter" node
14:30:57 Xml::getFloatArrayData(): can't read 0 array element in "parameter" node
14:30:57 Property::load_parameter_item(): can't read "color" parameter in "showcase" property
14:30:57 Xml::getBoolData(): unknown bool value "" in "parameter" node
14:30:58 Engine properties loaded: 2 / 2 (0ms)
14:30:58 User properties loaded: 294 / 294 (107ms)

 

Edited by aargh
Link to comment
2 hours ago, aargh said:

Ad 2 the editor generates the same guid for the file. This has led me to question whether the guid is deterministic.

My bad, I forgot to mention that it also could be restored from guids.db file. If the required string is absent - you'll get a new GUID.

1 hour ago, demostenes said:

1) It seems, that GUID generation algorithm was changed recently, correct?

Nope, it remained the same.

1 hour ago, aargh said:

I started looking for the cause and discovered that some files are not up to date on the latest version. I reported this elsewhere.

Okay, please check the corresponding topic.

1 hour ago, aargh said:

you need to find out why the file editor creates errors in files (duplicates or invalid xml)

Could you explain how exactly you're copying the assets? Copy-paste in Editor? Or in file manager?

I've started your project migration from 2.7.1 to 2.7.2, we'll try to give you feedback tomorrow.

Thank you!

How to submit a good bug report
---
FTP server for test scenes and user uploads:

Link to comment
6 minutes ago, morbid said:

Could you explain how exactly you're copying the assets? Copy-paste in Editor? Or in file manager?

Only via editor. Copy paste/duplicate function. Never ever outside editor.

Link to comment

Hi,

we've reproduced random crash with world loading you reportedearlier. The issue with lost mesh that @aargh described in the first place didn't show up. I've checked your nodes in asset_library folder.

I'd like to note that migration from 2.7.1 went smoothly.

We still investigating "unknown GUID" problem.

Thanks!

How to submit a good bug report
---
FTP server for test scenes and user uploads:

Link to comment

Thanks, thats good news.

I ve taken revision of the project 463, deleted guids.db and runtime directory and run editor. I have this guid error in the log:

FileSystem::mount_file(): can't mount "asset_library/rocks/rock_for_landscapes/mesh/Pile_Ramp_Small.mesh" file, because of GUID collision

I ve fulltext searched this guid through project (.meta, .node, .world,.mat) but it is only in Pile_Ramp_Small.mesh.meta

How is this possible? Guid DB is fresh and meta is only 1x, which is OK.

Thanks.

 

 

 

 

 

Link to comment
×
×
  • Create New...