GUID and the search for the missing link

[Amerio.Stephane]

Hello,

"GUID and the search for the missing link" sounds a bit like an adventurous journey... and it is!

More than once, for whatever reasons, some GUID link get broken. It can be because of an SDK upgrade, moving a folder between projects, or any other incorrect or incomplete work. But we get invalid GUID all the times.

Not a big deal if we know/remember what it was referring to. But it would be SO MUCH simpler with these simple additions:

• In all files using a guid::// url, please-pretty-please also add a human readable text that shows the file referenced by this guid. You could also use this text to explain at runtime why a guid could not be loaded instead of the cryptic current messages. This alone would be of tremendous help.
• In the Editor, have a way to highlight/search for all missing/incorrect guid. Not to clean them, but just quickly find them. Even inside NodeReference please!
• At run-time, when a guid is missing please don't just crash (eg. when a node property's guid is incorrect, the component is not correctly loaded and the app crash). We lost a lot of time finding this out.
• When upgrading a project, if the project uses "core" materials, meshes, properties, the new sdk may reference them with a new guid (or maybe they just don't exist any more). Please highlight this better in the upgrade process, or when the guid has changed, just auto-fix it. This happened to us with some surfaces that referenced the mesh_base.mat. The mesh simply disappeared from the editor (became invisible and was not highlighted in red).

Thanks !

[ExBemined]

We are facing the same issue now, after multiple SDK upgrades we are seeing errors on startup about a particular GUID that cannot be resolved, and we have no clue which file it is trying to find as there is no mention of it in any .meta nor in guids.db. Guids can be very tricky working with a large team and multiple branches, specially when upgrading SDKs since we cannot force all branches to be merged and then stop the entire development of new content for the duration of the upgrade.

[Amerio.Stephane]

Hi,

Any feedback, or feed for thought ?

[silent]

Hello!

Thank you for the waiting. Busy days before the release :)

We moved from path-based XMLs to GUID-based approach to provide an artists-friendly workflow. To get rid of the constant errors and broken scenes when texture / material / mesh / fbx files were moved by the artists. Only a single guids.db file needs to be changed, instead of changing multiple other files that have a link to the modified files.

GUIDs are working good enough in combination with an Editor (it handles guid <-> path relations and tries to keep all the content in a good shape). Once you will try to manually copy files from another projects (with different version) you may have a bad times. There is no any possible solution to handle such cases. We hope that package creation tool will allow smooth content transfers between different projects (will be available in 2.14) and will reduce amount of human-based errors that causes GUIDs to be broken.

 

Quote

• In all files using a guid::// url, please-pretty-please also add a human readable text that shows the file referenced by this guid. You could also use this text to explain at runtime why a guid could not be loaded instead of the cryptic current messages. This alone would be of tremendous help.

That's highly unlikely. Human readable path will be always in conflict if you will use some kind of VCS system and can be very misleading. We think more in the way of introducing some kind of additional database for GUIDs history that will simply collect all the possible guids and names starting from the initial development stage. Unfortunately, it will be always hard to merge and this file will be often in conflict stage. Another solution (very stupid, but conflict-free) - is to generate and store unique guids file that somehow ties to a current PC only. As a result - if you have a team of 100 you will get something like 100 guids history files that will continuously grow in size and Editor in theory can somehow parse all these 100 files on startup and show you all the possible previous real file names for specific GUIDs. These all solutions are not perfect, but at least it may allow to understand what's going on with this GUID.

 

Quote

• In the Editor, have a way to highlight/search for all missing/incorrect guid. Not to clean them, but just quickly find them. Even inside NodeReference please!

Can you please give us more detailed info about this? You simply want to see a broken nodes with incorrect GUIDs? How this should look like and what kind of things you want to do with this list of broken guids?

 

Quote

• At run-time, when a guid is missing please don't just crash (eg. when a node property's guid is incorrect, the component is not correctly loaded and the app crash). We lost a lot of time finding this out.

That's sounds like a 100% bug to me. Missing guids should not cause the application crashes (at least if there are only couple of them, not hundreds and thousands). Do you have any test scene where you can get quick and easy crash simply because of missing guid? That would help us to find and fix the root cause faster.

 

Quote

• When upgrading a project, if the project uses "core" materials, meshes, properties, the new sdk may reference them with a new guid (or maybe they just don't exist any more). Please highlight this better in the upgrade process, or when the guid has changed, just auto-fix it. This happened to us with some surfaces that referenced the mesh_base.mat. The mesh simply disappeared from the editor (became invisible and was not highlighted in red).

That also sounds very weird and shouldn't in theory happens. Do you have a reproduction project that we can download and check on our side? We are testing each migration using all of our content projects (including some internal) and never experienced similar behavior. It would be interesting to see what is really caused this and do a proper fix for that.

Thank you!

[demostenes]

11 hours ago, silent said:

That's highly unlikely. Human readable path will be always in conflict if you will use some kind of VCS system and can be very misleading.

Wouldnt be possible to just show actual path, where these problematic assets are? I thing question was not about hardcoding path somewhere, but just show actual path of file with problematic guid. So in case of error you recive something like error guid:://xy (in \data\..\material.mat). Currently it is really tedious to solve these issues, only way is to take total commander, full text search all text files in project, find which files are affected and correct it. And it simply happens from time time, no matter what processes in team you set, somebody always make mistake (typical case is to move files locally, forget to commit and do update) and you have to clean this several times per month. So knowing directly from log where is the issue would save lots of time. Thanks.

Edited April 7, 2021 by demostenes

[Amerio.Stephane]

Hi Si1lent,

Thanks for taking the time to write a detailed answer :) I know this is not a simple matter and GUID implementation and management is complex.

1 hour ago, silent said:

We think more in the way of introducing some kind of additional database for GUIDs history that will simply collect all the possible guids and names starting from the initial development stage.

I don't think a complete history would be necessary, maybe just the last known name/path will be enough. This is just so a console warning would be more explicit. As the guid.db is written by the Editor, maybe the Editor could update the guid.db file to contain the last known path too. It's just so frustrating when you import an object and you get the dreaded "unknown GUID", just because you forgot a mount point (but which one???).

1 hour ago, silent said:

Another solution (very stupid, but conflict-free) - is to generate and store unique guids file that somehow ties to a current PC only

IMHO this would be useless, especially because we use a lot of different PC to work on the same database. So the GUID would keep changing, without providing any benefit regarding the name/guid association.

1 hour ago, silent said:

Can you please give us more detailed info about this?

This could be a "simple" table view, listing objects/material/references with invalid GUID. Then the user could double click an item to open it. Then you highlight in bright red the relevant error. No more searching through .node or .mat text file the relevant invalid guid copied from the log.txt....

You can improve on it later. Just this 1st would be a win.

 

As for the other issues, I'll try to reproduce them.

Thanks!