Mount point umount for transnational collaboration

[Lales.Charles]

Hi,

In addition to VCS (Git), we're trying to take profit of mount point (umount) as suggested in forum (2.7.2) and doc.
It is working fine.. but few adjustments are required and I'm not sure to use it properly with terrain.

The fact is we have a little specificity to deal with: transnational collaboration, e.g. between France and Germany.
Latency between two locations impose to consider "local" shared folder, that have to be mirrored daily somehow.

Let's try to explain it bit more, with these 3 roles:

• Producer create project and 3d scene, dev src code, compile to generate exec bin
• Contributor can update 3d scene (out of terrain), then exec bin
• User can only exec bin

 ... over 2 transnational locations:

• France
  • Shared folder / server1
    • Source data (e.g. to generate terrain)
    • Shared data (this has to be properly mirrored)
      • Entities
      • Landscapes (with each landscape folder containing its arrays and multiple .landscape files with proper input/output "local" paths..)
  • Producer1 / clone repo
  • Contributor1 / clone repo
• Germany
  • Shared folder / server2
    • Source data
    • Shared data (entities, landscapes)
  • Producer2 / clone repo
  • User1 / only assembled dir

Umount let terrain to be generated once, or is suitable way to share entities.
At first attempt, I put the bunch on umount files in versioning, but 2 servers -> shared folder path have to be updated depending of location.
To deal with that, we dev an umountGenerator, put it in versioning with its config file, in order to generate proper umount files when needed.

Q. Would you have considered such context? Would see / suggest another approach?

When generating a terrain, the current world file will explicitly link to shared folder instead of using umount.
Guess it is normal since landscape tool doesn't allow umount in output path, but accept shared folder path, at least when you're editing directly the .landscape file.. 
NB: no mapped drives, but UNC Linux path
But.. if you're editing world file, removing beginning of path, letting only umount, it does work, and we get then a world file location independent that can properly be under versioning.
But.. the next terrain generation will setup again complete UNC path.. and manual update is again required.

Q. Are you aware of these points? Is it fine? Is there any possible improvement?

Finally, assemble project seems to ignore umount and shared data, declared in world file both with umount or explicit UNC path.
A quick fix consist in manual copy of shared data.

Q. Is it normal?

Kind regards,
Charles

[morbid]

Hi Charles,

This sounds like a sophisticated case.

Mount points were designed to serve as a filesystem extension, one of the major cases we wanted to cover is managing content libraries that are deployed on the same computer. We also tested this functionality with network shared folders, and it worked but had some limitations that we couldn't overcome due to the independent of us reasons (Windows/Linux specifics).

TerrainGlobal works with paths only (unfortunately), while everything else is using GUIDs. In some sense, the terrain is unique, because for one LOD it uses a number of files, that's why it "looks" into a folder, not a specific file. So we can't simply link each LOD to a file. Probably, the best practice would be using the same approach as we do for the other assets: while the terrain data is linked with the unique ID you can store it anywhere you want.

The manual correction of a path will work, but we can't guarantee the correct behavior in all cases. Probably, you can fix path in the final landscape stage.

Now to your questions.

18 hours ago, Lales.Charles said:

Q. Would you have considered such context? Would see / suggest another approach?

As I tried to explain it before, we didn't cover such a scheme. As a workaround, I see the following option: store terrain in the data folder. This will ease all troubles with paths.

18 hours ago, Lales.Charles said:

Q. Are you aware of these points? Is it fine? Is there any possible improvement?

Yes, we're aware of this. Improvement will come with ObjectPlanet, where we'll try to bring better integration with asset system and file system. Another option could be keeping consistent UNC paths, but I'm not sure if this possible considering the complexity of transnational synchronization.

As I understood you're using VCS for source code only. Have you considered using additional VCS for the content (databases, entities, etc)? I think it can solve the mirroring issue - just commit and checkout data folder. We use for this purpose SVN, some of our clients stick to Plastic SCM.

18 hours ago, Lales.Charles said:

Finally, assemble project seems to ignore umount and shared data, declared in world file both with umount or explicit UNC path.
A quick fix consist in manual copy of shared data.

Q. Is it normal?

Yep, manual copying is the best workaround for this. In 2.9+ we planned major changes to the SDK Browser and UnigineEditor. Assemble functionality will be integrated into UnigineEditor because Browser simply doesn't work with engine filesystem (that's why it doesn't work in 2.7.3).

Thanks!

[Lales.Charles]

Hi,

"Sophisticated case".. yes, but real one for us..
Since we're international company, numerous projects may have to deal with such constraint.

Terrain work around -> store in data folder.
But terrain are range 100 x 100 km, georeference with elevation, texture, vegetation, building and entities.
The idea of umount usage with terrain is to generate them once, and provide them to other dev (contributors, users) in read only.
On below diagram, what is then the purpose of Terrain and Vegetation in Shared Content Storage?
We're testing such setup, but mirrored with another location.

Regarding VCS, it is not for source code only.
We partially Git versioned the complete project folder, but with ignore list as described in this other post:

In fact the umount files are not versioned (because of different shared folder / locations) but config file for a umount genetor.

Not a big deal for SDK browser and project assemble issue.
Thanks for Plastic SCM proposition, we have no exp internally on this VCS.
I'd give a try to umount options, just to make versioned projects lighter.
And right now it is working fine with landscapes (loading more or less fast is well handled), providing an easy way for the contributors (c.f. roles previously defined) to quickly get smthg running for them to focus on e.g. entities edition.

Kind regards,
Charles

[morbid]

Charles, thanks for the diagram. It's much easier to understand now.

I didn't mean anything offensive by saying "sophisticated", probably not the best wording from my side, sorry.

The case itself is really useful for us. Right now the scheme that you came up with (i mean the one on the picture) looks clear. The only thing I'd like to warn you is a possible combination of RW rights on one mount point for two developers. We can't guarantee a stable assets workflow if one file will be modified simultaneously.

And let me summarize everything, so nothing will be lost in this thread:

• The first issue, and as I see, the most disturbing. Terrain doesn't work with mount points. We're going to fix this with new ObjectPlanet, which will come in 2.9. For now, there's only one workaround that you'd already found - manual path change.
• Mount point generator works for you and it fits your needs. I guess there's no request for any changes to the engine filesystem or tools, right?
• Assembling Project with proper mount points will be available after 2.9 SDK release.

As for the git and version control.

I've familiar with an opinion that git isn't the best VCS for "heavy" projects that include a lot of textures, geometry, etc. That's why I said about SVN or Plastic. These systems could be more friendly for massive databases.

Please, let me know if you have more questions.

[Lales.Charles]

Hi,

Do not worry, I don't take offense for such wording, that was just to point out that it is not experimental for us but really operational / business need.

Don't worry for umount RW with concurrent access, we will clearly define rules on this.

Terrain not working with umount... why? It is nicely of us. Not by default since terrain generator will write complete path in current world file, but after manual update the editor is fine and use properly rel path / umount declared in data.
Fact is, I tried to update .landscape file, the output path (root_folder) with umount.. but terrain generator doesn't accept it.
But, it does accept the data_path of the umount, e.g. //<server>/<shared_folder>/shared_content_storage/landscape/aMountain
In other words, from landscape generator, we generate terrain in shared folder loaded as umount.
Everything works fine, but since world file are under versioning that we want to keep location independent, we have unload the world, update world file to short all terrain path:
- from //<server>/<shared_folder>/shared_content_storage/landscape/aMountain/[...]
- to landscape/aMountain/[...]
.. and reload the world.

Umount generator, it is a idea / suggestion for others / very basic:


Generate button will generated selected umount directly in data > entities / landscape
You can switch from one server to another one (list defined in config file under version), depending on your location.

Regarding VCS, Plastic seems powerful and flexible, can understand the motivation.
But a clone of what is currently in Git -> 150MB (requires SDK browser reconfigure for bin folder + umount generator to be run)
If we consider all data included -> more than 10GB.. 

Kind regards,
Charles

[morbid]

18 hours ago, Lales.Charles said:

Terrain not working with umount... why? It is nicely of us

I meant it could be done better and we'll look into future terrain integration with engine filesystem. The other thing is "teaching" Landscape Tool to work with mount points. It should be done to keep a consistent design.

Your umount generator looks really good, I did once a simple script that generated .umount files, but it was much much simpler than your tool :)

Thanks again for the feedback.