Feedback required: New multiple directories support (outside of data_path) for Editor2 / Engine

[silent]

Hello!

We are currently researching solutions to add multiple extern directories (e.g. from external repos) for convenient assets workflow inside Editor (and Engine itself).

We are looking forward to your suggestions! Please, write a comment if you are feeling that something important is currently missing  and will be very useful in your projects.

After initial discussions we've came up with a mounting points idea, that will work as follows:

• You can mount almost any directory that available on filesystem. If you need to mount another project's content, it's possible only by adding the root data directory of this project. Please, refer to the project scheme below.
• You can only mount directories to the root data directory of the current project.
• Editor should be able to work without guids.db, .meta files will be enough to generate correct paths and GUIDs for the internal filesystem on-fly.
• You can add new mounts via command line options / API (in runtime) and via Editor 2 GUI.
• By default all shares are mounted as read only.
• If you need to change some assets from external paths you will need to request an read-write access. .lock file will be generate to prevent other running editors to modify the same file (force edit is also available).
• All the runtimes for external sources will be generated in external mount directories.
• Assemble project feature will be moved to Editor2. It will be mandatory to manually include files / folders (via GUI) that should be kept in final build. No automatic validation is possible (due to some assets may be accessed from different projects). guids.db for final project will be generate at this step.
• All the selected content of external directories will be copied to a single data directory on assemble.
• Core engine files inside data will be moved to the another directory that should be specified as -core_path. These files are (including their unpacked versions):
  • core.ung
  • editor2.ung
  • scripts.ung
  • *.cache files (editor.cache, unigine.cache, shaders.cache)
  • *.cfg ???

so, as a result you will have only project-related files inside data directory.

That will basically allow to:

• Create a single content repository that you can re-use across multiple projects by simply adding it to the data directory (without need to copy-paste actual content)
• Protect (in case of .lock file implementations) data from accidental loss when two or more editors are trying to write to the same files at once.
• Mix in a single project different mount types - CVS / Share and different access modes (RO / RW).

What will not work with multiple external mounts:

• Validation (we simply can’t delete unused assets since they may be used in another project).
• Simultaneous editing of a single asset.
• Mount different paths inside a single mount point to a multiple mounts (e.g., you can’t mount <REPO1>/data/props/table if you already mounted <REPO1> or <REPO1>/props/chair).

So the project structure may look something similar to this after this changes:

• Project_Dir
  • bin
    • project-name.exe
    • unigine_x64_double.dll
    • editor2_x64_double.dll
  • data
    • common [--> link to the data of common SVN Repo]
      all content is already generated, fast validation required only
      • .runtimes
      • asset1.meta
      • asset1.fbx
      • ...
    • buildings [--> link to the network share]
      all content is already generated, fast validation required only
      • .runtimes
      • asset2.meta
      • asset2.tga
      • my_template_world.world
      • my_template_world.meta
      • ...
    • props [local files unique to this project only]
    • .runtimes [local runtimes generated from local files only]
  • core [-core_path ../core]
    • core.ung
    • editor2.ung
    • shader_d3d11.cache
    • unigine.cfg
    • project.cfg
  • source
    • project.vcxproj
    • project.cpp

The drawbacks are:

• In case of the multiple network shared folders: validation, streaming and overall performance degradation.
• Spontaneous crashes may appear in Editor when some assets that currently in use will be deleted by another user on network drive.
• .ulink, .ulist, .uignore will be removed completely.

Looking forward to your feedback!

[joseph.artsimovich]

This suggestion sounds reasonable. It should serve our purposes nicely. The ability to mount extra data directories at runtime is particularly welcome.

[aargh]

Hi

Great idea. Still think about working with the virtual filesystems

[sergej.koptev]

Hi. Looks nice, but there are some considerations:
1. It's necessary to be able to save/load the list of mounted directories to file (xml-based, maybe). 
Use-case:
- Somebody creates a new project, and list of mounted assets. All this stuff uploaded to CVS.
- Other users will now be able to download the project along with the configured assets mount list (patches in this list should be relative, to prevent manual correcting after project download).

2. "By default all shares are mounted as read only" - great idea. Maybe, these "read-only" data should have visualisation in editor (to differentiate them from editable by current user in current project).
Attributes "read/write" should be saved at described "mount list" too.
Seemings that there will some projects for assets import and edit, and some - only for assets placing on scene.

3. The most important.
"You can only mount directories to the root data directory of the current project." - it's unusable.
Actually, you already described "bad case": " you can’t mount <REPO1>/data/props/table if you already mounted <REPO1> or <REPO1>/props/chair "

In these system, there will be hell trash in /data root folder. Chairs, tanks, tables, machineguns,.... All of them in /data.
Actually, structure will be  more complicated, and looks like
objects/static/buildings/town
objects/static/buildings/military
objects/static/buildings/metro_stations
... <10-20 more>

For example, we are have to include first 3 patches. How? Dump the to the heap, at /data ? Unusable.
Of course, it's necessary to keep structure, and mount them like
data/objects/static/buildings/town
data/objects/static/buildings/military
...