This page has been translated automatically.
Video Tutorials
Interface
Essentials
Advanced
How To
UnigineEditor
Interface Overview
Assets Workflow
Settings and Preferences
Working With Projects
Adjusting Node Parameters
Setting Up Materials
Setting Up Properties
Lighting
Landscape Tool
Using Editor Tools for Specific Tasks
Extending Editor Functionality
Built-in Node Types
Nodes
Objects
Effects
Decals
Light Sources
Geodetics
World Nodes
Sound Objects
Pathfinding Objects
Players
Programming
Fundamentals
Setting Up Development Environment
Usage Examples
C++
C#
UnigineScript
UUSL (Unified UNIGINE Shader Language)
Plugins
File Formats
Rebuilding the Engine Tools
GUI
Double Precision Coordinates
API
Containers
Common Functionality
Controls-Related Classes
Engine-Related Classes
Filesystem Functionality
GUI-Related Classes
Math Functionality
Node-Related Classes
Objects-Related Classes
Networking Functionality
Pathfinding-Related Classes
Physics-Related Classes
Plugins-Related Classes
IG Plugin
CIGIConnector Plugin
Rendering-Related Classes
Content Creation
Content Optimization
Materials
Art Samples
Tutorials

Distributed Computing

Warning
The functionality described in this article is not available in the Community SDK edition.
You should upgrade to Engineering / Sim SDK edition to use it.

Terrain generation is a complex task and may require a substantial amount of time depending on the area size and data resolution. Sandworm allows using distributed computing to generate terrain.

Notice
Distributed computing makes sense when used for speeding up terrain generation that takes more than an hour. Applying distributed computing to a small terrain area with low-detail data (10 meters or more per pixel) may be unprofitable at all.

The concept is to use several computers united into a network, one of which is Master and assigns tasks, and the other are Workers that perform portions of work (processing of source data) assigned to them. UNIGINE Editor with the Sandworm tool and a terrain project should be open on Master, while Workers run the console application. Source files (geodata) and common cache shall be stored in a shared folder. If a Worker still has available resources, additional processes may be run on it and participate in terrain generation immediately, i.e. without restarting the generation process. Master creates the asset based on the result.

Notice
All workstations performing computations for a certain terrain generation job must be running on the same operating system (Windows or Linux). UNIGINE SDK version should also be the same.

Participants of the process:

Master

Computer that manages the whole generation process, forms the task pool, and distributes tasks between Workers. Master tracks all connected Workers and assigns tasks to them. Master itself also performs calculations. Is run via Sandworm.

Worker

Computer connected over the network to Master and performing certain tasks assigned by Master. Is enabled by running bin/SandwormNode_x64.exe.

Notice
The number of Workers connected to one Master is not limited (as long as the network bandwidth allows that).

Launching Order#

You can launch Master and Workers in any sequence.

You can run additional Workers after the terrain generation process has started, and Master will distribute jobs for them.

One machine can have several Workers running. Worker can have several processes enabled (forks). Forks are activated via the menu on Master.

Setting Up the Master#

To set up Master, select the Distributed option in the Generation section. In the Parameters, toggle on the Enabled option.

Master settings:

Enabled If toggled on, distributed computing is enabled for the terrain generation, and the machine running Editor with Sandworm is Master.
Broadcast Port Port for listening to Sandworm by Workers.
Server Port Port used for information exchange.
Workload on Master Percentage of terrain generation performed by Master.
Notice
All sources and project cache should be stored in a shared location accessible by both Master and all Workers. The path to this location should be identical on Master and Workers.

When the project is ready for generation, and you click Generate, the window with the list of available Workers appears. Right-clicking a Worker allows the following:

Worker options available on Master

Worker options available on Master
Create Fork Creates one more process on the same Worker. We estimate a reasonable number of forks as 4 to 6. Higher values reduce the Worker's performance.
Shutdown Removes the fork or Worker from the list.
Clear Cache Clears the Worker's local cache.

Setting Up a Worker#

Notice
Worker should have the same version of OS and UNIGINE SDK as Master.

To set up a Worker:

  1. Run the SandwormNode_x64.exe console application stored in the bin/ folder of your project.
  2. Set the path for storing the local cache using the --sw_local_cache_path console command.
  3. Use --sw_broadcast_port, if you want to change the default broadcast port.
  4. Worker detects Master on the network, connects to it, downloads the scene, prepares all layers (checks their accessibility) and is ready for terrain generation.

Worker application is running and ready for terrain generation

Worker application is running and ready for terrain generation

You can run multiple console applications on one machine, and/or fork one Worker via Master to create more processes.

Notice
  • Operating System versions on Master and Workers should be identical.
  • The SDK version of the Editor run on Master should be the same as the version of the console app (including precision).
  • Development/Release version of SDK does not matter.

When the terrain generation is completed and you don't need Workers anymore, close the console application on Workers or shutdown processes via Master.

Using Distributed Computing#

Workflow:

  1. Prepare your environment.

    Connect all computers to the network. It is recommended to use a network with at least 1 Gb bandwidth. Otherwise, you may experience network lags (see Troubleshooting section). Broadcasting should not be blocked on your network, otherwise distributed computation would be unavailable.

    Notice
    It is recommended to use an SSD drive to store the generation cache and output files. If an HDD is used, the generation performance is significantly lower due to multiple read/write and head positioning operations.
  2. On each Worker computer, run the Worker console application.

    Notice
    The versions of binary executable files must be the same on all computers.
  3. On the Master computer:

    • Run UnigineEditor and open the Sandworm tool (Tools -> Sandworm).
    • Open the *.sworm asset you are going to generate and configure the output settings.
    • Click the Distributed option and specify the Master's settings. Save the project configuration.
    • Click the Generate button. The window with the list of available Workers will open.

      List of Workers

      List of Workers

      Set the number of forks you need and click Generate.

    • During the generation process you can connect more Workers, if necessary: run the console application on a Worker (even it is already running one or several). It will automatically connect to Master, and Master will assign a process to this Worker.

    • When the terrain generation is completed, disable Worker processes.

Troubleshooting#

If the network latency is too large despite 1Gb bandwidth or higher, this can be caused by a 100 Mb or 10 Mb device connected to the network. Data exchange rate drops down to the maximum rate supported by such device, slowing down the generation speed.

  • Some 100 Mb or 10 Mb devices can have a working network interface when they are turned off.
  • Some 1 Gb devices, when turned off, have a network interface working at 100 Mb rate, which slows down connection on LAN.

Other recommendations:

  • Use SSD to store shared source data.
  • Add the application to the antivirus exceptions, as it may reduce performance.
  • Check that network ports used by Master and Workers are allowed by the firewall.
Last update: 2021-09-29
Build: ()