Syncker Plugin
Syncker is a multi-node rendering system that makes it easy to create an immersive CAVE system or a large-scale visualization wall on a computer cluster synchronized over the network in real-time.
Simply connect multiple computers into one high-resolution seamless display using LAN. And it does not matter if these computers are running on Windows and Linux at the same time, as Syncker works even across different platforms. Moreover, created virtual environments can have any monitor configuration, since all viewports are fully configurable and multi-monitor grids on slave computers are supported.
See Also
- The Running Syncker for a custom project article for more details on Syncker initialization and synchronization code.
- The Syncker-specific options article for more details on Syncker command-line options and console commands.
- The Video Wall Output with AppWall Plugin article for more details on using the AppWall plugin for multi-monitor slave configurations.
Syncker API:
- The Manager interface article for more details on managing Syncker via code (C++, UnigineScript).
- The Master interface article for more details on managing Syncker master via code (C++, UnigineScript).
- The Slave interface article for more details on managing Syncker slave via code (C++, UnigineScript).
Structure and Principles
Syncker allows you to render a world across different computers synchronized over LAN. These computers can be of two types:
| Master | Master application is an application running on the main computer apart from rendering it also performs calculations of physics, animations and controls slaves. Notice
The main computer should have the most advanced hardware configuration, as it performs most of the calculations.
All other applications are synchronized with a master. |
|---|---|
| Slaves | Slave applications are all other applications running on computers connected over the network. There can be an unlimited number of slaves connected to one master (as long as network bandwidth allows it). The purpose of such applications is to render all nodes that are seen in their viewports.
Configuration of slaves is set up to fit the configuration of monitors. |
Launching Order
The order of launching master and slave applications does not matter: you can launch several slaves, then the master, and then the rest of the slaves - synchronization will start automatically. This is achieved by having a buffer of object creation/deletion calls. Each time a new slave connects to the network, the master tells it, which objects are to be created or deleted.
Interpolation and Extrapolation
Unigine Syncker uses Interpolated Snapshots (IS) to tackle with the problem of lost packets. It works by taking two old, but known positions and interpolating the object between them. It is accomplished by having a buffer of received positions and rotations, along with the time they represent. We usually take our current local time minus some predefined amount - interpolation period (40 ms by default), then go into our buffer, find the two indices which are just before and just after this time and interpolate.
If we don't have a received position and rotation for the time we're looking for, the extrapolation (guessing) is used. It also has a limited time - extrapolation period (200 ms by default). If the extrapolation period is over, but there are still no packets received and all objects will freeze.
In most cases this method provides a very accurate representation of the world to each of the slaves, as in general only already known positions of the remote objects are rendered and in rare cases the system will try to extrapolate (guess) where an object is. This, however, comes at a cost, as we always render 40 ms (interpolation period) behind current time, so that new packets have time to arrive with data.
Optimized Synchronization
The scene contains static and dynamic objects, and "dynamic" doesn't mean that object's states are changed constantly. Taking this into account Syncker uses the following strategy:
- for static objects synchronization packets are sent on their creation and deletion.
- for dynamic objects the packets are sent only when their state has changed.
Moreover, Syncker offers you a flexible customization of the whole synchronization process, based on custom user TCP/UDP messages. Instead of sending the whole bunch of transformation data for objects that can only be rotated or sending huge amounts of transformation data for all parts of complex objects, when their positions can be determined using just a handful of simple parameters, you can send a single quaternion or a set of parameters.
All packets are compressed before sending (LZ4 or ZLIB). You can choose the desired algorithm and compression quality.
Thus, the excessive load on the network is minimized, while keeping performance high.
Two-Way Communication
Slaves are not silent anymore. If a slave fails to receive a message via UDP protocol, it immediately reports to the master (via TCP) and requests to resend the message again. In this case the master sends the current state of all synchronized objects via UDP to perform update.
Moreover, a slave can control the master and other slaves (e.g. run the profiler on all computers or shutdown all applications) by sending the syncker_console command.
Multiple Cameras and Multi-Monitor Slaves
Syncker allows to synchronize views from multiple cameras. There are two types of cameras:
- Main master camera - a single camera that corresponds to the main viewer's position. The screen configuration determines viewports relative to this camera. Example: a camera in the plane's cockpit, corresponding the the pilot's point of view.
- Auxiliary camera - an additional camera (static or dynamic), that can be set anywhere on the scene. You can have as many cameras of this type as necessary. Example: a ground-based surveillance camera or a thermal imaging camera mounted on the plane's wing.
Syncker offers you an extreme flexibility of viewport configuration. You can use up to 6 monitors on a single slave, each having it own viewport assigned. For this purpose you will have to use the AppWall plugin.
Screen Configurations
You can set up the desired screen / projection configuration for the Syncker in the screen/projection setup mode. To activate this mode use the syncker_setup console command from the master or a slave:
syncker_setup 1Screen configuration for the Syncker has to meet the requirements.
Two types of configuration can be used:
- A multi-monitor setup formed by displays, where each slave renders its viewport on the corresponding monitor (or several monitors).
NoticeThe following picture shows the screen configuration for the multi-monitor setup, where each slave renders its viewport to a single monitor.Projection and modelview matrices are automatically calculated on the basis of the viewer's position.
Screen Configuration (the left picture) for Multi-Monitor Setup (the right picture) - A multi-projector setup formed also by displays, where each slave renders its viewport via the corresponding projector as if a surface which the viewport was projected onto was a display.
A multi-projector setup. The screens are considered as displays.NoticeA special case of the multi-projection setup is a CAVE (Cave Automatic Virtual Environment):Projection and modelview matrices are automatically calculated on the basis of the viewer's position.
Projection Configuration for CAVE
Syncker Pipeline
To connect computers and synchronize them, the following pipeline is used.
- After being launched the master starts sending broadcast messages that it is available using a UDP protocol.
- All slaves in LAN after being launched listen to the selected UDP port. Upon receiving this message, they connect to a master via a TCP protocol.NoticeActually 2 TCP ports are used:
- TCP port - for messages.
- TCP ping port - for latency measurement (to ensure correct interpolation).
- After the connection was established, both UDP and TCP protocols are used to exchange data between the master and slaves.
- Over UDP protocol non-critical messages are sent. This includes messages on render parameters (for example, ambient color or HDR exposure), position of a player and all nodes and materials that are synchronized. It also includes frame-related information (if the game is enabled, the current frame number, frame duration, a time scale and even the game seed for random number generators).
- Over TCP protocol critical messages are sent. These are messages on creation and deletion of nodes and user messages.
Using Syncker
The basic workflow is as follows:
- Implement Syncker logic for your master and slave application (you can use the same application on both master and slave sides).
- Set up screen / projection configuration.
-
Prepare your environment.
It is recommended to use a 100 Mb LAN. Otherwise, you may experience network lags (see Troubleshooting section).
All applications you use must have access to all Unigine files and project data. So, you should copy your project to all computers. If some nodes are missing in the world file on a local computer, they will not be rendered.
-
Run the master application.
To run a master application you should substitute the System Runtime Script with the special one: core/systems/syncker/unigine.cpp and provide necessary startup command-line options, e.g.:
Shell commands<your_app_name> -system_script core/systems/syncker/unigine.cpp -extern_plugin "Syncker" -sync_master 1 -sync_broadcast_address xxx.xxx.xxx.xxx -sync_view_config <config_name>.sync -sync_view <display_name>NoticeThe order of launching master and slave applications does not matter. -
Run a slave application.
To run a slave application you should also substitute the System Runtime Script with the special one: core/systems/syncker/unigine.cpp and provide necessary startup command-line options, e.g.:
Shell commands<your_app_name> -system_script core/systems/syncker/unigine.cpp -extern_plugin "Syncker" -sync_master 0 -sync_slave_name <slave_name> -sync_view_config <config_name>.sync -sync_view <display_name>If you want a slave with a multi-monitor configuration to render several viewports, you should use the AppWall plugin and assign viewports to monitors via the sync_view_n options, e.g.:
Shell commands<your_app_name> -system_script core/systems/syncker/unigine.cpp -extern_plugin "AppWall,Syncker" -sync_master 0 -sync_slave_name <slave_name> -sync_view_config <config_name>.sync -sync_view_0 <display_0> -sync_view_1 <display_1>NoticeThe order of plugins in the list matters: AppWall must be specified before Syncker.
Debug Window
Syncker's debug window enables you to monitor the hierarchy of objects at run time. The whole hierarchy means all objects (not only the ones registered in the Editor) including objects in cache, internal structure of node references, etc. When a node is selected, all necessary debug information is displayed. This information is updated each frame.
Search option allows you to find any node by its name or ID (on the Master or on a Slave), so a faulty node can be found quickly.
To show the debug window, run the syncker_debug console command from the master or a slave as follows:
syncker_debug 1- Open the System Menu by pressing Esc.
- Go to the Plugins tab and enable the Show debug window option in the Syncker group:
The debug window will open:
Troubleshooting
If the network latency is too large despite 1Gb bandwidth or higher, it can be caused by a 100 Mb or 10 Mb device connected to a network. Data exchange rate will drop down to the maximum rate supported by such device, slowing down Syncker connection speed.
- Some 100 Mb or 10 Mb devices can have a working network interface when they are turned off.
- It is also possible that when turned off, 1 Gb devices have a network interface working at 100 Mb rate, which will slow down connection in LAN.
Useful Tool
If you have a source SDK, you can use a simple and useful tool to monitor the network messages exchange speed. It is server.usc found in source/tools/Interpreter/scripts/network/.