Syncker Plugin
Syncker is a multi-node rendering system that makes it easy to create an immersive CAVE system or a large-scale visualization wall on a computer cluster synchronized over the network in real-time.
Simply connect multiple computers into one high-resolution seamless display using LAN. And it does not matter if these computers are running on Windows and Linux at the same time, as Syncker works even across different platforms. Moreover, created virtual environments can have any monitor configuration, since all viewports are fully configurable and multi-monitor grids on slave computers are supported.
Moreover, Syncker offers you a flexible customization of the whole synchronization process, based on custom user messages. Instead of sending the whole bunch of transformation data for objects that can only be rotated or sending huge amounts of transformation data for all parts of complex objects, when their positions can be determined using just a handful of simple parameters, you can send a single quaternion or a set of parameters.
See Also#
- The Running Syncker for a custom project article for more details on Syncker initialization and synchronization code.
- The Syncker-specific options article for more details on Syncker command-line options and console commands.
- The Video Wall Output with AppWall Plugin article for more details on using the AppWall plugin for multi-monitor slave configurations.
Syncker API:
- The Manager interface article for more details on managing Syncker via code (C++, C#, UnigineScript).
- The Master interface article for more details on managing Syncker master via code (C++, C#, UnigineScript).
- The Slave interface article for more details on managing Syncker slave via code (C++, C#, UnigineScript).
Structure and Principles#
Syncker allows you to render a world across different computers synchronized over LAN. These computers can be of two types:
Master | Master application is an application running on the main computer. In addition to rendering, it also performs animations and calculations of physics, and controls slaves. The main computer should have the most advanced hardware configuration, as it performs most calculations.
All other applications are synchronized with the master. |
---|---|
Slaves | Slave applications are all other applications running on computers connected over the network. There can be an unlimited number of Slaves connected to one master (as long as the network bandwidth allows). The purpose of such applications is to render all nodes that are seen in their viewports.
Configuration of slaves is set up to fit the configuration of monitors. |
Launching Order#
The order of launching master and slave applications does not matter: you can launch several Slaves, then the Master, and then the rest of the Slaves — synchronization will start automatically. Master starts the session as soon as all Slaves are connected.
Synchronization Modes#
Syncker is able to work in both, synchronous and asynchronous mode. In synchronous mode all computers work with the current frame, the Master proceeds to the next frame only when the current one is rendered on all Slaves. This mode is a recommended one and is enabled by default. In case there are a lot of Slaves used, resulting in significant growth of the network load, it is recommended to try and reduce load by enabling broadcast or multicast addressing modes. If changing the addressing mode does not solve the problem, you are recommended to switch Syncker to the asynchronous mode and adjust send rate, interpolation/extrapolation parameters.
In synchronous mode there are two synchronization points in each frame:
- Data Sync Point - In the postUpdate() the Master sends all changes for the current frame to Slaves and suspends the thread and waits until all Slaves report that packets were received successfully. After reporting all Slaves are also put on hold until the last one reports to the Master, which in turn resumes its own thread as well as the threads of all the Slaves.
This point guarantees that all computers render the same frame. Frame rate will be the same as the FPS on slowest host. So, in case there is a spike on one computer it will affect all others. This ensures seamlessness of the image across all displays unlike the asycnhronous mode where spikes result in tears between the channels observed here and there. - Swap Sync Point - No data is transferred here only thread synchronization is performed for Master and Slaves.
This point guarantees that DRAW (GPU bound) processes on all computers swap buffers simultaneously. This is especially important for applications running at frame rates lower than the vertical retrace rate (e.g., 30 FPS). This sync point reduces the so-called "accordion effect" of objects in motion seeming to grow and shrink as they move across from one screen to the next, keeping adjacent displays "synchronized".
Advantages (as compared to asynchronous mode):
- Works with physics, as all computers work with the current frame.
- Best synchronization quality pleasing the eye
- No resources wasted on calculating interpolation, all data is sent "as is".
Disadvantages (as compared to asynchronous mode):
- Increased network load, as this mode does not have options for load reduction unlike the async one.
- At least 1-2 milliseconds are spent on synchronization (waiting for other threads). Async mode can provide higher frame rate than the synchronous one. In case one Slave goes unresponsive for a long time all others shall remain suspended until disconnected by timeout (30 seconds by default).
Asynchronous Mode#
Asynchronous mode provides synchronization between the channels by means of reconstruction using the data of previous frames. In asynchronous mode all computers draw their own frames, each with its own frame rate. So, unlike the synchronous mode where we synchronize frames, in the asynchronous we synchronize time. This mode should be used when the synchronous mode cannot provide stable smooth synchronization.
UNIGINE Syncker uses Interpolated Snapshots (IS) to tackle with the problem of lost packets between the Master and Slaves. It works by taking two old, but known positions and interpolating the object between them. It is accomplished by having a buffer of received positions and rotations, along with the time they represent. We usually take our current local time minus some predefined amount — interpolation period (40 ms by default), then go into our buffer, find the two indices that are just before and just after this time and interpolate.
If we don't have a received position and rotation for the time we're looking for, the extrapolation (guessing) is used. It also has a limited time — extrapolation period (200 ms by default). If the extrapolation period is over but there are still no packets received, all objects will freeze.
In most cases, this method provides a very accurate representation of the world to each Slave, as in general only already known positions of remote objects are rendered and in rare cases the system will try to extrapolate (guess) where an object is. This, however, comes at a cost, as we always render 40 ms (interpolation period) behind current time, so that new packets have time to arrive with data.
Two-Way Communication#
Slaves are not silent, they can send messages to the Master, they can even control it as well as they can control other Slaves (e.g., run the profiler on all computers or shutdown all applications) by sending the sync command.
For sending and receiving messages between the Master and Slaves Syncker uses a single UDP socket with the following five technologies:
- Reliable - guarantees packet delivery using the so-called acknowledge packets. When the Master sends a message, the Slave reports that the message is received. In case a Slave does not reply in a certain period of time the Master tries to re-send the message.
- Sequenced - guarantees that all packets are received in the right order, finding all duplicates. Each packet has a number and until a Slave receives the packet, the Master shall not send next ones to it (with greater numbers).
- Merged - small messages are combined into a greater one instead of sending them all one right after the other (send a single 100-bytes packet rather than sending 10 packets 10-bytes each). This makes sending acknowledge packets to the sender simpler and faster.
- Fragmented - a large message exceeding the MTU size (Maximum Transmission Unit), shall be split into several packets having the MTU size. This ensures delivery of all packets protecting Slaves from buffer overflow caused by a single large packet.
- Compressed - all messages are compressed using the LZ4 algorithm before sending. This reduced network load, but slightly increases the load on the computer itself.
So, here's how the packets are delivered:
- During the frame up to the Data Sync Point (in case of a synchronous mode) the Master prepares messages for sending and places them to a queue. This "big message" is compressed, split into parts according to the MTU size, wrapped into packets with the following three numbers specified for each of them: sequence number (packet group), fragment number, and total number of fragments in this sequence.
- Then the packets are sent.
- Upon receiving any of the packets a Slave changes the size of its sliding window in accordance with the number of fragments and adds the received fragment to this window.
- In case of missing/lost packets a Slave adds the corresponding information on the missing part it requests to be sent again to the acknowledge packet dispatched to Master.
Delivery Modes#
Basically, you can use the following delivery modes for sending/receiving messages:
- Reliable - Reliable and sequenced mode, enabled by default. All packets shall be delivered to the recipient in the exact order they were sent.
- Unreliable - Pure UDP. Packets may be lost, duplicated, or received in an order that differs from the one they were sent. Packets are not compressed, fragmented, or merged. Everything is sent "as is". This mode is the fastest one as it requires no additional time for processing. You can use this mode for sending timestamped auxiliary data (e.g., for interpolation) having the size close to MTU. Example: sending 10 times a second positions of 10 aircrafts with the information on the current time, status of flaps, slats, ailerons, landing gears, etc.
- Sequenced - Packets may be lost, but never duplicated, they arrive in the exact order they were sent. This mode can be used for sending auxiliary data having the size close to MTU. Example: sending each frame positions of 10 aircrafts with the information on the status of flaps, slats, ailerons, landing gears, etc.
Multiple systems may use Syncker's network simultaneously (e.g. IG and a users App application). For convenience, all messages are sent and received via named channels. If the specified channel does not exist, it shall be created.
Addressing Modes#
The following addressing modes for communication are available:
Depending on your project's requirements, you can choose the one that fits best: ensures the lowest possible load, or provides additional features like easy testing on a single PC or communication between hosts from different subnets using different ports.
Multiple Cameras and Multi-Monitor Slaves#
Syncker allows synchronizing views from multiple cameras. There are two types of cameras:
- Main master camera — a single camera that corresponds to the main viewer's position. The screen configuration determines viewports relative to this camera. Example: a camera in the plane's cockpit, corresponding the pilot's point of view.
- Auxiliary camera — an additional camera (static or dynamic) that can be set anywhere in the scene. You can have as many cameras of this type as necessary. Example: a ground-based surveillance camera or a thermal imaging camera mounted on the plane's wing.
By default, the main master camera is used. You can create auxiliary cameras and specify their viewports to be displayed by selected Slaves. Such cameras are synchronized automatically.
Syncker offers you an extreme flexibility of viewport configuration. You can use up to 6 monitors on a single slave, each having it own viewport assigned. For this purpose, you should use the AppWall plugin.
Screen Configurations#
You can set up the desired screen / projection configuration for the Syncker in the screen/projection setup mode. To activate this mode, use the syncker_setup console command from the master or a slave:
syncker_setup 1
Screen configuration for the Syncker has to meet the requirements.
Two types of configuration can be used:
- A multi-monitor setup formed by displays, where each slave renders its viewport on the corresponding monitor (or several monitors).
The following picture shows the screen configuration for the multi-monitor setup, where each slave renders its viewport to a single monitor.Projection and modelview matrices are automatically calculated on the basis of the viewer's position.Screen Configuration (the left picture) for Multi-Monitor Setup (the right picture)
- A multi-projector setup formed by projectors, where each slave renders its viewport via the corresponding projector. By default, the projectors display images as seen from the viewer's position.
Projector Configuration (the left picture) for Multi-Projector Setup (the right picture)A special case of the multi-projection setup is CAVE (Cave Automatic Virtual Environment):Projection and modelview matrices are automatically calculated on the basis of the viewer's position.Projection Configuration for CAVE
Syncker Pipeline#
Basically the Syncker has 4 states:
- Wait for connections - waiting until all Slaves are connected.
- Starting... - MTU calculation and startup synchronization of all hosts.
- Session Started - the session is started.
- Session Finished - the session is over, all Slaves are disconnected.
Let us consider Syncker running in the synchronous mode using broadcast addressing with the Master and a single Slave:
- After being launched, the Master opens the port (-sync_port), and starts listening to it. The Syncker status is switched to "Wait for connections".
- The Slave gets the list of all available network adapters (Wifi, Ethernet), finds a broadcast address for each of them, and periodically sends connection request packets (broadcast mode). The Syncker status is "Wait for connections".
- The Master receives the connection request packet, registers the Slave and sends a connection response packet (unicast mode), where it indicates the currently loaded world. When all Slaves are connected, the Syncker status is switched to "Starting..." and Master sends projection settings to all of them.
- As soon as the Slave receives the confirmation from the Master the status is switched to "Starting..." and starts loading the world from the Master.
- While the Syncker status remains "Starting..." both, Master and Slave, are trying to find the maximum possible MTU value (currently limited to 1432 bytes).
- Upon completion the Slave sends a message that it is ready to continue to the Master. After receiving such messages from all Slaves (and finding an MTU for each Slave), the Master changes the status to "Session Started" notifying all hosts.
- The session has started, everything is ready for work.
Using Syncker#
Simple Synchronized Demonstration#
Suppose, you have prepared a "static" project (meaning that no new objects are created in it and no existing ones are deleted or moved) with some cinematics. This corresponds to the majority of archviz projects or various types of demonstrations.
Suppose, you'd like to demonstrate this project using a wall or a CAVE system. Syncker should do the job here, but there is nothing in your project about it, not a single line of code! Well, you don't need it, simply use a special system script that launches the Syncker! That's all you should do, as the only thing moving during the cinematic sequence is the camera (which is synchronized automatically) with all other objects being static (no need to sync them).
This is the simplest mode, you don't have to know IP addresses or ports neither do you have to write any code. All you need to know is:
- Who shall be the Master.
- What is the number of Slaves.
Then you simply do the following:
- Run a Master application substituting the System Runtime Script with the special one (core/systems/syncker/unigine.cpp) and providing necessary startup command-line options to define it as the Master and set the total number of host computers:
<your_app> -system_script core/systems/syncker/unigine.cpp -extern_plugin "Syncker" -sync_master 1 -sync_count 2
-sync_count here sets the total number of computers (Slaves + Master). In this simplest case -sync_count 2 - means a single Master and a single Slave.
- Run each Slave application using the same system script and indicating that the host is a Slave (not Master):
<your_app> -system_script core/systems/syncker/unigine.cpp -extern_plugin "Syncker" -sync_master 0
That's it! Run Master and Slave(s) in any order (it doesn't matter). Slaves shall automatically find the Master in the network and connect to it.
Basic Workflow#
The basic workflow is as follows:
- Implement Syncker logic for your Master and Slave application (you can use the same application on both Master and Slave sides).
-
Prepare your environment.
It is recommended to use a 100 Mb LAN. Otherwise, you may experience network lags (see Troubleshooting section).
All applications you use must have access to all Unigine files and project data. So, you should copy your project to all computers. If some nodes are missing in the world file on a local computer, they will not be rendered.
-
Run the Master application.
To run a Master application provide necessary startup command-line options, e.g.:
<your_app> -extern_plugin "Syncker" -sync_master 1 -sync_view <display_name> -sync_count <number_of_hosts>
The order of launching Master and Slave applications does not matter. -
Run a Slave application.
To run a Slave application provide necessary startup command-line options, e.g.:
<your_app> -extern_plugin "Syncker" -sync_master 0 -sync_view <display_name>
If you want a Slave with a multi-monitor configuration to render several viewports, you should use the AppWall plugin and assign viewports to monitors via the sync_view_n options, e.g.:
<your_app> -extern_plugin "AppWall,Syncker" -sync_master 0 -sync_view_0 <display_0> -sync_view_1 <display_1>
The order of plugins in the list matters: AppWall must be specified before Syncker. - Set up screen / projection configuration. This step is performed when all Syncker hosts are successfully connected and working.
Debug Window#
Syncker's debug window enables you to monitor the hierarchy of objects at run time. The whole hierarchy means all objects (not only the ones registered in the Editor) including objects in cache, internal structure of node references, etc. When a node is selected, all necessary debug information is displayed. This information is updated each frame.
Search option allows you to find any node by its name or ID (on the Master or a Slave), so a faulty node can be found quickly.
To open the debug window, run the syncker_debug console command from the master or a slave as follows:
syncker_debug 1
- Open the System Menu by pressing Esc.
- Go to the Plugins tab and enable the Show debug window option in the Syncker group:
The debug window will open:
Troubleshooting#
Basically there are two types of problems:
Slave Cannot Connect to Master#
The console shall look like this:
A Slave periodically sends broadcast messages when trying to find the Master that responds, but there's no reply. Messages do not reach the Master.
Solutions:
- Make sure that the port used by the Slave for sending/receiving messages (Slave UDP port) is not blocked by the firewall on the Master. Try disabling firewalls on the Master and all the Slaves.
- Make sure that broadcast / multicast messages are not blocked by the router/switch. It is unlikely, but still possible.
- Try to explicitly specify Master's IP address via the sync_master_address command, for example:
In this case a Slave shall skip the phase of searching for the Master via broadcast messages and shall use unicast mode for direct connection.
-sync_master_address 192.168.0.1
- Make sure that Master really waits for new connections. Check the sync_count value, is it greater than 1?
Master Cannot Reply to Slave#
In this case the console shall look like this:
Connection requests from a Slave were received by the Master, the connaction was opened and the session was started, but connection response did not reach the Slave.
Solutions:
- Make sure that the port used by the Slave for sending/receiving messages (Slave UDP port) is not blocked by the firewall. Try disabling firewalls on the Master and all the Slaves.
- Make sure that broadcast / multicast messages are not blocked by the router/switch. It is unlikely, but still possible.
- Try using a unicast connection:
-sync_method unicast
- Check that the broadcast address found is valid, try setting up a broadcast channel for all hosts:
-sync_broadcast_address 255.255.255.255
- Try setting another multicast address on the Master and Slaves via the sync_multicast_address command.
- Check IP/port of the Slave connected to the Master. Is this the right Slave, or maybe there is another instance of the Slave looking for the Master on the same port? Try changing port number on the Master and Slaves (e.g., -sync_port 25100) and restart applications.
Network Latency#
If the network latency is too large despite 1Gb bandwidth or higher, it can be caused by a 100 Mb or 10 Mb device connected to a network. Data exchange rate will drop down to the maximum rate supported by such device, slowing down Syncker connection speed.
- Some 100 Mb or 10 Mb devices can have a working network interface when they are turned off.
- It is also possible that when turned off, 1 Gb devices have a network interface working at 100 Mb rate, which slows down connection in the LAN.
Integration with the Microprofile tool enables you to monitor performance, detect bottlenecks, and eliminate them.
Useful Tool#
If you have a source SDK, you can use a simple and useful tool to monitor the network messages exchange speed. It is server.usc found in source/tools/Interpreter/scripts/network/.