Syncker Plugin

• The Running Syncker for a custom project article for more details on Syncker initialization and synchronization code.
• The Syncker-specific options article for more details on Syncker command-line options and console commands.
• The Video Wall Output with AppWall Plugin article for more details on using the AppWall plugin for multi-monitor slave configurations.

Syncker API:

• The Manager interface article for more details on managing Syncker via code (C++, C#, UnigineScript).
• The Master interface article for more details on managing Syncker master via code (C++, C#, UnigineScript).
• The Slave interface article for more details on managing Syncker slave via code (C++, C#, UnigineScript).

The order of launching master and slave applications does not matter: you can launch several Slaves, then the Master, and then the rest of the Slaves — synchronization will start automatically. Master starts the session as soon as all Slaves are connected.

In case only the Master is present (-sync_count 1) and connection of additional slaves on the fly is enabled (-sync_allow_extra_slaves 1), the session starts immediately and lasts forever (until the Master is on).

Syncker is able to work in both, synchronous and asynchronous mode. In synchronous mode all computers work with the current frame, the Master proceeds to the next frame only when the current one is rendered on all Slaves. This mode is a recommended one and is enabled by default. In case there are a lot of Slaves used, resulting in significant growth of the network load, it is recommended to try and reduce load by enabling broadcast or multicast addressing modes. If changing the addressing mode does not solve the problem, you are recommended to switch Syncker to the asynchronous mode and adjust send rate, interpolation/extrapolation parameters.

In synchronous mode there are two synchronization points in each frame:

• Data Sync Point - In the postUpdate() the Master sends all changes for the current frame to Slaves and suspends the thread and waits until all Slaves report that packets were received successfully. After reporting all Slaves are also put on hold until the last one reports to the Master, which in turn resumes its own thread as well as the threads of all the Slaves.
  This point guarantees that all computers render the same frame. Frame rate will be the same as the FPS on slowest host. So, in case there is a spike on one computer it will affect all others. This ensures seamlessness of the image across all displays unlike the asycnhronous mode where spikes result in tears between the channels observed here and there.
• Swap Sync Point - No data is transferred here only thread synchronization is performed for Master and Slaves.
  This point guarantees that DRAW (GPU bound) processes on all computers swap buffers simultaneously. This is especially important for applications running at frame rates lower than the vertical retrace rate (e.g., 30 FPS). This sync point reduces the so-called "accordion effect" of objects in motion seeming to grow and shrink as they move across from one screen to the next, keeping adjacent displays "synchronized".

Advantages (as compared to asynchronous mode):

• Works with physics, as all computers work with the current frame.
• Best synchronization quality pleasing the eye
• No resources wasted on calculating interpolation, all data is sent "as is".

Disadvantages (as compared to asynchronous mode):

• Increased network load, as this mode does not have options for load reduction unlike the async one.
• At least 1-2 milliseconds are spent on synchronization (waiting for other threads).
  Notice

  Async mode can provide higher frame rate than the synchronous one. In case one Slave goes unresponsive for a long time all others shall remain suspended until disconnected by timeout (30 seconds by default).

Asynchronous Mode#

Slaves are not silent, they can send messages to the Master, they can even control it as well as they can control other Slaves (e.g., run the profiler on all computers or shutdown all applications) by sending the sync command.

For sending and receiving messages between the Master and Slaves Syncker uses a single UDP socket with the following five technologies:

• Reliable - guarantees packet delivery using the so-called acknowledge packets. When the Master sends a message, the Slave reports that the message is received. In case a Slave does not reply in a certain period of time the Master tries to re-send the message.
• Sequenced - guarantees that all packets are received in the right order, finding all duplicates. Each packet has a number and until a Slave receives the packet, the Master shall not send next ones to it (with greater numbers).
• Merged - small messages are combined into a greater one instead of sending them all one right after the other (send a single 100-bytes packet rather than sending 10 packets 10-bytes each). This makes sending acknowledge packets to the sender simpler and faster.
• Fragmented - a large message exceeding the MTU size (Maximum Transmission Unit), shall be split into several packets having the MTU size. This ensures delivery of all packets protecting Slaves from buffer overflow caused by a single large packet.
• Compressed - all messages are compressed using the LZ4 algorithm before sending. This reduced network load, but slightly increases the load on the computer itself.

So, here's how the packets are delivered:

• During the frame up to the Data Sync Point (in case of a synchronous mode) the Master prepares messages for sending and places them to a queue. This "big message" is compressed, split into parts according to the MTU size, wrapped into packets with the following three numbers specified for each of them: sequence number (packet group), fragment number, and total number of fragments in this sequence.
• Then the packets are sent.
• Upon receiving any of the packets a Slave changes the size of its sliding window in accordance with the number of fragments and adds the received fragment to this window.
• In case of missing/lost packets a Slave adds the corresponding information on the missing part it requests to be sent again to the acknowledge packet dispatched to Master.

Delivery Modes#

Addressing Modes#

Syncker allows synchronizing views from multiple cameras. There are two types of cameras:

• Main master camera — a single camera that corresponds to the main viewer's position. The screen configuration determines viewports relative to this camera. Example: a camera in the plane's cockpit, corresponding the pilot's point of view.
• Auxiliary camera — an additional camera (static or dynamic) that can be set anywhere in the scene. You can have as many cameras of this type as necessary. Example: a ground-based surveillance camera or a thermal imaging camera mounted on the plane's wing.

By default, the main master camera is used. You can create auxiliary cameras and specify their viewports to be displayed by selected Slaves. Such cameras are synchronized automatically.

Syncker offers you an extreme flexibility of viewport configuration. You can use up to 6 monitors on a single slave, each having it own viewport assigned. For this purpose, you should use the AppWall plugin.

You can set up the desired screen / projection configuration for the Syncker in the screen/projection setup mode. To activate this mode, use the syncker_setup console command from the master or a slave:

syncker_setup 1

Screen configuration for the Syncker has to meet the requirements.

Two types of configuration can be used:

• A multi-monitor setup formed by displays, where each slave renders its viewport on the corresponding monitor (or several monitors).
  Notice

  Projection and modelview matrices are automatically calculated on the basis of the viewer's position.
  The following picture shows the screen configuration for the multi-monitor setup, where each slave renders its viewport to a single monitor.

  

  Screen Configuration (the left picture) for Multi-Monitor Setup (the right picture)
• A multi-projector setup formed by projectors, where each slave renders its viewport via the corresponding projector. By default, the projectors display images as seen from the viewer's position.

  

  Projector Configuration (the left picture) for Multi-Projector Setup (the right picture)
  Notice

  Projection and modelview matrices are automatically calculated on the basis of the viewer's position.
  A special case of the multi-projection setup is CAVE (Cave Automatic Virtual Environment):

  

  Projection Configuration for CAVE

Basically the Syncker has 4 states:

• Wait for connections - waiting until all Slaves are connected.
• Starting... - MTU calculation and startup synchronization of all hosts.
• Session Started - the session is started.
• Session Finished - the session is over, all Slaves are disconnected.

Let us consider Syncker running in the synchronous mode using broadcast addressing with the Master and a single Slave:

1. After being launched, the Master opens the port (-sync_port), and starts listening to it. The Syncker status is switched to "Wait for connections".
2. The Slave gets the list of all available network adapters (Wifi, Ethernet), finds a broadcast address for each of them, and periodically sends connection request packets (broadcast mode). The Syncker status is "Wait for connections".
3. The Master receives the connection request packet, registers the Slave and sends a connection response packet (unicast mode), where it indicates the currently loaded world. When all Slaves are connected, the Syncker status is switched to "Starting..." and Master sends projection settings to all of them.
4. As soon as the Slave receives the confirmation from the Master the status is switched to "Starting..." and starts loading the world from the Master.
5. While the Syncker status remains "Starting..." both, Master and Slave, are trying to find the maximum possible MTU value (currently limited to 1432 bytes).
6. Upon completion the Slave sends a message that it is ready to continue to the Master. After receiving such messages from all Slaves (and finding an MTU for each Slave), the Master changes the status to "Session Started" notifying all hosts.
7. The session has started, everything is ready for work.

Suppose, you have prepared a "static" project (meaning that no new objects are created in it and no existing ones are deleted or moved) with some cinematics. This corresponds to the majority of archviz projects or various types of demonstrations.

Suppose, you'd like to demonstrate this project using a wall or a CAVE system. Syncker should do the job here, but there is nothing in your project about it, not a single line of code! Well, you don't need it, simply use a special system script that launches the Syncker! That's all you should do, as the only thing moving during the cinematic sequence is the camera (which is synchronized automatically) with all other objects being static (no need to sync them).

This is the simplest mode, you don't have to know IP addresses or ports neither do you have to write any code. All you need to know is:

• Who shall be the Master.
• What is the number of Slaves.

Then you simply do the following:

• Run a Master application substituting the System Runtime Script with the special one (core/systems/syncker/unigine.cpp) and providing necessary startup command-line options to define it as the Master and set the total number of host computers:
  Shell commands

  <your_app> -system_script core/systems/syncker/unigine.cpp -extern_plugin "Syncker" -sync_master 1 -sync_count 2

  -sync_count here sets the total number of computers (Slaves + Master). In this simplest case -sync_count 2 - means a single Master and a single Slave.

• Run each Slave application using the same system script and indicating that the host is a Slave (not Master):
  Shell commands

  <your_app> -system_script core/systems/syncker/unigine.cpp -extern_plugin "Syncker" -sync_master 0

That's it! Run Master and Slave(s) in any order (it doesn't matter). Slaves shall automatically find the Master in the network and connect to it.

The basic workflow is as follows:

1. Implement Syncker logic for your Master and Slave application (you can use the same application on both Master and Slave sides).

2. Prepare your environment.

   It is recommended to use a 100 Mb LAN. Otherwise, you may experience network lags (see Troubleshooting section).

   All applications you use must have access to all Unigine files and project data. So, you should copy your project to all computers. If some nodes are missing in the world file on a local computer, they will not be rendered.

3. Run the Master application.

   To run a Master application provide necessary startup command-line options, e.g.:

   Shell commands

   <your_app> -extern_plugin "Syncker" -sync_master 1 -sync_view <display_name> -sync_count <number_of_hosts>

   Notice

   The order of launching Master and Slave applications does not matter.

4. Run a Slave application.

   To run a Slave application provide necessary startup command-line options, e.g.:

   Shell commands

   <your_app> -extern_plugin "Syncker" -sync_master 0 -sync_view <display_name>

   If you want a Slave with a multi-monitor configuration to render several viewports, you should use the AppWall plugin and assign viewports to monitors via the sync_view_n options, e.g.:

   Shell commands

   <your_app> -extern_plugin "AppWall,Syncker" -sync_master 0 -sync_view_0 <display_0> -sync_view_1 <display_1>

   Notice

   The order of plugins in the list matters: AppWall must be specified before Syncker.
5. Set up screen / projection configuration. This step is performed when all Syncker hosts are successfully connected and working.

The console shall look like this:

A Slave periodically sends broadcast messages when trying to find the Master that responds, but there's no reply. Messages do not reach the Master.

Solutions:

• Make sure that the port used by the Slave for sending/receiving messages (Slave UDP port) is not blocked by the firewall on the Master. Try disabling firewalls on the Master and all the Slaves.
• Make sure that broadcast / multicast messages are not blocked by the router/switch. It is unlikely, but still possible.
• Try to explicitly specify Master's IP address via the sync_master_address command, for example:
  Source code

  -sync_master_address 192.168.0.1

  In this case a Slave shall skip the phase of searching for the Master via broadcast messages and shall use unicast mode for direct connection.
• Make sure that Master really waits for new connections. Check the sync_count value, is it greater than 1?

In this case the console shall look like this:

Connection requests from a Slave were received by the Master, the connaction was opened and the session was started, but connection response did not reach the Slave.

Solutions:

• Make sure that the port used by the Slave for sending/receiving messages (Slave UDP port) is not blocked by the firewall. Try disabling firewalls on the Master and all the Slaves.
• Make sure that broadcast / multicast messages are not blocked by the router/switch. It is unlikely, but still possible.
• Try using a unicast connection:
  Source code

  -sync_method unicast

• Check that the broadcast address found is valid, try setting up a broadcast channel for all hosts:
  Source code

  -sync_broadcast_address 255.255.255.255

• Try setting another multicast address on the Master and Slaves via the sync_multicast_address command.
• Check IP/port of the Slave connected to the Master. Is this the right Slave, or maybe there is another instance of the Slave looking for the Master on the same port? Try changing port number on the Master and Slaves (e.g., -sync_port 25100) and restart applications.

If the network latency is too large despite 1Gb bandwidth or higher, it can be caused by a 100 Mb or 10 Mb device connected to a network. Data exchange rate will drop down to the maximum rate supported by such device, slowing down Syncker connection speed.

• Some 100 Mb or 10 Mb devices can have a working network interface when they are turned off.
• It is also possible that when turned off, 1 Gb devices have a network interface working at 100 Mb rate, which slows down connection in the LAN.

Integration with the Microprofile tool enables you to monitor performance, detect bottlenecks, and eliminate them.

Useful Tool#