IG and Syncker

[Amerio.Stephane]

Hello,

The documentation about the IG and Syncker in 2.10 is a bit obscure to me, so please could you clarify the following points:

• IG and Syncker plugins can no longer be both specified in the extern_plugin command line. This raises an error dialog at runtime. Ok, so far so good, as IG now has its own internal sync mechanism
• But if the Syncker DLL are present in the bin folder, they are loaded, even though I'm using only extern_plugin "IG". This is a bit confusing. Yet, it allows me to use console commands with the syncker, like "syncker_console" so its very useful. Maybe the doc should be clarified here.
• As the IG now have its own "master/slave" sync mechanism, does it automatically matches the Syncker master/slaves if the DLL is loaded, or do I need to add some Syncker-specific command line option?
• How should we sync the creation of new nodes/materials with the IG? Is the Syncker still required here?
• The IG must now be configured at start-up with a fixed number of slaves. What if a slave crashes? Or if I need to add a slave mid-session? (this was actually a use-case, we can work without it, but I'd like to know)
• Is there a specific start-up sequence, with specific timings or anything, to be enforced at the IG start-up with multiple slaves? We actually have a lot of "hang-on" situation, where the master does not acknowledge some slaves and it just "hang" there, and we have to restart the whole cluster (this should probably be a ticket of its own)

Thanks!

[cash-metall]

Hello Stephane!

we experienced a lot of problems working with the syncker inside IG. More and more often it is necessary to execute some specific logic on the slave(it is the limitation of the syncker). Also, in most cases Syncker is not the best solution, because It synchronizes the internal state of the nodes.

IG now loads the syncker itself, to avoid problems with the initialization order. You cannot manually initialize the syncker now. There are two arguments for this.

"-ig_master 1" for syncker & ig master initialization
"-ig_master 0" for syncker & ig slave initialization
"-ig_master -1" (or witout this argument) for use IG without syncker

you can also use all syncker-specific arguments such as sync_view, sync_name, sync_broadcast_address... except sync_master. 

Now ig has its own synchronization system, based on udp (with the ability to send with a reliable).  This system synchronizes what has changed only. However, we still use a Syncker for some areas, such as clouds and particle systems, cause they require permanent full state synchronization. And also we use a syncker for a multi-projection system.

3 hours ago, Amerio.Stephane said:

How should we sync the creation of new nodes/materials with the IG? Is the Syncker still required here?

Entities, articulated parts, components are now synchronized through IG. Syncker does not know about them.
Of course you can continue to use the syncker. or use ig usermessage and synchronize only those parameters that you need and only when they really change.

 

3 hours ago, Amerio.Stephane said:

• The IG must now be configured at start-up with a fixed number of slaves. What if a slave crashes? Or if I need to add a slave mid-session? (this was actually a use-case, we can work without it, but I'd like to know)
• Is there a specific start-up sequence, with specific timings or anything, to be enforced at the IG start-up with multiple slaves? We actually have a lot of "hang-on" situation, where the master does not acknowledge some slaves and it just "hang" there, and we have to restart the whole cluster (this should probably be a ticket of its own)

The requirement to indicate the number of slaves is necessary only for the first run. Master will not perform any action until it finds all the slaves. So you can run them in any order.
if someone turns off, it's not scary. However, when connecting a new slave in the process, it will not receive the entire message history.

[Amerio.Stephane]

Let's say I have to load a specific node at runtime (NOT a CIGI entity). I used to call in 2.9 (on the master) synckerMaster->createNode(). Now, how should I do that? Do I need to use the synckerMaster returned by IG singleton? or should I use Syncker Manager? What if I want to create a new material? Same as before?

(I just want to make sure I use correctly the API, in the way you intended it to be used)

About the start-up order: Our setup have 13 slaves. About 75% of the time, at startup, the master does not "see" all slaves. Master is started first, then all slaves are started at the same time. Some connect OK, other are just not "seen" by the master. I may be doing something wrong here (timings? ports?) or something may need to fixed?

To alleviate this, you mean I should just start with ig_count 1 and then start all slaves a few seconds later? (in this case, the current doc should be more explicit because it feels like no more slave can connect after the master has finished its setup)

[morbid]

15 minutes ago, Amerio.Stephane said:

About the start-up order: Our setup have 13 slaves. About 75% of the time, at startup, the master does not "see" all slaves. Master is started first, then all slaves are started at the same time. Some connect OK, other are just not "seen" by the master. I may be doing something wrong here (timings? ports?) or something may need to fixed?

Hi Stephane,

May I ask you if this happens with 2.10 only?

[Amerio.Stephane]

Yes this is 2.10 only, because only 2.10 requires the ig_count at startup. In 2.9, we used to just start the master first and then the slaves in any order, and connect/disconnect slaves at will (of course, the new slaves would not receive the message history, but that is expected).

[cash-metall]

13 hours ago, Amerio.Stephane said:

Let's say I have to load a specific node at runtime (NOT a CIGI entity). I used to call in 2.9 (on the master) synckerMaster->createNode(). Now, how should I do that? Do I need to use the synckerMaster returned by IG singleton? or should I use Syncker Manager? What if I want to create a new material? Same as before?

it does not matter. you can take as a plug-data (Plugins::Syncker::Manager::get()->getMaster())  as ig_manager->getMaster(). they are identical. Yes you can use a syncker for such cases. For example, we use syncker to animate the environment - some light sources, background traffic...  

 

13 hours ago, Amerio.Stephane said:

To alleviate this, you mean I should just start with ig_count 1 and then start all slaves a few seconds later? (in this case, the current doc should be more explicit because it feels like no more slave can connect after the master has finished its setup)

Yes, ig_count 1 is a valid case. it will work as before.

we’ll check why slaves may not connect. but we need more information. there may be an approximate network topology - how many computers are on the network, what IP addresses and masks they have. please also attach logs (bin/log.html) from the slaves that could not connect the first time.

[Amerio.Stephane]

 

7 hours ago, cash-metall said:

Yes, ig_count 1 is a valid case. it will work as before.

Just tried this, and the behavior is strange: the ownship is not moving on slaves that connect after the master has fully started, but the time of day does changes correctly on these slaves. So they are connected, but don't receive/apply the entity position/attitude.

log_master.html log_slave.html

Edited February 26, 2020 by Amerio.Stephane
(logs attached)