lag after call addMeshSurface function on mesh

[honya]

Hi,

 

when i call addMeshSurface function on mesh, then there is a lag that take 1.5-2 second (version 2.4.1). It is visible, because i see mesh, then mesh with added surfaces and new added surfaces have load low res texture (it fuzzy) and when lag end, then i can see high res texture on new added surface.

 

I attach test scene, that can simulate it in version 2.5.

After push button "l" it create create new mesh with added surfaces.

 

Here you can see lag around 0.07 second, it lines 6 and 7.

17:55:24 update clock: 5.50977
17:55:24 megred
17:55:24 megred clock: 5.51241
17:55:24 render clock: 5.51254
17:55:24 flush clock: 5.51277
17:55:24 flush clock: 5.51308
17:55:25 update clock: 5.58026
17:55:25 render clock: 5.58047
17:55:25 flush clock: 5.58069

In our code is lag around 1.7 second, it lines 15 and 16.

20:27:02 flush clock: 17.0518
20:27:02 Create time ext: 0.000812 second
20:27:02 update clock: 17.0654
20:27:02 Create time: 0.088793 second
20:27:02 Load AnimationWrapper male unarmed time: 0.059082 second
20:27:02 Load skinner time: 0.059368 second
20:27:02 Reload animations time: 2.2e-005 second
20:27:02 Merge skinned meshs time: 0.149476 second
20:27:02 update clock: 17.2151
20:27:02 character clock: 17.2153
20:27:02 character clock: 17.216
20:27:02 render clock: 17.2161
20:27:02 render clock: 17.2162
20:27:02 flush clock: 17.2164
20:27:02 flush clock: 17.2165
20:27:04 update clock: 18.9156
20:27:04 update clock: 18.9158
20:27:04 character clock: 18.916
20:27:04 character clock: 18.9168
20:27:04 render clock: 18.9171
20:27:04 render clock: 18.9172
20:27:04 flush clock: 18.9174
20:27:04 flush clock: 18.9175 

Can you check, what cause this lag?

Thanks

 

Honya

atlas.rar

[silent]

Hi Honya,

 

In the test scene that you send us lag is very minimal and I haven't seen any fuzzy textures.

If in your case you have already heavy VRAM usage, GPU driver will need some time to upload resources (such as mesh + textures) to GPU.

 

Right now we have no idea what can cause such behavior on your side, sorry.

[honya]

Hi Silent,

 

thanks for give me some directions, that can cause it.

[honya]

What i mean with fuzzy texture check lags_after_add_surface.mp4 video, it simply add surfaces to mesh and it freeze whole application.

lags_after_add_surface.mp4

[silent]

If you will check profiler, you will see that you have increased memory usage:

• Memory: 375 > 407 MB
• Meshes: 56 > 60 MB
• Textures: 22 > 25 MB

Allocating memory in run time is not a fast operation, I'm afraid. Also in your case there might be some shader compilation occurs. 

As I said before, there is no such behavior on attached test scene (mostly because we don't have the most part of the code that causing such behavior on your side). 

 

Maybe moving some heavy jobs to dedicated thread or Async job will make any difference, but it's hard to say. 

[honya]

The  only thing that is not in the test scene is loading around 160 animations to created skinned mesh.

But the load from my code is finished before the big lag and time for load is around 100 ms.

 

I will try remove some operations, but the main problem with big lag is that happen after finishing my code.

 

So it look as some depended load outside my code.

[honya]

Ok i found cause, it is Editor that create lag around 1.5 second.

 

Without Editor the lag is around 120 ms, that is still big enough to notice it.

[demostenes]

This lag happens even in case addMeshSurface function is not used. For example when adding simple mesh with addNode, there is cca 90ms lag after first call, after second call 50ms (it is probably cached). Such duration is no problem, but problem is lag, when engine is not responding. It causes visual lags, can cause desync with network, etc...

 

Equipping (adding objects on character, or into character mesh) is crucial part of any RPG game, so we cant afford any lag. Especially when more players at once do such action. Duration is ok, but not lag. 

 

If I am correct, addnode can be called in parallel thread, but addMeshSurface must be in main thread. What is proper solution for this?

[honya]

Hi,

 

i finally found root of this lag.

It happen, when i load node that need to load material library with 689 materials.

 

I attach scene equip_lag.rar, that can reproduce this behavior, for do it you must push button "l".

 

It don't matter if textures are founded or don't exist, see this ss:

 

 

 

For first ss (with existing textures) is first update time 3.70189 and second update 5.19246.

For second ss (without existing textures) is first update time 5.57539 and second update 6.94893.

 

The material library using folder with 1264 textures, that have 875 MB size.

 

Thanks.

Honya

[silent]

honya

 

Have you tried to reload all the material libraries on start, so adding new node will not cause such lag? I'm afraid. we can't do much with that on our side since loading 700+ materials on fly is not an atomic operation and will always take some time.

 

Thanks!

[demostenes]

honya

 

Have you tried to reload all the material libraries on start, so adding new node will not cause such lag? I'm afraid. we can't do much with that on our side since loading 700+ materials on fly is not an atomic operation and will always take some time.

 

Thanks!

 

We have no problem with duration. It is absolutelly OK, that such operation takes some time. But lag, when engine is stuck (even fps counter is frozen) is problem.

[honya]

silent

 

I try now all combination for load material library with 700+ materials, see table below:

.world | .node | lag |
   -   |   X   |  X  |
   X   |   X   |  X  |
   X   |   -   |  X  |
   -   |   -   |  -  |  

The only combination without lag is that i don't use this material library, all other always have lags with editor.

 

When i add node with material library in node, not in world, via editor, then there is only 0.130 ms lag.

 

When i try this combination without editor, then there was not notable lags, it was around 20-35 ms.

In our scene without editor is lag around 200 ms, but there is more code around.

 

So main problem is run with editor.

[demostenes]

Preloading does not help, there is always small lag. So what to do? 35ms duration would be great time, but lag is problem. Players hate lags, because lags are very often cause of death and if somebody dies because of lags more ofen, he will quit playing the game and creates hate thread on the forum.

[silent]

Hi Guys,

 

We understand that it can affect the user experience, but you are creating and modifying meshes in runtime - that will always cause some lags. The only way to avoid it - create predefined set of meshes (in all possible combinations) and always keep them in memory (that's always not a very easy task and possibly overkill). In that case when player will equip some new cloth it will appear immediately.

 

Maybe when you load your character (in world init) you can pre-combine all the meshes and then use already combined to get the speed-up.

 

We will also try to answer ASAP in the following thread: https://developer.unigine.com/forum/topic/4335-runtime-add-surfaces-on-staticskinned-mesh/- maybe that will also give some boost and reduce the lag.

 

Thanks!

[demostenes]

Hi Guys,

 

We understand that it can affect the user experience, but you are creating and modifying meshes in runtime - that will always cause some lags. The only way to avoid it - create predefined set of meshes (in all possible combinations) and always keep them in memory (that's always not a very easy task and possibly overkill). In that case when player will equip some new cloth it will appear immediately.

Problem is not duration, it is perfectly OK, if player will wait for example 1-2 seconds for equip. Problem is lag, freezing whole engine. And it is not only about creating meshes in runtime, but adding existing meshes into scene (add node, or enabling/disabling surface of existing mesh). According to our tests it seems,that in all cases (add node, add surface, enable/disable surface) lag is caused by work with material library. With smaller library lag is smaller. We are now trying to implement some solution using async threads, so i will give you more feedback, when we try various approaches. Btw, we cant use "all combination mesh", it would mean to have mesh with cca 2000 surfaces (number now and it will grow even more), this is absolutely unmaintable and also a performance killer. MMORPG "industry standard" is to merge equip meshes on the fly according to player needs, there is really no other option and this was also verified by various performance tests we did, maybe you remember that.

[demostenes]

There are new findings, lag is happening during load_node operation. There is something happening with mat library used in such node at the end of frame and it causes lag (freeze of engine). The bigger material library is, bigger lag (node is typically using 1-2 materials from this library) To reduce this lag, we can split this big library into 50 small ones (to have only several materials in each library), but this does not solve root cause of it. What is exactly happening in engine during this lag? Preload of mat library has no effect.

[unclebob]

Hi demostenes,

 

According to the video, it looks like a shader compilation issue. We are aware of this issue yet there's no quick solution.

 

I guess the only workaround you can do now is to warmup shader cache for each material for each item to be equipped. Try to create an instance of ObjectMeshSkinned, assign proper material to it and render for one frame. Also take notice that shader cache depend on global render state: shaders will be recompiled if any of the graphic features (like camera effects, scattering, haze mode, screen space effects and so on) was changed. That may affect how you want to warmup shaders.

 

Also try to set "render_manager_create_shaders" to 1. It'll compile material shaders for all passes with current global render defines. It still may not help especially if you're changing render features frequently.

 

[honya]

unclebob:

We try this seting what you recomend, but without effect.

[honya]

Even after we divided big material library to smaller, the lag still exist with same time.

Check equip_lag_2.rar(Unigine 2.5, float precision, only unigine script), after push "l" button, it load node to the scene and there is 1.28461 second lag between frames with editor.

In the scene is 31 materials libraries, when i remove some materials library, the lag between frames is smaller.

 

So basically we are in same situation, that is don't matter if have one big materials library, or we have lot of small materials libraries.

[silent]

Honya,

Just checked in release build (without editor) in a newly created project. After pressing 'l' I have a lag about 0.05 seconds, which really can't be noticed:

10:47:20 update clock: 4.68; timediff: 0.00
10:47:20 loadNode clock: 4.68
10:47:20 update clock: 4.73; timediff: 0.05

What should we do to increase this lag? Add more materials into the scene?

Please, never test performance inside editor, it working with nodes list and each update is about 2ms only for editor stuff, that will not be present in a final build.

Thanks!

[demostenes]

1,28 second lag was in editor. 0,05 second lag is not noticable, it should be more noticable in complex scenes (more materials, more nodes...). We have reasonable times without editor in empty scene, but this is only one player, no environment and not connected to server.

My current concern is about synchronization of client to our server. If clients lags (client is not responding to anything), it will cause desynchronization with server. Situation, when several players change equip at once in some complex scene (almost any) can rise such lag above 200-300ms (which starts to be noticable) and this can cause nasty sync issues with server. We are now in the middle of connecting server to client, so we will practical experience soon (lets see what we can solve by code).

Anyway, wouldnt be possible to eliminate this lag somehow (async doing of whatever is happening inside engine)? 

 

 

 

[silent]

demostenes

The problem is that rendering is always happens in main thread (single thread), so adding new mesh / material will anyway cause stall. The only solution here will be to always keep in memory models and textures of the equipment (which can be hard to achieve, especially if you have very detailed and memory consuming objects).

Devs will check what else can be improved on the engine side, but it will not solve your issue completely (even if we can somehow speedup the node load process).

Thanks!

 

[demostenes]

3 hours ago, silent said:

demostenes

The problem is that rendering is always happens in main thread (single thread), so adding new mesh / material will anyway cause stall. The only solution here will be to always keep in memory models and textures of the equipment (which can be hard to achieve, especially if you have very detailed and memory consuming objects).

Devs will check what else can be improved on the engine side, but it will not solve your issue completely (even if we can somehow speedup the node load process).

Thanks!

 

Actually speed of load is excellent. Even object with lots of textures is loaded very fast and duraton of such action is no problem at all. Only problem is lag (stall). 

How exactly should such caching work? Having all equip nodes loaded into scene on the start and just disable surfaces? I can imagine having all meshes in memory (equip meshes are quite small, so this is can be easily done, equip textures are bigger problem, this can go up to several GBs and it will grow. Maybe we could be able to preload with some 256*256 texture and after equip change material to high res texture. But I am afraid, change of material will cause lag - this must be tested, maybe not).  Thanks!