[SOLVED] 2.17 crash after 2.16.1 migration

[romain.janil]

Hi,

a just migrated a project from 2.16.1 (working, no crashes) to 2.17, to test purposes, seemed ok but whenever I try to run editor or run the project, it crashes a few seconds before getting an stabilized image :((

end of editor_log says:

15:45:10 ERROR:    Failed to compile fragment shader
15:45:10 ERROR:    Compilation log:
15:45:10 ERROR:        (5106,105-118): error X3004: undeclared identifier 'var_dirt_scale'
15:45:10 ERROR:        hlsl 5106: dirt = lerp(dirt, s_texture_tex_dirt.SampleLevel(s_sampler_tex_dirt, DATA_UV.zw, 0).rgb, toFloat3(var_dirt_scale));
15:45:10 ERROR:        (5106,96-119): error X3013: 'toFloat3': no matching 1 parameter function
15:45:10 ERROR:        hlsl 5106: dirt = lerp(dirt, s_texture_tex_dirt.SampleLevel(s_sampler_tex_dirt, DATA_UV.zw, 0).rgb, toFloat3(var_dirt_scale));
15:45:10 ERROR:    
15:45:10 ERROR:    Material::compileShader(): can't compile pass:"full_compose" material:"core/materials/base/render/composite/render_composite.basemat"
15:45:10 ERROR:    
15:45:10 ERROR:    
15:45:10 ERROR:    

I'm trying to see what could be the cause but hard to tell...I created a blank new 2.17 project, no problem.

What should I do/deactivate or where should I start to look to prevent this crash ? Last ressort option would be to create a new project and import whole content, logic etc but I'd rather find the problem directly.

thanks ?

[romain.janil]

created also a default .world in this migrated project,- it loads correctly-, so it might be world content related weirdly. I'm in the process of commenting-decommenting everything in the world xml to see what setting or node could be the guilty one...

progressing, it might related to some materials (mesh_base)using spatial noise or detail tex apparently...still digging

Edited June 23, 2023 by romain.janil

[silent]

Hi Romain,

That's related to the missing dirt scale slider from composite base material.

You can fix this by copying attached file into <Project>/data/core/materials/base/render/composite directory (you need to create all the subfolders manually).

This bug will be fixed in the upcoming update (2.17.1 or 2.17 hotfix).

Thanks!

render_composite.basemat

[romain.janil]

yes I just figured that out too! thanks!

 

[romain.janil]

but doesn't solve the crash

[silent]

That's strange. Try also to delete all the shader cache from the project's data dir.

Also make sure that you don't have any other unpacked core shaders (from 2.16.x SDK).

Thanks!

[silent]

Don't think also that this shader error can cause and engine crash, there might be something else in your project that causing that.

Could you please also send us an editor_log_* files after crashing? Also we would need to understand how do you launch an Editor in your project? Are you using bat files in root directory or using SDK Browser (Projects tab)?

Thanks!

[romain.janil]

I'll do that tomorrow (shader cache). Anyway just for about your last question I tried both ways from browser and .bat, same result after  a few seconds crash

[romain.janil]

yes there's something else not shader related (deleted cache). I just tried to run (from browser) the default world I copied from a blank 2.17 project into the problematic project  and it closes (shutdown) after a couple of seconds, while it loads apparently fine in editor. strange. I'll do tomorrow a full copy of content/logic to a new project ans see how things behave. 

[romain.janil]

Still digging but I think it's more to do with new/updated mesh streaming process. Exact same project (with same settings applies somehow) crashes 2.17 when it runs flawlessly in2.16.1. i'll try to migrate another similar project and so if problem is consistent.

[romain.janil]

and related to some nvidia drivers maybe too.. :(: I made a build which crashes (as so editor) on a computer with rtx titan 535.98 drivers (dx11), and same build is running correctly on another one (A6000 517.40, old ones)...

 

[silent]

If you can send us an example project with that crash - it would be interesting to look at it.

ATM on our internal projects we don't have any crashes or whatsoever, so we might missing something critical here.

[romain.janil]

I'll see what I can do, if I can isolate the problem, and share something...

I need also to investigate a bit more what's happening on this specific hardware/software combination on that machine, because, I copied all asset data/project etc on another machine and running editor or build on this one works just fine, no crash.

It might be a data corruption somewhere maybe in some cache (drive ?). I let you know if I manage to package a minimal example/identify the problem/and finally manage to get things running as they should do (no crash) on this specific machine.

[romain.janil]

continued. it doesn't seem related to a source data problem I guess, because I regenerated everything, created a new project, reimported etc no problem in 2.16, but still crashing 2.17.

Other really weird thing I've noticed is that on the problematic machine rig (titan rtx) I can't have shadows! (screenspace shadows work, but not dynamic shadow map: here's a screenshot from default scene (template_render_high)...

 

Should I reinstall sdk ? (if that makes sense...)

[silent]

Hi Romain,

Reinstalling SDK will not help, I'm afraid. Could you please try to do clean install of the latest available Game-ready drive from the nVidia website (I believe it's 536.23)?

Also if you have some Editor / App crash dumps from 2.17.x, could you please upload them to us? They are usually located in %LOCALAPPDATA%\CrashDumps directory.

If there is no crash dumps, you can enable their generation via Windows Registry editing. Use DumpType 2 (full dump).

Thanks!

[romain.janil]

ok thanks for the insight, I'll do that asap.

[romain.janil]

just uploaded latest editor crashdump to the ftp (RJ_crashDump)

and btw, installed latest gameReady, still no shadows; and crash in 2.17

Edited June 27, 2023 by romain.janil

[silent]

According to crash dump Editor is running with DX12 API. Is that correct? In 2.16.x Editor only supports DX11 API.

Could you please try to change API for 2.17 Editor to DX11 and see if that changes anything (you can also try to change to Vulkan, but no guarantees here).

Unfortunately, we don't have RTX Titan on our test stand, but everything works fine with RTX 2080Ti (with similar TU102 die), so we are really running out of clues at this moment. Also we have no shadows issues on any graphics API, so either there is a bug in the Titan driver path or we did something terrible :) If you have some GPU to swap - it would be interesting to see the resutls (for example, on 2080Ti as well).

After checking the crash dump we also found that crash occurs in Object::updateLods(), but also there are a lot of background threads that do Landscape, Meshes, Grassess streaming, so there is no obvious and exact spot at this time that we could take a look at and fix. 

Thanks!

[romain.janil]

Hi Silent,

tried in with both dx11 and dx12 with the 2.17 version. same thing. The one I sent you was indeed in dx12, for 2.17. I can re-check with dx11 just to be sure, but imho it had same issue.

As i told you, I have another machine which runs on a quadro rtx A6000, and everything's fine!. I have another one which runs on a 3080ti, but it's a lpatop, I haven't tested yet. I'll do that. too.

Regarding content I'm trying to get a minimal working project vs a crashy one, enabling/disabling content one after another and monitor. I keep digging.

thanks,

 

 

 

[silent]

Can you also upload dxdiag and editor / engine logs from that crashy PC and the second one with RTX A6000? Maybe we can spot something here as well.

And please correct me if I wrong, the current state on this PC:

• 2.16.1 project "A" is running fine on with DX11 API (Editor / Runtime); No DX12 tests;
• 2.17.x project "A" is crashing:
  • DX11: Editor / Runtime
  • DX12: Editor / Runtime(???)
  • Vulkan: ???
• 2.17.x empty project (project "B") - shadow bug (at least with DX11).

At the same time on another PC (with RTX A6000):

• 2.16.1 / 2.17.x project "A" running fine (DX11 - Editor / Runtime);
• 2.17.x project "B" - no issues (at least with DX11).

Thanks!

 

[romain.janil]

7 minutes ago, silent said:

Can you also upload dxdiag and editor / engine logs from that crashy PC and the second one with RTX A6000? Maybe we can spot something here as well.

And please correct me if I wrong, the current state on this PC:

• 2.16.1 project "A" is running fine on with DX11 API (Editor / Runtime); No DX12 tests; YES
• 2.17.x project "A" is crashing:
  • DX11: Editor / Runtime YES
  • DX12: Editor / Runtime(???) YES both
  • Vulkan: ??? NOT TESTED
• 2.17.x empty project (project "B") - shadow bug (at least with DX11). INDEED

At the same time on another PC (with RTX A6000):

• 2.16.1 / 2.17.x project "A" running fine (DX11 - Editor / Runtime); YES
• 2.17.x project "B" - no issues (at least with DX11). NO ISSUE WITH DX11 AND DX12

Thanks!

i'll upload dxdiag and logs soon.

thanks,

Romain

 

 

[romain.janil]

Fy, just tested new project"B" on the 3080ti (laptop) and i have shadows, as expected, so this is really related to one PC only...

[romain.janil]

uplaoded dxfiag from crashy pc. At the end lots of tracks of crashing editor_double, uploaded also one .wer and edotir log (which doesn't say a word about crash)

Edited June 27, 2023 by romain.janil

[silent]

Thanks for the dxdiag. Looks like you have a workstation with a dual Xeon CPU, is that correct? I think the main culprit is currently here.

If you can disable a single CPU from BIOS / UEFI and test again - that would be great.

[romain.janil]

Hi, indeed this is a dual cpu configuration. I don't know if the bios allows to disable one processor, I'll have a look.