.cache_textures out of control

[sebastian.vesenmayer]

Hi,

 

since we are using the new version 2.8 of the engine, we observed that the new .cache_textures folder is growing fast.

I had over 250000 files in there today and it is ongoing poluting our customers installation.

It seems like every time, we are loading ressources from our file system, new cache files will be generated with different hashes but never deleted.

We would like an option to disable the creation of cached textures.

 

Regards

Sebastian

[silent]

Hi Sebastian,

There is no option to disable texture caching. This small dummy textures are displayed on objects while the original files are still loading in separate thread.

Could you please tell us how do you load your textures? If you are using GUIDs all over your files the amount of cached textures should be equal to the amount of textures that you are using in your project. This number should not increase uncontrollable.

It would be nice to get a reproduction where you have endless textures cache generation in order to suggest some different approach to fix this.

We are also considering to move the .cache_textures directory outside of the data to the user directory (which created when you run engine with -project_name <Name> argument).

Thanks!

[sebastian.vesenmayer]

We are loading our Textures via Image class from tga/png files.

Convert them to dds.

Create mipmaps.

Save it to the filesystem and assign the path to the Material.

Reproducer will follow.

We don't need this feature, since our textures are uploaded to vram anyway. We are not using streaming.

 

Regards Sebastian.

Edited October 7, 2019 by sebastian.vesenmayer

[silent]

Right now it's not possible to disable textures cache creation. We will check your sample and see what can we suggest for such use-case.

 

Quote

Save it to the filesystem and assign the path to the Material.

When you are setting path to the texture in materials you automatically using streaming. In short, if you are dealing with saving / loading files it's better to use GUIDs. Don't forget to save all the GUIDs to the guids.db file when you finished saving all the changes.

Alternatively, you can directly set the required texture to the slot (avoiding streaming) via setTexture(int num, const Ptr<Texture> & texture). No saving to disk is required in that case. But in that case you need to control manually the lifetimes of your pointers.

If you already have an OpenGL texture loaded into VRAM you can convert it with Texture::fromGLTexture2D() methods or directly copy via following methods:

• int getGLInternalFormat( )
• int getGLPixelFormat( )
• int getGLPixelType( )
• int getGLTarget( )
• int getGLTextureID( )

Thanks!

[sebastian.vesenmayer]

We have no guids.db because content can change every time the programm is running.

We are using Unigine only as a render library in our projects.

Our customers get a final executable from us, with engine data, without Unigine Editor.

They are doing parts of the content by their own. We have a converter from our own file formats which translates into Unigine structures during runtime.

So how can we combine our workflows the best possible way?

 

6 hours ago, silent said:

Alternatively, you can directly set the required texture to the slot (avoiding streaming) via setTexture(int num, const Ptr<Texture> & texture). No saving to disk is required in that case. But in that case you need to control manually the lifetimes of your pointers.

When using setTexture with a Texture pointer and calling "render_streaming_reload textures" , or similiar in older version, it was crashing, so we have to save the ImageFiles.

We are not using streaming. You can upload textures too vram by calling getTexture on the material Object. Streaming is turned off by world file.

 

I did test it in a small example project, and it seems like when meta files are missing when loading a node byUnigine::World::get()->loadNode() it generates new cache files.

Between, how much is the collision rate between two generated hashes in your engine?

Thanks.

 

 

Edited October 8, 2019 by sebastian.vesenmayer

[silent]

Quote

I did test it in a small example project, and it seems like when meta files are missing when loading a node byUnigine::World::get()->loadNode() it generates new cache files.

Do you mean that for the same .node (let's say it has 3 textures and 1 material file assigned) you will get at each new load a new set of 3 cache textures? Or first call will generates only 3 textures and additional calls after will not trigger cache re-creation?

Now I can see that all the roads are really leading to the one issue that you described here:

• https://developer.unigine.com/forum/topic/5849-vanishingbroken-meshes/

It would be nice to have a test scene where we can reproduce all these issues with render_streaming_reload meshes / texture. We tried to achieve these results several times, but on our side everything is working as expected.

If we would be able to find and fix these crashes you probably can just start to use setTexure and pass the pointer directly to the material files. At least we can attach a debugger and see if there is something to do with engine itself or there is a way to fix this in logic.

 

Quote

So how can we combine our workflows the best possible way?

The best option is to emulate the Editor work: generate GUIDs for each new file, register them in filesystem, maitain the guids.db file (save on exit from application). When texture is deleted in runtime delete the texture cache for it and so on.

If the main concern is the a big amount of files in .cache_textures directory you can simply delete it at shutdown: https://developer.unigine.com/en/docs/2.9/api/library/rendering/class.render#destroyCacheTextures_void

 

Quote

Between, how much is the collision rate between two generated hashes in your engine?

Will post as soon as I will get the info from the dev team.

Thanks!

[sebastian.vesenmayer]

5 minutes ago, silent said:

Do you mean that for the same .node (let's say it has 3 textures and 1 material file assigned) you will get at each new load a new set of 3 cache textures? Or first call will generates only 3 textures and additional calls after will not trigger cache re-creation?

We are loading nodes only once per execution of the program. So in your described case, every time I start the program, 3 cache textures will be generated, with their corresponding metafile, which means 6 files. In our case it is creating arround 1500 new files, when starting the program again. When meta files for the loaded node/mesh/texture are available, they are generated only at the first start. So I guess i have to copy all meta files, to stop adding more cache textures.

 

14 minutes ago, silent said:

I believe that maitain guids.db in actual condition and generate determined GUIDs generation is the key for the correct interaction between engine and content.

Well thats the key, but how can we achieve it?

 

29 minutes ago, silent said:

Now I can see that all the roads are really leading to the one issue that you described here:

• https://developer.unigine.com/forum/topic/5849-vanishingbroken-meshes/

It would be nice to have a test scene where we can reproduce all these issues with render_streaming_reload meshes / texture. We tried to achieve these results several times, but on our side everything is working as expected.

If we would be able to find and fix these crashes you probably can just start to use setTexure and pass the pointer directly to the material files. At least we can attach a debugger and see if there is something to do with engine itself or there is a way to fix this in logic.

This is a different problem I guess which only affects meshes.

I would like to have a test scene too, to reproduce it but i am still on it.

I will try the setTexture. I am curious if it is working now.

 

Thanks

[sebastian.vesenmayer]

1 hour ago, sebastian.vesenmayer said:

I will try the setTexture. I am curious if it is working now.

Seems to work fine now. What is happening when I call render_streaming_reload texture, when there is only a texture object without a path? Is it just skipping?

[sebastian.vesenmayer]

1 hour ago, silent said:

If the main concern is the a big amount of files in .cache_textures directory you can simply delete it at shutdown: https://developer.unigine.com/en/docs/2.9/api/library/rendering/class.render#destroyCacheTextures_void

Should be sufficient for now. Thanks.

[silent]

Sebastian,

Good to hear that reloading texture is working right now. If I understand you correctly, current approach with directly setting texture and clearing cache directory on shutdown allows you to continue application development?

 

Quote

What is happening when I call render_streaming_reload texture, when there is only a texture object without a path? Is it just skipping?

Yes, it should be skipped.

 

Quote

Between, how much is the collision rate between two generated hashes in your engine?

I've checked on my old laptop and test was able to generate 29M hashes without collisions until the RAM limit is reached and application crashed. Test app:

#pragma once
#include <gtest/gtest.h>

#include <UnigineGUID.h>
#include <UnigineSet.h>

TEST(UGUID, TestCollision) {
	using namespace Unigine;
	Unigine::Set<UGUID> guids;
	UGUID guid;
	for(int i = 0; i < 10000000; ++i) {
		guid.generate();

		ASSERT_FALSE(guids.contains(guid));
		guids.append(guid);
	}

Regarding the maintaining guids.db file we can give you more useful advises if you will be able to provide a minimal example (that we can build on our side) that emulates your application work (starting engine -> modifying 3rd party assets in runtime -> adding assets to .node -> adding .node to scene -> load textures). I do understand that preparing such scene is very time consuming operation, so no rush here.

Thanks!