Filesystem performance optimization

[ulf.schroeter]

Problem

 

In case of large numbers of raw files (e.g. large worlds with a lot of textures/meshes) in the data directory rendering shows significant frame drops on file streaming.

 

Cause

 

Current Filesystem::loadFile() implemention does linear search in raw file list for file path lookup.

 

int FileSystem::loadFile(const char *n,int priority,float weight) {
....
for(int i = 0; i < files.size(); i++) {
   if(is_file(files[i], processed_file.file) == 0) continue;
   .....
   // found file store file for background loading
   ....
}

 

In case of larger file numbers linear filepath search and is_file() check for thousands of files takes significant time especially in case of multi-files per frame loading. This leads to frame drops as method is called by render thread.

 

Proposed Optimization

 

Move full path lookup and is_file() check to Filesystem::update() which is called by background filesystem thread.

 

Doing so leads to much more stable render frame rate in case of file streaming as render thread now only stores new file load task which takes minimal time. Full file path lookup and check in background thread doesn't affect render performance (Required access synchronization to files vector between render and filesystem thread is negligible)

[alexei.garbuzenko]

I'd better suggest using hashtable instead of linear search. Some time ago we tried optimizing this part and mostly failed - there was no performance increase at all. The problem is in modern CPU performance - even 10000 file names are enumerated in less than ~0.01s. Also it often happens that file name search took N milliseconds, while file data read takes from N*10 to N*100ms cause of slow storage system. Only when 100% of data are in HDD/OS cache there is small perfrormance increase.

[ulf.schroeter]

In our test case with approx. 5000 files in data folder filepath lookup took up to 0.5 ms per single file. In case of multi-file requestes per frame total time required for rendering thread summed up to some [ms] which is very bad if total frame time is just 16 [ms] for 60Hz frame rate.

 

In our case moving lookup/processing to filesystem thread reduced frame drops significantly. If alternative hash map implementation can reduces overhead for rendering thread to nearly nothing -> ok.

[frustum]

Thanks, added.

[steve3d]

another question, why not make filesystem thread automatically refresh when the engine window becomes activated when it's in window mode or editor loaded?

[frustum]

Current Unigine SDK 2010-11-01 can't load meshes and textures from zip or ung archives. This is because FileSystem::loadFile() function always returns 1 without checking of archives. So I have reverted filesystem code. To fix performance issue special tree for back string sorting scheme have been added. File searching is not a bottleneck anymore.

[ulf.schroeter]

good solution