FileWatcherThread or component method


photo

Recommended Posts

Dear Unigine team,

Here is a proposal of FileWatcherThread, self-explaining.
It is not proposed by your SDK, but Editor exbibits such kind of behavior monitoring folder content..
Here for my need only one callback is managed, shouldn't be complicated to add multiple callbacks with addCallback/removeCallBacky.

######## FileWatcherThread.h ######## 

#pragma once
#include <UnigineCallback.h>
#include <UnigineString.h>
#include <UnigineThread.h>

#ifdef DLL_EXPORT
#define DLLEI __declspec(dllexport)
#else
#define DLLEI __declspec(dllimport)
#endif

using namespace Unigine;

DLLEI enum class FileStatus { created, modified, erased };

class DLLEI FileWatcherThread : public Thread
{
public:
	FileWatcherThread(String _s, unsigned int _d);
	~FileWatcherThread();

public:
	void process();

	//CallbackBase* fileChanged; //no info
	CallbackBase2<String, FileStatus>* fileChanged; //with info

private:
	String file_to_watch;
	unsigned int delay;
	bool file_existed;
	long long last_modif_time;
};
  
######## FileWatcherThread.cpp ######## 
#include "FileWatcherThread.h"

#include <UnigineFileSystem.h>

using namespace Unigine;

FileWatcherThread::FileWatcherThread(String _s, unsigned int _d) :
	file_to_watch(_s), delay(_d)
{
	file_existed = FileSystem::isFileExist(file_to_watch);
	last_modif_time = file_existed ? FileSystem::getMTime(file_to_watch) : -1;

	fileChanged = nullptr;
}

FileWatcherThread::~FileWatcherThread()
{

}

void FileWatcherThread::process()
{
	while (isRunning())
	{
		sleep(delay);

		try { //try catch file access..
			if (file_existed)
			{
				if (!FileSystem::isFileExist(file_to_watch))
				{
					file_existed = false;

					//propagate
					if (fileChanged)
						fileChanged->run(file_to_watch, FileStatus::erased);
				}
				else
				{
					auto mt = FileSystem::getMTime(file_to_watch);
					if (last_modif_time != mt)
					{
						last_modif_time = mt;

						//propagate
						if (fileChanged)
							fileChanged->run(file_to_watch, FileStatus::modified);
					}
				}
			}
			else if (FileSystem::isFileExist(file_to_watch))
			{
				file_existed = true;

				//propagate
				if (fileChanged)
					fileChanged->run(file_to_watch, FileStatus::created);
			}
		}
		catch (std::exception& e) {
			Log::error("Fail to watch file %s, err: %s\n", file_to_watch.get(), e.what());
		}
	}
}

Would you see smthg wrong in this code?

The need is a component checking continuously if a file has any update impacting UI.
I initially put the FileSystem calls in its update method... but with notable impact on performance (at least visible with microprofile).

Now with a dedicated separate thread dealing with file watching and parsing, it is more efficient and a bool signal the component to rebuild impacted UI.
But would it be possible to use COMPONENT_UPDATE_SYNC_THREAD or COMPONENT_UPDATE_ASYNC_THREAD instead?
You improved doc on component methods, with now indication of optional order value, but bit more explanation on e.g. both previous meth would be great, an example even better ;-)

Kind regards,
Charles

Link to post

Hi Charles,

Not quite sure that I've understand the main issue. You need a way to check the file for modifications, right?

You wrote a C++ component for that and what next? Do you want us to include it somewhere? Can you please elaborate?

Thanks!

How to submit a good bug report
---
FTP server for test scenes and user uploads:

Link to post

Hi,

Sorry, you're right, bite more explanation will help.

First, your editor is monitoring folder content, right?
Is there any thread dealing with that?
Would it be possible to expose it in SDK API?

Waiting for that, I implemented our own FileWatcherThread specializing your Thread class, c.f. code given.
Feel free to integrate it in SDK, and especially to check it.

Second point, this FileWatcherThread first usage is a component watching a file to detect when a new parse is required to update UI.
Such FileSystem::isFileExist and FileSystem::getMTime seems to be slightly too expensive to put them in component::update method.
Therefor, another thread checking times to times the file, parsing it, and running callback to propagate the change.
But, question: what about component::update_sync_thread or component::update_async_thread, wouldn't it be recommended for such // file check?
Could you maybe please add example in doc? Currently it is light...

Kind  regards,
Charles

Link to post

I would be happy if there are any examples of this updateSync and updateAsync methods. 

Last time when I wrote component handling through editor, I tried to use those methods. 

But I think I also did not understand the usage of those methods correctly with respect to engine loop.

Then I too put my I/O hardware waiting calls in another thread and sync it with mutexes on main thread. 

Eager to have some examples. 

Rohit

Link to post

sync update and async update executed within one frame

main thread sequence

  1. run sync methods
  2. run async methods
  3. wait sync 
  4. run update
  5. run post update
  6. wait async
  7. run swap methods

image.png

on picture example of profile for 16 components.

you can check execution sequence in microprofile. just make a lot of TestComponents in scene and open microprofile

class TestComponent : public Unigine::ComponentBase
{
public:
	COMPONENT_DEFINE(TestComponent, Unigine::ComponentBase);
	COMPONENT_UPDATE(update);
	COMPONENT_UPDATE_SYNC_THREAD(update_sync);
	COMPONENT_UPDATE_ASYNC_THREAD(update_async);
	COMPONENT_SWAP(swap);
	COMPONENT_UPDATE_PHYSICS(update_physics);
	
	
	void update() { wait(5, __FUNCTION__); }
	void swap() { wait(1, __FUNCTION__); }
	void update_sync() { wait(10, __FUNCTION__); }
	void update_async() { wait(60, __FUNCTION__); }
	void update_physics() { wait(10, __FUNCTION__); }
	
	void wait(int ms, const char * name)
	{
			int prid = Profiler::beginMicro(name);
		Thread::sleep(ms);
			Profiler::endMicro(prid);
	}
};

 

Update sync can be used for quick calculations. you will get the result in the update.
Async update can be used for lengthy calculations, but you will get the results only in a swap.

https://developer.unigine.com/en/docs/2.13/principles/component_system/component_system_cpp/?rlang=cpp#structure_logic

 

Link to post

I might add that these methods are deprecated. We are planning to remove them. As we can see UpdateSyncThread and UpdateAsyncThread are not so useful because they are executed before WorldLogic::update() and WorldLogic::postUpdate() functions. I highly recommend to use CPUShader class for this (UpdateSyncThread and UpdateAsyncThread based on it). It gives you more flexibility and you can choose when exactly to start and stop your threads. As the UpdateAsyncThread, Engine::swap() always waits for all CPUShaders: https://developer.unigine.com/en/docs/2.13/api/library/rendering/class.cpushader?rlang=cpp

When the CPUShader will be implemented on C#, we will think about removing UpdateSyncThread/UpdateAsyncThread or, at least, mark them as deprecated.

Best regards,
Alexander

Link to post

Hi,

Thanks for all the info and links.

So, if I do understand right update_sync and update_async are exec in 1 frame, and somehow blocking but at different step.. right?
Nothing to do with C# asynchronous programming async and await...
If task is long, it can impact rendering perf right?

I gave a try to implement a FileWatcherCPUShader... but probably in very wrong way since UNG starts but stuck right after run() called.
Probably because FileWatcherCPUShader::process is infinite loop blocking the first frame...

DLLEI enum class FileStatus { created, modified, erased };

class DLLEI FileWatcherCPUShader : public CPUShader
{
public:
	FileWatcherCPUShader(String _s, unsigned int _d);
	~FileWatcherCPUShader();

public:
	void process(int thread_num, int threads_count) override;
	void run();
	void stop();

	CallbackBase2<String, FileStatus>* fileChanged; //with info

private:
	String file_to_watch;
	unsigned int delay;
	bool file_existed;
	long long last_modif_time;

	volatile bool process_again;
};

So at the end, if you'd like a component dealing in // with an external resource e.g. a file but not impacting main loop, then your own FileWatcherThread as written above is fine.. 

Kind regards,
Charles

Link to post