Jump to content

openCV slow frame reading


photo

Recommended Posts

I'm using openCV in Unigine project. Function cv::VideoCapture::read() works pretty fast alone and within Unreal Engine. But calling it from Unigine::WorldLogic::update() is very slow (15 FPS). What are the possible reasons and how can I fix it?

 

unreal.jpg

unigine.jpg

Link to comment
#include <UnigineApp.h>
#include <UnigineLogic.h>
#include <opencv2\opencv.hpp>

class AppWorldLogic : public Unigine::WorldLogic
{
public:
	AppWorldLogic() {}
	virtual ~AppWorldLogic() {}

	virtual int init()
	{
		cap.open(0);
		return 1;
	}

	virtual int update()
	{
		if (!cap.isOpened()) return 1;
		cv::Mat frame;
		cap.read(frame); //this works much faster out of Unigine
		return 1;
	}

	virtual int shutdown()
	{
		cap.release();
		return 1;
	}

private:
	cv::VideoCapture cap;
};

int main(int argc, char **argv)
{
	Unigine::EnginePtr engine(UNIGINE_VERSION, argc, argv);

	AppWorldLogic world_logic;
	engine->main(NULL, &world_logic, NULL);

	return 0;
}

here is minimal example in Unigine. Maybe I can make read(frame) function async?

Link to comment

Vasilij,

If I remember correctly, read() operation is blocking one, so the performance could be really low here. Do you have sources for the UE4 OpenCV example that you stated at the first post? It would be interesting to see how such framerate is achieved?

Thanks!

How to submit a good bug report
---
FTP server for test scenes and user uploads:

Link to comment

it is actually the same

//cpp file, AWebCam is inherited from AActor 
//cv::Mat frame and cv::VideoCapture cap are private members declared in header

void AWebCam::Tick(float DeltaTime)
{
	Super::Tick(DeltaTime);
	updateFrame();
	updateTexture();
}

void AWebCam::updateFrame()
{
	if (!cap.isOpened()) return;
	cap.read(frame);
}


AWebCam::updateTexture() does not strongly affect performance (no difference in fps)  and contains a copying of pixeldata from cv::Mat to UTexture2D with FMemory::Memcpy()

Link to comment
  • 4 weeks later...

According to that article: https://wiki.unrealengine.com/Integrating_OpenCV_Into_Unreal_Engine_4 OpenCV reading is updated with a fixed framerate of 15 frames per second. You can do it the same way in Unigine to avoid frame drops:

//

float RefreshTimer = 0.0f
float RefreshRate = 15.0f;

//

virtual int update()
{
	RefreshTimer += App::getIfps();
	
	if (RefreshTimer >= 1.0f / RefreshRate && cap.isOpened())
	{
		RefreshTimer -= 1.0f / RefreshRate;
		cv::Mat frame;
		cap.read(frame);
	}
	
	return 1;
}

The best solution will be to move all the OpenCV related code to the separate thread (no sample code for this, sorry) :)

Thanks!

  • Like 1

How to submit a good bug report
---
FTP server for test scenes and user uploads:

Link to comment

Merging such API's are always don't go side by side. But in our case we do the similar thing like the first pictures. We are not only reading 4 input sources from other hardware but pushing 1-2 HD media read from hard disk to Unigine.

After all processing we take back two rendering from GPU and put them back to show on TV or to stream or to Record.

We achieve this with the frame rates required for PAL and NTSC. Till 60 FPS. Everything works great.

But the workflow and color space conversions added to achieve this are most important.

We generally push YUV frames to GPU than RGB. This helps with half bandwidth of PCIe is used than RGB transfers for any dimension of buffer. Once in GPU we convert YUV ro RGBA.

Process it and convert back to YUV for GPU downloading, to send to other hardware or to record on Hard disk. This is archived by Writing custom software using DirectX. Unigine API are added to it  to get video textures mapped on to Unigine materials and then taking back Unigine final rendering to System memory from GPU.

IMHO no other way will give you desired FPS. Such techniques will only help.

As Silent suggested fixing the frame rate for opencv capture. This way Unigine rendering will be faster but the captured texture will only update 15 times a frame. Will this work for your product? Will variable frame processing timings with above approach works for your product?

Rohit

Link to comment
×
×
  • Create New...