Koksharov.Vasilij Posted April 5, 2018 Share Posted April 5, 2018 I'm using openCV in Unigine project. Function cv::VideoCapture::read() works pretty fast alone and within Unreal Engine. But calling it from Unigine::WorldLogic::update() is very slow (15 FPS). What are the possible reasons and how can I fix it? Link to comment
silent Posted April 5, 2018 Share Posted April 5, 2018 Hi, Is there any code sample available? If you are copying GPU memory to CPU it can be painfully slow. Could you please provide minimal working example of such app implementation with OpenCV for investigation? Thanks! How to submit a good bug report --- FTP server for test scenes and user uploads: ftp://files.unigine.com user: upload password: 6xYkd6vLYWjpW6SN Link to comment
Koksharov.Vasilij Posted April 6, 2018 Author Share Posted April 6, 2018 #include <UnigineApp.h> #include <UnigineLogic.h> #include <opencv2\opencv.hpp> class AppWorldLogic : public Unigine::WorldLogic { public: AppWorldLogic() {} virtual ~AppWorldLogic() {} virtual int init() { cap.open(0); return 1; } virtual int update() { if (!cap.isOpened()) return 1; cv::Mat frame; cap.read(frame); //this works much faster out of Unigine return 1; } virtual int shutdown() { cap.release(); return 1; } private: cv::VideoCapture cap; }; int main(int argc, char **argv) { Unigine::EnginePtr engine(UNIGINE_VERSION, argc, argv); AppWorldLogic world_logic; engine->main(NULL, &world_logic, NULL); return 0; } here is minimal example in Unigine. Maybe I can make read(frame) function async? Link to comment
silent Posted April 9, 2018 Share Posted April 9, 2018 Vasilij, If I remember correctly, read() operation is blocking one, so the performance could be really low here. Do you have sources for the UE4 OpenCV example that you stated at the first post? It would be interesting to see how such framerate is achieved? Thanks! How to submit a good bug report --- FTP server for test scenes and user uploads: ftp://files.unigine.com user: upload password: 6xYkd6vLYWjpW6SN Link to comment
Koksharov.Vasilij Posted April 9, 2018 Author Share Posted April 9, 2018 it is actually the same //cpp file, AWebCam is inherited from AActor //cv::Mat frame and cv::VideoCapture cap are private members declared in header void AWebCam::Tick(float DeltaTime) { Super::Tick(DeltaTime); updateFrame(); updateTexture(); } void AWebCam::updateFrame() { if (!cap.isOpened()) return; cap.read(frame); } AWebCam::updateTexture() does not strongly affect performance (no difference in fps) and contains a copying of pixeldata from cv::Mat to UTexture2D with FMemory::Memcpy() Link to comment
silent Posted April 9, 2018 Share Posted April 9, 2018 Hi Vasilij, Thanks for the info. We have plans to release an OpenCV integration plugin in 2.7.1. As soon as we will push the 2.7 release our devs will take a look at this issue. 1 How to submit a good bug report --- FTP server for test scenes and user uploads: ftp://files.unigine.com user: upload password: 6xYkd6vLYWjpW6SN Link to comment
silent Posted May 3, 2018 Share Posted May 3, 2018 According to that article: https://wiki.unrealengine.com/Integrating_OpenCV_Into_Unreal_Engine_4 OpenCV reading is updated with a fixed framerate of 15 frames per second. You can do it the same way in Unigine to avoid frame drops: // float RefreshTimer = 0.0f float RefreshRate = 15.0f; // virtual int update() { RefreshTimer += App::getIfps(); if (RefreshTimer >= 1.0f / RefreshRate && cap.isOpened()) { RefreshTimer -= 1.0f / RefreshRate; cv::Mat frame; cap.read(frame); } return 1; } The best solution will be to move all the OpenCV related code to the separate thread (no sample code for this, sorry) :) Thanks! 1 How to submit a good bug report --- FTP server for test scenes and user uploads: ftp://files.unigine.com user: upload password: 6xYkd6vLYWjpW6SN Link to comment
rohit.gonsalves_ Posted May 9, 2018 Share Posted May 9, 2018 Merging such API's are always don't go side by side. But in our case we do the similar thing like the first pictures. We are not only reading 4 input sources from other hardware but pushing 1-2 HD media read from hard disk to Unigine. After all processing we take back two rendering from GPU and put them back to show on TV or to stream or to Record. We achieve this with the frame rates required for PAL and NTSC. Till 60 FPS. Everything works great. But the workflow and color space conversions added to achieve this are most important. We generally push YUV frames to GPU than RGB. This helps with half bandwidth of PCIe is used than RGB transfers for any dimension of buffer. Once in GPU we convert YUV ro RGBA. Process it and convert back to YUV for GPU downloading, to send to other hardware or to record on Hard disk. This is archived by Writing custom software using DirectX. Unigine API are added to it to get video textures mapped on to Unigine materials and then taking back Unigine final rendering to System memory from GPU. IMHO no other way will give you desired FPS. Such techniques will only help. As Silent suggested fixing the frame rate for opencv capture. This way Unigine rendering will be faster but the captured texture will only update 15 times a frame. Will this work for your product? Will variable frame processing timings with above approach works for your product? Rohit Link to comment
Recommended Posts