WIP - converting to OpenCL - 1 test #3

kalwalt · 2019-09-02T14:37:18Z

i'm testing the conversion to OpenCL: basically converting the part of the code where OpenCL can improve the performances, it should only necessary change some cv::Mat to cv::UMat.
This PR is most a trial and error, i will report in the comments the problems that may arise.

kalwalt · 2019-09-02T14:44:14Z

Whit the latest i get this error:

/home/walter/kalwalt-github/artoolkitX_em_2d/Source/ARX/OCVT/PlanarTracker.cpp:415:20: error: 
      no matching constructor for initialization of 'cv::UMat'
  ...colorFrame(_frameSizeY, _frameSizeX, CV_8UC4, frame, cv::USAGE_DEFAULT);
     ^          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/walter/kalwalt-github/artoolkitX_em_2d/Source/depends/emscripten/opencv-4.1/modules/core/include/opencv2/core/mat.inl.hpp:3600:7: note: 
      candidate constructor not viable: no known conversion from
      'unsigned char *' to 'const cv::Scalar' (aka 'const Scalar_<double>') for
      4th argument
UMat::UMat(int _rows, int _cols, int _type, const Scalar& _s, UMatUsageF...
      ^

cv::Mat instead may accept void* data

Mat (int rows, int cols, int type, void *data, size_t step=AUTO_STEP)

from the OpenCV docs

kalwalt · 2019-09-02T15:25:58Z

I read this article https://www.learnopencv.com/opencv-transparent-api/ and i solved sending the data initially to a cv::Mat and after getting them with the getUmat function. Any way this is fixed only for the Emscripten part, won't work for the other platforms.

 void ProcessFrameData(unsigned char * frame)
   {
     cv::Mat data(_frameSizeY, _frameSizeX, CV_8UC4, frame);
     // When using emscripten the image comes in as RGB image from the browser
     // Convert it to Gray
     #if ARX_TARGET_PLATFORM_EMSCRIPTEN
     cv::UMat colorFrame = data.getUMat(cv::ACCESS_READ);

kalwalt · 2019-09-02T16:13:16Z

This error with the latest commit:

/home/walter/kalwalt-github/artoolkitX_em_2d/Source/ARX/OCVT/PlanarTracker.cpp:724:52: error: 
      no member named 'ptr' in 'cv::UMat'
                memcpy(data, _trackables[i]._image.ptr(), _trackables[i]...
                             ~~~~~~~~~~~~~~~~~~~~~ ^
In file included from /home/walter/kalwalt-github/artoolkitX_em_2d/Source/ARX/OCVT/PlanarTracker.cpp:43:
In file included from /home/walter/kalwalt-github/artoolkitX_em_2d/Source/ARX/OCVT/OCVConfig.h:43:
In file included from /home/walter/kalwalt-github/artoolkitX_em_2d/Source/depends/emscripten/opencv-4.1/modules/core/include/opencv2/core.hpp:60:

Needs to think how to solve...

kalwalt · 2019-09-02T16:58:26Z

Source/ARX/OCVT/PlanarTracker.cpp

@@ -721,7 +721,10 @@ class PlanarTracker::PlanarTrackerImpl
                info.fileName = _trackables[i]._fileName;
                // Copy the image data and use a shared_ptr to refer to it.
                unsigned char *data = (unsigned char *)malloc(_trackables[i]._width * _trackables[i]._height);
-                memcpy(data, _trackables[i]._image.ptr(), _trackables[i]._width * _trackables[i]._height);
+                /// this needs to be verifyed!!


this is only a test, completely not sure of.

kalwalt · 2019-09-02T17:04:31Z

Source/ARX/OCVT/PlanarTracker.cpp

@@ -504,7 +504,7 @@ class PlanarTracker::PlanarTrackerImpl
                    fs << "trackableId" + index << _trackables[i]._id;
                    fs << "trackableFileName" + index << _trackables[i]._fileName;
                    fs << "trackableScale" + index << _trackables[i]._scale;
-                    fs << "trackableImage" + index << _trackables[i]._image;
+                    //fs << "trackableImage" + index << _trackables[i]._image;


i commented out because an error (no viable conversion from UMat to another format, don't remember exactly)

kalwalt · 2019-09-02T17:05:54Z

Source/ARX/OCVT/PlanarTracker.cpp

@@ -543,7 +543,7 @@ class PlanarTracker::PlanarTrackerImpl
                    fs["trackableId" + index] >> newTrackable._id;
                    fs["trackableFileName" + index] >> newTrackable._fileName;
                    fs["trackableScale" + index] >> newTrackable._scale;
-                    fs["trackableImage" + index] >> newTrackable._image;
+                    //fs["trackableImage" + index] >> newTrackable._image;


same as in the other comment, in this way the code compile completely, but save and load Trackables Database won't work.

kalwalt · 2019-09-02T17:06:58Z

@ThorstenBux I have my Artoolkitx.js and Artoolkitx.wasm ready!

kalwalt · 2019-09-02T17:25:10Z

The code do not detect the marker but i get an decisive increase in fps, probably the reason is in the changes in passing the data.
If you want to try i let here the compiled libs.
artoolkitxCL.zip

kalwalt · 2019-09-02T18:16:24Z

sure the issue is in the PlanarTracker.cpp

TrackedImageInfo GetTrackableImageInfo(int trackableId)
    {
        TrackedImageInfo info;
        for(int i=0;i<_trackables.size(); i++) {
            if(_trackables[i]._id==trackableId) {
                info.uid = _trackables[i]._id;
                info.scale = _trackables[i]._scale;
                info.fileName = _trackables[i]._fileName;
                // Copy the image data and use a shared_ptr to refer to it.
                unsigned char *data = (unsigned char *)malloc(_trackables[i]._width * _trackables[i]._height);
                /// this needs to be verifyed!!
                cv::Mat dataPtr(_trackables[i]._width, _trackables[i]._height, CV_8UC4, data);
                _trackables[i]._image = dataPtr.getUMat(cv::ACCESS_READ);
                //memcpy(data, _trackables[i]._image.ptr(), _trackables[i]._width * _trackables[i]._height);
                info.imageData.reset(data, free);
                info.width = _trackables[i]._width;
                info.height = _trackables[i]._height;
                info.fileName = _trackables[i]._fileName;
                return info;
            }
        }
        return info;
    }

before was:

TrackedImageInfo GetTrackableImageInfo(int trackableId)
    {
        TrackedImageInfo info;
        for(int i=0;i<_trackables.size(); i++) {
            if(_trackables[i]._id==trackableId) {
                info.uid = _trackables[i]._id;
                info.scale = _trackables[i]._scale;
                info.fileName = _trackables[i]._fileName;
                // Copy the image data and use a shared_ptr to refer to it.
                unsigned char *data = (unsigned char *)malloc(_trackables[i]._width * _trackables[i]._height);
                memcpy(data, _trackables[i]._image.ptr(), _trackables[i]._width * _trackables[i]._height);
                info.imageData.reset(data, free);
                info.width = _trackables[i]._width;
                info.height = _trackables[i]._height;
                info.fileName = _trackables[i]._fileName;
                return info;
            }
        }
        return info;
    }

the problem is we can not have a ptr() for the UMat, that Mat instead has. If we fix this maybe the code will works correctly.
Proabably we can do other optimizations with Umat as for example in CameraPoseFromPoints and some other part of the code too.

ThorstenBux · 2019-09-02T19:34:58Z

That is amazing work! I had done some UMat things a way back. Will have a look at what you did. It might probably be next week though, we are planning to head out for a long weekend and there is a lot todo this week. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Walter Perdan <[email protected]> Sent: Tuesday, September 3, 2019 6:16:24 AM To: ThorstenBux/artoolkitX_em_2d <[email protected]> Cc: Thorsten Bux <[email protected]>; Mention <[email protected]> Subject: Re: [ThorstenBux/artoolkitX_em_2d] WIP - converting to OpenCL - 1 test (#3) sure the issue is in the PlanarTracker.cpp TrackedImageInfo GetTrackableImageInfo(int trackableId) { TrackedImageInfo info; for(int i=0;i<_trackables.size(); i++) { if(_trackables[i]._id==trackableId) { info.uid = _trackables[i]._id; info.scale = _trackables[i]._scale; info.fileName = _trackables[i]._fileName; // Copy the image data and use a shared_ptr to refer to it. unsigned char *data = (unsigned char *)malloc(_trackables[i]._width * _trackables[i]._height); /// this needs to be verifyed!! cv::Mat dataPtr(_trackables[i]._width, _trackables[i]._height, CV_8UC4, data); _trackables[i]._image = dataPtr.getUMat(cv::ACCESS_READ); //memcpy(data, _trackables[i]._image.ptr(), _trackables[i]._width * _trackables[i]._height); info.imageData.reset(data, free); info.width = _trackables[i]._width; info.height = _trackables[i]._height; info.fileName = _trackables[i]._fileName; return info; } } return info; } before was: TrackedImageInfo GetTrackableImageInfo(int trackableId) { TrackedImageInfo info; for(int i=0;i<_trackables.size(); i++) { if(_trackables[i]._id==trackableId) { info.uid = _trackables[i]._id; info.scale = _trackables[i]._scale; info.fileName = _trackables[i]._fileName; // Copy the image data and use a shared_ptr to refer to it. unsigned char *data = (unsigned char *)malloc(_trackables[i]._width * _trackables[i]._height); memcpy(data, _trackables[i]._image.ptr(), _trackables[i]._width * _trackables[i]._height); info.imageData.reset(data, free); info.width = _trackables[i]._width; info.height = _trackables[i]._height; info.fileName = _trackables[i]._fileName; return info; } } return info; } the problem is we can not have a ptr() for the UMat, that Mat instead has. If we fix this maybe the code will works correctly. Proabably we can do other optimizations with Umat as for example in CameraPoseFromPoints and some other part of the code too. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FThorstenBux%2FartoolkitX_em_2d%2Fpull%2F3%3Femail_source%3Dnotifications%26email_token%3DAD765PGMUQ5TCMCJVE46UFTQHVJ7RA5CNFSM4IS6DJK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5WLK6A%23issuecomment-527218040&data=02%7C01%7C%7C6605fd082de941e1156708d72fd1aa7f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637030449856984036&sdata=vnxImNDWLPdjgE1FQZADlqmYOjkOfA8AfXdRNVPcecw%3D&reserved=0>, or mute the thread<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PCYZCRAB65WSIU2K2TQHVJ7RANCNFSM4IS6DJKQ&data=02%7C01%7C%7C6605fd082de941e1156708d72fd1aa7f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637030449856984036&sdata=4%2F7TYhaW7MoTP8GXqnrMrQQcUQzxnir31p0qu9QT398%3D&reserved=0>.

kalwalt · 2019-09-02T21:14:27Z

Yes, but i was very lucky too! The Transparent Api helps a lot!
We will see how to solve, no problem when. Good holiday!

ThorstenBux · 2019-09-02T21:24:01Z

I've pushed my branch of UMat things that I had done previously:
https://github.com/ThorstenBux/artoolkitX_em_2d/tree/perf_OpenCL_UMat

You can have a look at them if you like :).

kalwalt · 2019-09-02T21:37:36Z

Ok thank you very much! For sure i will look at!

kalwalt · 2019-09-03T19:11:33Z

Source/ARX/OCVT/PlanarTracker.cpp

-                memcpy(data, _trackables[i]._image.ptr(), _trackables[i]._width * _trackables[i]._height);
+                /// this needs to be verifyed!!
+                cv::Mat dataPtr(_trackables[i]._width, _trackables[i]._height, CV_8UC4, data);
+                _trackables[i]._image = dataPtr.getUMat(cv::ACCESS_READ);


we should do the reverse!, we will have to fill in the data from the trackables to image data, so it doesn't work

kalwalt · 2019-09-03T22:05:10Z

Source/ARX/OCVT/PlanarTracker.cpp

          cv::UMat grayImage(_frameSizeY, _frameSizeX, CV_8UC1);
-          cv::cvtColor(colorImage, grayImage, cv::COLOR_RGBA2GRAY);
+          std::cout << "grayImage ok!" << std::endl;


the problem seems located here i don't recieve this message: grayImage ok!

The image should be coming in as luma image from the API directly as the conversion happens inside the API in JS for the video stream.

Not sure if the code refers to the JPG image used as trackable.

It refers to 'newTrackable' i think the jpg image. It seems strange to me that fails at this point. Unlucky i will not have time to try other changes...

ThorstenBux · 2019-09-04T07:30:12Z

Did you compare to my umat changes? I thought they did work at some stage Get Outlook for iOS<https://aka.ms/o0ukef>

________________________________ From: Walter Perdan <[email protected]> Sent: Wednesday, September 4, 2019 6:53:48 PM To: ThorstenBux/artoolkitX_em_2d <[email protected]> Cc: Thorsten Bux <[email protected]>; Mention <[email protected]> Subject: Re: [ThorstenBux/artoolkitX_em_2d] WIP - converting to OpenCL - 1 test (#3) @kalwalt commented on this pull request.

________________________________ In Source/ARX/OCVT/PlanarTracker.cpp<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FThorstenBux%2FartoolkitX_em_2d%2Fpull%2F3%23discussion_r320598028&data=02%7C01%7C%7C673a3bd706ad406b458808d73104a3d7%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637031768301731204&sdata=%2FgjA6ic9oUWMzEzhRM4sXn15t2gljambsJTb943DIfM%3D&reserved=0>:

cv::UMat grayImage(_frameSizeY, _frameSizeX, CV_8UC1);

- cv::cvtColor(colorImage, grayImage, cv::COLOR_RGBA2GRAY); + std::cout << "grayImage ok!" << std::endl; It refers to 'newTrackable' i think the jpg image. It seems strange to me that fails at this point. Unlucky i will not have time to try other changes... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FThorstenBux%2FartoolkitX_em_2d%2Fpull%2F3%3Femail_source%3Dnotifications%26email_token%3DAD765PHPK7OXOVLTGXNHGQ3QH5LPZA5CNFSM4IS6DJK2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCDSHRNY%23discussion_r320598028&data=02%7C01%7C%7C673a3bd706ad406b458808d73104a3d7%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637031768301741222&sdata=4HrBB7iPWGEICfzdTcbc5gI4sH%2FvlqzpMUiFV2XU5Tg%3D&reserved=0>, or mute the thread<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PHO727FUULWYDALCHDQH5LPZANCNFSM4IS6DJKQ&data=02%7C01%7C%7C673a3bd706ad406b458808d73104a3d7%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637031768301761232&sdata=5TzK1GtdgiZFo44sEtobutyKZKx5mFnLmty3efUWonk%3D&reserved=0>.

kalwalt · 2019-09-04T07:33:14Z

i did but very quickly and not for all the code, i should do indeed !

…olkitX_em_2d into feature-test-OpenCL

kalwalt · 2019-09-10T12:25:22Z

The code actually seems to fail when loading the data image to be tracked (in the addMarker function) not understand why it can not loaded, i don't see such critical code in addMarker or maybe it fails silently in another part but i can not get where.

kalwalt · 2019-09-10T17:17:39Z

Source/ARX/OCVT/PlanarTracker.cpp

-          newTrackable._image = grayImage;
+          cv::cvtColor(colorImage, grayImage, cv::COLOR_RGBA2GRAY);
+          std::cout << "grayImage ok 2!" << std::endl;
+          newTrackable._image = grayImage.getUMat(cv::ACCESS_RW);


Now it stops at this point: Issue while converting to UMat.

or the step before...

ThorstenBux · 2019-09-12T00:53:29Z

Hi,

I start to feel that OpenCL won't work on mobile. Some things that I'm reading point into the direction that the drivers need to be written for the specific GPU and things. Which might explain why it starts failing. Basically, the browser doesn't have access to it but we tell the transpiled OpenCV that OpenCL is available so it tries to call it but fails because it isn't really there. The web counterpart would be WebCL which in turn isn't implemented inside any browser yet and the Emscripten polyfill is reportedly slower than not using it at all.

We might need to find another solution to tweak the performance

kalwalt · 2019-09-12T20:33:17Z

yes i can understand this. But could be an issue the fact i build the OpenCV lib with "-DWITH_OPENCL=ON -DWITH_OPENCL_SVM=ON" with OpenCL enabled but also with OpenCL and the Shared Virtual Memory enabled? I can try to test without this last one, just to understand if it make a difference.

ThorstenBux · 2019-09-12T20:35:06Z

Already tried that without success.

ThorstenBux · 2019-09-12T20:35:26Z

We would need to find some other idea to improve performance. I fear

kalwalt · 2019-09-12T20:37:58Z

ok good to know, and basically have you some ideas for other improvements?

ThorstenBux · 2019-09-12T20:40:18Z

Well, first would be to get it running profile it to see where the bottleneck is.

One idea would be to run a rectangle detection over the image first to reduce the size of the image that needs to run the recognition and tracking.

ThorstenBux · 2019-09-12T20:42:44Z

I see you are working with https://github.com/kalwalt/MarkerlessARJS maybe they did something different which could improve performance?

I recall that the performance might be solvePnP but not entirely sure. I'd say let us aim for a profiling run? Could you do that?

kalwalt · 2019-09-12T20:59:25Z

Well, first would be to get it running profile it to see where the bottleneck is.

yes you are right, this is important.

One idea would be to run a rectangle detection over the image first to reduce the size of the image that needs to run the recognition and tracking.

Do you mean a rectangle detection thanks to OpenCV, like a blob detection algorithm?

I see you are working with https://github.com/kalwalt/MarkerlessARJS maybe they did something different which could improve performance?

yes i'm working on this but i'am far to make something that can be tested, The original code was developed with a oldest version of OpenCV. I hope that it can be something better in performances, i hope to go on tihs to compare with ArtoolkitX.js, The original code was only for windows I think, didn't tested the original one. The author says also in this issue ahmetozlu/augmented_reality#1 about Dimensionality Reduction maybe could be an idea?

I recall that the performance might be solvePnP but not entirely sure. I'd say let us aim for a profiling run? Could you do that?

I will try it i f i have time tomorrow or in the next days. Actually i am in France for job, i arrived today.

kalwalt · 2019-09-12T21:04:13Z

also read this issue: ahmetozlu/augmented_reality#2

ThorstenBux · 2019-09-12T21:07:14Z

also read this issue: ahmetozlu/augmented_reality#2

Detection and tracking is already separated inside the C function as far as I know but it is not running as two threads that would be an idea.

kalwalt · 2019-09-12T21:19:25Z

Detection and tracking is already separated inside the C function as far as I know but it is not running as two threads that would be an idea.

Can we do seperate threads with Emscripten? we can but enabling -s USE_PTHREADS=1 right?
as i did for jsartoolkit5 in kalwalt/jsartoolkit5#2 but it is needed to enable the shared memory option on the phone. Or maybe you mean something else.

ThorstenBux · 2019-09-12T21:54:43Z

I've compiled with PTHREADS before but didn't know about the shared memory option?

kalwalt · 2019-09-14T20:27:39Z

I've compiled with PTHREADS before but didn't know about the shared memory option?

yes it is. maybe you used before chrome and other browsers disabled the default. But maybe in a near future it will become a default option again.
I will try to do the profiling tomorrow.

kalwalt · 2019-09-15T10:33:16Z

i m doing the profiling i will open a separate issue for this.

kalwalt added 2 commits August 31, 2019 21:43

convert cv::Mat to cv::UMat, not finished

1e8dde0

last cv::MAt to cv::UMat translation

67527bf

kalwalt added enhancement New feature or request OpenCV all regarding OpenCV OpenCL all about OpenCL C/C++ about C and C++ code labels Sep 2, 2019

this fix the issue inside ProcessFrameData() - only for EMSCRIPTEN

04b6592

kalwalt added 4 commits September 2, 2019 17:47

UMat in FeatureDetector

746a87b

UMat for HarrisDetector

d5de60d

addMarker needs to copy the image

201b12f

UMat conversion for DetectAndCompute inside AddMarker

4ef2829

trying to convert the shared pointer

622cb01

kalwalt commented Sep 2, 2019

View reviewed changes

commenting te save and load images, for a succesfull build!

41fe28e

kalwalt commented Sep 2, 2019

View reviewed changes

kalwalt commented Sep 3, 2019

View reviewed changes

kalwalt added 2 commits September 3, 2019 21:25

passing the data correctly now

74ddef1

simpler passing of data to convert to gray

42f379e

kalwalt commented Sep 3, 2019

View reviewed changes

kalwalt added 5 commits September 4, 2019 10:08

image restored in Load and SaveDatabase

aaa4c77

adding Umat in minMaxLoc

5b7fe39

better ProcessFrameData

108358c

better ProcessFrameData

7321c4e

Merge branch 'feature-test-OpenCL' of https://github.com/kalwalt/arto…

5cb8794

…olkitX_em_2d into feature-test-OpenCL

convert to UMat at the end

42ca820

kalwalt commented Sep 10, 2019

View reviewed changes

This was referenced Sep 15, 2019

Testing the 2d tracking to improve performances #4

Open

Improving performances with mutlithreading #5

Open

WIP - converting to OpenCL - 1 test #3

Are you sure you want to change the base?

WIP - converting to OpenCL - 1 test #3

Conversation

kalwalt commented Sep 2, 2019

kalwalt commented Sep 2, 2019

kalwalt commented Sep 2, 2019 • edited Loading

kalwalt commented Sep 2, 2019

Choose a reason for hiding this comment

kalwalt Sep 2, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kalwalt commented Sep 2, 2019

kalwalt commented Sep 2, 2019

kalwalt commented Sep 2, 2019

ThorstenBux commented Sep 2, 2019 via email

kalwalt commented Sep 2, 2019

ThorstenBux commented Sep 2, 2019

kalwalt commented Sep 2, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThorstenBux commented Sep 4, 2019 via email

kalwalt commented Sep 4, 2019

kalwalt commented Sep 10, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThorstenBux commented Sep 12, 2019

kalwalt commented Sep 12, 2019

ThorstenBux commented Sep 12, 2019 via email

ThorstenBux commented Sep 12, 2019 • edited Loading

kalwalt commented Sep 12, 2019

ThorstenBux commented Sep 12, 2019

ThorstenBux commented Sep 12, 2019

kalwalt commented Sep 12, 2019 • edited Loading

kalwalt commented Sep 12, 2019

ThorstenBux commented Sep 12, 2019

kalwalt commented Sep 12, 2019 • edited Loading

ThorstenBux commented Sep 12, 2019

kalwalt commented Sep 14, 2019

kalwalt commented Sep 15, 2019

kalwalt commented Sep 2, 2019 •

edited

Loading

kalwalt Sep 2, 2019 •

edited

Loading

ThorstenBux commented Sep 12, 2019 •

edited

Loading

kalwalt commented Sep 12, 2019 •

edited

Loading

kalwalt commented Sep 12, 2019 •

edited

Loading