Template_Matching_RealTime

Abstract

Recognize the image from the image to determine if it exists.
Images that are part of an image may have been rotated, moved, or changed in brightness.
It presents a variety of methodologies for this.
The final goal is process on real-time & Incremental the image to recognize.

I record various attempts in this repo .

Check Basic_CV repo for a detailed Theory.

Outline

• Matching Template - opencv
• Feature-point Detection & Matching - opencv
  • SIFT
  • Result
• Homography (A part of Feature Matching)
  • Result
  • BFmatching

• In Videos
  • Program Scenarios
  • Test Result

• Supplementation
  • Applied
  • Not yet
• In Real-Time GUI Tool
  • Interface
  • Program Scenarios
  • An approach from a different perspective

• Reference

Matching Template - opencv

Use opencv's matchTemplate, get loc, normalize ..

Check template_matching.py

In Smart_Camera(Navigation) project [ Easy case ]

In this case, performance is very nice.
But, size or rotational transformations (hard cases) do not work well.
So, I can't use it

Feature-point Detection & Matching - opencv

Use SIFT, SURF, ORB, FAST, BRISK, AKAZE ..
Comparative analysis paper

If feature-points are simple, use FAST, BRISK ..
else(complex), use SIFT, SURF, AKAZE ..

This project's cases are complex. So I use SIFT.

SIFT

Paper

1. Scale-space extrema detection: The first stage of computation searches over all scales and image locations. It is implemented efficiently by using a difference-of-Gaussian function to identify potential interest points that are invariant to scale and orientation.

2. Keypoint localization: At each candidate location, a detailed model is fit to determine location and scale. Keypoints are selected based on measures of their stability.

3. Orientation assignment: One or more orientations are assigned to each keypoint lo- cation based on local image gradient directions. All future operations are performed on image data that has been transformed relative to the assigned orientation, scale, and location for each feature, thereby providing invariance to these transformations.

4. Keypoint descriptor: The local image gradients are measured at the selected scale in the region around each keypoint. These are transformed into a representation that allows for significant levels of local shape distortion and change in illumination.

If you want more detailed information, check This docs

In code,

sift = cv2.xfeatures2d.SIFT_create()
kp, des = sift.detectAndCompute(img1, None)

kp: keypoints, < cv2.KeyPoint 0x10d4137b0 >
des: descriptors, ndarray [[n, n, n, .. n], [n, n, n, .. n], [n, n, n, .. n] ..]

matches = bf.match(des1, des2)

matches: < cv2.DMatch 0x10d4137b0 >

Result

Check Feature_DetectMatch.py

In Smart_Camera(Navigation) project [ Easy case ]

In Smart_Camera(Navigation) project [ Hard case ]

This case's performance is not good yet..

This project is demanded working robustly(whether a small image is rotated, moved, or changed in brightness)
So, I'm doing middle processing to boolean the result from feature detection & matching.

Homography (A part of Feature Matching)

I think if i use this one, my app work robustly.

As far as, I know homography works for planar objects
So, I use before, detect planar objects() .. in small image

How can I detect ?

Maybe, I can use Clipseg, which will be shown below

Result

So.. If I get detected shop_sign image, I could see the good performance by using homography

1. shop_sign - 1 (True data)

Ratio = 0.6, Good matches:122/53093

Ratio = 0.5, Good matches:20/53093

2. shop_sign - 2 (True data)

Ratio = 0.6, good matches:67/53093

Ratio = 0.5, Good matches:20/53093

3. Iris (False data)

Ratio = 0.6, good matches:13/53093

Ratio = 0.5, Good matches:0/53093

Check Homography.py

Based on the above figures,
the matching results were good when ratio = 0.5 and good matches > 5

BFmatching

BFmatching is BruteForce matching.
I get the boolean result by using BFmatching

If Length of matching >= threshold is True
else(Length of matching < threshold), False.

I used BFmatching even when I could use FLANN because in our case accuracy is more important than speed. (I am using SIFT, not ORB for the similar reason.)

Check BFmatching.py

In Videos

Program Scenarios

Based on the above contents, I would like to write it as a program

Test Result

Points

Original

Preprocessed
Only used two points temporarily.

Test_Img

Test_Video

In terminal (Before)

In terminal (After)

Supplementation

Applied

Resize (speed)

Which kind of interpolation best for resizing image

Not yet

In my test above, Point image is not a wide(panorama, 360 ..) image.
I'll check performance in additional test (Point image is wide image)

When points are a lot, Maybe This app doesn't work well enough on real-time.

I want to test with various processing techniques as well as CLIPSeg. (I want to see a change in performancd according to preprocessing)

In Real-Time GUI Tool

Interface

Program Scenarios

So, I completed the program.

---

An approach from a different perspective

Check Optional_processing.md
Check One_shot_learning.md

Reference

https://en.wikipedia.org/wiki/Homography
https://ieeexplore.ieee.org/document/8346440
https://arxiv.org/pdf/2103.00020.pdf
https://paperswithcode.com/paper/prototypical-networks-for-few-shot-learning
https://proceedings.neurips.cc/paper_files/paper/2017/filecb8da6767461f2812ae4290eac7cbc42-Paper.pdf
https://keras.io/examples/vision/siamese_contrastive