Image Feature Matching

[chrisflesher]

I was wondering if JAX has any functionality for sparse feature matching, kind of like OpenCV's SIFT / KNN functionality? How would one go about implementing all of the things needed to do something similar in JAX?

I saw you had an extract_patches function but am not sure how this works exactly, could this be used to develop a SIFT like feature somehow?

I was having trouble finding any kind of nearest neighbors library... think the best advice I've found so far is to try to re-use some existing CUDA KDTree or just use a CPU. Any ideas here?

Anyways I'm not very familiar with machine learning equivalent of this functionality... so any pointers would be appreciated.

[claudiofantacci]

Hi @chrisflesher, thanks for reaching out!

dm_pix does not have any functionalities for feature detection like SIFT, SURF etc, neither has KNN.
The main goal of dm_pix is to provide image processing functionalities written in JAX.

As written in the doc, extract_patches extracts portion of images. This is usually used in machine learning or image processing pipelines to apply specific functions on a limited, well-defined portion of an image. I guess you can use this to implement classic feature detectors on top.

For general recommendation, I'm afraid I don't know what to recommend for what you ask right now. I can ask around or wait if someone around here can make any suggestion for you 😄

[chrisflesher]

If you ever happen to find someone with some suggestions please let me know! Maybe we will just stick with OpenCV for now but it would be nice to speed up this computation using accelerators eventually.

For KNN I saw JAX has jax.lax.top_k and jax.lax.approx_max_k, maybe we could write an O(N^2) search using these.

[claudiofantacci]

Sure will do! Yes those looks like good candidates to begin with 😄