[Feature] Machine Learning using Rockchip NPU on supported boards.

[yoni13]

I have searched the existing feature requests to make sure this is not a duplicate request.

• Yes

The feature

Since immich already supports Hardware Transcoding with hardware acceleration via rkmpp, maybe we can make machine learning working on rockchip NPUs.

Links:
rknn-toolkit2
frigate's implantation
How to test

Platform

• Server
• Web
• Mobile

[yoni13]

Update: I am able to run "ViT-B-32__openai" using rknn-toolkit2 on emulator, I haven't verify the accuracy but I will say it got some progress.

[mertalev]

AnnSession uses None for the name and only provides the shape, so this can do the same (or provide None for both if the shape is also unavailable).

[yoni13]

I am out of ideas now 🫠.
I think I need to know more about insightface to solve this.

[mertalev]

Can you clarify what the issue is? Is it that the InsightFace models don't work with RKNN, or is it that you're not sure how to handle the get_inputs / get_outputs interface?

[yoni13]

Can you clarify what the issue is? Is it that the InsightFace models don't work with RKNN, or is it that you're not sure how to handle the get_inputs / get_outputs interface?

I am not sure how to handle the get_inputs and get_outputs, the example in rknn model zoo just rknn.inference(inputs=[infer_img]) and yes I don't know how to handle that😐

[yoni13]

Can you clarify what the issue is? Is it that the InsightFace models don't work with RKNN, or is it that you're not sure how to handle the get_inputs / get_outputs interface?

I am not sure how to handle the get_inputs and get_outputs, the example in rknn model zoo just rknn.inference(inputs=[infer_img]) and yes I don't know how to handle that😐

For this, just call onnx runtime to get the correct faces and unload it.

[yoni13]

(machine-learning-py3.11) root@orangepi3b:~/immich/machine-learning# ./start.sh 
ERROR: ld.so: object '/usr/lib/aarch64-linux-gnu/libmimalloc.so.2' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
[12/03/24 12:15:23] INFO     Starting gunicorn 23.0.0                                                                                                                                        
[12/03/24 12:15:23] INFO     Listening at: http://[::]:3003 (5620)                                                                                                                           
[12/03/24 12:15:23] INFO     Using worker: app.config.CustomUvicornWorker                                                                                                                    
[12/03/24 12:15:23] INFO     Booting worker with pid: 5621                                                                                                                                   
[12/03/24 12:15:32] I        Started server process [5621]                                                                                                                                   
[12/03/24 12:15:32] I        Waiting for application startup.                                                                                                                                
[12/03/24 12:15:32] I        Created in-memory cache with unloading after 300s of inactivity.                                                                                                
[12/03/24 12:15:32] I        Initialized request thread pool with 4 threads.                                                                                                                 
[12/03/24 12:15:32] I        Application startup complete.                                                                                                                                   
[12/03/24 12:15:34] I        Loading textual model 'ViT-B-32__openai' to memory                                                                                                              
W rknn-toolkit-lite2 version: 2.3.0
[12/03/24 12:15:34] I        Loading RKNN model from /cache/clip/ViT-B-32__openai/textual/model.rknn                                                                                         
--> Load RKNN model
I RKNN: [12:15:35.403] RKNN Runtime Information, librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
I RKNN: [12:15:35.403] RKNN Driver Information, version: 0.9.7
I RKNN: [12:15:35.404] RKNN Model Information, version: 6, toolkit version: 2.3.0(compiler version: 2.3.0 (c949ad889d@2024-11-07T11:39:30)), target: RKNPU lite, target platform: rk3566, framework name: ONNX, framework layout: NCHW, model inference type: static_shape
W RKNN: [12:15:35.728] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes
W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.

Can someone help me with dynamic_shapes&static_shapes? I have no idea what are them.
Or we can ignore that warning?

[mertalev]

If it is a static shape RKNN model, please ignore the above warning message

I think you can ignore it if there are no errors.

[yoni13]

Small update: I tested the model in Immich, and the ViT-B-32 shows a latency difference of 200 ms after using RKNN(from cpu bcs I am on dietpi).

[yoni13]

outputs :  list[NDArray[np.float32]] = self.rknn.inference(inputs=inputs, data_format='nchw')

and

[01/06/25 09:12:35] I        Running inference on RKNN model                                                                                                                                         
W The input[0] need NHWC data format, but NCHW set, the data format and data buffer will be changed to NHWC.

But


Am I missing something here?🤔
Wasn't the model NCHW?

[mertalev]

Some engines prefer one or the other and will internally only use one because all their optimizations require that format. It looks like this one accepts NCHW, but only by converting it to NHWC internally.

[yoni13]

Some engines prefer one or the other and will internally only use one because all their optimizations require that format. It looks like this one accepts NCHW, but only by converting it to NHWC internally.

So, input being NCHW should not be an issue?
🤔

[yoni13]

I used a random array to test the diff between the result run by onnx and rknn.
LOG: https://pastebin.com/26JGWZUE
DIFF: https://www.diffchecker.com/sLmjBdfZ/

The model seems to be working, might there be an issue with our inputs?

[yoni13]

No, not inputs.

from rknnlite.api import RKNNLite
import numpy as np
import onnxruntime as ort


# Model paths
rknn_model_path = "/cache/clip/ViT-B-32__openai/visual/model.rknn"
onnx_model_path = "/cache/clip/ViT-B-32__openai/visual/model.onnx"

rknn = RKNNLite()
rknn.load_rknn(rknn_model_path)
rknn.init_runtime()

session = ort.InferenceSession(onnx_model_path, providers=['CPUExecutionProvider'] )

def check_for_inf(array):
  """Checks if an array contains inf or -inf values."""
  return np.isinf(array).any()

def test_models(input_data, rknn_model_path, onnx_model_path):
  """Tests the RKNN and ONNX models with the given input data."""
  # RKNN model inference
  rknn_result = rknn.inference(inputs=[input_data], data_format='nchw')[0]

  # ONNX model inference
  onnx_result = session.run(None, {"image": input_data}, None)[0]

  return rknn_result, onnx_result


# Generate random input data and test until inf is found
iteration = 0
while True:
  iteration += 1
  input_data = np.random.rand(1, 3, 224, 224).astype(np.float32)
  rknn_result, onnx_result = test_models(input_data, rknn_model_path, onnx_model_path)

  if check_for_inf(rknn_result) or check_for_inf(onnx_result):
    print(f"Model produced inf/-inf at iteration {iteration}")
    print("RKNN result:", rknn_result)
    print("ONNX result:", onnx_result)
    break

  print(f"Iteration {iteration}: No inf/-inf found yet.")

I asked Gemini to write a script to test, but after 500 iteration, no inf,-infs found.

Hope this leads us to learn more what happened.

[sgofferj]

I don't know anything about machine learning development but I do have a FriendlyElec CM3588 NAS which is mostly idling at the moment. I was anyways planning to move my Immich to that from my 15 years old Xeon home server.
I can't offer much coding help but I can offer CPU/NPU time if that helps.

[yuukh]

build镜像失败了，网上解决这个报错一般通过unset PYTHONHOME 和 PYTHONPATH 来解决，但看日志输出PYTHONHOME 和 PYTHONPATH本就没有值，你有碰到过这个问题吗？

machine translation：
I failed to build image , and the common solution to this error online is to unset PYTHONHOME and PYTHONPATH. However, according to the log output, PYTHONHOME and PYTHONPATH don't have any values. Have you encountered this issue before?

[+] Building 0.7s (14/23)                                                              docker:default
 => [internal] load build definition from Dockerfile                                             0.0s
 => => transferring dockerfile: 4.33kB                                                           0.0s
 => [internal] load metadata for docker.io/library/python:3.11-slim-bookworm@sha256:6ed5bff4d7d  0.1s
 => [internal] load metadata for docker.io/library/python:3.11-bookworm@sha256:adb581d8ed80edd0  0.1s
 => [internal] load .dockerignore                                                                0.0s
 => => transferring context: 58B                                                                 0.0s
 => [builder-cpu 1/1] FROM docker.io/library/python:3.11-bookworm@sha256:adb581d8ed80edd03efd4d  0.0s
 => [internal] load build context                                                                0.0s
 => => transferring context: 1.27kB                                                              0.0s
 => [prod-cpu 1/1] FROM docker.io/library/python:3.11-slim-bookworm@sha256:6ed5bff4d7d377e2a27d  0.0s
 => [prod-rknn 1/1] ADD --checksum=sha256:73993ed4b440460825f21611731564503cc1d5a0c123746477da6  0.0s
 => CACHED [prod-rknn 1/1] ADD --checksum=sha256:73993ed4b440460825f21611731564503cc1d5a0c12374  0.0s
 => CACHED [prod 1/8] RUN apt-get update &&     apt-get install -y --no-install-recommends tini  0.0s
 => CACHED [prod 2/8] WORKDIR /usr/src/app                                                       0.0s
 => CACHED [prod 3/8] RUN echo "hard core 0" >> /etc/security/limits.conf &&     echo "fs.suid_  0.0s
 => CACHED [builder 1/6] RUN apt-get update && apt-get install -y --no-install-recommends g++    0.0s
 => ERROR [builder 2/6] RUN pip install --upgrade pip && pip install poetry                      0.5s
------
 > [builder 2/6] RUN pip install --upgrade pip && pip install poetry:
0.405 Python path configuration:
0.405   PYTHONHOME = (not set)
0.405   PYTHONPATH = (not set)
0.405   program name = '/usr/local/bin/python3.11'
0.405   isolated = 0
0.405   environment = 1
0.405   user site = 1
0.405   safe_path = 0
0.405   import site = 1
0.405   is in build tree = 0
0.405   stdlib dir = '/usr/local/lib/python3.11'
0.406   sys._base_executable = '/usr/local/bin/python3.11'
0.406   sys.base_prefix = '/usr/local'
0.406   sys.base_exec_prefix = '/usr/local'
0.406   sys.platlibdir = 'lib'
0.406   sys.executable = '/usr/local/bin/python3.11'
0.406   sys.prefix = '/usr/local'
0.406   sys.exec_prefix = '/usr/local'
0.406   sys.path = [
0.406     '/usr/local/lib/python311.zip',
0.406     '/usr/local/lib/python3.11',
0.406     '/usr/local/lib/python3.11/lib-dynload',
0.406   ]
0.406 Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
0.406 Python runtime state: core initialized
0.406 LookupError: no codec search functions registered: can't find encoding
0.406
0.406 Current thread 0x0000ffff9d7c3020 (most recent call first):
0.406   <no Python frame>
------
Dockerfile:31
--------------------
  29 |     RUN apt-get update && apt-get install -y --no-install-recommends g++
  30 |
  31 | >>> RUN pip install --upgrade pip && pip install poetry
  32 |     RUN poetry config installer.max-workers 10 && \
  33 |         poetry config virtualenvs.create false
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install --upgrade pip && pip install poetry" did not complete successfully: exit code: 1

[yoni13]

build镜像失败了，网上解决这个报错一般通过unset PYTHONHOME 和 PYTHONPATH 来解决，但看日志输出PYTHONHOME 和 PYTHONPATH本就没有值，你有碰到过这个问题吗？

machine translation：
I failed to build image , and the common solution to this error online is to unset PYTHONHOME and PYTHONPATH. However, according to the log output, PYTHONHOME and PYTHONPATH don't have any values. Have you encountered this issue before?

[+] Building 0.7s (14/23)                                                              docker:default
 => [internal] load build definition from Dockerfile                                             0.0s
 => => transferring dockerfile: 4.33kB                                                           0.0s
 => [internal] load metadata for docker.io/library/python:3.11-slim-bookworm@sha256:6ed5bff4d7d  0.1s
 => [internal] load metadata for docker.io/library/python:3.11-bookworm@sha256:adb581d8ed80edd0  0.1s
 => [internal] load .dockerignore                                                                0.0s
 => => transferring context: 58B                                                                 0.0s
 => [builder-cpu 1/1] FROM docker.io/library/python:3.11-bookworm@sha256:adb581d8ed80edd03efd4d  0.0s
 => [internal] load build context                                                                0.0s
 => => transferring context: 1.27kB                                                              0.0s
 => [prod-cpu 1/1] FROM docker.io/library/python:3.11-slim-bookworm@sha256:6ed5bff4d7d377e2a27d  0.0s
 => [prod-rknn 1/1] ADD --checksum=sha256:73993ed4b440460825f21611731564503cc1d5a0c123746477da6  0.0s
 => CACHED [prod-rknn 1/1] ADD --checksum=sha256:73993ed4b440460825f21611731564503cc1d5a0c12374  0.0s
 => CACHED [prod 1/8] RUN apt-get update &&     apt-get install -y --no-install-recommends tini  0.0s
 => CACHED [prod 2/8] WORKDIR /usr/src/app                                                       0.0s
 => CACHED [prod 3/8] RUN echo "hard core 0" >> /etc/security/limits.conf &&     echo "fs.suid_  0.0s
 => CACHED [builder 1/6] RUN apt-get update && apt-get install -y --no-install-recommends g++    0.0s
 => ERROR [builder 2/6] RUN pip install --upgrade pip && pip install poetry                      0.5s
------
 > [builder 2/6] RUN pip install --upgrade pip && pip install poetry:
0.405 Python path configuration:
0.405   PYTHONHOME = (not set)
0.405   PYTHONPATH = (not set)
0.405   program name = '/usr/local/bin/python3.11'
0.405   isolated = 0
0.405   environment = 1
0.405   user site = 1
0.405   safe_path = 0
0.405   import site = 1
0.405   is in build tree = 0
0.405   stdlib dir = '/usr/local/lib/python3.11'
0.406   sys._base_executable = '/usr/local/bin/python3.11'
0.406   sys.base_prefix = '/usr/local'
0.406   sys.base_exec_prefix = '/usr/local'
0.406   sys.platlibdir = 'lib'
0.406   sys.executable = '/usr/local/bin/python3.11'
0.406   sys.prefix = '/usr/local'
0.406   sys.exec_prefix = '/usr/local'
0.406   sys.path = [
0.406     '/usr/local/lib/python311.zip',
0.406     '/usr/local/lib/python3.11',
0.406     '/usr/local/lib/python3.11/lib-dynload',
0.406   ]
0.406 Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
0.406 Python runtime state: core initialized
0.406 LookupError: no codec search functions registered: can't find encoding
0.406
0.406 Current thread 0x0000ffff9d7c3020 (most recent call first):
0.406   <no Python frame>
------
Dockerfile:31
--------------------
  29 |     RUN apt-get update && apt-get install -y --no-install-recommends g++
  30 |
  31 | >>> RUN pip install --upgrade pip && pip install poetry
  32 |     RUN poetry config installer.max-workers 10 && \
  33 |         poetry config virtualenvs.create false
--------------------
ERROR: failed to solve: process "/bin/sh -c pip install --upgrade pip && pip install poetry" did not complete successfully: exit code: 1

I don't face that problem, have you tried to rebuild without cache?

[yuukh]

same error with --no-cache option.I'll try to figure it out later.

[yoni13]

same error with --no-cache option.I'll try to figure it out later.

maybe try docker system prune (REMOVE ALL UNUSED IMAGES,DATAS etc) and rebuild
or maybe get your pi on a VPN because looks like you are in China(?

[yuukh]

非常感谢，执行了docker system prune后可以顺利构建镜像了，不知道之前是什么奇怪的bug。网络问题我早已用代理解决了，网络一般不会造成bug，只是apt update之类的命令会很慢。可以让npu发挥作用了，谢谢你的付出。

machine translation:
Thank you very much, after executing docker system prune I can build the image without any problem. I can get the npu to work, thanks for the effort.

[yoni13]

Oh, wait!
After changing Smart Search concurrency to 1, it works! (clip) 🎉🎉🎉

It looks like it got itself race-conditioned or something.

[yoni13]

Search: (CPU)2s -> (NPU)750 ms
I just found out that the previous result's CPU model was not preloaded, there are 100~500ms improvements sometimes.

[yoni13]

https://github.com/leafqycc/rknn-multi-threaded
I wonder how this works.

[devinzhang91]

https://github.com/leafqycc/rknn-multi-threaded I wonder how this works.

Initialize different NPU cores (only rk3588 and rk3576 have multiple NPUs) into an NPU pool, similar to a thread pool. Put Fream into a queue, which is equivalent to asynchronous processing. Get Fream is the result of the previous Fream. If Fream is not processed (queue is empty), skip this loop.

[yoni13]

https://github.com/leafqycc/rknn-multi-threaded I wonder how this works.

Initialize different NPU cores (only rk3588 and rk3576 have multiple NPUs) into an NPU pool, similar to a thread pool. Put Fream into a queue, which is equivalent to asynchronous processing. Get Fream is the result of the previous Fream. If Fream is not processed (queue is empty), skip this loop.

So people might actually knows the "inf" issue, but I don't see anyone documenting it(?

[yoni13]

Ok, now all models (clip ViT-B-32 and buffalo_l) work with concurrency 1.
I will push updates and models to my repo later.
Also, I don't think concurrency 1 is acceptable, will investigate it further when I have time.

[yoni13]

We need an export script that accepts CLI arguments for the Immich model name and SoC, then exports the model and uploads it to Hugging Face. Then we can run that CLI for each model.

I think I can try to make one.

[mertalev]

Yes, that would work

also, is it appropriate to set the immich-app/ViT-B-32__openai/textual/model.rknn a dummy (to mark that this model supports RKNN) and instead load the model from path.replace('model.rknn',f'rknn/{soc_name}.rknn')? or do we have to rewrite the model_path.suffix logic 🤔

And for the file, any way to make Hugging Face not download rknn files if not using rknn? I see, line 71@app/models/base.py

They can just go in the same folder without the rknn folder.

[yoni13]

Yes, that would work

also, is it appropriate to set the immich-app/ViT-B-32__openai/textual/model.rknn a dummy (to mark that this model supports RKNN) and instead load the model from path.replace('model.rknn',f'rknn/{soc_name}.rknn')? or do we have to rewrite the model_path.suffix logic 🤔

And for the file, any way to make Hugging Face not download rknn files if not using rknn? I see, line 71@app/models/base.py

They can just go in the same folder without the rknn folder.

OK👍.

[yoni13]

Yes, that would work

also, is it appropriate to set the immich-app/ViT-B-32__openai/textual/model.rknn a dummy (to mark that this model supports RKNN) and instead load the model from path.replace('model.rknn',f'rknn/{soc_name}.rknn')? or do we have to rewrite the model_path.suffix logic 🤔
And for the file, any way to make Hugging Face not download rknn files if not using rknn? I see, line 71@app/models/base.py

They can just go in the same folder without the rknn folder.

ViT-B-32__openai

https://github.com/yoni13/immich_to_rknn2/tree/unificli
This script takes the soc name and soc name and builds model to that folder(/buffalo_l/detection/rk3566.rknn)

It hasn't generated the model.rknn dummy yet, probably later.
Done.

[yoni13]

I am adding it to rknn/export later

[yoni13]

We only load the onnx model when the get_input and get_output are required now, no more 1 gig memory usage for a textual model!

[devinzhang91]

Great job, I have an Orangepi CM4 and just made a PCB circuit board that converts PCIe (m2 adapter) to 5 SATA. I happen to need this!

[yoni13]

#15241 btw
Porting tests yesterday (my first time writing tests, really tired) and found a unintended(or not idk) bugs that some values passed to the rknn is diff from inputs, taking a look later.

But seems like everything still works(?

[mertalev]

Nice, I'll take a look at it soon

[yoni13]

After this commit yoni13@4b0f93c#diff-11bd5eb879036c49a92372828a94f25a483e31f60e829255bcee33b87ca0260b , we have

Run poetry run mypy --install-types --non-interactive --strict app/
error: --install-types failed (no mypy cache directory)

idk what happened

edit: probably duplicate-module error, just found that mypy issue thread
python/mypy#10863
wait that was fixed??
it only fails in CI

[yoni13]

That test works on my PC, so um........?

[yoni13]

I solved that issue! Wish me luck with the exams tomorrow!

[Harris-Shelby]

可以打包一个rk3566的镜像测试一下吗

[Harris-Shelby]

我非常乐意测试，明天给你答复
I'm very happy to test it and reply to you tomorrow.

[mertalev]

Did you pull the latest main? asVector is not used anymore so it should be an old commit.

[Harris-Shelby]

git checkout dd52c2d68ce9829424bfc15c5ecb09efdadb1ac0 should make the rknn works again

Working well, the NPU of the RK3566 is finally useful. Thank you very much. It's awesome!

[yoni13]

Did you pull the latest main? asVector is not used anymore so it should be an old commit.

Oh we were testing it with the latest(at that time) release, I will begin testing using main later.

[yoni13]

Yep, testing with v1.125.1 works

[devinzhang91]

I would like to ask which version of rknn-toolkit2 are you using? I encountered an issue while using version 1.6, where the onnx model cannot be opened. The reason is that onnx runtime==1.16.0 and onnx==1.14.1 cannot support higher versions of opset in onnx (from immich-app/ViT-B-32_openi), but I upgraded onnx runtime==1.19.0 and onnx==1.17.0 and cannot meet the dependency of rknn-toolkit2==1.6.0. This makes me very distressed.

[yoni13]

I would like to ask which version of rknn-toolkit2 are you using? I encountered an issue while using version 1.6, where the onnx model cannot be opened. The reason is that onnx runtime==1.16.0 and onnx==1.14.1 cannot support higher versions of opset in onnx (from immich-app/ViT-B-32_openi), but I upgraded onnx runtime==1.19.0 and onnx==1.17.0 and cannot meet the dependency of rknn-toolkit2==1.6.0. This makes me very distressed.

We dont use rknn-toolkit2 in the app because of the dependencies issue, we convert those model first at github action.

[yoni13]

I would like to ask which version of rknn-toolkit2 are you using? I encountered an issue while using version 1.6, where the onnx model cannot be opened. The reason is that onnx runtime==1.16.0 and onnx==1.14.1 cannot support higher versions of opset in onnx (from immich-app/ViT-B-32_openi), but I upgraded onnx runtime==1.19.0 and onnx==1.17.0 and cannot meet the dependency of rknn-toolkit2==1.6.0. This makes me very distressed.

for version, we are using rknn-toolkit2==2.3.0

[yoni13]

V1.6 is a really old ! Maybe you didnt know that they changed their repo to https://github.com/airockchip/rknn-toolkit2 ?

[devinzhang91]

V1.6 is a really old ! Maybe you didnt know that they changed their repo to https://github.com/airockchip/rknn-toolkit2 ?V1.6真的很老了！也许您不知道他们将存储库更改为https://github.com/airockchip/rknn-toolkit2 ？

Yes, I have switched to version 2.3.0 and the dependencies have been resolved. Thank you for your reply.In addition, I found a slight difference in accuracy when running the rknn model in the simulator, but I believe this is a normal situation.