Help with TensorRT in 13.0 dev

[jonadis]

I'm trying to follow the docs here:
https://deploy-preview-4055--frigate-docs.netlify.app/configuration/detectors
and am getting stuck on the step where it says:

mkdir trt-models
wget https://raw.githubusercontent.com/blakeblackshear/frigate/docker/tensorrt_models.sh
chmod +x tensorrt_models.sh
docker run --gpus=all --rm -it -v `pwd`/trt-models:/tensorrt_models -v `pwd`/tensorrt_models.sh:/tensorrt_models.sh nvcr.io/nvidia/tensorrt:22.07-py3 /tensorrt_models.sh

because I get a 404 on https://raw.githubusercontent.com/blakeblackshear/frigate/docker/tensorrt_models.sh
the URL from the stable docs seems to work but I'm wondering if that's compatible with 13.0 dev because I get an error when I try to follow the 13.0 dev instructions with the sh script from the master branch from the stable docs. This is the error:

2023-09-11 13:59:11.854613606  [2023-09-11 13:59:11] frigate.detectors.plugins.tensorrt ERROR   : 1: [stdArchiveReader.cpp::StdArchiveReader::42] Error Code 1: Serialization (Serialization assertion stdVersionRead == serializationVersion failed.Version tag does not match. Note: Current Version: 232, Serialized Engine Version: 213)
2023-09-11 13:59:11.868463258  Process detector:tensorrt:
2023-09-11 13:59:11.868524736  [2023-09-11 13:59:11] frigate.detectors.plugins.tensorrt ERROR   : 4: [runtime.cpp::deserializeCudaEngine::66] Error Code 4: Internal Error (Engine deserialization failed.)
2023-09-11 13:59:11.869671673  Traceback (most recent call last):
2023-09-11 13:59:11.869687775    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-09-11 13:59:11.869688981      self.run()
2023-09-11 13:59:11.869690195    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-09-11 13:59:11.869691277      self._target(*self._args, **self._kwargs)
2023-09-11 13:59:11.869706849    File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
2023-09-11 13:59:11.869708052      object_detector = LocalObjectDetector(detector_config=detector_config)
2023-09-11 13:59:11.869709076    File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
2023-09-11 13:59:11.869710099      self.detect_api = create_detector(detector_config)
2023-09-11 13:59:11.869711177    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2023-09-11 13:59:11.869712121      return api(detector_config)
2023-09-11 13:59:11.869715085    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 221, in __init__
2023-09-11 13:59:11.869716926      self.input_shape = self._get_input_shape()
2023-09-11 13:59:11.869718048    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 93, in _get_input_shape
2023-09-11 13:59:11.869740082      binding = self.engine[0]
2023-09-11 13:59:11.869741198  TypeError: 'NoneType' object is not subscriptable
2023-09-11 13:59:11.869747192  Exception ignored in: <function TensorRtDetector.__del__ at 0x7f410d7cb790>
2023-09-11 13:59:11.869748207  Traceback (most recent call last):
2023-09-11 13:59:11.869749272    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 239, in __del__
2023-09-11 13:59:11.869986055      if self.outputs is not None:
2023-09-11 13:59:11.869987898  AttributeError: 'TensorRtDetector' object has no attribute 'outputs'

[NickM-27]

You are referencing outdated docs for 0.13, you should be referencing https://deploy-preview-6262--frigate-docs.netlify.app/configuration/object_detectors#nvidia-tensorrt-detector

[solarsparq]

This also worked for me (running 0.13-beta3).. I was accidentally referencing stable/old documentation to create my own trt-models under /trt-models .. it appears the project went with nesting it under /config & self-creation of missing models .. which I definitely appreciate. Downloading a 6GB NVIDIA docker image to generate the models seemed a little overkill.. especially for people in bandwidth-constrained situations. Thank you for all the work on Frigate.. I really like this project.

[jonadis]

Thanks Nick, that's the solution.
Separately, I see the new docs mention CodeProject.AI
Do you have any feeling of which is better between TensorRT and CodeProject.AI ?

[NickM-27]

it entirely depends on the model that is used, both offer different models. Also, CodeProject.AI works via http/rest so TensorRT will offer lower inference times