[HW Accel Support]: CPU and GPU extremely high on 0.13.2 #10412
-
Describe the problem you are havingI attempted to update Frigate from .12.1 to .13.2 after updating nvidia drivers to 535-server from 525-server I have flapped between .13.2 and .12.1 and can repeat the results. .13.2 pegs my GPU and CPUs and causes the server to go 90+ PWM on fans to keep the Tesla P4 cool, while running .12.1 keeps the GPU cool and server PWM fans ~20% Changing detectors to sub streams do not change GPU use, it's at 100% for 2 cameras or 11 cameras on 640x480 sub streams. While this limits ram use from 4GB down to ~1.6GB, this causes 100% gpu load and high temperatures. Screenshot of a final roll-back to .12.1 showing gpu use ~average at 25%
Version0.13.2 Frigate config filemqtt:
host: $homeassistantIP
user: $user
password: $password
database:
path: /db/frigate.db
ffmpeg:
global_args: -hide_banner -loglevel warning
input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -use_wallclock_as_timestamps 1
hwaccel_args: preset-nvidia-h264
birdseye:
enabled: True
width: 1280
height: 720
quality: 16
restream: true
go2rtc:
streams:
camera01: # <- for RTSP streams
- rtsp://user:pass@IP:554/Streaming/Channels/1
...
camera11: # <- for RTSP streams
- "ffmpeg:rtsp://user:pass@IP:554/Streaming/Channels/1#video=h264#audio=aac"
webrtc:
listen: ":8555"
candidates:
- fqdn:8555
cameras:
camera01:
ffmpeg:
output_args:
record: preset-record-generic
inputs:
- path: rtsp://127.0.0.1:8554/camera01
input_args: preset-rtsp-restream
roles:
- record
detect:
enabled: True
... #(all cameras configured the same)
detect:
fps: 5
record:
enabled: True
retain:
days: 21
mode: motion
events:
pre_capture: 5
post_capture: 5
retain:
objects:
person: 31
car: 31
objects:
track:
- person
- cat
- car
- dog
- bird
filters:
person:
# Optional: minimum width*height of the bounding box for the detected object (default: 0)
min_area: 5000
# Optional: maximum width*height of the bounding box for the detected object (default: 24000000)
max_area: 100000
# Optional: minimum width/height of the bounding box for the detected object (default: 0)
min_ratio: 0.5
# Optional: maximum width/height of the bounding box for the detected object (default: 24000000)
max_ratio: 2.0
# Optional: minimum score for the object to initiate tracking (default: shown below)
min_score: 0.5
# Optional: minimum decimal percentage for tracked object's computed score to be considered a true positive (default: shown below)
threshold: 0.7
snapshots:
enabled: True
timestamp: False
bounding_box: True
crop: False
retain:
default: 21
objects:
person: 60
detectors:
tensorrt:
type: tensorrt
model:
# path: /trt-models/yolov7-tiny-416.trt
path: /config/model_cache/tensorrt/yolov7-320.trt
input_tensor: nchw
input_pixel_format: rgb
# width: 416
# height: 416
width: 320
height: 320
rtmp:
enabled: Falsedocker-compose file or Docker CLI commandversion: "3.9"
services:
frigate:
environment:
- YOLO_MODELS=yolov4-608,yolov7x-640
- USE_FP16=false
container_name: frigate-rt
restart: unless-stopped
image: ghcr.io/blakeblackshear/frigate:stable
# image: ghcr.io/blakeblackshear/frigate:0.12.1-tensorrt
shm_size: 2gb
# devices:
# - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
deploy: # <------------- Add this section
resources:
reservations:
devices:
- driver: nvidia
capabilities: [gpu]
volumes:
- /etc/localtime:/etc/localtime:ro
- /home/kajer/docker/frigate-rt/db:/db
- /home/kajer/docker/frigate-rt/config/:/config/
- /home/kajer/docker/frigate-rt/trt-models:/trt-models
- /CamStore:/media/frigate
- type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
target: /tmp/cache
tmpfs:
size: 1000000000
ports:
- "8554:8554"
- "8555:8555/tcp"
- "8555:8555/udp"
- "5000:5000"Relevant log outputN/AFFprobe output from your cameraN/AOperating systemDebian Install methodDocker Compose Network connectionWired Camera make and modeltrendnet/generic rtsp Any other information that may be helpfulNo response |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 19 replies
-
|
this is most likely due to motion detection being more sensitive, I'd suggest tuning motion detection |
Beta Was this translation helpful? Give feedback.
-
|
As stated above. I had 12.1 running detection on the 30fps primary streams just fine, including on debian/ubuntu driver version 535-server. I appreciate the memory savings by moving detection to sub streams, but I feel I am not getting the reliable object detection that the primary streams were giving me. In a few more days, I'll have a box with a Quadro RTX5000 and a 2080Ti free for use, I can move the frigate config over there and see what happens for kicks. |
Beta Was this translation helpful? Give feedback.
-
not sure what you mean by that, that wasn't something that I suggested you change |
Beta Was this translation helpful? Give feedback.
-
|
In trying to get GPU use under control, the use of substreams and GPU memory usage was a byproduct. I have further tweaked the motion masks and object filters to get GPU use back under control. I have since removed the If this ticket goes stale, that's fine, but It was not a pleasant experience needing to rewrite my config to use substreams, add masks, and tune all of the things between 12.1 and 13.2. I absolutely want the PTZ features, and will make 13.2 work, but it wasn't as plug and play as I was used to on previous major version changes. I have another Frigate system on 12.1 right now, also using a Tesla P4, that I am waiting to upgrade to 13.2. I dont have PTZ cams at that site, but most cameras are 3-5MP range. I'm willing to take detailed logs if you want to diag that update as well. The Frigate instance is running ubuntu server in an esx7 VM with a Tesla P4 in pcipassthru mode. |
Beta Was this translation helpful? Give feedback.
-
|
13.2 + 11 cameras worth of trees seems to have reached the limits of the Tesla P4 When I moved frigate over to a Quadro RTX 5000, all utilization problems go away. I will keep my detectors on the sub streams, but I can remove the motion masks with little penalty. I know throwing overkill levels of GPU horsepower is not the right way to solve my initial concern of less-than-optimal detection sensitivity post 12.1 migration. I still think the .12.1 to .13.2 migration could have been more "plug and play." I have one more frigate instance with ver .12.1 and ~10 cameras detecting on the main stream to migrate over, and will report on that when I get around to doing that work. As that site doesn't have any PTZ cams, I have little incentive to move away from .12.1 |
Beta Was this translation helpful? Give feedback.
-
|
While tuning further motion masks, can this be explained?
The neighbor came up the drive with a truck, and object detection was triggered while behind the motion mask. The video pipeline documentation says this shouldn't happen, since object detection is AFTER motion detection. no motion boxes are observed on this view, but objects bounding boxes are, as well as my home assistant automation regarding vehicles on the driveway. |
Beta Was this translation helpful? Give feedback.
-
|
motion mask will not stop objects being detected, motion can happen outside the mask or object detection can be run due to another object being tracked |
Beta Was this translation helpful? Give feedback.
13.2 + 11 cameras worth of trees seems to have reached the limits of the Tesla P4
When I moved frigate over to a Quadro RTX 5000, all utilization problems go away. I will keep my detectors on the sub streams, but I can remove the motion masks with little penalty. I know throwing overkill levels of GPU horsepower is not the right way to solve my initial concern of less-than-optimal detection sensitivity post 12.1 migration. I still think the .12.1 to .13.2 migration could have been more "plug and play."
I have one more frigate instance with ver .12.1 and ~10 cameras detecting on the main stream to migrate over, and will report on that when I get around to doing that work. As that site does…