USB Frigate inference time ... 30-60 ms then drops to approx. 10 ms

[ElectronicBattle]

Hello

Question:
Why does my inference time change when I start using the UI and what is poor about my config which I could alter in order to get the inference time back to its original low value?

Background:
My eight cameras were processed in 0.13 with consistently short (10-20 ms) inference times as per the title. I have recently u/g to 0.14 and then 0.14.1 and in doing so I have adjusted my config and tried a few ideas. Something has changed because

• 0.13 to 0.14: first the inference time rose to be four or five times slower

• then overnight, with (I think) no changes, it settled down and remained at 10 ms or so (IR greyscale images, lower data rates, fewer motion/object activations/detections? Or my imagination and I did change things but forgot ...)

• 0.14 and also 0.14 to 0.14.1: I fiddled and messed up the config such that everything is working but the inference time has again changed to those higher figures

• I now see that when Frigate is running and left alone the inference time is short but when I start using the webUI to look at things, it rises (please see metrics screenshot below).

In amongst these config alterations I have lost track of a couple of changes (I ought to try to work out a decent version control system rather than my memory) and now I can't see how to get back to where I was.

It's all working beautifully so I suppose I oughtn't to worry, but returning to the days of constantly <15 ms inference time is something I would like to do before I start experimenting with a new graphics card.

Here's my "slow" config for 0.14.1:

#BASELINE8_150125_0.14.1 based on but updated for version
#BASELINE8_170824_013 this works OK on Frigate and on frigateviewer: 9ms inference and <50% TN CPU load fastinference8cameras013_config_310724_NO_RESTREAMING_FIXED taken from #config_and_audio_110724# # this works with the #FrigateViewer tablet app - events playback OK
# improved detection (height & width added) # also added ffmpeg out args record audio and logger entries at EOF

objects:
  filters:
    dog:
      min_score: .8
      threshold: .9
    cat:
      min_score: .65
      threshold: .8
    face:
      min_score: .7
    package:
      min_score: .65
      threshold: .9
    license_plate:
      min_score: .6
    amazon:
      min_score: .75
    ups:
      min_score: .75
    fedex:
      min_score: .75
    person:
      min_score: .65
      threshold: .85
    car:
      min_score: .65
      threshold: .85


ui:
  time_format: 24hour

mqtt:
  enabled: false

model:
  path: plus://e570a5fa60ccfce5aca76ad55c4ffddd  #mobiledet edgeTPU 2024.3 7/11/24



go2rtc:
  streams:
    #reference format from NickM-27 @ https://github.com/blakeblackshear/frigate/discussions/12376 
    #              "ffmpeg:rtsp://...                                               #video=h264#audio=copy#hardware" note the quotes and hashes
    greenhouse:
      - ffmpeg:rtsp://192.168.1.123:554/media=0/channel=0&level=0/track0#video=h264#audio=copy
      - ffmpeg:greenhouse#audio=opus
    upper_patio:
      - ffmpeg:rtsp://192.168.1.123:554/media=0/channel=1&level=0/track0#video=h264#audio=copy
      - ffmpeg:upper_patio#audio=opus
    ponds:
      - ffmpeg:rtsp://192.168.1.123:554/media=0/channel=2&level=0/track0#video=h264#audio=copy
      - ffmpeg:ponds#audio=opus
    blue_gate:
      - ffmpeg:rtsp://192.168.1.123:554/media=0/channel=3&level=0/track0#video=h264#audio=copy
      - ffmpeg:blue_gate#audio=opus
    lower_steps:
      - ffmpeg:rtsp://192.168.1.123:554/media=0/channel=4&level=0/track0#video=h264#audio=copy
      - ffmpeg:lower_steps#audio=opus
    lower_patio:
      - ffmpeg:rtsp://192.168.1.123:554/media=0/channel=5&level=0/track0#video=h264#audio=copy
      - ffmpeg:lower_patio#audio=opus
    shed:
      - ffmpeg:rtsp://192.168.1.123:554/media=0/channel=6&level=0/track0#video=h264#audio=copy
      - ffmpeg:shed#audio=opus
    drive:
      - ffmpeg:rtsp://192.168.1.123:554/media=0/channel=7&level=0/track0#video=h264#audio=copy
      - ffmpeg:drive#audio=opus
#    doorbell:
#      - "ffmpeg:rtsp://admin:[email protected]:554/Streaming/Channels/101#raw -vf lenscorrection=k1=-0.450:k2=0.1"
#      - "ffmpeg:doorbell#audio=opus"

####rtsp is 30060 and webRTC is 30061 (TN Frigate app interrnals)
  webrtc:
    candidates:
      - 192.168.1.194:30061
      - stun:30061


ffmpeg:
  #####you can only have one line here#####
  #hwaccel_args: preset-nvidia-h264
  #hwaccel_args: preset-nvidia-hevc #gives black screens on cameras
  #hwaccel_args: preset-nvidia-cuvid #gives black screens on cameras
  output_args:
    record: preset-record-generic-audio-aac

detectors:
  coral:
    type: edgetpu
    device: usb
    detect:
      enabled: true

birdseye:
  enabled: true
  mode: continuous #continuous | objects | motion

# record: # <----- Enable recording # was working
#  enabled: true # was working

# added from https://fixtse.com/blog/coral-ia-frigate start#
record:
  enabled: true
  retain:
    days: 4
    mode: motion
  events:
    retain:
      default: 7
      mode: active_objects
# added from https://fixtse.com/blog/coral-ia-frigate #

snapshots:
  # Optional: Enable writing jpg snapshot to /media/frigate/clips (default: shown below)
  enabled: true
  # Optional: save a clean PNG copy of the snapshot image (default: shown below)
  clean_copy: true #false  #true
  # Optional: print a timestamp on the snapshots (default: shown below)
  timestamp: false
  # Optional: draw bounding box on the snapshots (default: shown below)
  bounding_box: true
  # Optional: crop the snapshot (default: shown below)
  crop: false
  # Optional: height to resize the snapshot to (default: original size)
  height: 768
  #height: 800
  # Optional: Restrict snapshots to objects that entered any of the listed zones (default: no required zones)
  required_zones: []
  # Optional: Camera override for retention settings (default: global values)
  retain:
    # Required: Default retention days (default: shown below)
    default: 3
    # Optional: Per object retention days
    objects:
      person: 7
      cat: 4
      dog: 4
      bird: 1
      fox: 4
      badger: 4
      car: 7
      license_plate: 7

  # Optional: quality of the encoded jpeg, 0-100 (default: shown below)
  quality: 100

##########################################################
##########################################################
# go2rtc: # this worked from https://docs.frigate.video/configuration/live/#setting-stream-for-live-ui
#  streams:
#    upper_patio:
#    - rtsp://192.168.1.123:554/media=0/channel=1&level=0/track0 # <- stream which supports video & aac audio.
#    - "ffmpeg:upper_patio#audio=opus" # <- copy of the stream which transcodes audio to opus for webrtc
#    restream: true        
#    upper_patio_sub:
#    - rtsp://192.168.1.123:554/media=0/channel=1&level=1/track0 # <- stream which supports video & aac audio.
#    - "ffmpeg:upper_patio_sub#audio=opus"
##########################################################
##########################################################

motion:
  improve_contrast: true

detect:
  width: 640
  height: 360

#record:
#  width: 1920
#  height: 1080

########################################################################################################################################################################################################################################

cameras: #########GREENHOUSE##################GREENHOUSE##################GREENHOUSE##################GREENHOUSE##################GREENHOUSE#########GREENHOUSE#########
  greenhouse:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.123:554/media=0/channel=0&level=1/track0 #level1 = SD 640x360 (checked CORRECT); track0 media=0 are the only working options tested in ffplay
          roles:
            - detect


        - path: rtsp://192.168.1.123:554/media=0/channel=0&level=0/track0 #level0 = HD 1920x1080(checked CORRECT);  track0 media=0 are the only working options tested in ffplay
          roles:
            - record

    objects:
      track:
        - face
        - person
        - cat
        - dog
        - bird
        - deer
        - fox #placeholder
        - badger #placeholder

    birdseye:
      mode: continuous #continuous | objects | motion
    # masks are black: they block out areas where you aren't interested in detecting things
      order: 1




  upper_patio: #########UPPER_PATIO###############################################UPPER_PATIO###########################UPPER_PATIO###########################UPPER_PATIO###########################
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.123:554/media=0/channel=1&level=1/track0 #level1 = SD 640x360 (checked CORRECT); track0 media=0 are the only working options tested in ffplay
          roles:
            - detect


        - path: rtsp://192.168.1.123:554/media=0/channel=1&level=0/track0 #level0 = HD 1920x1080(checked CORRECT);  track0 media=0 are the only working options tested in ffplay
          roles:
            - record

    objects:
      track:
        - face
        - person
        - cat
        - dog
        - bird
        - deer
        - fox #placeholder
        - badger #placeholder

    birdseye:
      mode: continuous #continuous | objects | motion
    # masks are black: they block out areas where you aren't interested in detecting things
      order: 2

    motion:
      mask: 
        0,219,72,169,140,139,146,79,218,73,291,72,362,133,430,143,450,121,488,120,523,98,640,95,640,0,0,0

    zones:
      upper patio:
        coordinates: 
          640,360,640,337,610,312,603,297,594,279,564,256,524,255,510,242,487,220,451,203,435,201,421,187,409,166,392,152,334,146,274,138,215,142,122,145,78,162,23,202,0,360




  ponds: #################PONDS#######################################PONDS#########################PONDS#########################PONDS#########################PONDS#########################
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.123:554/media=0/channel=2&level=1/track0 #level1 = SD 640x360 (checked CORRECT); track0 media=0 are the only working options tested in ffplay
          roles:
            - detect


        - path: rtsp://192.168.1.123:554/media=0/channel=2&level=0/track0 #level0 = HD 1920x1080(checked CORRECT);  track0 media=0 are the only working options tested in ffplay
          roles:
            - record
    objects:
      track:
        - face
        - person
        - cat
        - dog
        - bird
        - deer
        - fox #placeholder
        - badger #placeholder

    birdseye:
      mode: continuous  #continuous | objects | motion
     # masks are black: they block out areas where you aren't interested in detecting things
      order: 3



  blue_gate: #########BLUE_GATE###################################################BLUE_GATE#############################BLUE_GATE#############################BLUE_GATE#############################BLUE_GATE#############################
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.123:554/media=0/channel=3&level=1/track0 #level1 = SD; track0 media=0 are the only working options tested in ffplay
        #- path: http://truenas.local:8082/media=0/channel=3&level=1/track0 # didn't work: two streams avalble from .123 not from motioneye
        #- path: http://truenas.local:8082 # didn't work: two streams avalble from .123 not from motioneye
      #  input_args: preset-rtsp-restream

          roles:
            - detect
        - path: rtsp://192.168.1.123:554/media=0/channel=3&level=0/track0 #level0 = HD ;  track0 media=0 are the only working options tested in ffplay
          roles:
            - record

    objects:
      track:
        - face
        - person
        - cat
        - dog
        - bird
        - deer
        - fox #placehlder
        - badger # placeholder
        - car
        - license_plate

    birdseye:
      mode: continuous   #continuous | objects | motion
      # masks are black: they block out areas where you aren't interested in detecting things
      order: 4

    motion:
      mask:
        - 0,153,85,148,140,139,239,140,300,142,354,145,398,141,429,141,486,145,518,213,450,226,605,360,640,360,640,0,0,0
        - 479,360,423,337,411,310,390,288,369,268,352,248,340,228,297,223,267,247,231,253,212,204,188,200,152,218,107,227,88,222,73,234,63,250,54,266,45,282,27,292,0,287,0,360

    zones:
      gravel:
        coordinates: 
          411,273,471,360,523,360,585,360,474,227,473,151,388,141,283,146,228,142,0,151,0,301,296,233



  lower_steps: #########LOWER_STEPS#########LOWER_STEPS#######LOWER_STEPS#######LOWER_STEPS#######LOWER_STEPS#######LOWER_STEPS#######LOWER_STEPS#######LOWER_STEPS#######LOWER_STEPS#######
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.123:554/media=0/channel=4&level=1/track0 #level1 = SD; track0 media=0 are the only working options tested in ffplay
        #- path: http://truenas.local:8082/media=0/channel=5&level=1/track0 # didn't work: two streams avalble from .123 not from motioneye
        #- path: http://truenas.local:8082 # didn't work: two streams avalble from .123 not from motioneye
       #input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://192.168.1.123:554/media=0/channel=4&level=0/track0 #level0 = HD ;  track0 media=0 are the only working options tested in ffplay
          roles:
            - record

    objects:
      track:
        - face
        - person
        - cat
        - dog
        - bird
        - deer
        - fox #placeholder
        - badger #placeholder

    birdseye:
      mode: continuous   #continuous | objects | motion
      order: 5


  lower_patio: #########LOWER_PATIO######LOWER_PATIO######LOWER_PATIO######LOWER_PATIO######LOWER_PATIO######LOWER_PATIO######LOWER_PATIO######LOWER_PATIO######LOWER_PATIO######LOWER_PATIO
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.123:554/media=0/channel=5&level=1/track0 #level1 = SD; track0 media=0 are the only working options tested in ffplay
        #- path: http://truenas.local:8082/media=0/channel=5&level=1/track0 # didn't work: two streams avalble from .123 not from motioneye
        #- path: http://truenas.local:8082 # didn't work: two streams avalble from .123 not from motioneye
       #input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://192.168.1.123:554/media=0/channel=5&level=0/track0 #level0 = HD ;  track0 media=0 are the only working options tested in ffplay
          roles:
            - record

    objects:
      track:
        - face
        - person
        - cat
        - dog
        - bird
        - deer
        - fox #placeholder
        - badger #placeholder

    birdseye:
      mode: continuous   #continuous | objects | motion
      order: 6

      # masks are black: they block out areas where you aren't interested in detecting things
    motion:
      mask:
        - 0,148,92,104,184,55,253,0,269,154,272,305,318,125,354,107,345,56,406,60,417,131,504,240,570,360,640,360,640,0,0,0
        #- 0,219,72,169,140,139,146,79,218,73,291,72,362,133,430,143,450,121,488,120,523,98,640,95,640,0,0,0

    zones:
      lower patio:
        coordinates: 
          284,360,303,183,333,157,344,139,357,118,352,99,359,74,382,73,384,110,403,128,499,246,580,360




  shed: #########SHED##########################SHED#########################SHED#########################SHED#########################SHED#########################SHED#########################
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.123:554/media=0/channel=6&level=1/track0 #level1 = SD; track0 media=0 are the only working options tested in ffplay
        #- path: http://truenas.local:8082/media=0/channel=5&level=1/track0 # didn't work: two streams avalble from .123 not from motioneye
        #- path: http://truenas.local:8082 # didn't work: two streams avalble from .123 not from motioneye
       #input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://192.168.1.123:554/media=0/channel=6&level=0/track0 #level0 = HD ;  track0 media=0 are the only working options tested in ffplay
          roles:
            - record

    objects:
      track:
        - face
        - person
        - cat
        - dog
        - bird
        - deer
        - fox #placeholder
        - badger #placeholder

    birdseye:
      mode: continuous   #continuous | objects | motion
      order: 7



  drive: #########DRIVE#########DRIVE#########DRIVE#########DRIVE#########DRIVE#########DRIVE#########DRIVE#########DRIVE#########DRIVE#########DRIVE#########DRIVE#########DRIVE#########DRIVE
    enabled: true

    objects:
      track:
        - face
        - person
        - cat
        - dog
        - bird
        - deer
        - fox #placeholder
        - badger #placeholder
        - car
        - license_plate

    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.123:554/media=0/channel=7&level=1/track0 #level1 = SD; changing track0 to track1 didn't work)
      #input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://192.168.1.123:554/media=0/channel=7&level=0/track0 #level0 = HD ;(changing track0 to track1 didn't work)
          roles:
            - record

    birdseye:
      mode: continuous #continuous | objects | motion
      #restream: True
      order: 8

    motion:
      # masks are black: they block out areas where you aren't interested in detecting things
      mask:
        - 98,129,170,113,189,0,0,0,0,0,0,142
        - 273,0,170,0,178,67,258,117,256,191,414,181,640,307,640,0

    # zones are red: they select areas where monitoring of activity is wanted
    zones:
      driveway:
        coordinates: 
          78,360,248,360,381,360,640,360,640,276,449,177,303,183,298,129,200,64,144,113,0,127,0,360

#  doorbell: #####DOORBELL##########################DOORBELL##########################DOORBELL##########################DOORBELL#####################DOORBELL#####################DOORBELL#####################
#    enabled: true #remember that EZViz won't stream to more than thee (four?) destinations
#    ffmpeg:
#      inputs:
#        - path: rtsp://admin:[email protected]:554/Streaming/Channels/101 # H264 1536x1536
#          roles:
#            - detect
#      output_args:
#        detect: -vf lenscorrection=k1=-0.450:k2=0.1 -f rawvideo -pix_fmt yuv420p
#        - path: rtsp://127.0.0.1:8554/doorbell
#          roles:
#            - record
#      output_args:
#          detect: -vf lenscorrection=k1=-0.450:k2=0.1
#         
#    objects:
#      track:
#        - person
#        - cat
#        - dog
#        - bird
#        - deer
#        - fox #placeholder
#        - badger #placeholder
#    birdseye:
#      mode: continuous     #continuous | objects | motion
#        # masks are black: they block out areas where you aren't interested in detecting things
#      order: 9

# for the doorbell camera. this is what ffplay said (once I'd set the lens correction):

#$  ffplay -i rtsp://admin:[email protected]:554/Streaming/Channels/101 -vf lenscorrection=k1=-0.450:k2=0.095
#  Stream #0:0: Video: h264 (Main), yuv420p(progressive), 1536x1536, 15 fps, 15 tbr, 90k tbn
#  Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp



logger:
  logs:
      #frigate.record.maintainer: debug
    frigate.object_detection: debug
      #watchdog.blue_gate: debug


#version: 0.14
#camera_groups:
#  birdseye:
#    order: 1
#    icon: LuActivity
#    cameras: birdseye
version: 0.14

and here's the metrics page:

which shows about 7 ms until 6:40pm this evening which is when i clicked on the browser tab to look at the day's events in frigate, and the amber bar occurs (60 ms) followed by the greens at 40+ ms.

[NickM-27]

This is likely just a CPU issue, when the server has to spend time doing other operations it is struggling to keep the USB running at full speed.

[ElectronicBattle]

but a few hours of ordinary uniterrupted running later, and the inference time has increased once again:

[hawkeye217]

Did you try swapping the cable?

[ElectronicBattle]

No - it's the standard USB3.0 cable which came with the Coral. It's certainly something I can try but it seems unlikely to be the culprit (seeing as it's a static installation, no vibration, it gets unplugged very rarely). But still worth swapping, yes.

[hawkeye217]

Definitely give that a try. We've had reports from users of the included cable going bad.

[ElectronicBattle]

That's interesting to note. In that case, I shall order a couple of suitable cables later today to experiment with.

In the mean time I will stop the app, move to the other USB3 port, and I'll monitor what happens.

I'll report back as I see results to my experiments; please keep any eye on this thread and I'd welcome your comments if you have any ideas.

[ElectronicBattle]

stop|move to other USB3 port|start:

The associated cpu load c/o btop, where the 50% utilisation drops to half that when the container is stopped and, several minutes after starting it again, it doesn't climb back up:

I noted that the webUI page showed the event video clips but the live feeds were all dead. Reloading the page in the browser gives me "This site can’t be reached 192.168.1.194 refused to connect" so I went to restart the Frigate app once again, only to find it had changed from "running" down to "deploying" in the mean time.

I stopped it and restarted it again:

and the system cpu use now says it's gone back up to 50%(btop):

and the metrics page looks like this:

So it looks like during the first restart the inference time is short (as expected) but Frigate is stuck in a "stale session", and it then rises once restarted the second time at which point the CPU demand increases and the Frigate session is running properly.

I have seen this pattern three times now in the last week or so , with the same cable in two different USB3 ports. I will also change the cable but it appears to me to be a software matter (not necessarily related to Frigate alone).

Going back months, I remember trying a config change on 0.13 which caused long inference times but further edits caused the time to reduce back to < 10 ms and I wonder if that is happening again.

Time to try a new cable (quality unknown, longer than it needs to be, but I already have it ready to use whereas the other cables are not yet ordered).

[ElectronicBattle]

to be clear, until an object detection happens, it will show 10 as that is the starter average. So this may not actually be that the coral is running well but rather that it has not been used until a certain point

OK, useful to know. Certainly there does seem to be some sort of start/restart anomaly and perhaps my inference times need to be monitored for longer to make sure there are definitely detections happening to "create work" for the Coral TPU (coincidentally, I think there have been people & dogs moving in and out of shot several times today so there has been a good degree exercising but I accept it might be hit and miss).

I'll give it an hour and I'll report back. Thanks for this continued guidance.

[ElectronicBattle]

Unfortunately, waiting longer shows me that the long inference times have returned:

[ElectronicBattle]

numerous experiments later, mainly focussed on config changes, and I got nowhere. I decided to stop the frigate app in TrueNAS and install a second version alongside it with identical settings:

What's more is that I saw the Nvidia being properly detected and utilised. Running overnight the Coral is still showing <10 ms inference time and the overall CPU use has dropped

[NickM-27]

That is good to see

[ElectronicBattle]

Yes, a relief (I have documented it here, as best I can, to hopefully help others in the future and to remind me the next time I find something "amiss"!).

Also, my GPU is now showing up at long last, which is a definite bonus.

[ElectronicBattle]

Problem was fixed by my stopping the Frigate app in TrueNAS and installing a second version alongside it with identical settings.