Recordings copy speed from cache to NAS - Unable to keep up with recording segments in cache errors

[LaurenceGough]

Hi all,

I am looking for a bit of help.

Thanks to the recent Frigate BETAs I have identified what seems like a very intermittent issue writing to my NAS. The NAS is dedicated to Frigate with a single 2TB WD20PURX-64PFUY0 (the purple surveillance drive) HDD. Only /recordings/ is stored on the NAS. DB etc is local to the host.

Example of the error: "WARNING : Unable to keep up with recording segments in cache for Cam1. Keeping the 5 most recent segments out of 6 and discarding the rest..." etc

This causes a loss of recording during the duration of this, usually around 30-40 seconds is lost.

This can happen quite a bit in an hour or it can go half a day or more without showing once!

Thanks to Blake's recommendation I added the following to my config:

logger:
  logs:
    frigate.record: debug

This then shows the copy procedures from the cache to the NAS in the logs, on average this takes around 1 second, but close to where it goes wrong it can go up to 5-15 seconds for a few clips after.

One example:

2023-02-05 19:17:02.316665692  [2023-02-05 19:17:02] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-02-05/19/Cam2/15.16.mp4 in 2.9474120140075684 seconds.
2023-02-05 19:17:06.411911579  [2023-02-05 19:17:06] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-02-05/19/Cam2/15.24.mp4 in 3.513415813446045 seconds.
2023-02-05 19:17:06.570395745  [2023-02-05 19:17:06] frigate.record                 WARNING : Unable to keep up with recording segments in cache for Cam1. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-02-05 19:17:06.579165134  [2023-02-05 19:17:06] frigate.record                 WARNING : Unable to keep up with recording segments in cache for Cam2. Keeping the 5 most recent segments out of 9 and discarding the rest...
2023-02-05 19:17:13.828748463  [2023-02-05 19:17:13] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-02-05/19/Cam1/16.08.mp4 in 6.398522853851318 seconds.
2023-02-05 19:17:21.282820385  [2023-02-05 19:17:21] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-02-05/19/Cam1/16.18.mp4 in 6.570835828781128 seconds.

This NAS is old and only has 256MB of RAM but it is dedicated to only this, it is connected to a gigabit link through a gigabit switch to the Raspberry Pi host which Frigate is running on as a docker container. The HDD passes SMART tests with 0 warnings but I haven't done anything more extensive on it.

The CPU usage on the Pi never goes above 50%, checked with Portainer and with htop. RAM usage is around 50%.

Does anyone have any ideas on what the issue could be? I've turned on all of the options on my NAS to increase performance such as no journaling, write caching on etc. I have restarted my NAS and I have fully updated and restarted my Raspberry Pi. I have tried to accessing events, seeking in these events etc and this doesn't cause the issue. I was thinking maybe during this that was slowing writes.

I am leaving a continuous ping running from the Pi to the NAS and it was always 2ms or less and never dropped or had any packet loss. The switch ports confirm no packet errors, or loss. Even during this issue pings are below 2ms and no packets are lost.

Thank you.

[NickM-27]

If your NAS only has 256MB RAM I doubt it has a fast HDD. Probably just slow, outdated hardware.

[blakeblackshear]

It could also just be that you are running up against the limits of what a Pi can send over a network interface. It's reading from the cameras and writing to the NAS. It could also be that the thread which writes to the NAS is being interrupted to do other things for Frigate. It may just be more than the little Pi can handle.

[blakeblackshear]

The Coral USB3 and ethernet may also share a bus and compete with each other.

[NickM-27]

The Coral USB3 and ethernet may also share a bus and compete with each other.

looks like this is the case for Pi 3 but was addressed in the pi 4: https://raspberrypi.stackexchange.com/a/99897

[LaurenceGough]

Hmm that is a really great point.

Doing a bit of digging it seems the Pi 4 is restricted to a 1x 2.0 PCIE lane for all 4x USB ports. That is 500MB/s or 4 Gbps shared.

So I'm not sure how much the Coral uses (I've no idea!). The SSD running the Pi is also running from USB 3.

However, as Nick mentioned it looks like the Ethernet runs from a separate bus now. I know that was an issue before with the Pi 3.

Pi 4 graph found here: https://heise.cloudimg.io/width/2000/q75.png-lossy-75.webp-lossy-75.foil1/_www-heise-de_/ct/imgs/04/2/7/4/3/7/8/1/RPi4-Block-e4359fdb4be588cf.png

So by purchasing a USB caddy to use as storage I might be just causing a different issue...

I've started looking at the ODROID and other alternatives to the Pi again now. I of course something that will do the job but also not something that drains a ton of power. Something with proper SATA and/or NVME connections sounds ideal...

[NickM-27]

AFAIK the odroid doesn't have working hwaccel with ffmpeg. I'd personally suggest some Intel based small form factor device. Just because it has a higher TDP than a pi doesn't mean it is going to use that much power as it's just a (loosely defined) max draw.

[LaurenceGough]

As this issue comes and goes for me, having a very brief look at the record.py code that runs in Frigate which performs this function I have made a suggested pull request of changing the value of 5 to 20, which I don't think mine has gone quite as high yet but it gives a bit of a buffer. Perhaps 15 might be a middle ground?

The only downside that I could see, in my very, very limited knowledge is a potential for a slightly increased cache usage, but I don't think by that much. However, perhaps merging the change would completely break Frigate, I really don't know enough about it :D. I think it'd fix the issue for me though. Maybe one day when I have the time I'll give it a try building it and testing it myself.

#5419

I see this is why so many products have analytics so they can easily see if this is an issue for some of their users or not. (I'm not suggesting this at all! haha! )

[LaurenceGough]

Just for info it also does it when set to record motion on both cameras rather than record always on both.

    record:
      enabled: True
      retain:
         days: 20
         mode: motion

[DrSpaldo]

I'm getting similar log messages. Except mine is Unraid with SSD cache & spinning drives for storage as well as 64 gigs ram. I have bumped up the shm to 1gb & tmpfs to 16gig. See if that makes a difference..

[NickM-27]

If it keeps happening you should include the times it takes for moving the files via debug logs for frigate.record

[LaurenceGough]

Is it still an issue with your updated settings? As Nick said above if you turn on debug with:

logger:
  logs:
    frigate.record: debug

What times are you seeing?

Thanks

[DrSpaldo]

Is it still an issue with your updated settings? As Nick said above if you turn on debug with:

logger:
  logs:
    frigate.record: debug

What times are you seeing?

Thanks

I changed the settings as above and also turned on the debug. It eventually almost stopped my entire Unraid machine. Everything was so slow I couldn't even load the Frigate or Unraid web interface. So I had to SSH in and kill the Frigate process, that fixed the slow issue. Then I slightly reduced the shm & tmpfs and turned off the debugging. I can't work out where the old logs are kept, so I can't see around that time exactly what was happening.

I am still getting a couple of the same "WARNING : Unable to keep up with recording segments in cache " messages in the logs, but they are pretty limited to a bunch every day or so. I am actually thinking that it could (and this is a guess) be the same time that the mover is activating and moving the files from my cache drive to the main array in Unraid. I have no basis for the theory, other than it may be a time when HDD access is slower. Because when I had the debug log on at first, the move operation was 0.x seconds, so it was pretty fast...

[DrSpaldo]

Actually, I just checked the Unraid log. Can't see the mover doing anything at that time. At another time I can see this error in the log:

Feb 20 22:30:11 SpaldoUnraid kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=emulator,mems_allowed=0,global_oom,task_memcg=/docker/341b48023b9789f52e371f8e0bf598ebfbb91868321d8137e94f41908eafa8f3,task=frigate.capture,pid=32553,uid=0
Feb 20 22:30:11 SpaldoUnraid kernel: Out of memory: Killed process 32553 (frigate.capture) total-vm:40479300kB, anon-rss:36674540kB, file-rss:0kB, shmem-rss:52kB, UID:0 pgtables:72128kB oom_score_adj:0

[Red-ington]

Did we have a fix for this??

[LaurenceGough]

Hi all @Red-ington (tagging as I saw you posted above).

I have found a workaround that almost resolves the issue for me but not quite fully.

The segment (recording) drop issue was on average happening for 15-25 minutes of elevated segments at completely random times, 4-6 times a day. The build that Nick kindly created was better, but still during these times I think I recall the count going up to 50-60 at one point!! Other times around 30-40.

I checked the full syslogs on my Raspberry Pi and NAS and any other logs, any cron jobs that I could find. There was absolutely nothing apart from Frigate segment drop messages. CPU usage never above 50% on either device. RAM usage was high on the NAS but kind of expected as it only has 256MB RAM and is probably caching a lot. Nothing really obvious using the RAM, SMB wasn't using much.

Anyway, I thought I'd change the NAS to use NFS rather than SMB/CIFS/SAMBA. The difference is night and day. Exactly the same setup otherwise. Netgear decided to make the share path quite different when using NFS so that caught me out for a little bit... PS for anyone reading, use showmount -e to check what NFS path is being advertised, it's not always what you set. nfs-common package will need to be installed to be used as NFS client and to run showmount.

However, there were still about 5 times it went to a high of 6-8 segments for a few minutes since I performed this fix ~20 hours ago. The duration is clearly lower and it went no higher than a max of 8 segments at one time, a big improvement. Best of all, no recording losses for the first time in ages!!!!

Perhaps the issue was with the oplocks of SMB/SAMBA, although I believe this is requested by the client, which would be either Frigate, Docker or Portainer, I am not quite sure what the client would be considered in this case. The version of NFS I use doesn't support oplocks.

Hopefully this helps at least one other person.

Nick kindly created a new build of Frigate which has an extended cache size before discarding and it has additional logging (please note this was a one off build for testing):

I (Nick did, this is a quote) have created a build at crzynik/frigate:rec-seg-20 which sets the segment count to 20 and also adds a log to put how many segments are currently stored.

Also tagging this issue in as it's a similar issue. #5818

There is a lot of useful information in that issue too, it's worth a read.

Thanks

[f1d094]

For anyone who is still wrestling with this issue, the solution for me was to completely disable swap inside the container. Despite a very capable system I had had loads of "Unable to keep up with recording segments in cache..." issues, which brought me here. Previsously I had an already conservative --memory-swappiness=1 which is now --memory-swappiness=0.

Please note that I have plenty of RAM dedicated to the container as well as hard limits on utilization so that any memory leaks leading to OOM don't bring down the system. Be sure to check your nvidia-smi, vmstat 1, iotop, iostat -sxy --human 1 1, and podman stats, etc. to make sure you don't have an unrelated issue before trying.

For anyone interested, below is my current podman run command for initializing the container. We are now humming along with 21 cameras and barely breaking a sweat:

podman run -d \
  --name frigate \
  --privileged \
  --memory=6g \
  --memory-swap=6g \
  --memory-swappiness=0 \
  --ipc=private \
  --restart unless-stopped \
  --mount type=tmpfs,target=/tmp/cache,tmpfs-size=5000000000 \
  --device /dev/bus/usb:/dev/bus/usb \
  --device nvidia.com/gpu=0 \
  --security-opt=label=disable \
  --cap-add=CAP_PERFMON \
  --shm-size 1g \
  -v /mnt/Media:/media/frigate \
  -v /opt/Frigate/config:/config \
  -v /etc/localtime:/etc/localtime:ro \
  --ip=172.30.2.2 \
  -p 8555:8555/tcp \
  -p 8555:8555/udp \
  --network=daleks-net \
  ghcr.io/blakeblackshear/frigate:stable

[NickM-27]

if you want to put up a PR to add a note of this in the troubleshooting docs that would be great

[f1d094]

Hmm...wouldn't let me create a PR because I am not a collaborator. :-/

[NickM-27]

you'll need to fork the repo and make a PR via that

[f1d094]

Ah yes. Need coffee.

[f1d094]

PR#10021 change as you see fit. Thank you for all your hard work.

[lillrynkan]

I had the same issue using an external (USB) spinning disk (2TB) that has been used to max capacity for some 2 years. I did some tests on the disk and found it did not perform fast at all (~5Mb/sec). I ended up reformatting the partition and performance was then back to normal. The disk was originally formatted with exfat. I am suspecting that the issue was heavy fragmentation on the disk. A suggestion that might be preventing this is to have an option to control the deletion of old recording so that this happens based on % instead of 1h recording since 1h recording on large volumes is a very low % and will increase fragmentation. However I think this problem is mostly affecting exfat file systems on spinning disks.

[f1d094]

I think a nice performance improvement would be to use lazy deletion regardless of FS, assuming it is supported.

@blakeblackshear: Is there a mechanism to ionice the deletes?

[Cytomax55]

##############################

EDIT
I just found out my storage is full lol... let me see if that fixes it first.. sorry

####################################

Frigat is running on
Intel Core i3-7320 @ 4x 4.1GHz [47.0°C]
16 gigs of ram
GO2RTC 1.9.4

The storage is running on NFS to my nas running Truenas with 4 spinning HD in raid 1

the 2 computers are connected via a 10 gig cable, cameras are on a 1 gig switch

htop shows
50% across all 4 cores
and 12.5 out of 15.5 RAM usage

I have 10 cameras running
detect at 720p 5 fps
record at 4 k

I am currently getting

2024-06-30 10:22:01.422915458 [2024-06-30 10:22:01] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for camera_50. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-30 10:22:11.412490898 [2024-06-30 10:22:11] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for camera_42. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-30 10:22:11.413380651 [2024-06-30 10:22:11] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for camera_45. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-30 10:22:11.414257115 [2024-06-30 10:22:11] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for camera_43. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-30 10:22:11.415045961 [2024-06-30 10:22:11] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for camera_46. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-30 10:22:11.415747803 [2024-06-30 10:22:11] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for camera_41. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-30 10:22:11.416801003 [2024-06-30 10:22:11] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for camera_47. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-30 10:22:11.417794581 [2024-06-30 10:22:11] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for camera_44. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-30 10:22:11.418777928 [2024-06-30 10:22:11] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for camera_48. Keeping the 6 most recent segments out of 7 and discarding the rest...
2024-06-30 10:22:11.419547516 [2024-06-30 10:22:11] frigate.record.maintainer WARNING : Unable to keep up with recording segments in cache for camera_50. Keeping the 6 most recent segments out of 7 and discarding the rest...

here is my docker container

frigate:
container_name: frigate
privileged: true # this may not be necessary for all setups
restart: unless-stopped
image: ghcr.io/blakeblackshear/frigate:stable
shm_size: "200mb" # update for your cameras based on calculation above
devices:
#- /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
- /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
- /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
volumes:
- /etc/localtime:/etc/localtime:ro
- /media/user/frigate/config:/config
- /media/user/cameras/frigate/storage:/media/frigate
- type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
target: /tmp/cache
tmpfs:
size: 1000000000
ports:
- "5000:5000"
- "8554:8554" # RTSP feeds
- "8555:8555/tcp" # WebRTC over tcp
- "8555:8555/udp" # WebRTC over udp
environment:
FRIGATE_RTSP_PASSWORD: "redacted"

what say you?

[JonGilmore]

What version of frigate are you running?

edit: nevermind, i see it in your docker compose, stable.

[Cytomax55]

I deleted everything...
ill monitor the logs but hopefully its just because my HD was full
ill look into making sure it deletes thing automatically

[zacheryph]

I am running into this now also after upgrading from 0.12 to 0.14. I've never seen this error before. Same hardware, same [number of] cameras, same everything. Why is it having a problem now that never was before :(

[NickM-27]

You should create your own support issue and provide config and logs. There's no way to help without this info

[LaurenceGough]

Just FYI, hopefully you saw my "fix" above to switch the NAS to use NFS instead? Anyway I've long since moved to a smaller X86 based computer and I couldn't be happier. The onboard Intel iGPU is able to work just as well as the USB Coral saving even more power. I've never seen one of these errors since I just use SATA storage now. It's significantly faster and the power consumption is the same as the Pi. As cool as the Pi's I feel they are easily beat for a home server.

[fgonza2]

i am on NFS and started to see this error when migrating from 14.0 to 14.1 - i went back to 14.0 and the issue is gone. Confirmed that for me, it happens only in 14.1 but not 14.0 any ideas why?

[photomoose]

@fgonza2 @zacheryph Try disabling the swap memory, as @f1d094 mentioned above in #5392 (comment)

It's a bit early for me to confirm, but I did this a couple of hours ago and have noticed a significant improvement in performance and no sight of those errors since.

I am running Frigate in Docker on a Proxmox VM. The VM has an NFS drive mounted in /etc/fstab as follows:

10.0.0.4:/volume1/frigate       /mnt/frigate    nfs     defaults        0 0

The memory and NFS drive are configured in my docker compose as follows:

services:
  frigate:
    container_name: frigate
    privileged: true
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    mem_swappiness: 0
    memswap_limit: 7.5gb
    deploy:
      resources:
        limits:
          memory: 7.5gb
    cap_add:
      - CAP_PERFMON
    shm_size: "128mb"
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - /dev/dri/renderD128:/dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware. See video on how to find this
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /opt/frigate/config/config.yml:/config/config.yml:rw # This is where config.yml is stored to hold camera info, etc.
      - /mnt/frigate:/media/frigate
...

[fgonza2]

i had instabilities in my NFS connection, finally rootcaused that was the issue for me.

[LaurenceGough]

This has happened now 2 or 3 times with 14.1 (0.14.1-f4f3cfa) running in Docker on a far more power than my original post Intel x86 platform.

It clears up after restarting Frigate but then seems to reappear eventually (with all cameras).

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 1  0 1018112 786584 284716 3976860    0    1    18   854 3390   11  9  5 85  1  0  0
 0  0 1018112 785564 284716 3977632    0    0     0     0 2547 2453  1  2 97  0  0  0
 0  0 1018112 785564 284716 3979172    0    0     0     0 3054 2464  1  2 97  0  0  0

free -h
               total        used        free      shared  buff/cache   available
Mem:           7.5Gi       3.0Gi       776Mi        17Mi       4.1Gi       4.5Gi
Swap:          2.0Gi       997Mi       1.0Gi

I believe this is all healthy looking from my limited Linux knowledge... I/O seemed normal.

I have loads of free disk space and it's all correctly mounted, I checked in Frigate container itself also - so it's not that.

However Frigate is constantly complaining of being unable to keep up with recording segments every few seconds non-stop for days when this happens. Since Frigate was last restarted was 5 days before the error this time.

I do have a 5120MB memory limit on the Frigate docker (out of the 8GB) perhaps this is the cause?

I'm now adding a 6.5GB limit, 1GB reservation and mem_swappiness: 0

Thanks to f1d094 for sharing this potential fix with us!

What is odd is how well it worked for quite a while since 14.1 release, which makes you think that perhaps it was another update that caused it, either that or my most likely constant Frigate config tinkering and therefore Frigate restarting for the first few months was enough to prevent it, haha...

[bmatthewshea]

This has happened now 2 or 3 times with 14.1 (0.14.1-f4f3cfa) running in Docker on a far more power than my original post Intel x86 platform.

@LaurenceGough Yeah. Just to add my 2 cents - I just had this happen for the first time on basically same specs you have, but only 1 camera. I never saw this error previously on older version and have been running 0.14.1 for many months w/ no probs. My files are saved to a NAS on 1gbit line. NAS is only used by me - no backups running or anything else that would account for any 'lagged' read/writes.. and ran older version to same nas for like a year and never saw this. I will try the fix for swap if it happens again.

[Cytomax55]

i have been running frigate for a few months and i just started receiving this error with the latest docker version
running on

manjaro
5800X
64 gigs of ecc ram

Frigate (0.14.1-f4f3cfa)
go2rtc platform=linux/amd64 revision=199fdd6 version=1.9.8

"WARNING : Unable to keep up with recording segments in cache for Cam1. Keeping the 5 most recent segments out of 6 and discarding the rest..."