[Tutorial] Dewarping 360 Video

[edalquist]

I have a Ubiquiti AI 360 camera that I wanted to feed into Frigate to have it along side all my other non-Ubiquiti cameras. The AI 360 was the cheapest path for a "good" 360 camera since I already had a UDM SE.

Figuring out a "good" way to dewarp the video for consumption involved:

1. A LOT of time reading the ffmpeg v360 filter documentation
2. Using ffmpeg's ffplay to quickly test various filter options

• ex: ffplay -i rtsps://USER:[email protected]:7441/CAMERA_KEY -vf "v360=fisheye:output=equirect:ih_fov=180:iv_fov=180:pitch=90,crop=in_w:in_h/2:0:in_h/2"

Here is what the camera outputs and a few output options:

• Camera Output: https://imgur.com/iJpenDb
• 6x1 Cube Map: https://imgur.com/Ge8vR7e
  • -vf "v360=fisheye:output=c6x1:ih_fov=180:iv_fov=180:pitch=90"
• Equirectangular: https://imgur.com/AaKWWUd
  • -vf "v360=fisheye:output=equirect:ih_fov=180:iv_fov=180:pitch=90,crop=in_w:in_h/2:0:in_h/2"
• Dual Fisheye: https://imgur.com/XwX91eo
  • -vf "v360=fisheye:output=dfisheye:ih_fov=180:iv_fov=180:pitch=90,crop=in_w:in_h/2:0:in_h/2"
• Half Equirectangular: https://imgur.com/X8jAvGO
  • -vf "v360=fisheye:output=hequirect:ih_fov=180:iv_fov=180:pitch=90,crop=in_w:in_h/2:0:in_h/2"

Ideally I would feed two Half Equirectangular streams into Frigate and treat them as independent cameras. Unfortunately the v360 dewap and crop filters do not support hardware acceleration so this process does end up being CPU intensive, using an entire core of a i5-12500.

So for a "single stream" option I went with the Dual Fisheye output format which Frigate's object detection seems to play well enough with.

Go2RTC is the real hero here since the entire dewarp and crop process can be configured on it's stream allowing Frigate to consume as many copies of that single stream as needed without increasing the CPU cost of the operations.

The next issue I ran into is the setup time for the stream. Unifi's re-streaming endpoint is slow to start AND the dewarp filter adds to the initialization time. I had to manually set up the input_args for the camera to override the default 5s timeout and increase it to 9s.

go2rtc:
  streams:
    under_deck:
      - ffmpeg:rtsps://USER:[email protected]:7441/CAMERA_KEY#video=h264#raw=-vf
        "v360=fisheye:output=dfisheye:ih_fov=180:iv_fov=180:pitch=90,crop=in_w:in_h/2:0:in_h/2"
under_deck:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/under_deck
          input_args:
            - -rtsp_transport
            - tcp
            - -timeout
            - '9000000'

Ideally I'd be able to use a camera like the HIKVISION DS-2CD63C5G0E-IVS which supports on-board ePTZ streams allowing multiple "virtual" dewarped camera streams. That camera is more than 2x the cost of the AI360 however.

Breaking down the ffmpeg filter config:

• v360 - 360 video filter
  • fisheye - input video format
  • output=dfisheye - output video format, dual fisheye
  • ih_fov=180 - input horizontal field of view, since the video is 360 and we are doing "dual" outputs needs to be at 180
  • iv_fov=180 - input vertical field of view, since the video is 360 and we are doing "dual" outputs needs to be at 180
  • pitch=90 - orients the "virtual view" 90 degrees up so the view is "to the side" vs "down at the ground"
• crop - crop filter, applied AFTER the 360 filter
  • in_w:in_h/2 - Width:Height - sets the width and height of the output image, in_w and in_h are references to the input width and height. Since we are cropping the black top half of the dewarped image we set the output width to match the input and the output height to half the input
  • 0:in_h/2 - X Position:Y Position - where to set the X,Y of the top left of the crop. So X is the left edge (0) and Y is half way down the image (in_h/2)

[ElectronicBattle]

This is very clever, and timely - I only found out recently that this sort of dewarping could be effectively done within Frigate.

I hope to try it with my doorbell camera which gives a "fisheye" image as seen in my other post to create a corrected image (the example is courtesy of the EZViz "app" but I don't know if this degree of success is feasible with ffmpeg).

[ElectronicBattle]

That's a good idea - my main machine is linux and with plenty of resource so I can develop on that, get it running and then implement it on my frigate host machine.

That host is a ten year old, quad core hyperthreaded Xeon which ought to cope with some ffmpeg demands but I have flattened it in recent weeks playing around with Frigate - it's also got a low spec Nvidia graphics card which can decode h265 etc. but not encode, so maybe it'll help.

[edalquist]

As far as I can tell frigate's ffmpeg process that reads data from go2rtc is still using hardware decoding. However the go2rtc ffmpeg process doesn't seem to be using hardware acceleration of any sort. intel_gpu_top shows 8 ffmpeg processes which match the 8 that frigate is running.

The ffmpeg process for go2rtc that does the dewarp doesn't have the -hwaccel flag and when I've tried enabling any sort of hardware acceleration for that process it fails to initialize.

I just posted https://stackoverflow.com/q/78911734/131192 to see if any ffmpeg experts can commit on if it is even feasible at all to use v360+crop with VAAPI

[ElectronicBattle]

That stackoverflow approach is also a good idea - I'm afraid I'm somewhat out of my depth with a lot of this.

I recall doing some ffmpeg work within Frigate a few weeks ago and all of my eight threads were at full tilt (CPU transcoding to be able to see the video-clip playback via HLS) when trying to decode a few rtsp streams.
I realised that doing all 8 cameras was a non-starter without GPU effort and I'd also need a better GPU than the one I have fitted in the limited slot height available to me(GT1030). The work you've done means it now looks like some of the processes might not suit GPU utilisation anyway.

I have two instances of Frigate: v0.13 is constantly running as described above and will be able to use an Nvidia card but not at the moment due to implementation limitations (not me, for once ! - it's to do with the docker hosting mechanism) and the other is on my main machine (Linux Mint 22) running v0.14 and using amdgpu Radeon RX580 and an m.2 single Coral via a single lane PCIe adaptor card.

(So I might be able to help test things if you'd like - I'd be happy to assist but you might need to spell it out if it gets too involved though!).

[edalquist]

What I have posted in the tutorial is as good as I think I'm going to get things until someone with more ffmpeg expertise comments on potential options for hardware acceleration and dewarp/crop.

The reason I worked to figure out how to do the dewarp in go2rtc is to have just once ffmpeg process that is using the CPU for decode/dewarp/crop/encode. go2rtc then just sends the output from that one ffmpeg process to as many clients as need it. Frigate's ffmpeg process that consumes the dewarped output IS using hardware acceleration to decode and grab frames for the detect process.

Here is what my per-process CPU usage looks like for 8 cameras using an Intel i5-12500, the iGPU, and a USB Coral:

[ElectronicBattle]

I've enjoyed a modicum of success using equirect and fisheye experiments but the best solution so far is that mentioned by @hawkeye217 in the other thread, namely "lenscorrection" along the lines of

-vf lenscorrection=cx=0.5:cy=0.5:k1=-0.290:k2=-0.012

I used that string as a baseline and I've been fiddling around according to here and here. I also note there are other options to explore c/o lensfun which contains a distortion calculator based on RAW images of a rectilinear map of some form.

I can get it to be about as good as the EZViz app:

My attempt:

which is decent enough. This is the final string:

ffplay -i rtsp://user:password@ip_addr:554/Streaming/Channels/101 -vf lenscorrection=k1=-0.450:k2=0.1

[Zuirdj]

Hi!

I went through the tortuous road of ffmpeg some weeks ago, not being an expert but a newbie in Frigate, following 2 or 3 post on reddit and here about dewarping. So in my case was trial and error.

I have a slim passageway outside my kitchen , where I put a turret Dahua-clone cam, but it was only focussing one side of the corridor. I found a fisheye Hikvision DS-2CD2955FWD-IS, as I didn't want to put two cameras focsing to the two point of interest:

Looking for the solution I found topics calling to use ffplay to experiment the best result. Although at first I thought the best solution was to use "output:dfisheye" as you, the use of the stream in HA and the detection wouldn't be the best for my use case.

Finally I decided to divide the stream and to use it as they were 2 cameras. Obviously this would increase the CPU usage, but until now my I7-8700 is being nice.

go2rtc:
  streams:
    cocina: 
      ffmpeg:rtsp://user:[email protected]:554/Streaming/channels/101#video=h264#audio=copy#raw=-vf v360=fisheye:flat:id_fov=200:d_fov=95:roll=-122:pitch=31:rorder=rpy:w=1920:h=1080 # DS-2CD2955FWD-IS fisheye
    cocina_sub: 
      ffmpeg:rtsp://user:[email protected]:554/Streaming/channels/102#video=h264#audio=copy#raw=-vf v360=fisheye:flat:id_fov=200:d_fov=95:roll=-122:pitch=31:rorder=rpy:w=640:h=360 # DS-2CD2955FWD-IS fisheye
    taller: 
      ffmpeg:rtsp://user:[email protected]:554/Streaming/channels/101#video=h264#audio=copy#raw=-vf v360=fisheye:flat:id_fov=200:d_fov=95:roll=-12:pitch=30:rorder=rpy:w=1920:h=1080 # DS-2CD2955FWD-IS fisheye
    taller_sub: 
      ffmpeg:rtsp://user:[email protected]:554/Streaming/channels/102#video=h264#audio=copy#raw=-vf v360=fisheye:flat:id_fov=200:d_fov=95:roll=-12:pitch=30:rorder=rpy:w=640:h=360 # DS-2CD2955FWD-IS fisheye

cameras:
  cocina:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/cocina
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/cocina_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    motion:
      mask: 0.416,0.001,0.418,0.103,0.629,0.286,0.996,0.852,0.997,0.004
    live:
      stream_name: cocina_sub

  taller:
    enabled: true
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/taller
          input_args: preset-rtsp-restream
          roles:
            - record
            - audio
        - path: rtsp://127.0.0.1:8554/taller_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    motion:
      mask: 0,0,0,0.851,0.314,0.321,0.676,0
    live:
      stream_name: taller_sub

The result:

I miss the OSD with the name, date, time of my other cameras, and sometimes the fan of the pc running Frigate speed up often, but for now is the best that I got. Audio is not working yet, but I will try to make it work if possible when having more time.

[ElectronicBattle]

Oh - well done! That's very pleasing.

I got as far as my "dewarping" string and then got too muddled up to work out where to put the ffmpeg directives inside Frigate's config.yaml. I can now see how you did it, so I can copy your approach when I get chance these next few evenings and hopefully it'll look "right".

[edalquist]

Does the DS-2CD2955FWD-IS support virtual PTZ streams? IE can you set up multiple streams to the camera and enable Frigate's PTZ controls to get a camera-side dewarped view?