GaussianBlurBackgroundProcessor doesn't work with offscreencanvas on Safari 16.4+; Significant memory usage error

[pzubar]

Although Safari 16.4 supports OffscreenCanvas API and the VirtualBackgroundProcessor does its job, after switching to GaussianBlurBackgroundProcessor video becomes unprocessed. Moreover, after some time of using any processor with inputFrameBufferType: 'offscreencanvas', Safari shows the "This web page was reloaded because it was using significant memory" error and reloads the page.

Also, sometimes "VirtualBackgroundProcessor" shows only the background, not the person (I have no specific steps to reproduce, it seems to happen randomly after changing the video quality or processor, etc).

• I have verified that the issue occurs with the latest twilio-video-processors.js release and is not marked as a known issue in the CHANGELOG.md.
• I reviewed the Common Issues and open GitHub issues and verified that this report represents a potentially new issue.
• I am not sharing any Personally Identifiable Information (PII)
  or sensitive account information (API keys, credentials, etc.) when reporting this issue.

Code to reproduce the issue:

...
let blurProcessor = new GaussianBlurBackgroundProcessor({
  assetsPath: '/tflite',
  blurFilterRadius: 10,
  maskBlurRadius: 1,
  pipeline: Pipeline.Canvas2D,
  debounce: isSafari() // true
});
await blurProcessor.loadModel();
...
videoTrack.addProcessor(blurProcessor, {
    inputFrameBufferType: typeof OffscreenCanvas === 'undefined' ? 'canvas' : 'offscreencanvas',
    outputFrameBufferContextType: '2d',
};

Expected behavior:

Processed (blurred) video is shown. No error is shown. The page is not reloaded. A person is shown altogether with a virtual background.

Actual behavior:

The video is not blurred. The page is reloaded showing the "This web page was reloaded because it was using significant memory" error. A person is not shown.

Software versions:

• Browser(s): Safari 16.4 (18615.1.26.11.23)
• Operating System: macOS 13.3.1. (MacBook Pro, 2020)
• twilio-video-processors.js: 2.0.0
• Third-party libraries (e.g., Angular, React, etc.): Ember.js

[Talb2005]

Any update on this?

[luisrivas]

Hi @pzubar!

Thanks for reporting this issue.

We added some performance improvements to the version 3. To resolve this issue, we recommend updating to the newest video processor version. If you continue to experience problems after upgrading, please let us know, and we will be happy to assist you further.

https://github.com/twilio/twilio-video-processors.js/releases

Bests,
Luis