Camera Screen goes black when using Skia Frame Processors with VisionCamera (doesn't happen with normal frame processors) #2897

DragonDare · 2025-01-14T19:53:18Z

Description

I'm trying to implement pose detection by drawing stick-figure of the pose on my app. But with Skia Frame processor, the camera screen goes black with only the stick figure being visible. However, if i use a normal frame processor, the camera is visible and not black.

Packages:

"react-native-fast-tflite": "^1.5.1",
"@shopify/react-native-skia": "1.5.0",
"react-native-worklets-core": "^1.5.0",
"vision-camera-resize-plugin": "^3.2.0"
"react-native-vision-camera": "^4.6.3"

React Native Skia Version

1.5.0

React Native Version

0.76.6

Using New Architecture

Enabled

Steps to Reproduce

Reproduce Mrousavy's SkiaCameraPoseDetection example
Remove the black paint
Still, camera isn't visible, but Pose Skeleton is.
If we use normal frame processor instead of skia, camera is visible.

Snack, Code Example, Screenshot, or Link to Repository

Here is my camera screen code which is creating the problem.

import TopAppBar from '@/components/TopAppBar';
import { SlamdunkTheme } from '@/constants/Theme';
import { scaleHeight, scaleWidth } from '@/utils/scaling';
import React, { useEffect, useMemo } from 'react';
import { StyleSheet, View, Text, Platform } from 'react-native';
import { SafeAreaView } from 'react-native-safe-area-context';
import { Camera, useCameraDevice, useCameraPermission, useSkiaFrameProcessor } from 'react-native-vision-camera';
import { useResizePlugin } from 'vision-camera-resize-plugin';
import { TensorflowModel, useTensorflowModel } from 'react-native-fast-tflite';
import { PaintStyle, Skia } from '@shopify/react-native-skia';
import getBestFormat from './formatFilter';

function tensorToString(tensor: TensorflowModel['inputs'][number]): string {
return ${tensor.dataType} [${tensor.shape}];
}

const LINE_WIDTH = 1;
const MIN_CONFIDENCE = 0.2;

const VIEW_WIDTH = scaleWidth(375);

const CameraScreen = () => {
const device = useCameraDevice('back');

const { hasPermission, requestPermission } = useCameraPermission();

useEffect(() => {
if (!hasPermission) {
requestPermission().catch((err) => {
console.error('Camera permission request failed:', err);
});
}
}, [hasPermission, requestPermission]);

if (!device) {
return Loading camera...;
}

const { resize } = useResizePlugin();

const delegate = Platform.OS === 'ios' ? 'core-ml' : undefined;
const plugin = useTensorflowModel(
require('../assets/models/singlepose-lightning-tflite-float16.tflite'),
delegate,
);
const format = useMemo(
() => (device != null ? getBestFormat(device, 720, 1000) : undefined),
[device],
);
console.log(format?.videoWidth, format?.videoHeight);

const pixelFormat = Platform.OS === 'ios' ? 'rgb' : 'yuv';

useEffect(() => {
const model = plugin.model;
if (model == null) {
return;
}
console.log(
Model: ${model.inputs.map(tensorToString)} -> ${model.outputs.map( tensorToString, )},
);
}, [plugin]);

// to get from px -> dp since we draw in the camera coordinate system
const SCALE = (format?.videoWidth ?? VIEW_WIDTH) / VIEW_WIDTH;

const paint = Skia.Paint();
paint.setStyle(PaintStyle.Fill);
paint.setStrokeWidth(LINE_WIDTH * SCALE);
paint.setColor(Skia.Color('yellow'));

const lines = [
// left shoulder -> elbow
5, 7,
// right shoulder -> elbow
6, 8,
// left elbow -> wrist
7, 9,
// right elbow -> wrist
8, 10,
// left hip -> knee
11, 13,
// right hip -> knee
12, 14,
// left knee -> ankle
13, 15,
// right knee -> ankle
14, 16,

// left hip -> right hip
11, 12,
// left shoulder -> right shoulder
5, 6,
// left shoulder -> left hip
5, 11,
// right shoulder -> right hip
6, 12,

];

const rotation = Platform.OS === 'ios' ? '0deg' : '90deg'; // hack to get android oriented properly

const frameProcessor = useSkiaFrameProcessor(
frame => {
'worklet';

  if (plugin.model != null) {
    const smaller = resize(frame, {
      scale: {
        width: inputWidth,
        height: inputHeight,
      },
      pixelFormat: 'rgb',
      dataType: 'uint8',
      rotation: rotation,
    });
    const outputs = plugin.model.runSync([smaller]);

    const output = outputs[0];
    const frameWidth = frame.width;
    const frameHeight = frame.height;

    for (let i = 0; i < lines.length; i += 2) {
      const from = lines[i];
      const to = lines[i + 1];

      const confidence = output[from * 3 + 2];
      if (confidence > MIN_CONFIDENCE) {
        frame.drawLine(
          Number(output[from * 3 + 1]) * Number(frameWidth),
          Number(output[from * 3]) * Number(frameHeight),
          Number(output[to * 3 + 1]) * Number(frameWidth),
          Number(output[to * 3]) * Number(frameHeight),
          paint,
        );
      }
    }
  }
},
[plugin, paint],

);

return (

Make sure your body is positioned correctly

{hasPermission && (

)}
{/*

*/}

);
};

export default CameraScreen;

const styles = StyleSheet.create({
overlay: {
...StyleSheet.absoluteFillObject,
top: scaleHeight(161),
alignItems: 'center',
},
textBox: {
width: scaleWidth(290),
alignSelf: 'center',
},
camera: {
height: scaleHeight(638),
},
instructionText: {
fontFamily: SlamdunkTheme.fonts.poppinsRegular,
fontSize: 22,
fontWeight: 400,
lineHeight: 33,
color: SlamdunkTheme.colors.backgroundColor,
textAlign: 'center',
},
dottedBox: {
width: scaleWidth(319),
height: scaleHeight(514),
borderWidth: 2,
borderColor: SlamdunkTheme.colors.backgroundVariant3,
borderStyle: 'dashed',
backgroundColor: SlamdunkTheme.colors.backgroundVariant3,
borderRadius: 8,
},
});

The text was updated successfully, but these errors were encountered:

maxximee · 2025-01-15T01:58:12Z

Same issue here.
Can be reproduced with a very simple usecase:

 const defaultPaint = Skia.Paint()
 const frameProcessor = useSkiaFrameProcessor(frame => {
        'worklet'
        frame.render(defaultPaint)
    }, [defaultPaint])

-> in component:
frameProcessor={frameProcessor}

This will show black screen on some devices.

I think it's linked to devices who are using Vulkan and Angle for OpenGL support, including a lot of samsung devices as wcandillon stated.

@DragonDare are you on a samsung device/emulator?

Also I think this issue should be addressed in react-native-vision-camera

wcandillon · 2025-01-15T05:52:59Z

yes I would definitely recommend to try on 1.7.7 and above.

DragonDare · 2025-01-15T07:48:41Z

Same issue here. Can be reproduced with a very simple usecase:
 const defaultPaint = Skia.Paint()
 const frameProcessor = useSkiaFrameProcessor(frame => {
        'worklet'
        frame.render(defaultPaint)
    }, [defaultPaint])
-> in component: frameProcessor={frameProcessor}

This will show black screen on some devices.

I think it's linked to devices who are using Vulkan and Angle for OpenGL support, including a lot of samsung devices as wcandillon stated.

@DragonDare are you on a samsung device/emulator?

Also I think this issue should be addressed in react-native-vision-camera

Im using a Motorola Edge 40 physical device.

Also I did mention this issue in vision camera github, no comments so far. Could you bump up the issue there? Will get some recognition.

DragonDare · 2025-01-15T08:28:46Z

yes I would definitely recommend to try on 1.7.7 and above.

Tried with skia version 1.7.7 right now, same problem.

DragonDare · 2025-01-18T18:13:22Z

@wcandillon any updates on this issue? If possible, could we atleast debug if this is a vision camera issue or a skia issue? Thanks

wcandillon · 2025-01-21T16:42:05Z

@DragonDare Yes I think it would be interesting to have proper support for VisionCamera, would you be ok to add a small example of Skia/Vision Camera to the example app: https://github.com/Shopify/react-native-skia/tree/main/apps/paper
That way we can monitor if the integration breaks or not.

@mrousavy Is that something you monitor on your side as well? Maybe we need to do some touch up on the integration part.

DragonDare · 2025-01-21T18:32:24Z

@wcandillon I did try the example repo https://github.com/mrousavy/VisionCameraSkiaDemo, which uses Skia Frame processors and it did give me a black screen (Yes i removed the black paint in the code)

mrousavy · 2025-01-22T09:57:25Z

@mrousavy Is that something you monitor on your side as well? Maybe we need to do some touch up on the integration part.

Yea I added Skia as a dependency - but I don't update it everytime a new version gets released. It definitely helps to also add VisionCamera to the Skia example app to test this integration better!

DragonDare · 2025-01-23T10:31:12Z

Guys @mrousavy @wcandillon @maxximee

I just realised I didn't add the "frame.render()" in my skia frame processor. This fixed the issue. The Camera is visible now.
  const frameProcessor = useSkiaFrameProcessor((frame) => {
    'worklet';

    frame.render();
I don't know why I'm just coming across this. https://react-native-vision-camera.com/docs/guides/skia-frame-processors.
Completely missed this when doing my initial research.

Originally posted by @DragonDare in #3365

mrousavy · 2025-01-23T10:32:29Z

lol ok no worries happens to the best of us 😂

maxximee · 2025-01-23T11:22:35Z

Guys @mrousavy @wcandillon @maxximee
I just realised I didn't add the "frame.render()" in my skia frame processor. This fixed the issue. The Camera is visible now.
  const frameProcessor = useSkiaFrameProcessor((frame) => {
    'worklet';

    frame.render();
I don't know why I'm just coming across this. https://react-native-vision-camera.com/docs/guides/skia-frame-processors.
Completely missed this when doing my initial research.
Originally posted by @DragonDare in #3365

Did you try on a samsung s8 / s22+ / s24 ?

Got a list of 400+ crash events on a few particular samsung devices with simply:

const defaultPaint = Skia.Paint()

const frameProcessor = useSkiaFrameProcessor(frame => {
        'worklet'
            frame.render(defaultPaint)
    }, [defaultPaint])

wcandillon · 2025-01-23T12:11:45Z

I expect it to work on these models (s8 / s22+ / s24 ) with the latest RN Skia version, if not we can definitely take a look.

…

On Thu, Jan 23, 2025 at 12:22 PM maxximee ***@***.***> wrote: Guys @mrousavy @wcandillon @maxximee I just realised I didn't add the "frame.render()" in my skia frame processor. This fixed the issue. The Camera is visible now. const frameProcessor = useSkiaFrameProcessor((frame) => { 'worklet'; frame.render(); I don't know why I'm just coming across this. https://react-native-vision-camera.com/docs/guides/skia-frame-processors. Completely missed this when doing my initial research. Originally posted by @DragonDare in #3365 Did you try on a samsung s8 / s22+ / s24 ? — Reply to this email directly, view it on GitHub or unsubscribe. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS or Android.

maxximee · 2025-01-24T04:30:08Z

I expect it to work on these models (s8 / s22+ / s24 ) with the latest
RN Skia version, if not we can definitely take a look.
…

Got hands on a samsung s8 and it's clearly still crashing with "@shopify/react-native-skia": "^1.10.2":

2025-01-24 11:26:31.530 19443-19793 ReactNativeJS com.klim.klimfilmscanner E Frame Processor Error: Exception in HostFunction: Failed to convert NativeBuffer to SkImage!, js engine: VisionCamera
2025-01-24 11:26:31.547 19443-19792 skia com.klim.klimfilmscanner D Could not create EGL image, err = (0x3003)
2025-01-24 11:26:31.547 19443-19792 RNSkia com.klim.klimfilmscanner I Failed to convert HardwareBuffer to OpenGL Texture!

And on other specific devices, for example pixel 9 with api 35 it shows a black screen:

    const frameProcessor = useSkiaFrameProcessor(frame => {
        'worklet'
            frame.render()
    }, [])

If anyone can give it a try and give me a feedback that would be great. I'd happy to be told I made a mistake somewhere...

wcandillon · 2025-01-24T08:40:11Z

that's interesting, the NativeBuffer API is tested on samsung s8 but I will give it another look just to be sure. Then it might be related to the way it operate with vision-camera maybe, let's see. this is where having an example of using vision-camera on the skia example app would be useful for testing.

…

On Fri, Jan 24, 2025 at 5:30 AM maxximee ***@***.***> wrote: I expect it to work on these models (s8 / s22+ / s24 ) with the latest RN Skia version, if not we can definitely take a look. … Got hands on a samsung s8 and it's clearly still crashing with ***@***.***/react-native-skia": "^1.10.2": Screenshot_20250124-112705_Film_Scanner.1.jpg (view on web) 2025-01-24 11:26:31.530 19443-19793 ReactNativeJS com.klim.klimfilmscanner E Frame Processor Error: Exception in HostFunction: Failed to convert NativeBuffer to SkImage!, js engine: VisionCamera 2025-01-24 11:26:31.547 19443-19792 skia com.klim.klimfilmscanner D Could not create EGL image, err = (0x3003) 2025-01-24 11:26:31.547 19443-19792 RNSkia com.klim.klimfilmscanner I Failed to convert HardwareBuffer to OpenGL Texture! — Reply to this email directly, view it on GitHub or unsubscribe. You are receiving this email because you commented on the thread. Triage notifications on the go with GitHub Mobile for iOS or Android.

DragonDare added the bug Something isn't working label Jan 14, 2025

DragonDare closed this as completed Jan 23, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Camera Screen goes black when using Skia Frame Processors with VisionCamera (doesn't happen with normal frame processors) #2897

Camera Screen goes black when using Skia Frame Processors with VisionCamera (doesn't happen with normal frame processors) #2897

DragonDare commented Jan 14, 2025

maxximee commented Jan 15, 2025 •

edited

Loading

wcandillon commented Jan 15, 2025

DragonDare commented Jan 15, 2025

DragonDare commented Jan 15, 2025

DragonDare commented Jan 18, 2025

wcandillon commented Jan 21, 2025

DragonDare commented Jan 21, 2025 •

edited

Loading

mrousavy commented Jan 22, 2025

DragonDare commented Jan 23, 2025

mrousavy commented Jan 23, 2025

maxximee commented Jan 23, 2025 •

edited

Loading

wcandillon commented Jan 23, 2025 via email

maxximee commented Jan 24, 2025 •

edited

Loading

wcandillon commented Jan 24, 2025 via email

Camera Screen goes black when using Skia Frame Processors with VisionCamera (doesn't happen with normal frame processors) #2897

Camera Screen goes black when using Skia Frame Processors with VisionCamera (doesn't happen with normal frame processors) #2897

Comments

DragonDare commented Jan 14, 2025

Description

React Native Skia Version

React Native Version

Using New Architecture

Steps to Reproduce

Snack, Code Example, Screenshot, or Link to Repository

maxximee commented Jan 15, 2025 • edited Loading

wcandillon commented Jan 15, 2025

DragonDare commented Jan 15, 2025

DragonDare commented Jan 15, 2025

DragonDare commented Jan 18, 2025

wcandillon commented Jan 21, 2025

DragonDare commented Jan 21, 2025 • edited Loading

mrousavy commented Jan 22, 2025

DragonDare commented Jan 23, 2025

mrousavy commented Jan 23, 2025

maxximee commented Jan 23, 2025 • edited Loading

wcandillon commented Jan 23, 2025 via email

maxximee commented Jan 24, 2025 • edited Loading

wcandillon commented Jan 24, 2025 via email

maxximee commented Jan 15, 2025 •

edited

Loading

DragonDare commented Jan 21, 2025 •

edited

Loading

maxximee commented Jan 23, 2025 •

edited

Loading

maxximee commented Jan 24, 2025 •

edited

Loading