How can I access the video at the exact timestamps?

[VasilP202]

I'm using react-native-skia to preview the video and draw the skeleton pose data that was detected during the recording previously. Recording is saved and then played using the Video component. During the playback, the skeleton data detected earlier are drawn based on timestamps at which each pose was detected. How can I exactly match each timestamp and draw a pose at each timestamp during video playback?

I've tried using currentTime, but this value doesn't update often enough, causing the drawn skeleton and video to be desynchronized. I also tried useFrameCallback from react-native-reanimated which I had better results with, but the skeleton is still out of sync with the video.

const { currentFrame, currentTime, duration, framerate } = useVideo(
  params.path,
  {
    paused,
    looping: false,
    playbackSpeed: 1,
  }
);
const timestamps = params.motionSequence.data.map(
  (frame) => frame.frame_timestamp,
);
const skeletonData = params.motionSequence.data.map(
  (frame) => frame.objectsDetected,
);
const { joints, bones } = useSkeleton();

 // ....
  
<Canvas style={styles.canvas}>
  <Image
    image={currentFrame}
    x={0}
    y={0}
    width={SCREEN_WIDTH}
    height={SCREEN_HEIGHT}
    fit="cover"
/>
</Canvas>
<Canvas style={styles.canvas}>
  {joints.map((joint, index) => {
    return (
      <Circle
        key={index}
        cx={joint[0]}
        cy={joint[1]}
        r={JOINT_RADIUS}
        color="green"
      />
    );
  })}
  {bones.map((bone, index) => {
    return (
      <Line
        key={index}
        p1={bone[0]}
        p2={bone[1]}
        color="green"
        style="stroke"
        strokeWidth={LINE_STROKE_WIDTH}
      />
    );
  })}
</Canvas>

[wcandillon]

are you using 1.3.3? we've made substantial improvements there.

…

On Wed, Jun 12, 2024 at 10:35 AM Vasil Poposki ***@***.***> wrote: I'm using react-native-skia to preview the video and draw the skeleton pose data that was detected during the recording previously. Recording is saved and then played using the Video component. During the playback, the skeleton data detected earlier are drawn based on timestamps at which each pose was detected. How can I exactly match each timestamp and draw a pose at each timestamp during video playback? I've tried using currentTime, but this value doesn't update often enough, causing the drawn skeleton and video to be desynchronized. I also tried useFrameCallback from react-native-reanimated which I had better results with, but the skeleton is still out of sync with the video. const { currentFrame, currentTime, duration, framerate } = useVideo( params.path, { paused, looping: false, playbackSpeed: 1, });const timestamps = params.motionSequence.data.map( (frame) => frame.frame_timestamp,);const skeletonData = params.motionSequence.data.map( (frame) => frame.objectsDetected,);const { joints, bones } = useSkeleton(); // .... <Canvas style={styles.canvas}> <Image image={currentFrame} x={0} y={0} width={SCREEN_WIDTH} height={SCREEN_HEIGHT} fit="cover"/></Canvas><Canvas style={styles.canvas}> {joints.map((joint, index) => { return ( <Circle key={index} cx={joint[0]} cy={joint[1]} r={JOINT_RADIUS} color="green" /> ); })} {bones.map((bone, index) => { return ( <Line key={index} p1={bone[0]} p2={bone[1]} color="green" style="stroke" strokeWidth={LINE_STROKE_WIDTH} /> ); })}</Canvas> — Reply to this email directly, view it on GitHub <#2474>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKXVV5W5BFGY4KQZ6ZUTTZHAB45AVCNFSM6AAAAABJF3XACGVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZWHAYTCMZRGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[VasilP202]

Hello, I came back to this problem after some time.

I tried 1.3.3 and the latest (1.3.6) version. The desynchronization between the video and the skeleton seems to be even greater. Is there some bug in video processing? I noticed few issues appeared including 2493.