Improve orientation management

[luicfrr]

I'm not sure if orientation is well handled the way it is at this momment and don't know if it's gonna work on all devices.

As suggested on this comment, maybe the best way to determine device/camera orientation is using the bellow code:

private val ORIENTATIONS = SparseIntArray()

init {
    ORIENTATIONS.append(Surface.ROTATION_0, 0)
    ORIENTATIONS.append(Surface.ROTATION_90, 90)
    ORIENTATIONS.append(Surface.ROTATION_180, 180)
    ORIENTATIONS.append(Surface.ROTATION_270, 270)
}

/**
 * Get the angle by which an image must be rotated given the device's current
 * orientation.
 */
@RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
@Throws(CameraAccessException::class)
private fun getRotationCompensation(cameraId: String, activity: Activity, isFrontFacing: Boolean): Int {
    // Get the device's current rotation relative to its "native" orientation.
    // Then, from the ORIENTATIONS table, look up the angle the image must be
    // rotated to compensate for the device's rotation.
    val deviceRotation = activity.windowManager.defaultDisplay.rotation
    var rotationCompensation = ORIENTATIONS.get(deviceRotation)

    // Get the device's sensor orientation.
    val cameraManager = activity.getSystemService(CAMERA_SERVICE) as CameraManager
    val sensorOrientation = cameraManager
            .getCameraCharacteristics(cameraId)
            .get(CameraCharacteristics.SENSOR_ORIENTATION)!!

    if (isFrontFacing) {
        rotationCompensation = (sensorOrientation + rotationCompensation) % 360
    } else { // back-facing
        rotationCompensation = (sensorOrientation - rotationCompensation + 360) % 360
    }
    return rotationCompensation
}

Some adptations are needed for it to work with vision-camera but it's a good starting point.

[frodriguez-hu]

cameraId is coming on device.id from the JS thread (we have to send it and receive it on the frame processor plugin).
We could get the Activity from the VisionCameraFaceDetectorPluginPackage that implements ReactPackage

[luicfrr]

I created a branch orientation and pushed a commit passing cameraId to plugin.
When I have some time I'll try to implement your suggestion above.

[frodriguez-hu]

I did this to get the activity:

  override fun createNativeModules(reactContext: ReactApplicationContext): List<NativeModule> {
  FrameProcessorPluginRegistry.addFrameProcessorPlugin("detectFaces") {
      proxy, options -> VisionCameraFaceDetectorPlugin(proxy, options, reactContext.currentActivity)
  }
    return emptyList()
  }

  override fun createViewManagers(reactContext: ReactApplicationContext): List<ViewManager<*, *>> {
    return emptyList()
  }

And deleting the companion object init.

Not nice but I wasn't able to find another way

Then this:

class VisionCameraFaceDetectorPlugin(
  proxy: VisionCameraProxy,
  options: Map<String, Any>?,
  activity: Activity?
)

And this:

  private var mActivity: Activity? = null
  private var cameraId: String? = null
  init {
      mActivity = activity
  }

The rest of the implementation is quite straight forward

[luicfrr]

You can access reactContext (also the activity) from proxy inside the plugin

[frodriguez-hu]

Thanks you!!!!! Much better by doing that

[frodriguez-hu]

@luicfrr I was able to implement this, and making some logs, i am receiving always 270 degrees from this implementation and PORTRAIT from the frame coming from VC (which is 0 degrees). It is not changing when I rotate the phone, but i guess that is expected cause the app I working on is locked on portrait

[frodriguez-hu]

PORTRAIT should come as 270 as right value (-90 degrees).

if you do this map to send to the process from faceDetector

  private fun getOrientation(
    orientation: Orientation
  ): Int {
    return when (orientation) {
      // device is landscape left
      Orientation.PORTRAIT -> 0
      // device is portrait
      Orientation.LANDSCAPE_LEFT -> 270
      // device is landscape right
      Orientation.PORTRAIT_UPSIDE_DOWN -> 180
      // device is upside-down
      Orientation.LANDSCAPE_RIGHT -> 90
    }
  }

PORTRAIT will be 0 degrees when the right value should be 270, that is why the FaceDetector is not detecting anything

[luicfrr]

I'm using frame orientation to determine device orientation.

When phone is portrait: frame.orientation returns landscape_left (device's camera sensor orientation is -90deg), so the final image orientation should be 270.

[frodriguez-hu]

mm, the weird thing is that on my phone at least, VC is returning PORTRAIT instead of LANDSCAPE_LEFT, could be a bug from VC?

[luicfrr]

are you getting this portrait value from js side?

[frodriguez-hu]

I am getting it on my android logs. on the callback function from VisionCameraFaceDetectorPlugin on frame.orientation

I am logging both values at the same time and they are different (the one calculated by the function from MLKit docs, and the one coming from VisionCamera). The MLKit one is the right one

[luicfrr]

I have some notes in some tests I made:
1 - If app orientation is not locked in portrait mode the camera preview will be stretched at some time.
2 - If app orientation is locked so above code will not work because the screen is always in portrait mode so activity.windowManager.defaultDisplay.rotation will always return the same value and will not work on other device orientations.

[luicfrr]

I also pushed some new commits to orientation branch, please take a look into it

[frodriguez-hu]

Is the first case happening with the normal orientation config? I mean isn't the preview being handled by vision camera?

If the orientation is locked, don't we want it to return always the same value? I mean, the value should change if the camera has switched to landscape or other orientation.

Cause if it was created to work on portrait and you put your phone on landscape, the app will still remain in portrait and the orientation will be portrait and Face Detector process will fail, which makes sense on that case. The case we don't want is to show landscape when it is portrait or viceversa.

[luicfrr]

vision-camera have a device orientation mode. It will always follow your device orientation doesn't matter if device or app orientation is locked.

The first case/issue happens on example app if I remove orientation: "portrait" from app.config.js and unlock my phone orientation. I think I'll fill an issue on vision-camera's repo reporting this problem.

[frodriguez-hu]

I am going to update my react native version on my main project and then make tests with the last VC version and the last version of face detector, I wll make some prooves with this after that.

So on your tests you found more issues with above implementation rather than with the current implementation?

[frodriguez-hu]

Hey @luicfrr I think we could delete this since the last release of 4.5.0 of VC where orientations issues were handled

[luicfrr]

yes yes, I'll update vc's version

[luicfrr]

orientation is now correct and a new version will be relased soon