Off centre face pose accuracy and consistency (Augmented Faces)

[dylanmenzies]

Face center and region pose error increases significantly with distance from the camera field centre. For regions this is immediately obvious, for example the nose in the augmented face demo is pushed out to the sides in landscape mode (see images attached). Objects based on the face mesh and center pose register well across the field however (see images). The region and center pose orientations remain fixed relative to each other, however the mesh is changing in local coordinates (See the remaining images - the mesh is rendered with a fixed pose so the rotation of the mesh can be clearly seen for the off centre images. Also shown is a view from above showing the distortion and rotation of the mesh). The mesh change in combination with the center pose produces the global registration, not the centre pose alone.

VERSIONS USED

• Android Studio: 2020.3.1
• ARCore SDK for Android: 1.43
• Device manufacturer, model, and O/S: Motorola G6 (same results for arcore on IPhone 8)
• Google Play Services for AR (ARCore): versionName=1.43.240930403
• Output of adb shell getprop ro.build.fingerprint: motorola/ali_n/ali_n:8.0.0/OPS27.82-87/108:user/release-keys

STEPS TO REPRODUCE THE ISSUE

The errors are always present, and do not require special conditions.

The images are produced from the augmented faces demo, modified to render the canonical_face_texture.png from the sdk, or the nose object. The fixed orientation pose is generated using float[] modelMatrix = { 1f,0,0,0, 0,1f,0,0, 0,0,1f,0, 0,0,-0.5f,1f };

WORKAROUNDS (IF ANY)

I've had good results correcting the pose by:- calculating a local pose based on face mesh vertices, forehead, nosetip, and lip, then apply current face pose to get more accurate face pose for region poses and tracking. A shift is also applied to the local pose to shift the fixed point of the mesh to 0 (See the last images with mesh and nose combined)

ADDITIONAL COMMENTS

The error is frustrating for accurate render of region objects, and also for accurate head tracking with translation, which has good uses besides graphic rendering.

It appears that since face registration is achieved very well, the gross change in the face mesh coordinates could be shifted to the center pose (and to correct the region poses), so that a more accurate physical head pose is given for non central positions.

[15kingben]

Hi Dylan, I don't think I am able to reproduce this issue, although the effect is fairly subtle. Are you able to reproduce this with a different device?

[dylanmenzies]

Hi Ben, I have these results on several iPhone and android devices. The pose angle error at the sides in landscape is significant, 10+ degrees, eg the nose object is visually not aligned with the face. The face mesh itself always appears well aligned in the 2D image - although in 3D the surface is distorting. For mesh overlay applications this is not an issue, but for added objects / tracking purposes this can be limiting. One of the images I posted isolates the face mesh with a fixed pose to view from above, making the distortion clear including the skewing of the nose. This is a clear way to test if the issue is common across devices. It appears the distortion is fundamental to how the mesh fitting is set up - I expect with more contraints (eg symmetry / null rotation) it could be reduced. I have also tested ARKit faces which uses a depth camera. In that case the mesh does not distort or rotate, as you would expect because it uses a 3D sensor. It can be used for accurate tracking.

…

On 16 Jul 2024, at 18:02, Ben King ***@***.***> wrote: Hi Dylan, I don't think I am able to reproduce this issue, although the effect is fairly subtle. Are you able to reproduce this with a different device? — Reply to this email directly, view it on GitHub <#1658 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACXXWWGWTUZZOQF6WK2IFR3ZMVGYVAVCNFSM6AAAAABJUOIVQOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRGQYDQMBYGQ>. You are receiving this because you authored the thread.