Quest Display Access Demo

Developers want camera access on the Meta Quest. Meta hasn't let us have it yet. The next best thing is display access. Thanks to Android's MediaProjector API, you can copy the display image to a texture in your Unity project in 'near realtime' (several frames of latency) as demonstrated in this horribly architected Unity project. No PC, embedded browser, or dev mode required

Demo from AprilTagDemo scene

⚠️ Issues (please read)!

To fix

⚠️ This demo project only creates a single MediaProjector session on app launch. If the app is interrupted (such as by the headset going to sleep) the session will end and you'll need to restart the app.

Gotchas

⚠️ You may need to be on Quest system software v68 or higher

⚠️ This only works on-headset. This will not work through QuestLink

⚠️ You cannot video record the display 'normally' while this app's MediaProjector session is running. You can instead use scrcpy to record any prototypes or demos you make with this.

⚠️ This still isn't proper camera access. Any virtual elements will obscure physical objects in the image. If you need to track something, you must not render anything on top of it!

Other info

• The captured view is ~82 degrees in horizontal and vertical FOV on Quest 3
• The capture texture is 1024x1024, at least on Quest 3
• The left eye buffer is where captured frames come from
• Quest system camera settings do not affect the capture resolution, framerate, or eye side

Reference

• https://developer.oculus.com/documentation/native/native-media-projection/
• https://developer.android.com/media/grow/media-projection
• https://github.com/android/media-samples/tree/main/ScreenCapture