[Question] Questions regarding ControllerData and ControllerStateType structure

[yilin404]

Hi! I’m working with Vuer to process Quest 3 controller data, but I have a few questions that I hope you can help clarify.

/**
 * Significantly more accurate and stable than hand-tracking.
 */
export type ControllerData = {
  left?:       Matrix4Tuple;      // array with length==25
  right?:      Matrix4Tuple;     // array with length==25
  leftState?:  ControllerStateType;
  rightState?: ControllerStateType;
};

export type ControllerStateType = {
  trigger:    boolean;
  squeeze:    boolean;
  touchpad:   boolean;
  thumbstick: boolean;
  aButton:    boolean;
  bButton:    boolean;

  triggerValue:      number;
  squeezeValue:      number;
  touchpadValue:   [ number, number ];   // X and Y values for the touchpad
  thumbstickValue: [ number, number ]; // X and Y values for the thumbstick
  aButtonValue:      boolean;
  bButtonValue:      boolean;
};

1. Regarding left and right in ControllerData:
   The left and right fields are of type Matrix4Tuple, with a length of 25. This length seems to match the number of key points in hand tracking. Could you clarify if these left and right matrices represent the hand keypoint poses just same as hand tracking?

2. Regarding the state fields in ControllerStateType:
   What exactly do the fields trigger, squeeze, touchpad, thumbstick, aButton, and bButton represent? What are their specific functions?
   Is there any documentation or detailed reference available for these fields?

Thanks in advance for your help. I look forward to your response!

[geyang]

I made a mistake, the length should be 16 as opposed to 25. I have just fixed this in the current version of the doc.

The details of these buttons can be found here:

---

Button and Trackpad States

The Motion Controller API is covered by the XRInputSource.gamepad sectionin the web standard. You can find it through this MDN page.

attribute to get the button and trackpad states. The following code snippet shows how to extract the button and trackpad states from the gamepad:

The returned data looks like the following:

/**
 * Significantly more accurate and stable than controller-tracking.
 */
export type ControllerData = {
  left?:       Matrix4Tuple;      // array with length==16
  right?:      Matrix4Tuple;      // array with length==16
  leftState?:  ControllerStateType;
  rightState?: ControllerStateType;
};

export type ControllerStateType = {
  trigger:    boolean;
  squeeze:    boolean;
  touchpad:   boolean;
  thumbstick: boolean;
  aButton:    boolean;
  bButton:    boolean;

  triggerValue:      number;
  squeezeValue:      number;
  touchpadValue:   [ number, number ];   // X and Y values for the touchpad
  thumbstickValue: [ number, number ]; // X and Y values for the thumbstick
  aButtonValue:      boolean;
  bButtonValue:      boolean;
};

Button and Trackpad States

The webXR Motion Controller API uses the XRInputSource's gamepad.

For detailed API, refer to the link

attribute to get the button and trackpad states. The following code snippet shows how to extract the button and trackpad states from the gamepad:

I plan to add a more event driven API for this in the future to make it easier to 
register press events.

Buttons	xr-standard Mapping	Required
buttons[0]	Primary trigger	Yes
buttons[1]	Primary squeeze button	No
buttons[2]	Primary touchpad	No
buttons[3]	Primary thumbstick	No

Axes	xr-standard Mapping	Required
axes[0]	Primary touchpad X	No
axes[1]	Primary touchpad Y	No
axes[2]	Primary thumbstick X	No
axes[3]	Primary thumbstick Y	No

  const gamepad = inputSource.gamepad;
  const buttons = gamepad?.buttons || [];

  return {
    transform: Array.from(transform) as Matrix4Tuple,
    trigger: buttons[0]?.pressed  || false,
    squeeze: buttons[1]?.pressed  || false,
    touchpad: buttons[2]?.pressed || false,
    thumbstick: buttons[3]?.pressed || false,
    aButton: buttons[4]?.pressed || false,
    bButton: buttons[5]?.pressed || false,

    triggerValue: buttons[0]?.value || 0,
    squeezeValue: buttons[1]?.value || 0,
    touchpadValue: [gamepad?.axes[0] || 0, gamepad?.axes[1] || 0],
    thumbstickValue: [gamepad?.axes[2] || 0, gamepad?.axes[3] || 0],
    aButtonValue: buttons[4]?.pressed || false,
    bButtonValue: buttons[5]?.pressed || false,
  };

Matrix format

All 4x4 transform matrices used in WebGL are stored in 16-element Float32Arrays.
The values are stored in the array in column-major order; that is, each column is
written into the array top-down before moving to the next column to the right and
writing it into the array. Therefore, for the array [a0, a1, a2, …, a13, a14, a15],
the matrix looks like this:

                                  ⌈  a0 a4 a8 a12  ⌉
                                  |  a1 a5 a9 a13  |
                                  |  a2 a6 a10 a14 |
                                  ⌊  a3 a7 a11 a15 ⌋

For details, refer to the MDN documentation on XR Rigid Body Transformation

[yilin404]

Thank you for providing the documentation! I have successfully implemented a remote control task for the robotic arm using the API, and I truly appreciate your support.

During my usage, I encountered the following issues and would like to ask for your advice on possible solutions:

1. Controller Tracking Issues

   Tracking Performance: Compared to gesture tracking, the controller tracking appears less reliable. It often loses tracking, for example, when the controller is within the field of view or temporarily set down.
   Recovery: Once tracking is lost, it seems impossible to recover automatically, which affects task continuity. Do you have any recommendations to improve these tracking issues?

2. Compatibility Between Gesture and Controller Tracking
   I noticed that gesture tracking and controller tracking cannot run simultaneously. This limitation presents challenges in scenarios requiring mixed input methods. Is this a design restriction of the API? If so, is there any possibility to enable simultaneous tracking for both inputs?

Looking forward to your response, and thank you again!

[geyang]

I will test out 1. This might take longer to fix. If you can share a video it will make it easier for me to decide where it came from. @AdamRashid96 do you notice this?

regarding 2, I believe you are able to use the controller on one hand, and hand on another. you just cant have all four at the same time. Is this what you were looking for?

[hejia-zhang]

I have experienced the issue 1 as well. There were also many times, vuer stucks in loading or the controller handle function never got called.