How can I evaluate our VLA model in BEHAVIOR-1K datasets?

[RZFan525]

How can I evaluate our VLA model in BEHAVIOR-1K datasets?
Is there any interface to test it?

[ChengshuLi]

Can you provide more details about your VLA model?

• V: You can get the robot's onboard sensing from the observation (env.get_obs()), such as RGB, depth, segmentation, etc.
• L: You can find the task definition of 1000 BEHAVIOR-1K activities, written in BDDL, a predicate logic-based language inspired by PDDL. You can feed this to your model as a type of formal language.
• A: You can feed the low-level action to the environment with a typical gym interface (env.step(action)). For instance, with an IKController for the arm, the action might mean "moving the end-effector forward by 5cm". You can also check out the high-level action primitives that we provide, where you can specify a primitive action type and a target object.

Hope this helps!

[RZFan525]

Thank you for your detailed reply. I want to run OpenVLA, RT-2 and RT-1 models. I'm going to try it based on your instructions.