Accessibility backbone v1

[miguelg719]

Why

The simplicity of accessibility trees provides a cleaner and more understandable context for the LLM. The length of context is reduced by almost half, which lowers the cost of inference. Intuitive for both large and small models. LLM can do zero-shot gen accurately with limited prompting.

What changed

Includes a new extract method called accessibilityExtract. Same format as text extract, it overloads the extract handler with one more parameter enabled as follows: useAccessibilityTree: true.

Ran text extract evals up to 93%. Current implementation does not respect the hierarchy of the accessibility tree but rather treats it horizontally, which causes issues with some table representations. Fix will be pushed within the next week (might require chunking)

test plan

• Manual testing on evals
• Run full eval on text extract
• Run full extract eval
• Implement and test same workflow with observe (v2?)
• Implement and test same workflow with act (v3?)

[changeset-bot]

⚠️ No Changeset found

Latest commit: c84437a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[miguelg719]

got to review CI script for a11y trees. Should it be ran on both regular dom extract evals and text_extract evals?
@kamath