Roborazzi AI-Powered Image Verification

[takahirom]

Background

Advancements in AI: AI technologies are becoming increasingly sophisticated and affordable.
Manual Screenshot Testing: Traditional methods require manual verification, limiting scalability.

Problem

Scalability Limitations: Manual checks hinder large-scale screenshot testing.
High Maintenance Costs: Human intervention increases effort and costs.

Proposal

Integrate AI for Image Comparison: Use AI to automate screenshot verification in tests.
Enhance Roborazzi: Add AiOptions to enable AI-based image evaluation.

Current code

@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
@Config(
  sdk = [30],
  qualifiers = RobolectricDeviceQualifiers.NexusOne
)
class AiTest {
  @get:Rule
  val composeTestRule = createAndroidComposeRule<MainActivity>()

  @get:Rule
  val roborazziRule = RoborazziRule(
    options = RoborazziRule.Options(
      roborazziOptions = RoborazziOptions(
        compareOptions = RoborazziOptions.CompareOptions(
          aiOptions = AiOptions(
            aiModel = AiOptions.AiModel.Gemini(
              apiKey = System.getenv("gemini_api_key")!!
            ),
          )
        )
      )
    )
  )

  @Test
  fun captureWithAi() {
    ROBORAZZI_DEBUG = true
    onView(ViewMatchers.isRoot())
      .captureRoboImage(
        roborazziOptions = provideRoborazziContext().options.addedCompareAiAssertions(
          AiOptions.AiAssertion(
            assertPrompt = "it should have PREVIOUS button",
            requiredFulfillmentPercent = 90,
          ),
          AiOptions.AiAssertion(
            assertPrompt = "it should show First Fragment",
            requiredFulfillmentPercent = 90,
          )
        )
      )
  }
}

Changes for Existing Projects

We won't introduce any new dependencies for existing projects. I think I'll add the roborazzi-ai module to manage dependencies like Gemini or others, so you don't have to worry about disrupting existing dependencies.

Discussion Points

When We Execute This

The current AI implementation runs only when:

• AiOptions are set
  • Because we can't run without prompts and API keys
• Comparing or verifying tasks
• The image is changed with ImageComparator
  • Because I think we can save a lot of money if we check only when images are changed.

And the test fails only when:

• All above conditions are met
• requiredFulfillmentPercent is set
• It doesn't fulfill
• (These tests fail even if it's a compare task.)

How We Set Prompts

Currently, it is a little difficult to set a prompt because we need to change RoborazziOptions.CompareOptions.AiOptions.prompt.

Things to Do

• Get feedback from someone
• Decide on the name of this feature
• Decide if it is okay to have prompt in RoborazziOptions.CompareOptions. If we create a new function for this, like captureRoboImageWithAI, it will require bigger changes. Alternatively, provide a way to change these options easily for RoborazziOptions.

[github-actions]

Snapshot diff report

File name	Image
com.github.takahirom .roborazzi.sample.bo xed.AiManualTest.whe nAiReturnSuccess_com pare.png
com.github.takahirom .roborazzi.sample.bo xed.AiManualTest.whe nAiReturnError_compa re.png
com.github.takahirom .roborazzi.sample.Ma nualTest.captureWith Ai_compare.png

[takahirom]

TODO:

• Add a test using AiModel.Manual
• remove loadRoboAi()
• Separate roborazzi-ai into roborazzi-ai and roborazzi-ai-gemini