How to properly use idb view hierarchy dump with Selector ?

[ziadsarour]

Hello,
First of all, thank you for all the great work !

My problem is that I can't pick an image from the photos library in iOS (not tried with Android).
It seems native Selector is lacking some docs.

So i dump the view hierarchy using idb (in a simulator) : IDB_UDID={udid} idb ui describe-all | jq .

Weird output : cancel button, images, etc. doesn't appear, only the root element is visible (I've also tried with --nested and --json options, but same output) :

[
  {
    "AXFrame": "{{0, 0}, {430, 932}}",
    "AXUniqueId": null,
    "frame": {
      "y": 0,
      "x": 0,
      "width": 430,
      "height": 932
    },
    "role_description": "application",
    "AXLabel": "myApp",
    "content_required": false,
    "type": "Application",
    "title": null,
    "help": null,
    "custom_actions": [],
    "AXValue": null,
    "enabled": true,
    "role": "AXApplication",
    "subrole": null
  }
]

It's okay, I can still find informations about images using describe-point, so I run IDB_UDID={udid} idb ui describe-point 0 150 | jq .. (0 150 are the coordinates of the first image).

Here is the output :

{
  "AXFrame": "{{0, 124.66666666666666}, {142, 141.99999999999997}}",
  "AXUniqueId": null,
  "frame": {
    "y": 124.66666666666666,
    "x": 0,
    "width": 142,
    "height": 141.99999999999997
  },
  "role_description": "image",
  "AXLabel": "Photo",
  "content_required": false,
  "type": "Image",
  "title": null,
  "help": null,
  "custom_actions": [],
  "AXValue": null,
  "enabled": true,
  "role": "AXImage",
  "subrole": null
}

It's great, I've some informations about the image element like "AXLabel", "role", "role_description", ...
The problem is, I don't how to use those informations. I guess "AXLabel" correspond to the "text" field but it's not working in my case :

await patrol.native.tap(
  Selector(text: 'Photo'),
  appId: 'com.apple.mobileslideshow',
);

await patrol.native.tap(
  Selector(text: 'Photo'),
  appId: 'com.apple.springboard',
);

(You will notice that I've tried with two different appId).

So here are my questions :

• How to use the fields provided by idb ? What is the mapping between "role", "role_description", ... from idb and "text", "contentDescription", "className", ... from Selector class ?
• How should I do if my Selector matches multiple elements but want to tap on 2nd or 3rd one ? The Selector class doesn't provide a similar at(n) method as the PatrolFinder class.

I've searched and found those 2 issues mentioning the photos library but they were not really helpful :

• How can I interact with the image picker and file picker? #354
• Add example file picker screen and test #514

Thank you !

[ziadsarour]

I'm still stuck with this :/

[ziadsarour]

@bartekpacia

[bartekpacia]

How to use the fields provided by idb ? What is the mapping between "role", "role_description", ... from idb and "text", "contentDescription", "className", ... from Selector class ?

We should have this written down somewhere - but we don't, so very sorry about this. IIRC, Selector.text should work for text and maybe contentDescription.

Note for myself: this shows how we can implement searching by more view properties on iOS.

How should I do if my Selector matches multiple elements but want to tap on 2nd or 3rd one ? The Selector class doesn't provide a similar at(n) method as the PatrolFinder class.

You can use Selector.instance. Please bear in mind though that it doesn't have the same behavior across platforms (but of course it should, and we'll fix that). See also #663 to learn more about this problem.

[ziadsarour]

Thank you very much and thank you for this awesome testing framework 👍

[abhiche]

Hi @ziadsarour ,
I am stuck with the same challenge.
How did you solve it?

[ziadsarour]

Hi @ziadsarour ,

I am stuck with the same challenge.

How did you solve it?

Hi @abhiche

I have not resolved this challenge because I am not using Patrol anymore. I've decided to test everything but the native parts for now.

It seems Patrol v3 makes it easier to test native parts using the develop command.