This repository was archived by the owner on Feb 24, 2025. It is now read-only.
Switch to using maestro action #3783
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
14 tasks
alessandroboron
approved these changes
Jan 19, 2025
|
@alessandroboron yup, sure. You can run maestro locally. See Writing and running UI tests with maestro on iOS |
loremattei
added a commit
that referenced
this pull request
Jan 20, 2025
<!-- Note: This checklist is a reminder of our shared engineering expectations. Feel free to change it, although assigning a GitHub reviewer and the items in bold are required.⚠️ If you're an external contributor, please file an issue first before working on a PR, as we can't guarantee that we will accept your changes if they haven't been discussed ahead of time. Thanks! --> Task/Issue URL: https://app.asana.com/0/1204165176092271/1209158713404042/f Tech Design URL: CC: **Description**: Sync e2e tests were failing randomly, especially during nightly runs, with an error that the server was not reachable. After some investigation, I suspect the error is misleading and the actual issue is that the recovery code was invalid when used because the tests got queued or run slower than usual and the GHA timed out, causing the clean up job to run and invalidate the recovery code. This PR: - Changes the schedule of the test, to avoid it overlaps with the other e2e tests and gets queued. - Increases the timeout (needed anyway by the switch to Robin which is much slower than Maestro Cloud) - Switches to Maestro Action for better reliability of the connection between GHA and Robin (see #3783 for more context about this) - <!-- If at any point it isn't actively being worked on/ready for review/otherwise moving forward strongly consider closing it (or not opening it in the first place). If you decide not to close it, use Draft PR while work is still in progress or use `DO NOT MERGE` label to clarify the PRs state and comment with more information. --> **Steps to test this PR**: 1. Run the tests from this branch and validate it's green (you can also check a recent successful run here: https://github.com/duckduckgo/iOS/actions/runs/12828988947 ) <!-- Before submitting a PR, please ensure you have tested the combinations you expect the reviewer to test, then delete configurations you *know* do not need explicit testing. Using a simulator where a physical device is unavailable is acceptable. --> **Definition of Done (Internal Only)**: * [X] Does this PR satisfy our [Definition of Done](https://app.asana.com/0/1202500774821704/1207634633537039/f)? **Copy Testing**: * [ ] Use of correct apostrophes in new copy, ie `’` rather than `'` **Orientation Testing**: * [ ] Portrait * [ ] Landscape **Device Testing**: * [ ] iPhone SE (1st Gen) * [ ] iPhone 8 * [ ] iPhone X * [ ] iPhone 14 Pro * [ ] iPad **OS Testing**: * [ ] iOS 15 * [ ] iOS 16 * [ ] iOS 17 **Theme Testing**: * [ ] Light theme * [ ] Dark theme --- ###### Internal references: [Software Engineering Expectations](https://app.asana.com/0/59792373528535/199064865822552) [Technical Design Template](https://app.asana.com/0/59792373528535/184709971311943)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Task/Issue URL: https://app.asana.com/0/1204165176092271/1209158713404043/f
Tech Design URL:
CC:
Description:
Maestro CLI fail quite often because it drops its connection to Robin CI. The result is that we get failures in GHA for tests that actually complete successfully in Robin.
This PR updates our GHA workflow to use Maestro Cloud Action that doesn't seem to have this issue.
The problem with the official Maestro Action is that it doesn't consider timeouts or other infrastructure failures as failures and marks tests that fall into that bucket as successful. This is known problem that they haven't solved yet. To mitigate this, I've forked the action and introduced a fix for the most prominent issue related to timeouts.
I plan to open a PR with these fixes against the official repository asap. In the meantime, this PR uses my fork.
Steps to test this PR:
Definition of Done (Internal Only):
Copy Testing:
’rather than'Orientation Testing:
Device Testing:
OS Testing:
Theme Testing:
Internal references:
Software Engineering Expectations
Technical Design Template