Add smoke-test workflow #103

uuf6429 · 2025-01-08T23:29:17Z

No description provided.

uuf6429 · 2025-01-09T07:36:21Z

In conclusion:

BrowserKit (http client)
- fails the checkbox test (throws InvalidArgumentException instead of DriverException)
BrowserKit (http kernel)
- same as BrowserKit (http client)
Chrome
- tonne of failures without a seemingly good reason
- (it's worth noting that their custom tests use data uris instead of a webserver)
Selenium2
- right-click test failed, no idea why
Classic
- all good on firefox, crashes on chrome because of window test

aik099 · 2025-01-09T07:51:52Z

@uuf6429 you can ignore Zombie driver because of minkphp/MinkZombieDriver#206 . This driver should be archived.

aik099 · 2025-01-09T07:56:22Z

Interesting workflow. You've combined a minimal version of the workflow for each of the drivers.

Similar stuff was done in https://github.com/instaclick/php-webdriver/blob/master/.github/workflows/ci.yml as well, but for MinkSelenium2Driver only. Maybe that will help you.

aik099 · 2025-01-09T07:58:09Z

.github/workflows/smoke.yml

+          - name: Chrome
+            repo: https://gitlab.com/behat-chrome/chrome-mink-driver.git
+            php: 7.4
+            setup: |
+              mkdir ./logs
+              ./vendor/bin/mink-test-server &> ./logs/mink-test-server.log &
+              docker run --net host --name chrome --volume /dev/shm:/dev/shm --shm-size 2g --volume ./vendor/mink/driver-testsuite/web-fixtures:/fixtures zenika/alpine-chrome:latest \
+                "--remote-debugging-address=0.0.0.0 --remote-debugging-port=9222 --disable-gpu --headless=new --disable-extensions --no-sandbox --use-gl=swiftshader --disable-software-rasterizer --disable-dev-shm-usage" &> ./logs/chrome.log &
+              sed "s#http://localhost/#http://localhost:8002/#" phpunit.xml.dist > phpunit.xml
+              while ! nc -z localhost 9222 </dev/null; do echo Waiting for chrome to start...; sleep 1; done
+              while ! nc -z localhost 8002 </dev/null; do echo Waiting for PHP server to start...; sleep 1; done
+            test: |
+              export WEB_FIXTURES_BROWSER=chrome
+              export DRIVER_MACHINE_BASE_PATH=/fixtures/
+              ./vendor/bin/phpunit --colors=always --testdox


Should we even test the 3rd party drivers?

It's listed in our documentation.. (just as zombie and sahi), so I added it. This is in line with the "testing active drivers" comment from earlier.

.github/workflows/smoke.yml

aik099 · 2025-01-09T08:04:23Z

Selenium2
right-click test failed, no idea why

That's why it fails: minkphp/MinkSelenium2Driver#381 . I have no idea how to fix it using https://github.com/instaclick/php-webdriver . That is the only blocker before that driver's release.

aik099 · 2025-01-09T09:15:57Z

Failures for the MinkBrowserKitDriver are fixed in minkphp/MinkBrowserKitDriver#183 .

stof · 2025-01-09T10:37:05Z

I suggest removing the job for the third-party driver. I don't think it makes sense to make our CI job fail based on their behavior which is out of our control, especially given that they fail many tests currently.

uuf6429 · 2025-01-09T22:51:29Z

I suggest removing the job for the third-party driver. I don't think it makes sense to make our CI job fail based on their behavior which is out of our control, especially given that they fail many tests currently.

@stof @aik099 I'm not strongly in favour of keeping it, but hear out my arguments:

we probably care that our changes do not impact them
there are numerous ways to configure the workflow in a way it does not reflect bad on us, eg:
- make the workflow as manually-triggered only so we only run it on demand (this was my original intention, but github doesn't let me run workflows manually before they have been merged to master, so I had to add the "pull_request" trigger)
- marking jobs as allowed-to-fail
we may request the vendor to maintain that part (with normal PRs) and if we see it doesn't happen, we can eventually just remove it
adding to point 3, we can explicitly ask vendors to participate, and if they don't, they won't be guaranteed backward compatibility (which makes maintenance easier for us)
the main objective of this workflow is to avoid making a local setup that is time consuming, non-reproducible and probably replicates other maintainers' efforts on their own machines.

Let me know if any of that convinces you and if still not convinced, I'll gladly remove this.

aik099 · 2025-01-10T08:25:47Z

We can talk to 3rd party driver manufacturers to see if they're willing for their driver to be compatible by our test suite changes.

Previously, GitHub Actions wasn't allowing ignoring build failures, which affected the CI badge in general on the README.md file. If you know how to allow a specific job within the build to fail without making the whole build fail but still showing that that job has failed (like in Travis CI times), then please do.

So let's try. Trying doesn't cost us anything.

uuf6429 · 2025-01-10T23:57:43Z

Previously, GitHub Actions wasn't allowing ignoring build failures, which affected the CI badge in general on the README.md file.

@aik099 apparently it's still not possible without some workarounds: actions/runner#2347 😞

Then the remaining option is (as said) a manual job not triggered by CI (or workaround in the linked issue).

aik099 · 2025-01-11T09:41:00Z

Previously, GitHub Actions wasn't allowing ignoring build failures, which affected the CI badge in general on the README.md file.

@aik099 apparently it's still not possible without some workarounds: actions/runner#2347 😞

Then the remaining option is (as said) a manual job not triggered by CI (or workaround in the linked issue).

@uuf6429 , then we'll do 2 jobs:

one job for drivers located in the minkphp organization, that will run automatically
another job for 3rd-party drivers (one per driver or maybe a combo job), that driver maintainers are welcome to adjust (we need to inform driver maintainers about such feature)

aik099 · 2025-01-11T11:48:41Z

It's strange that MinkSelenium2Driver job fails with Drag drop onto hidden itself test, when it doesn't fail on MinkSelenium2Driver repo test.

uuf6429 · 2025-01-11T12:22:08Z

I think I've seen that failure in the past...could it be transient/random? Unfortunately I can't retry that job without pushing something to run the whole workflow.

aik099 · 2025-01-11T14:02:21Z

... Unfortunately I can't retry that job without pushing something to run the whole workflow.

@uuf6429 , I've granted you the same permissions for this repository as on the webdriver-classic-driver repository. Now you should be able to retry builds, assign reviewers, merge, etc.

Enjoy.

uuf6429 · 2025-01-11T14:18:18Z

@uuf6429 , I've granted you the same permissions for this repository as on the webdriver-classic-driver repository. Now you should be able to retry builds, assign reviewers, merge, etc.

Awesome, I can stop that job that has been running for 5 hours 😅
https://github.com/minkphp/driver-testsuite/actions/runs/12722948823/job/35467568743

uuf6429 added 2 commits January 9, 2025 00:28

Add smoke-test workflow

e384263

Minor adjustments

c9568b6

uuf6429 added 3 commits January 9, 2025 08:39

Use oldest supported zombie driver

526cd83

Set node version

5156733

Use even older crap

85524bf

Roll back 9 year old downgrade

22a2728

Remove zombie driver

53c41cc

aik099 reviewed Jan 9, 2025

View reviewed changes

uuf6429 added 2 commits January 9, 2025 09:21

Various fixes

b5ce5bc

dvfgsafgnhdbgdbfndnfghz

6fa5925

aik099 mentioned this pull request Jan 9, 2025

PHPStan improvements #100

Merged

uuf6429 added 2 commits January 11, 2025 10:08

Limit minkse2driver to selenium 2

fbf3233

Fix infinite loops; fix selenium image

3827d67

uuf6429 added 2 commits January 11, 2025 11:09

Fix many problems

168e58d

Fix job name

719d090

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add smoke-test workflow #103

Add smoke-test workflow #103

uuf6429 commented Jan 8, 2025

uuf6429 commented Jan 9, 2025 •

edited

Loading

aik099 commented Jan 9, 2025

aik099 commented Jan 9, 2025

aik099 Jan 9, 2025

uuf6429 Jan 9, 2025

aik099 commented Jan 9, 2025

aik099 commented Jan 9, 2025

stof commented Jan 9, 2025

uuf6429 commented Jan 9, 2025 •

edited

Loading

aik099 commented Jan 10, 2025

uuf6429 commented Jan 10, 2025 •

edited

Loading

aik099 commented Jan 11, 2025 •

edited

Loading

aik099 commented Jan 11, 2025

uuf6429 commented Jan 11, 2025

aik099 commented Jan 11, 2025

uuf6429 commented Jan 11, 2025 •

edited

Loading

Add smoke-test workflow #103

Are you sure you want to change the base?

Add smoke-test workflow #103

Conversation

uuf6429 commented Jan 8, 2025

uuf6429 commented Jan 9, 2025 • edited Loading

aik099 commented Jan 9, 2025

aik099 commented Jan 9, 2025

aik099 Jan 9, 2025

Choose a reason for hiding this comment

uuf6429 Jan 9, 2025

Choose a reason for hiding this comment

aik099 commented Jan 9, 2025

aik099 commented Jan 9, 2025

stof commented Jan 9, 2025

uuf6429 commented Jan 9, 2025 • edited Loading

aik099 commented Jan 10, 2025

uuf6429 commented Jan 10, 2025 • edited Loading

aik099 commented Jan 11, 2025 • edited Loading

aik099 commented Jan 11, 2025

uuf6429 commented Jan 11, 2025

aik099 commented Jan 11, 2025

uuf6429 commented Jan 11, 2025 • edited Loading

uuf6429 commented Jan 9, 2025 •

edited

Loading

uuf6429 commented Jan 9, 2025 •

edited

Loading

uuf6429 commented Jan 10, 2025 •

edited

Loading

aik099 commented Jan 11, 2025 •

edited

Loading

uuf6429 commented Jan 11, 2025 •

edited

Loading