run UI tests in drone on Firefox #31325

individual-it · 2018-05-01T11:16:49Z

Description

run UI tests in drone on Firefox without using saucelabs

by using the selenium docker containers we are able to run tests in Firefox same as we do in Chrome. This PR

sets the environment for it
un-skips a couple of tests that actually do run in this version of selenium and firefox
fixes one input. setting the Email address didn't work correctly in Firefox if the field was pre-filled

Limitations:

this might break FF tests in travis, because it un-skips some tests. Question: do we need the travis runs with FF if we run them in drone
currently we are limited to selenium 3.8.1, because 3.9.0 removed the enablePassThrough option. So without changing the docker container we are limited to Firefox 58.0

Motivation and Context

get more independent of travis & saucelabs

How Has This Been Tested?

🤖

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

codecov · 2018-05-02T03:49:48Z

Codecov Report

Merging #31325 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #31325   +/-   ##
=========================================
  Coverage     64.84%   64.84%           
  Complexity    18324    18324           
=========================================
  Files          1198     1198           
  Lines         69580    69580           
  Branches       1283     1283           
=========================================
  Hits          45120    45120           
  Misses        24086    24086           
  Partials        374      374

Flag	Coverage Δ	Complexity Δ
#javascript	`53.1% <ø> (ø)`	`0 <ø> (ø)`	⬇️
#phpunit	`66.2% <ø> (ø)`	`18324 <ø> (ø)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1665fef...a5f69f6. Read the comment docs.

phil-davis

LGTM
I suggest that we remove Firefox test matrix entries from Travis.
chrome, IE11 and Edge on nightly Travis should be plenty, and Firefox will happen with every PR.

phil-davis · 2018-05-02T05:59:40Z

This will make another 14 matrix jobs in drone PR builds.
So that is something to think about.
IMO it is easy to have lots of drone bees working, we just need a solar-powered drone hive so the bees are green :)
@patrickjahns your thoughts?

phil-davis · 2018-05-02T06:00:43Z

drone says it passed, but no codecov status has come. what happened???

individual-it · 2018-05-02T06:22:17Z

@ownclouders rebase

ownclouders · 2018-05-02T06:22:19Z

Hey! I'm GitMate.io! This pull request is being rebased automatically. Please DO NOT push while rebase is in progress or your changes would be lost permanently ⚠️

ownclouders · 2018-05-02T06:22:38Z

Automated rebase with GitMate.io was successful! 🎉

patrickjahns

Please do not merge - see further comments

patrickjahns · 2018-05-02T19:46:44Z

This will make another 14 matrix jobs in drone PR builds.

Exactly - I am against having both browsers being tested during a PR test run. It will just add unnecessary build time to pull requests.

I see testing different browser being deferred to async runs (nightly / weekly). The mechanism for this is in the works. You might be just a little bit ahead of your time ;-)

Personally I would keep this open and revive once the mechanisms are in place. We can at the same time investigate how we can also add browserstack onto drone for the nightly/weekly scenarios

individual-it · 2019-02-14T04:31:35Z

rebased, made Firefox tests run on HTTP and skipped failing tests
@PVince81 @phil-davis @davitol what to do with that PR now?

phil-davis

Firstly, just 2 questions.

phil-davis · 2019-02-14T05:51:24Z

.drone.yml

@@ -515,6 +526,18 @@ services:
    when:
      matrix:
        USE_SERVER: true
+        SERVER_PROTOCOL: https
+
+  server:


This has the same pipeline step name as server: above.
Only 1 of them ever gets started, but does it cause some confusion for drone.yml parsing?

no confusion for drones, only for humans
we need to have the same name because that becomes the hostname for that service and it should be the same for HTTP and HTTPS, to have less confusion

phil-davis · 2019-02-14T05:51:53Z

.drone.yml

@@ -529,6 +552,18 @@ services:
    when:
      matrix:
        USE_FEDERATED_SERVER: true
+        SERVER_PROTOCOL: https
+
+  federated:


Same here, repeating the federated: step

phil-davis · 2019-02-14T06:04:28Z

Maybe for now we could merge all the:

drone pipeline additions
putting the SERVER_PROTOCOL variable into the existing chrome webUI and other API acceptance... matrix entries
skipOnFirefox tag changes
select a couple of webUI test suites to also run all the time on Firefox (that will ensure that we know if something obvious regresses on Firefox) - maybe webUILogin and webUIRenameFiles - that will ensure that it can login and do something basic on the files page.

individual-it · 2019-02-14T06:17:07Z

or running all @smokeTest tagged tests on FF?

Lots of code changes and discussion, so review is needed again

phil-davis

LGTM
And we can have a followup issue to investigate the test scenarios tagged skipOnFirefox and try to find out why they fail (real problem on Firefox or Firefox webDriver "feature"...)

patrickjahns · 2019-02-14T11:13:53Z

Just a minor note - the firefox container seems to be quite old

core/.drone.yml

Line 507 in a5f69f6

image: selenium/standalone-firefox-debug:3.8.1

And it was last built 12 months ago:

$ docker images | grep firefox
selenium/standalone-firefox-debug                        3.8.1                                      c2ebdfdb9e31        12 months ago       792MB

patrickjahns

Please use a more recent firefox version

individual-it · 2019-02-15T03:27:38Z

@patrickjahns see original description a newer version is not possible because of not finished implementations in the libraries

2. currently we are limited to selenium 3.8.1, because 3.9.0 removed the `enablePassThrough` option. So without changing the docker container we are limited to Firefox 58.0

not possible currently due to limitations in libraries

individual-it · 2019-02-15T03:44:28Z

@phil-davis issue opened for skipped tests #34491

individual-it · 2019-02-15T03:48:39Z

@patrickjahns issue to update to a newer version of Firefox #34493

individual-it · 2019-02-15T05:44:47Z

backport in #34494

individual-it force-pushed the ui-tests-firefox-drone branch 2 times, most recently from 3d866f0 to 63d342e Compare May 2, 2018 04:16

individual-it changed the title ~~[WIP] run UI tests in drone on Firefox~~ run UI tests in drone on Firefox May 2, 2018

individual-it requested review from phil-davis and DeepDiver1975 and removed request for phil-davis May 2, 2018 05:23

individual-it self-assigned this May 2, 2018

individual-it requested review from patrickjahns and removed request for DeepDiver1975 May 2, 2018 05:23

individual-it added 3 - To Review backport-request dev:acceptance-tests labels May 2, 2018

individual-it added this to the development milestone May 2, 2018

individual-it requested review from DeepDiver1975 and phil-davis May 2, 2018 05:23

phil-davis reviewed May 2, 2018

View reviewed changes

ownclouders force-pushed the ui-tests-firefox-drone branch from 63d342e to 146cae0 Compare May 2, 2018 06:22

patrickjahns previously requested changes May 2, 2018

View reviewed changes

ownclouders added the status/STALE label May 10, 2018

individual-it mentioned this pull request Aug 23, 2018

UI Tests with various set of browsers owncloud/QA#584

Closed

PVince81 added the QA:team label Feb 8, 2019

ownclouders removed the status/STALE label Feb 8, 2019

individual-it mentioned this pull request Feb 12, 2019

[wip] docker for saucelabs in travis #31874

Closed

9 tasks

individual-it force-pushed the ui-tests-firefox-drone branch from af3547a to bea7b3a Compare February 14, 2019 03:36

phil-davis reviewed Feb 14, 2019

View reviewed changes

individual-it force-pushed the ui-tests-firefox-drone branch 4 times, most recently from 2ebc4bb to 3c4ce3d Compare February 14, 2019 06:47

only run smoke tests on Firefox

a5f69f6

individual-it force-pushed the ui-tests-firefox-drone branch from 3c4ce3d to a5f69f6 Compare February 14, 2019 07:27

phil-davis approved these changes Feb 14, 2019

View reviewed changes

patrickjahns self-requested a review February 14, 2019 11:14

patrickjahns previously requested changes Feb 14, 2019

View reviewed changes

individual-it merged commit 1d00fbd into master Feb 15, 2019

delete-merged-branch bot deleted the ui-tests-firefox-drone branch February 15, 2019 03:38

individual-it mentioned this pull request Feb 15, 2019

debug skiped UI tests on Firefox #34491

Open

individual-it added 4 - To release and removed 3 - To Review labels Feb 15, 2019

ownclouders mentioned this pull request Feb 15, 2019

test on newer version of firefox #34493

Open

individual-it mentioned this pull request Feb 15, 2019

[stable10] run UI tests in drone on Firefox #34494

Merged

individual-it removed the backport-request label Feb 15, 2019

lock bot locked as resolved and limited conversation to collaborators Feb 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

run UI tests in drone on Firefox #31325

run UI tests in drone on Firefox #31325

individual-it commented May 1, 2018 •

edited

Loading

codecov bot commented May 2, 2018 •

edited

Loading

phil-davis left a comment

phil-davis commented May 2, 2018 •

edited

Loading

phil-davis commented May 2, 2018

individual-it commented May 2, 2018

ownclouders commented May 2, 2018

ownclouders commented May 2, 2018

patrickjahns left a comment

patrickjahns commented May 2, 2018 •

edited

Loading

individual-it commented Feb 14, 2019

phil-davis left a comment

phil-davis Feb 14, 2019 •

edited

Loading

individual-it Feb 14, 2019

phil-davis Feb 14, 2019 •

edited

Loading

phil-davis commented Feb 14, 2019

individual-it commented Feb 14, 2019

phil-davis left a comment

patrickjahns commented Feb 14, 2019

patrickjahns left a comment

individual-it commented Feb 15, 2019

individual-it commented Feb 15, 2019

individual-it commented Feb 15, 2019

individual-it commented Feb 15, 2019

run UI tests in drone on Firefox #31325

run UI tests in drone on Firefox #31325

Conversation

individual-it commented May 1, 2018 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

codecov bot commented May 2, 2018 • edited Loading

Codecov Report

phil-davis left a comment

Choose a reason for hiding this comment

phil-davis commented May 2, 2018 • edited Loading

phil-davis commented May 2, 2018

individual-it commented May 2, 2018

ownclouders commented May 2, 2018

ownclouders commented May 2, 2018

patrickjahns left a comment

Choose a reason for hiding this comment

patrickjahns commented May 2, 2018 • edited Loading

individual-it commented Feb 14, 2019

phil-davis left a comment

Choose a reason for hiding this comment

phil-davis Feb 14, 2019 • edited Loading

Choose a reason for hiding this comment

individual-it Feb 14, 2019

Choose a reason for hiding this comment

phil-davis Feb 14, 2019 • edited Loading

Choose a reason for hiding this comment

phil-davis commented Feb 14, 2019

individual-it commented Feb 14, 2019

phil-davis left a comment

Choose a reason for hiding this comment

patrickjahns commented Feb 14, 2019

patrickjahns left a comment

Choose a reason for hiding this comment

individual-it commented Feb 15, 2019

individual-it commented Feb 15, 2019

individual-it commented Feb 15, 2019

individual-it commented Feb 15, 2019

individual-it commented May 1, 2018 •

edited

Loading

codecov bot commented May 2, 2018 •

edited

Loading

phil-davis commented May 2, 2018 •

edited

Loading

patrickjahns commented May 2, 2018 •

edited

Loading

phil-davis Feb 14, 2019 •

edited

Loading

phil-davis Feb 14, 2019 •

edited

Loading