QE-11002 easier html reports (#323)

1. show tags on flat → to make searchable / filterable 2. pretty print md tables - for tables in a `the following:` steps 3. style logs more legible -> now with normal "light" colors 4. include full step text in tooltip hover - i.e. title attribute on 5. on scenario report - give more room for long names 6. use relative offsets instead of timestamps - full timestamps still in 7. remove external ansi parser -> for better licensing
dominodatalab · Apr 12, 2023 · 49fc02c · 49fc02c
1 parent 4d90848
commit 49fc02c
Show file tree

Hide file tree

Showing 15 changed files with 305 additions and 155 deletions.
diff --git a/.gitignore b/.gitignore
@@ -76,7 +76,6 @@ cover/
 *.pot
 
 # Django stuff:
-*.log
 local_settings.py
 db.sqlite3
 db.sqlite3-journal

diff --git a/data/unit/ansi.log b/data/unit/ansi.log
@@ -0,0 +1,39 @@
+
+  [35mScenario[0m: Just a scenario that opens a web page
+[90m    Given I start a webserver at directory "data/www" and save the port to the variable "PORT"
+[0m[1A
+    [32mGiven[0m I start a webserver at directory "data/www" and save the port to the variable "PORT"[90m     # started at 2023-04-10T15:59:01.540 took 0.013s[0m
+[90m      And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/buttons.html"
+[0m2023-04-10 15:59:01,555 DEBUG opening browser chrome
+2023-04-10 15:59:01,956 DEBUG webdriver.Chrome init
+2023-04-10 15:59:03,568 DEBUG executing page check "wait for document.readyState"
+2023-04-10 15:59:03,572 DEBUG executed page check "wait for document.readyState" in 0.004s
+2023-04-10 15:59:03,572 DEBUG executing page check "broken image checker"
+2023-04-10 15:59:03,576 DEBUG executed page check "broken image checker" in 0.003s
+2023-04-10 15:59:03,576 DEBUG navigating to url #http://192.168.86.23:52639/buttons.html
+2023-04-10 15:59:03,709 DEBUG executing page check "wait for document.readyState"
+2023-04-10 15:59:03,713 DEBUG executed page check "wait for document.readyState" in 0.004s
+2023-04-10 15:59:03,713 DEBUG executing page check "broken image checker"
+2023-04-10 15:59:03,718 DEBUG executed page check "broken image checker" in 0.004s
+[1A
+    [32m  And[0m I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/buttons.html"[90m                  # started at 2023-04-10T15:59:01.554 took 2.194s[0m
+[90m      # HOST_ADDRESS="192.168.86.23" PORT="52639"
+[0m[90m      And I should see the text "inexistent"
+[0m2023-04-10 15:59:04,035 DEBUG switched browser to the default frame
+2023-04-10 15:59:06,549 DEBUG switched browser to the default frame
+[1A
+[31m      And I should see the text "inexistent"[0m[90m                                                       # started at 2023-04-10T15:59:04.033 took 2.566s[0m
+
+... <removed stacktrace> ...
+
+2023-04-10 15:59:07,190 DEBUG quitting browser between sessions
+
+
+Failing scenarios:
+  data/features/feature_with_failing_scenario_with_web.feature:4  Just a scenario that opens a web page
+
+0 features passed, 1 failed, 0 skipped
+0 scenarios passed, 1 failed, 0 skipped
+2 steps passed, 1 failed, 0 skipped, 0 undefined
+Took 0m4.773s
+Error: test run failed, see above for details
diff --git a/data/unit/ansi.log.html b/data/unit/ansi.log.html
@@ -0,0 +1,42 @@
+<pre>
+
+  <span style="color: magenta;">Scenario</span>: Just a scenario that opens a web page
+<span style="color: grey;">    Given I start a webserver at directory "data/www" and save the port to the variable "PORT"
+</span>
+    <span style="color: green;">Given</span> I start a webserver at directory "data/www" and save the port to the variable "PORT"<span style="color: grey;">     # started at 2023-04-10T15:59:01.540 took 0.013s</span>
+<span style="color: grey;">      And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/buttons.html"
+</span>2023-04-10 15:59:01,555 DEBUG opening browser chrome
+2023-04-10 15:59:01,956 DEBUG webdriver.Chrome init
+2023-04-10 15:59:03,568 DEBUG executing page check "wait for document.readyState"
+2023-04-10 15:59:03,572 DEBUG executed page check "wait for document.readyState" in 0.004s
+2023-04-10 15:59:03,572 DEBUG executing page check "broken image checker"
+2023-04-10 15:59:03,576 DEBUG executed page check "broken image checker" in 0.003s
+2023-04-10 15:59:03,576 DEBUG navigating to url #http://192.168.86.23:52639/buttons.html
+2023-04-10 15:59:03,709 DEBUG executing page check "wait for document.readyState"
+2023-04-10 15:59:03,713 DEBUG executed page check "wait for document.readyState" in 0.004s
+2023-04-10 15:59:03,713 DEBUG executing page check "broken image checker"
+2023-04-10 15:59:03,718 DEBUG executed page check "broken image checker" in 0.004s
+
+    <span style="color: green;">  And</span> I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/buttons.html"<span style="color: grey;">                  # started at 2023-04-10T15:59:01.554 took 2.194s</span>
+<span style="color: grey;">      # HOST_ADDRESS="192.168.86.23" PORT="52639"
+</span><span style="color: grey;">      And I should see the text "inexistent"
+</span>2023-04-10 15:59:04,035 DEBUG switched browser to the default frame
+2023-04-10 15:59:06,549 DEBUG switched browser to the default frame
+
+<span style="color: red;">      And I should see the text "inexistent"</span><span style="color: grey;">                                                       # started at 2023-04-10T15:59:04.033 took 2.566s</span>
+
+... &lt;removed stacktrace&gt; ...
+
+2023-04-10 15:59:07,190 DEBUG quitting browser between sessions
+
+
+Failing scenarios:
+  data/features/feature_with_failing_scenario_with_web.feature:4  Just a scenario that opens a web page
+
+0 features passed, 1 failed, 0 skipped
+0 scenarios passed, 1 failed, 0 skipped
+2 steps passed, 1 failed, 0 skipped, 0 undefined
+Took 0m4.773s
+Error: test run failed, see above for details
+
+</pre>
diff --git a/features/cli/report_basics.feature b/features/cli/report_basics.feature
@@ -21,8 +21,8 @@ Feature: Report basics
       And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
      Then I wait to click the link "Feature with passing scenario with web"
      When I wait to click the link "Just a scenario that opens a web page"
-     Then I wait to see the button "Given I start a webserver at directory "data/www" and save the port to the variable "PORT""
-      And I should see the button "And I open a browser at the url "http://\{HOST_ADDRESS\}:\{PORT\}/buttons.html""
+     Then I wait to see the button "Given I start a webserver at directory \"data/www\" and save the port to the variable \"PORT\""
+      And I should see the button "And I open a browser at the url \"http://\{HOST_ADDRESS\}:\{PORT\}/buttons.html\""
      When I run the command "ls "{SCENARIO_RESULTS_DIR}/"" and save stdout to "STDOUT", stderr to "STDERR" and expect exit code "0"
      Then I should see "{STDOUT}" matches the following
       """
@@ -36,21 +36,23 @@ Feature: Report basics
       And I run the command "cucu report {CUCU_RESULTS_DIR}/multi-scenario-browser-results --output {CUCU_RESULTS_DIR}/multi-scenario-browser-report" and expect exit code "0"
       And I start a webserver at directory "{CUCU_RESULTS_DIR}/multi-scenario-browser-report/" and save the port to the variable "PORT"
       And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
+
+     # Verify HTML report
      When I click the link "Multiple scenarios with browser steps"
       And I click the link "Open our test checkboxes page"
-      And I should not see the image with the alt text "Given I start a webserver at directory "data/www" and save the port to the variable "PORT""
-      And I should not see the image with the alt text "And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/checkboxes.html""
-      And I should not see the image with the alt text "Then I should see the checkbox "checkbox with inner label""
-     Then I click the button "Then I should see the checkbox "checkbox with inner label""
-      And I should see the image with the alt text "Then I should see the checkbox "checkbox with inner label""
-      And I should not see the image with the alt text "Given I start a webserver at directory "data/www" and save the port to the variable "PORT""
-      And I should not see the image with the alt text "And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/checkboxes.html""
+      And I should not see the image with the alt text "Given I start a webserver at directory \"data/www\" and save the port to the variable \"PORT\""
+      And I should not see the image with the alt text "And I open a browser at the url \"http://{HOST_ADDRESS}:{PORT}/checkboxes.html\""
+      And I should not see the image with the alt text "Then I should see the checkbox \"checkbox with inner label\""
+     Then I click the button "Then I should see the checkbox \"checkbox with inner label\""
+      And I should see the image with the alt text "Then I should see the checkbox \"checkbox with inner label\""
+      And I should not see the image with the alt text "Given I start a webserver at directory \"data/www\" and save the port to the variable \"PORT\""
+      And I should not see the image with the alt text "And I open a browser at the url \"http://{HOST_ADDRESS}:{PORT}/checkboxes.html""
      When I save the current url to the variable "CURRENT_URL"
       And I click the link "Index"
      Then I navigate to the url "{CURRENT_URL}"
-      And I wait to see the image with the alt text "Then I should see the checkbox "checkbox with inner label""
-      And I should not see the image with the alt text "Given I start a webserver at directory "data/www" and save the port to the variable "PORT""
-      And I should not see the image with the alt text "And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/checkboxes.html""
+      And I wait to see the image with the alt text "Then I should see the checkbox \"checkbox with inner label\""
+      And I should not see the image with the alt text "Given I start a webserver at directory \"data/www\" and save the port to the variable \"PORT\""
+      And I should not see the image with the alt text "And I open a browser at the url \"http://{HOST_ADDRESS}:{PORT}/checkboxes.html\""
 
   Scenario: User can run a feature with mixed results and has all results reported correctly without skips
     Given I run the command "cucu run data/features/feature_with_mixed_results.feature --results {CUCU_RESULTS_DIR}/mixed-results" and expect exit code "1"
@@ -62,11 +64,11 @@ Feature: Report basics
        | .*         | Feature with mixed results | 4     | 2      | 2      | 0       | failed | .*       |
      When I click the button "Feature with mixed results"
      Then I should see a table that matches the following:
-      | Started at | Scenario                            | Total Steps | Status  | Duration |
-      | .*         | Scenario that also passes           | 1           | passed  | .*       |
-      | .*         | Scenario that fails                 | 2           | failed  | .*       |
-      | .*         | Scenario that has an undefined step | 1           | failed  | .*       |
-      | .*         | Scenario that passes                | 1           | passed  | .*       |
+      | Offset | Scenario                            | Steps | Status  | Duration |
+      | .*     | Scenario that also passes           | 1     | passed  | .*       |
+      | .*     | Scenario that fails                 | 2     | failed  | .*       |
+      | .*     | Scenario that has an undefined step | 1     | failed  | .*       |
+      | .*     | Scenario that passes                | 1     | passed  | .*       |
       And I click the button "Scenario that fails"
      Then I should see the text "RuntimeError: step fails on purpose"
      When I click the button "Index"
@@ -85,12 +87,12 @@ Feature: Report basics
        | .*         | Feature with mixed results | 5     | 2      | 2      | 1       | failed | .*       |
      When I click the button "Feature with mixed results"
      Then I should see a table that matches the following:
-      | Started at | Scenario                            | Total Steps | Status  | Duration |
-      | .*         | Scenario that also passes           | 1           | passed  | .*       |
-      | .*         | Scenario that fails                 | 2           | failed  | .*       |
-      | .*         | Scenario that has an undefined step | 1           | failed  | .*       |
-      | .*         | Scenario that is skipped            | 1           | skipped | .*       |
-      | .*         | Scenario that passes                | 1           | passed  | .*       |
+      | Offset | Scenario                            | Steps | Status  | Duration |
+      | .*     | Scenario that also passes           | 1     | passed  | .*       |
+      | .*     | Scenario that fails                 | 2     | failed  | .*       |
+      | .*     | Scenario that has an undefined step | 1     | failed  | .*       |
+      | .*     | Scenario that is skipped            | 1     | skipped | .*       |
+      | .*     | Scenario that passes                | 1     | passed  | .*       |
       And I click the button "Scenario that fails"
      Then I should see the text "RuntimeError: step fails on purpose"
      When I click the button "Index"
@@ -108,8 +110,8 @@ Feature: Report basics
        | .*         | Feature with background    | 1     | 1      | 0      | 0       | passed | .*       |
      When I click the button "Feature with background"
      Then I should see a table that matches the following:
-      | Started at | Scenario                            | Total Steps | Status  | Duration |
-      | .*         | Scenario which now has a background | 2           | passed  | .*       |
+      | Offset | Scenario                            | Steps | Status  | Duration |
+      | .*     | Scenario which now has a background | 2     | passed  | .*       |
 
   @show-skips
   Scenario: User can run feature with background and has all results reported correctly
@@ -122,9 +124,9 @@ Feature: Report basics
        | .*         | Feature with background    | 2     | 1      | 0      | 1       | passed | .*       |
      When I click the button "Feature with background"
      Then I should see a table that matches the following:
-      | Started at | Scenario                            | Total Steps | Status  | Duration |
-      | .*         | Scenario that is skipped            | 2           | skipped | .*       |
-      | .*         | Scenario which now has a background | 2           | passed  | .*       |
+      | Offset | Scenario                            | Steps | Status  | Duration |
+      | .*     | Scenario that is skipped            | 2     | skipped | .*       |
+      | .*     | Scenario which now has a background | 2     | passed  | .*       |
 
   @workaround @QE-7075
   @disabled
@@ -177,13 +179,13 @@ Feature: Report basics
       And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
      When I click the button "Slow feature #1"
      Then I should see a table that matches the following:
-      | Started at | Scenario      | Total Steps | Status | Duration |
-      | .*         | Slow scenario | 1           | .*     |    >*    |
+      | Offset | Scenario      | Steps | Status | Duration |
+      | .*     | Slow scenario | 1     | .*     |    >*    |
      When I go back on the browser
       And I click the button "Slow feature #2"
      Then I should see a table that matches the following:
-      | Started at | Scenario      | Total Steps | Status | Duration |
-      | .*         | Slow scenario | 1           | .*     |    >*    |
+      | Offset | Scenario      | Steps | Status | Duration |
+      | .*     | Slow scenario | 1     | .*     |    >*    |
 
   Scenario: User can run results without skips in the HTML test report
     Given I run the command "cucu run data/features/feature_with_mixed_results.feature --results {CUCU_RESULTS_DIR}/report_without_skips --generate-report --report {CUCU_RESULTS_DIR}/report_without_skips_report" and expect exit code "1"
@@ -194,11 +196,11 @@ Feature: Report basics
       | .*         | Feature with mixed results | 4     | 2      | 2      | 0       | failed | .*       |
      When I click the button "Feature with mixed results"
      Then I should see a table that matches the following:
-      | Started at | Scenario                            | Total Steps | Status  | Duration |
-      | .*         | Scenario that also passes           | 1           | passed  | .*       |
-      | .*         | Scenario that fails                 | 2           | failed  | .*       |
-      | .*         | Scenario that has an undefined step | 1           | failed  | .*       |
-      | .*         | Scenario that passes                | 1           | passed  | .*       |
+      | Offset | Scenario                            | Steps | Status  | Duration |
+      | .*     | Scenario that also passes           | 1     | passed  | .*       |
+      | .*     | Scenario that fails                 | 2     | failed  | .*       |
+      | .*     | Scenario that has an undefined step | 1     | failed  | .*       |
+      | .*     | Scenario that passes                | 1     | passed  | .*       |
       And I click the button "Scenario that fails"
      Then I should see the text "RuntimeError: step fails on purpose"
      When I click the button "Index"
@@ -215,8 +217,8 @@ Feature: Report basics
       | .*         | Feature with background | 1     | 1      | 0      | 0       | passed | .*       |
      When I click the button "Feature with background"
      Then I should see a table that matches the following:
-      | Started at | Scenario                            | Total Steps | Status  | Duration |
-      | .*         | Scenario which now has a background | 2           | passed  | .*       |
+      | Offset | Scenario                            | Steps | Status  | Duration |
+      | .*     | Scenario which now has a background | 2     | passed  | .*       |
 
   Scenario: User can run results without skips in the JUnit results
     Given I run the command "cucu run data/features/feature_with_mixed_results.feature --results {CUCU_RESULTS_DIR}/junit_without_skips" and expect exit code "1"
@@ -262,8 +264,8 @@ Feature: Report basics
        | .*         | Feature with failing scenario with web | 1     | 0      | 1      | 0       | failed | .*       |
      When I click the button "Feature with failing scenario with web"
      Then I should see a table that matches the following:
-       | Started at | Scenario                              | Total Steps | Status | Duration |
-       | .*         | Just a scenario that opens a web page | 3           | failed | .*       |
+       | Offset | Scenario                              | Steps | Status | Duration |
+       | .*     | Just a scenario that opens a web page | 3           | failed | .*       |
      When I click the button "Just a scenario that opens a web page"
       And I wait to click the button "show images"
-      And I should see the image with the alt text "And I should see the text "inexistent""
+      And I should see the image with the alt text "And I should see the text \"inexistent\""