Core: Generate xUnit Reports for e2e tests #962

sawood14012 · 2024-05-03T15:46:23Z

a Wip PR which enables generation of junit/xunit reports to connect report-portal and Quality dashboard

openshift-ci · 2024-05-03T15:46:34Z

Hi @sawood14012. Thanks for your PR.

I'm waiting for a codeready-toolchain member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ranakan19 · 2024-05-03T17:14:36Z

/ok-to-test

openshift-ci · 2024-05-15T08:43:48Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sawood14012
Once this PR has been reviewed and has the lgtm label, please assign matousjobanek for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mfrancisc

Thanks for taking a look at this 🙏

I'm not familiar with go-junit-report but don't you need to pipe it to the go test ./... command as explained here:

go test -v 2>&1 ./... | go-junit-report -set-exit-code > report.xml

make/test.mk

Co-authored-by: Francisc Munteanu <[email protected]>

make/test.mk

Co-authored-by: Francisc Munteanu <[email protected]>

make/test.mk

MatousJobanek · 2024-05-31T14:24:03Z

make/test.mk

+	@command -v ./go-junit-report >/dev/null 2>&1 || { echo "go-junit-report is not installed. Installing..."; GOBIN=$(PWD) go install github.com/jstemmer/go-junit-report/v2@latest; }
+	@echo "go-junit-report version:" && ./go-junit-report -version


I would rather avoid installing the binary in the folder of the repository - let's install it either in /tmp folder or somewhere else.

MatousJobanek · 2024-05-31T14:37:24Z

make/test.mk

@@ -163,7 +165,8 @@ execute-tests:
 	@echo "Status of ToolchainStatus"
 	-oc get ToolchainStatus -n ${HOST_NS} -o yaml
 	@echo "Starting test $(shell date)"
-	MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} go test ${TESTS_TO_EXECUTE} -run ${TESTS_RUN_FILTER_REGEXP} -p 1 -parallel ${E2E_PARALLELISM} -v -timeout=90m -failfast || \
+	$(MAKE) check-go-junit-report
+	MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} go test ${TESTS_TO_EXECUTE} -run ${TESTS_RUN_FILTER_REGEXP} -p 1 -parallel ${E2E_PARALLELISM} -v -timeout=90m -failfast 2>&1 | $(MAKE) generate-report REPORT_NAME=${REPORT_NAME}  || \


you cannot use the pipe for the whole output; otherwise, we will lose the content/output of the tests when running it locally as well as in openshift-ci

sonarcloud · 2024-06-14T12:19:52Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
3.4% Duplication on New Code

See analysis details on SonarCloud

sawood14012 · 2024-06-20T12:24:27Z

@MatousJobanek i have resolved your comments and updated my approach

This approach ensures that the full output of the tests is retained and can be used both locally and in OpenShift CI.
Also changed the dir of binary to /tmp
the binary doesn't execute or generate a report locally.

MatousJobanek · 2024-06-24T12:56:53Z

make/test.mk

+	@echo "Skipping Go Junit report check and install"
+endif
+	@echo "Running tests"
+	MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} go test ${TESTS_TO_EXECUTE} -run ${TESTS_RUN_FILTER_REGEXP} -p 1 -parallel ${E2E_PARALLELISM} -v -timeout=90m -failfast 2>&1 | tee test_output.log


test_output.log should be stored in a /tmp folder, right?

MatousJobanek · 2024-06-24T14:01:36Z

make/test.mk

+	$(MAKE) generate-report REPORT_NAME=${REPORT_NAME} || \
 	($(MAKE) print-logs HOST_NS=${HOST_NS} MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} && exit 1)


this is executed as a separate command, which means that printing logs will depend on the result of the generate-report target, not on the result of the tests.
The original way it was written printed out the logs only when the test failed and also returned the exit code 1, so execution of the print-logs target depended on the results of the tests - this needs to stay the same.

so i am now running the generate report cmd after print logs hope this works

MatousJobanek · 2024-06-25T09:59:07Z

make/test.mk

+	go test ${TESTS_TO_EXECUTE} -run ${TESTS_RUN_FILTER_REGEXP} -p 1 -parallel ${E2E_PARALLELISM} -v -timeout=90m -failfast 2>&1 | tee /tmp/test_output.log || ( $(MAKE) print-logs && exit 1 )
+	$(MAKE) generate-report REPORT_NAME=${REPORT_NAME}


if the test fails, then the previous line exits with exit code 1, so the report won't be generated - see

|| ( $(MAKE) print-logs && exit 1 )

I guess that you need to do something like this:

Suggested change

go test ${TESTS_TO_EXECUTE} -run ${TESTS_RUN_FILTER_REGEXP} -p 1 -parallel ${E2E_PARALLELISM} -v -timeout=90m -failfast 2>&1 | tee /tmp/test_output.log || ( $(MAKE) print-logs && exit 1 )

$(MAKE) generate-report REPORT_NAME=${REPORT_NAME}

MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} \

(go test ${TESTS_TO_EXECUTE} -run ${TESTS_RUN_FILTER_REGEXP} -p 1 -parallel ${E2E_PARALLELISM} -v -timeout=90m -failfast 2>&1 | tee /tmp/test_output.log && $(MAKE) generate-report REPORT_NAME=${REPORT_NAME}) || ( $(MAKE) print-logs && $(MAKE) generate-report REPORT_NAME=${REPORT_NAME} && exit 1 )

but I haven't tested it.
Not sure if there is some simpler way to deal with it

this won't work because of the pipe and the returned value from tee :-/

MatousJobanek · 2024-06-26T14:31:05Z

make/test.mk

+	MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} HOST_NS=${HOST_NS} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} \
+	go test ${TESTS_TO_EXECUTE} -run ${TESTS_RUN_FILTER_REGEXP} -p 1 -parallel ${E2E_PARALLELISM} -v -timeout=90m -failfast 2>&1 | tee /tmp/test_output.log; \
+	STATUS=$$?; \
+	$(MAKE) generate-report REPORT_NAME=${REPORT_NAME}; \
+	if [ $$STATUS -ne 0 ]; then $(MAKE) print-logs && exit 1; fi


I just tested it and it doesn't work as expected - if one of the test suites fails, then it keeps running with the following test suites, but it should stop instead

my guess is that STATUS=$$? takes the exit value from the tee command that is executed "after" the test as part of the pipe

I was thinking of changing to STATUS=${PIPESTATUS[0]}; \ to capture the exit status of go test

that could work, I didn't know about PIPESTATUS 👍

sawood14012 added 2 commits May 3, 2024 21:08

added a way to generate reports

bba8838

install go-unit-report

754a54b

sawood14012 requested review from MatousJobanek, xcoulon, alexeykazakov, rajivnathan, ranakan19, sbryzak and mfrancisc as code owners May 3, 2024 15:46

openshift-ci bot added the do-not-merge/work-in-progress label May 3, 2024

openshift-ci bot requested a review from michaelkleinhenz May 3, 2024 15:46

Merge branch 'master' into add-reporting

45853c2

openshift-ci bot added the needs-ok-to-test label May 3, 2024

openshift-ci bot added ok-to-test and removed needs-ok-to-test labels May 3, 2024

sawood14012 added 7 commits May 13, 2024 11:42

Merge branch 'codeready-toolchain:master' into add-reporting

b62d260

check and install go-junit-report

994ee26

another try

9a4f036

another try 2

3608890

add another path

cfc77de

changed the base image

ba52cf0

Merge branch 'master' into add-reporting

89aa8f9

sawood14012 added 2 commits May 15, 2024 14:15

another try with path

54d88be

try again with different base

66d7085

mfrancisc reviewed May 15, 2024

View reviewed changes

make/test.mk Outdated Show resolved Hide resolved

sawood14012 and others added 2 commits May 15, 2024 17:49

Update make/test.mk

5e78253

Co-authored-by: Francisc Munteanu <[email protected]>

Merge branch 'master' into add-reporting

dd88f5b

sawood14012 added 3 commits May 16, 2024 16:01

Merge branch 'master' into add-reporting

c4c1e1f

remove check go-junit-report

c566e6c

reinstall before trigger

395097d

openshift-merge-robot added the needs-rebase label May 18, 2024

mfrancisc reviewed May 21, 2024

View reviewed changes

make/test.mk Outdated Show resolved Hide resolved

sawood14012 and others added 2 commits May 21, 2024 12:54

Update make/test.mk

8f42fb8

Co-authored-by: Francisc Munteanu <[email protected]>

Merge branch 'master' into add-reporting

bb08a5c

openshift-merge-robot removed the needs-rebase label May 21, 2024

sawood14012 added 4 commits May 21, 2024 19:03

add metrics report

bd89da5

tidy up

b6edfad

Revert Dockerfile change

a176b82

Merge branch 'master' into add-reporting

1202053

MatousJobanek reviewed May 31, 2024

View reviewed changes

Merge branch 'master' into add-reporting

1162d6d

sawood14012 added 4 commits June 19, 2024 16:05

move the binary to /tmp

da34df6

run only on openshift ci

40c634c

run only on openshift ci

9768327

Fix Pipe everyting

1ecbfe3

sawood14012 changed the title ~~WIP: Generate xUnit Reports for e2e tests~~ Core: Generate xUnit Reports for e2e tests Jun 20, 2024

openshift-ci bot removed the do-not-merge/work-in-progress label Jun 20, 2024

sawood14012 added 2 commits June 20, 2024 23:41

Merge branch 'master' into add-reporting

113cd51

Merge branch 'master' into add-reporting

466d6a5

MatousJobanek reviewed Jun 24, 2024

View reviewed changes

sawood14012 added 2 commits June 24, 2024 21:24

move test output to /tmp

0438d14

move test output to /tmp 2

0519d8e

MatousJobanek reviewed Jun 25, 2024

View reviewed changes

generate rp if failed

8732398

MatousJobanek reviewed Jun 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Core: Generate xUnit Reports for e2e tests #962

Core: Generate xUnit Reports for e2e tests #962

sawood14012 commented May 3, 2024

openshift-ci bot commented May 3, 2024

ranakan19 commented May 3, 2024

openshift-ci bot commented May 15, 2024

mfrancisc left a comment

MatousJobanek May 31, 2024

MatousJobanek May 31, 2024

sonarcloud bot commented Jun 14, 2024

sawood14012 commented Jun 20, 2024

MatousJobanek Jun 24, 2024

sawood14012 Jun 24, 2024

MatousJobanek Jun 24, 2024

sawood14012 Jun 24, 2024

MatousJobanek Jun 25, 2024

MatousJobanek Jun 25, 2024

MatousJobanek Jun 26, 2024 •

edited

Loading

MatousJobanek Jun 26, 2024

MatousJobanek Jun 26, 2024

rsoaresd Jul 24, 2024

MatousJobanek Jul 25, 2024

		@command -v ./go-junit-report >/dev/null 2>&1 \|\| { echo "go-junit-report is not installed. Installing..."; GOBIN=$(PWD) go install github.com/jstemmer/go-junit-report/v2@latest; }
		@echo "go-junit-report version:" && ./go-junit-report -version

		$(MAKE) generate-report REPORT_NAME=${REPORT_NAME} \|\| \
		($(MAKE) print-logs HOST_NS=${HOST_NS} MEMBER_NS=${MEMBER_NS} MEMBER_NS_2=${MEMBER_NS_2} REGISTRATION_SERVICE_NS=${REGISTRATION_SERVICE_NS} && exit 1)

		go test ${TESTS_TO_EXECUTE} -run ${TESTS_RUN_FILTER_REGEXP} -p 1 -parallel ${E2E_PARALLELISM} -v -timeout=90m -failfast 2>&1 \| tee /tmp/test_output.log \|\| ( $(MAKE) print-logs && exit 1 )
		$(MAKE) generate-report REPORT_NAME=${REPORT_NAME}

Core: Generate xUnit Reports for e2e tests #962

Are you sure you want to change the base?

Core: Generate xUnit Reports for e2e tests #962

Conversation

sawood14012 commented May 3, 2024

openshift-ci bot commented May 3, 2024

ranakan19 commented May 3, 2024

openshift-ci bot commented May 15, 2024

mfrancisc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Jun 14, 2024

Quality Gate passed

sawood14012 commented Jun 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MatousJobanek Jun 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MatousJobanek Jun 26, 2024 •

edited

Loading