Allure test report for Dart and Flutter with Patrol

[pandoras-toolbox]

There is a great e2e test reporting tool called "Allure", see the demo report.

It has no support for Dart, but they would be glad if someone contributes it. I am starting to learn Dart, so I cannot contribute that, it will take a long time until I become good enough for that.

I know it is not Patrol related, but you guys are very proactive, maybe someone of you can add support for the Allure test report?

[bartekpacia]

Hi @pandoras-toolbox! Thanks for this question – we know about Allure and have been experimenting with it. It's a great tool.

Our team member @fylyppo can share more about this :)

[fylyppo]

Hi @pandoras-toolbox, from Patrol 2.0 we are highly "native-friendly" and after patrol test on both platforms you can see native test reports.

iOS .xcresult is placed ~/Library/Developer/Xcode/DerivedData/**/Logs/Test/*.xcresult
Android JUnit .xml is placed in your flutter project /build/app/outputs/androidTest-results/connected/*.xml

And of course if your running your tests on some device farms they're also provide you some reports (mainly in xml).

Recently I was testing Allure and it just works for Android and patrol but there's no test body and stacktrace. Check below 👇

And for iOS there's no official support but I found a nice package for making it compatible with patrol called allure-xcresult and it works perfect! 👇

I'll dive into our JUnit XML in hope to find a solution to provide more info to allure reports so I'll keep you posted.

Also feel free to suggest more tools you would like to use with patrol 🙂

[pandoras-toolbox]

I have a Java backgroung where as I understand Allure has interceptors for test frameworks (I use JUnit 5 in Java) which you have to add as dependencies to your project. For its annotations like @step it uses aspects to intercept.

If possible I would suggest to skip supporting JUnit 4 because it is very outdated and to support just JUnit 5.

Allure creates during the test run its own output data which is the raw input for generating the Allure report. But maybe it also needs the XML file for something because there are some modules for this on Github. Dmitry Baev from Allure should know all the details. Maybe Allure reuses the XML file and adds some more information for its report, but to use the XML file as the only source for the Allure report would be wrong I think.

I do not think that it is the right way just to convert a test tool's output file into Allure format like "allure-xcresult" is doing. Then a lot of things would be missing, like the methods annotated with @step, test structure when using @Epic or @story, it would not have any attachments or links in the Allure report. The environment feature of Allure on its start page is useful to include if it was run on dev, test or pre-prod environment for instance, with which browser or device, which version of the app and so on. That all would be missing if only a test result file is converted to Allure I think. I think "allure-xcresult" is doing it the wrong way.

I do not know how you managed to create a Allure test report for Patrol and Android, so I can only speculate a bit. The body is maybe missing because nothing exists which intercepts the test execution, nothing creates Allure raw data on the fly for test steps and exceptions and for other things like attachments (for example screenshots), severity and links, which can also be in the body section.

The standard XML report was designed more for unit tests and lacks a lot of information needed for end-to-end test automation. JUnit 5 has introduced a own output format a while ago, but I do not now if that is relevant for Allure.

I am not sure if anything of that is helpful. I would suggest to contact Dmitry Baev who seems to be the long-time lead developer of Allure. If I can help with anything just let me know.

[bartekpacia]

Thanks @pandoras-toolbox for chipping in and sharing lots of interesting info.

If possible I would suggest to skip supporting JUnit 4 because it is very outdated and to support just JUnit 5.

Unfortunately, it's not that easy, because we're restricted by Android itself. It doesn't have official support for JUnit 5 (link 1, link 2). There's an unofficial library but so far we haven't had any time to research it.

I have a Java backgroung where as I understand Allure has interceptors for test frameworks (I use JUnit 5 in Java) which you have to add as dependencies to your project. For its annotations like https://github.com/step it uses aspects to intercept.

I didn't know that Allure has additional libraries (e.g. allure-java you mentioned). That's cool. The problem is that there's no Dart x Allure integration, so we're blocked right from the start.

The standard XML report was designed more for unit tests and lacks a lot of information needed for end-to-end test automation. JUnit 5 has introduced a own output format a while ago, but I do not now if that is relevant for Allure.

Thanks, we didn't know about this as well.

I am not sure if anything of that is helpful. I would suggest to contact Dmitry Baev who seems to be the long-time lead developer of Allure. If I can help with anything just let me know.

Very, very helpful. Thank you!

[pandoras-toolbox]

I did not know that Android still uses officially JUnit 4. I would then stick to it because for Patrol there would be no benefit with JUnit 5 I think. I thought it is outdated, but if it is official then it is another situation.

Dmitry Baev confirmed my assessment. Simply converting a report to Allure would only beautify it but not use the Allure features which go much beyond that. Maybe that is good enough for the start.

I might contribute such a Dart/Allure integration some time in the future if until then noone has has done it.

[bartekpacia]

Hi @pandoras-toolbox, we've made a small spike to better support Allure in Patrol. Here's the result: https://patrol.leancode.co/integrations/allure.

[timurTages]

please give an example for installation allure