Reporting for TEAM Engine 5.0

A solution for improving reporting in TEAM ENGINE

Reporting is expected to be one the major changes in TEAM Engine 5.0. Improving the reporting will allow users to better understand the errors and possible fixes. The report should contain the following information:

General Information about the test:

• Name of the test
• Revision of the test
• Version of TEAM Engine use to run the test
• Date and Time

Information about the resource being tested

• Test artifact (URL end of point of the server, name of the file, etc.)

Tests and Result

• Show final result of the test
• Show first what is the "basic" conformance class.
• About conformance classes
  • show the name of the conformance class.
  • show by conformance classes how many tests passed, failed or skipped
  • link to the conformance class to get more details about the test and direct reference to the specification Annex A or B, where the conformance classes were declared.

In the case of failure

• Show the input (e.g. request performed by the server, parameters, etc.)
• Show the output (e.g. response made by the server)
• Show the error
• Explain the error
• Provide list of suggestions

TestNG and CTL:

• The report should be the same for all type of tests (TestNG or CTL)
• The report should have the same output in XML when the test is run via:
  • command line
  • rest API
  • web user interface

Example

Approaches:

Convert results to EARL

Results of TestNG and CTL can be converted to EARL: http://www.w3.org/TR/EARL10-Schema/.

Convert CTL to TestNG

For the purpose of creating better and generic reporting for all test suites, some of the three reports should be used: CTL test, client test or TestNG test. TestNG seems the easiest path. If all the tests are transformed to TestNG results a transformer can then be used to converted it to HTML.

The major steps to be done in coding perspective:

1. Convert CTL test result and Client test result in TestNg format

   i. Create a Java code snippet in TECore.executeTest(TestEntry test, XdmNode params, XPathContext context) method which stores CTL test and client test result necessary information in Json format.

   ii. After completion of the test, then we use that Json result and convert it to TestNg project XML format, this thing run after the complete suite stop. This operation perform from TECore.execute_suite(String suiteName, List params)

2. Then write a code snippet (XslHtmlTransformer) in service provider which take input as an XML (final result) and transform the result in form of HTML using xslt stylesheet we create an html file in user log directory.

For diagram click here

From the above diagram steps can be listed as:

1. Get result from ctl/client test in form of json.
2. Convert that json result in form of xml which look same as testng result
3. Parse this xml result with xslt sheet (which is written in TEAMEngine service provider).
4. Generate another xml result which contain all information including all html tag.
5. Save this data into user log directory.

By this approach we get following advantage:

● When we ran the TestNgest via REST API at that time result comes in the form of XML so if we pass that XML result to new snippet which written in service provider part it converts it into an html.