Skip to content

EARL results

Jump to bottom Edit New page
rjmartell edited this page Jun 16, 2016 · 26 revisions

Sample EARL results

Listing 1 shows the basic elements of EARL test results using the RDF/XML syntax.

Listing 1

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:earl="http://www.w3.org/ns/earl#" 
         xmlns:http="http://www.w3.org/2011/http#"
         xmlns:ptr="http://www.w3.org/2009/pointers#" 
         xmlns:dc="http://purl.org/dc/terms/">

  <earl:Assertor rdf:about="http://tes.example.org/">
    <dc:title xml:lang="en">TES</dc:title>
    <dc:description xml:lang="en">A test execution service that runs conformance test suites.</dc:description>
  </earl:Assertor>

  <earl:TestSubject rdf:about="https://developers.google.com/kml/documentation/KML_Samples.kml">
    <dc:title xml:lang="en">KML Samples</dc:title>
    <dc:description xml:lang="en">Examples intended to support the KML tutorial.</dc:description>
  </earl:TestSubject>

  <earl:TestRequirement rdf:about="http://www.opengis.net/spec/KML/2.3/conf/level-1">
    <dc:title xml:lang="en">KML 2.3 - Conformance Level 1</dc:title>
    <dc:description xml:lang="en">Conformance Level 1 includes test cases that address 
    absolute requirements. A KML document must satisfy all assertions at this level to 
    achieve minimal conformance</dc:description>
    <dc:isPartOf rdf:resource="http://docs.opengeospatial.org/ts/14-068r2/14-068r2.html"/>
  </earl:TestRequirement>

  <earl:TestRequirement rdf:about="http://www.opengis.net/spec/KML/2.3/conf/level-2">
    <dc:title xml:lang="en">KML 2.3 - Conformance Level 2</dc:title>
    <dc:description xml:lang="en">Includes all tests in Level 1, plus test cases covering 
    requirements that should be satisfied by a KML document. Non-conformance at this 
    level may hinder the utility, portability, or interoperability of the document.</dc:description>
    <dc:hasPart rdf:resource="http://www.opengis.net/spec/KML/2.3/conf/level-1"/>
  </earl:TestRequirement>

  <earl:Assertion rdf:ID="assert-atc-101">
    <earl:subject rdf:resource="http://www.example.org/placemark1.kml"/>
    <earl:assertedBy rdf:resource="http://tes.example.org/"/>
    <earl:mode rdf:resource="http://www.w3.org/ns/earl#automatic"/>
    <earl:test>
      <earl:TestCase rdf:about="http://www.opengis.net/spec/KML/2.3/conf/level-1/atc-101">
        <dc:title xml:lang="en">Document element</dc:title>
        <dc:isPartOf rdf:resource="http://www.opengis.net/spec/KML/2.3/conf/level-1"/>
      </earl:TestCase>
    </earl:test>
    <earl:result>
      <earl:TestResult rdf:ID="result-atc-101">
        <earl:outcome rdf:resource="http://www.w3.org/ns/earl#fail"/>
        <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2015-09-29T11:50:00Z</dc:date>
        <dc:title xml:lang="en">XML Schema validation errors found: 2</dc:title>
        <dc:description xml:lang="en" rdf:parseType="Literal">
          <ul xmlns="http://www.w3.org/1999/xhtml">
            <li>Line 5 - cvc-datatype-valid.1.2.1: 'TRUE' is not a valid value for 'boolean'.</li>
            <li>Line 5 - cvc-type.3.1.3: The value 'TRUE' of element 'kml:visibility' is not
              valid.</li>
          </ul>
        </dc:description>
        <earl:pointer>
          <ptr:PointerCollection>
            <ptr:xpointer>element(/1/1/2)</ptr:xpointer>
          </ptr:PointerCollection>
        </earl:pointer>
        <earl:info rdf:parseType="Literal" xml:lang="en">
          <test-method status="FAIL" signature="validate()" name="validate" duration-ms="47"
            started-at="2015-09-29T11:50:00Z" finished-at="2015-09-29T11:50:00Z">
            <exception class="java.lang.AssertionError">
              <message>
                <![CDATA[Total validation errors found: 2]]>
              </message>
            </exception>
          </test-method>
        </earl:info>
      </earl:TestResult>
    </earl:result>
  </earl:Assertion>
</rdf:RDF>

Notes

  • conformance classes/levels are denoted by earl:TestRequirement statements.
  • the earl:Software class can describe either an assertor (e.g. TE v4.3), or a test subject.
  • each earl:TestCase is linked to (dc:isPartOf) the TestRequirement to which it belongs. Better to link via dc:hasPart. For example when defining level 2, you can relate to level 1 via dc:hasPart (note: relatively few conformance classes constitute a 'level' that includes some other class--most are independent).
  • include request/response message content for failing tests

From ISO 19105, A.3:

"A conformance level is a special kind of conformance class in which requirements of a higher level contain all the requirements of the lower levels."

For example, Level 2 contains Level 1 (i.e. L3 {dc:hasPart} L2 {dc:hasPart} L1)

Listing 2 shows the output produced by the EarlReportListener under development (in branch task/152). Each earl:TestCase belongs to a higher-level earl:TestRequirement that represents a conformance class.

Listing 2

<rdf:RDF
    xmlns:dct="http://purl.org/dc/terms/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:earl="http://www.w3.org/ns/earl#"
    xml:base="http://example.org/earl/">
  <earl:Assertion rdf:about="assert-1466098100416">
    <earl:test>
      <earl:TestCase rdf:about="org/opengis/cite/abc10/level1/Capability1Tests#isEmpty">
        <dct:isPartOf>
          <earl:TestRequirement rdf:about="Conformance-Level-1">
            <dct:title>Conformance Level 1</dct:title>
          </earl:TestRequirement>
        </dct:isPartOf>
        <dct:description>Implements ATC 1-1</dct:description>
        <dct:title>isEmpty</dct:title>
      </earl:TestCase>
    </earl:test>
    <earl:result>
      <earl:TestResult rdf:about="result-1466098100416">
        <earl:outcome rdf:resource="http://www.w3.org/ns/earl#failed"/>
        <dct:description>Expected empty string. expected [true] but found [false]</dct:description>
        <dct:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime"
        >2016-06-16T17:28:20.416Z</dct:date>
      </earl:TestResult>
    </earl:result>
    <earl:subject>
      <earl:TestSubject rdf:about="file:/W:/tmp/ets-abc10/target/test-classes/atom-feed-2.xml"/>
    </earl:subject>
    <earl:assertedBy>
      <earl:Assertor rdf:about="https://github.com/opengeospatial/teamengine">
        <dct:description xml:lang="en">Official test harness of the OGC conformance testing program (CITE).</dct:description>
        <dct:title xml:lang="en">OGC TEAM Engine</dct:title>
      </earl:Assertor>
    </earl:assertedBy>
    <earl:mode rdf:resource="http://www.w3.org/ns/earl#automatic"/>
  </earl:Assertion>
  <earl:Assertion rdf:about="assert-1466098100417">
    <earl:test>
      <earl:TestCase rdf:about="org/opengis/cite/abc10/level1/Capability1Tests#trim">
        <dct:isPartOf rdf:resource="Conformance-Level-1"/>
        <dct:description>Implements ATC 1-2</dct:description>
        <dct:title>trim</dct:title>
      </earl:TestCase>
    </earl:test>
    <earl:result>
      <earl:TestResult rdf:about="result-1466098100417">
        <earl:outcome rdf:resource="http://www.w3.org/ns/earl#passed"/>
        <dct:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime"
        >2016-06-16T17:28:20.417Z</dct:date>
      </earl:TestResult>
    </earl:result>
    <earl:subject rdf:resource="file:/W:/tmp/ets-abc10/target/test-classes/atom-feed-2.xml"/>
    <earl:assertedBy rdf:resource="https://github.com/opengeospatial/teamengine"/>
    <earl:mode rdf:resource="http://www.w3.org/ns/earl#automatic"/>
  </earl:Assertion>
  <earl:Assertion rdf:about="assert-1466098100413">
    <earl:test>
      <earl:TestCase rdf:about="org/opengis/cite/abc10/level1/Capability1Tests#docIsValidAtomFeed">
        <dct:isPartOf rdf:resource="Conformance-Level-1"/>
        <dct:description>Implements ATC 1-3</dct:description>
        <dct:title>docIsValidAtomFeed</dct:title>
      </earl:TestCase>
    </earl:test>
    <earl:result>
      <earl:TestResult rdf:about="result-1466098100413">
        <earl:outcome rdf:resource="http://www.w3.org/ns/earl#failed"/>
        <dct:description>1 schema validation error(s) detected.
Severity: ERROR
Message: element "Feed" not allowed here; expected element "entry" or "feed"
Location:  line=1 column=121 expected [false] but found [true]</dct:description>
        <dct:date rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime"
        >2016-06-16T17:28:20.413Z</dct:date>
      </earl:TestResult>
    </earl:result>
    <earl:subject rdf:resource="file:/W:/tmp/ets-abc10/target/test-classes/atom-feed-2.xml"/>
    <earl:assertedBy rdf:resource="https://github.com/opengeospatial/teamengine"/>
    <earl:mode rdf:resource="http://www.w3.org/ns/earl#automatic"/>
  </earl:Assertion>
</rdf:RDF>

References

Add a custom footer
Add a custom sidebar
Clone this wiki locally