Abstract

This document describes the formal schema of the Evaluation and Report Language (EARL) 1.0. EARL is a vocabulary, the terms of which are defined across a set of specifications and technical notes, and that is used to describe test results. The primary motivation for developing this vocabulary is to facilitate the exchange of test results between Web accessibility evaluation tools in a vendor-neutral and platform-independent format. It also provides reusable terms for generic quality assurance and validation purposes.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at https://www.w3.org/TR/.

This Evaluation and Report Language (EARL) 1.0 Schema document is published as a W3C Working Group Note because the Evaluation and Repair Tools Working Group (ERT WG) reached the end of its Charter. At this time there are not sufficient implementations to complete EARL 1.0 Schema as a W3C Recommendation.

EARL 1.0 Schema completed Last Call stage before this publication. It has some implementations and may be reused in part or in whole by other W3C groups in the future, such as the Accessibility Conformance Testing (ACT) Task Force.

EARL 1.0 is supported by the following documents:

If you wish to make comments regarding this Evaluation and Report Language (EARL) 1.0 Schema document, please send them to public-earl10-comments@w3.org (publicly visible mailing list archive).

Publication as a Working Group Note does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document has been produced by the Evaluation and Repair Tools Working Group (ERT WG) as part of the Web Accessibility Initiative (WAI) Technical Activity.

This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

This document is governed by the 1 September 2015 W3C Process Document.

1. Introduction

The Evaluation and Report Language (EARL) defines a vocabulary for expressing test results. It enables any person, software application, or organization to assert test results for any test subject tested against any set of criteria. The test subject might be a website, an authoring tool, a user agent, or some other entity. The set of criteria may be accessibility guidelines, formal grammars, or other types of quality assurance requirements. Thus, EARL is flexible with regard to the contexts in which it can be applied.

This document provides the core schema of EARL. Other parts of the EARL suite of specifications include:

The Developer Guide for Evaluation and Report Language (EARL) 1.0 explains how to implement and use EARL, including conformance requirements for software tools. An Evaluation and Report Language (EARL) Overview is also available.

EARL is not a comprehensive vocabulary for describing test procedures, test criteria, or test requirements but, rather, for describing the outcomes from such testing. EARL can be supplemented by test description vocabularies or other vocabularies for different aspects of the testing cycle.

1.1. Audience of this Document

The assumed audience of this specification is developers of software tools and processes who want to express test results in a machine readable format that is semantically rich. More introductory background about EARL as well as specific guidance for quality assurance tool developers, in particular for web accessibility evaluation tool developers, is provided in the Evaluation and Report Language (EARL) 1.0 Guide.

This document assumes that the reader is familiar with the Resource Description Framework (RDF) and can read its XML serialization. Readers who wish to understand more about RDF should read a general introduction or the RDF Primer [RDF-PRIMER].

1.2. Document conventions

The RDF representation of the vocabulary defined by this document uses the namespace http://www.w3.org/ns/earl#. The prefix earl is used throughout this document to denote this namespace. Other prefixes used throughout this document include:

2. Classes

This section describes the classes defined by this document. Every test result in EARL is expressed as an assertion. An EARL Assertion contains the following information:

Assertor
This can include information about who or what ran the test. For example human evaluators, automated accessibility checkers, or combinations of these.
Test Subject
This can include web content (such as web pages, videos, applets, etc.), software (such as authoring tools, user agents, etc.), or other things being tested.
Test Criterion
What are we evaluating the test subject against? This could be a specification, a set of guidelines, a test from a test suite, or some other testable statement.
Test Result
What was the outcome of the test? A test result could also include contextual information such as error messages or relevant locations within the test subject.

EARL provides flexibility to describe different types of assertions, such as those carried out by automated testing tools or by human evaluators, or those made about generic testing requirements or specific test cases.

Examples

Example 1: A person carries out a manual evaluation of a web page to an accessibility requirement.

Assertor
Bob B. Bobbington
Test Subject
A web page located at http://www.example.org/page.html
Test Criterion
Success Criterion 1.1.1 of the Web Content Accessibility Guidelines (WCAG) 2.0
Test Result
Passed

Example 2: A software application carries out automated validation of a web page to a technical specification.

Assertor
The W3C Markup Validator located at http://validator.w3.org/
Test Subject
The XHTML returned from a GET request to the URI http://www.example.org/page.html at 2004-04-14T14:00:04+1000
Test Criterion
The validity of the XHTML code
Test Result
Failed, the <li> element on line 53, char 7 was not closed.

2.1. Assertion Class

Assertion - a statement that embodies the results of a test.

Related Properties

Examples

Example 3: Instance of an assertion expressed as an RDF/XML fragment.

<earl:Assertion rdf:about="#assertion">
  <earl:assertedBy rdf:resource="#assertor"/>
  <earl:subject rdf:resource="http://www.example.org/"/>
  <earl:test rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/H36"/>
  <earl:result rdf:resource="#result"/>
</earl:Assertion>

2.2. Assertor Class

Assertor - an entity such as a person, a software tool, an organization, or any other grouping that carries out a test collectively.

Related Classes

Rather than specifying only an earl:Assertor type, it is recommended that one of the following types be employed in addition:

earl:Software
Software - the assertor is a piece of software.
foaf:Agent external link
Agent - the assertor is an agent, as defined by [FOAF].
foaf:Person external link
Person -the assertor is a person, as defined by [FOAF].
foaf:Organization external link
Organization - the assertor is an organization, as defined by [FOAF].
foaf:Group external link
Group - the assertor is a group of agents, as defined by [FOAF].

Related Properties

It is recommended to provide additional information about the Assertor by using the following properties from external vocabularies:

dct:title external link
Human readable title for the assertor.
dct:description external link
Human readable description of the assertor.
foaf:name external link
Name of the assertor. This could be supplemented with further refinements such as foaf:firstName external link or foaf:surname external link if the assertor is a person.
foaf:nick external link
Nick name of the assertor
foaf:mbox external link
E-mail address of the responsible assertor, which is preferably provided in an encrypted format using the foaf:mbox_sha1sum external link property.
foaf:homepage external link
Homepage of the assertor.
foaf:member external link
Member of the assertor, such as an individual in a group of testers or a tool used by an agent.

Examples

Example 4: An Assertor that is a person called Bob B. Bobbington.

<foaf:Person rdf:about="http://www.example.org/people/#bob">
  <foaf:name>Bob B. Bobbington</foaf:name>
  <foaf:mbox rdf:resource="mailto:bob@example.org"/>
  <foaf:mbox_sha1sum>1a9daad476f0158b81bc66b7b27b438b4b4c19c0</foaf:mbox_sha1sum>
</foaf:Person>

Example 5: An Assertor that is a piece of software called Cool Tool.

<earl:Software rdf:about="http://www.example.org/tools/#cooltool">
  <doap:name xml:lang="en">Cool Tool</doap:name>
  <doap:description xml:lang="en">My favorite tool!</doap:description>
  <doap:created>2011-04-27</doap:created>
  <doap:homepage rdf:resource="http://example.org/tools/cool/"/>
  <doap:release>
    <doap:revision>1.0.3</doap:revision>
  </doap:release>
</earl:Software>

Example 6: An Assertor that is the person from example 4 using the software tool from example 5.

<foaf:Group rdf:about="#assertor">
  <dct:title xml:lang="en">Bob using Cool Tool</dct:title>
  <dct:description xml:lang="en">Bob doing semi-automated testing</dct:description>
  <earl:mainAssertor rdf:resource="http://www.example.org/people/#bob"/>
  <foaf:member rdf:resource="http://www.example.org/tool/#cooltool"/>
</foaf:Group>

Note: According to this example, "Cool Tool" is a resource of type foaf:Agent. According to example 5, it is also a resource of type earl:Software. These are not contradictory statements and are valid RDF representations.

2.3. TestSubject Class

Test Subject - the class of things that have been tested against some test criterion.

Related Classes

Rather than specifying only an earl:TestSubject type, it is recommended that one of the following types be employed in addition:

earl:Software
Software - the test subject is a piece of software being tested.
cnt:Content external link
Content - the test subject is a representation of the content as defined by [Content].
http:Response external link
HTTP Response - the test subject is the response from an HTTP server as defined by [HTTP].
foaf:Document external link
Document - the test subject is a document, such as electronic file, as defined by [FOAF].

Related Properties

It is recommended to provide additional information about the Test Subject by using the following properties from external vocabularies:

dct:title external link
Human readable title for the subject.
dct:description external link
Human readable descriptions of the subject.
dct:date external link
Date on which the subject was created or identified.
dct:hasPart external link
Reference to another subject that are part of this subject.
dct:isPartOf external link
Reference to another subject of which this subject is a part of.

Examples

Example 7: A group of resources that have been tested together as a single test subject.

<earl:TestSubject rdf:about="http://www.example.org/">
  <dct:title xml:lang="en">example.org Web site</dct:title> 
  <dct:description xml:lang="en">Each page on the example.org Web site</dct:description> 
  <dct:hasPart rdf:resource="http://www.example.org/style.css"/> 
  <dct:hasPart rdf:resource="http://www.example.org/page1.html"/> 
  <dct:hasPart rdf:resource="http://www.example.org/page2.html"/> 
  <dct:hasPart rdf:resource="http://www.example.org/image1.png"/> 
  <dct:hasPart rdf:resource="http://www.example.org/image2.png"/> 
</earl:TestSubject>

2.4. TestCriterion Class

Test Criterion - a testable statement, usually one that can be passed or failed. It is a super class for all types of tests including things such as validation requirements, code test cases, checkpoints from guidelines such as Web Content Accessibility Guidelines [WCAG], or others.

Related Classes

Rather than specifying only an earl:TestCriterion type, it is recommended that one of the following types be employed in addition:

earl:TestRequirement
Test Requirement - a higher-level requirement that is tested by executing one or more sub-tests. For example WCAG 2.0 Success Criterion 1.1.1, which is evaluated using several