Involving Users in Evaluating Web Accessibility

Introduction

Web accessibility evaluation often focuses on conformance to accessibility standards such as WCAG. While conformance is important, there are many benefits to evaluating with real people. For example, to learn how your website or web tool really works for users and to better understand accessibility issues. Evaluating with users with disabilities and with older users identifies usability issues that are not discovered by conformance evaluation alone.

This page is part of a multi-page Evaluating Web Accessibility resource suite that outlines different aspects of evaluating web accessibility. It is the second of two pages on including users in web projects.

Please read Involving Users in Web Projects for Better, Easier Accessibility, which covers broader issues of including users early in website design, tool development, standards, and other web projects.

Initial Review

A first step in evaluating web accessibility is reviewing the website to check for any obvious accessibility problems. For example, some checks are in Easy Checks – A First Review of Web Accessibility.

Even web developers with little accessibility knowledge can find some accessibility issues through an initial review. An accessibility expert with first-hand experience of how people with different disabilities interact with the web can:

• evaluate accessibility issues for a broad range of users, which might not be found by individual users
• help fix any known barriers before bringing in users
• focus the evaluation with users on potential areas of concern

The initial review identifies any significant accessibility barriers to fix before evaluating with users. It also helps define what to focus on for evaluation with users.

Range of User Evaluation

Users with disabilities and older users can be included in a wide range of evaluation activities, from brief consultations to large-scale usability studies. There are many options in between these extremes:

• Informal evaluations of specific accessibility issues can be very simple. For example, ask someone you know who uses a screen reader or someone with other disabilities to find some data in an early draft of a data table. Observe their interaction and discuss the issues.
• Formal usability testing of a website follows established protocols to gather quantitative and qualitative data from representative users performing specific tasks. Formal usability tests can be optimized to focus on accessibility issues.

What type of evaluation you do depends on factors such as the stage in your project, for example, initial investigation of design ideas, testing specific areas of prototypes, or reviewing final designs.

Conducting informal evaluations throughout development is more effective than only formal usability testing at the end of a project.

Basics

In most cases, including users in evaluation involves:

• getting a few people with disabilities - and depending on your target audience, older users
• including them throughout the development process to complete sample tasks on prototypes so you can see how the different aspects of the design and coding could be improved
• discussing accessibility issues with them

See Involving Users in Web Projects for Better, Easier Accessibility for guidance on getting a range of users and users’ experience interacting with the web

Just as with any evaluation with users, whether you include novice, average, or advanced users depends on your target users. For example, if you are developing a web application for accountants inside a company, you probably want advanced assistive technology users. And for a public website to apply for disability benefits, you want novice assistive technology users.

Caution: Carefully consider all input. Avoid assuming that input from one person with a disability applies to all people with disabilities.

A person with a disability does not necessarily know how other people with the same disability interact with the web. They might not know enough about other disabilities to provide valid guidance on other accessibility issues. Getting input from a range of users is best.

Note: In addition to finding accessibility problems, evaluating with users with disabilities usually reveals general usability problems that impact all users, including those without disabilities.

Analyzing Accessibility Issues

Web accessibility depends on several components of web development and interaction working together, including web browsers, assistive technologies (AT), and web content.

Accessibility problems can be caused by one or more different components. For example, if a user who cannot use a mouse has trouble with keyboard access, it could be because:

• the developer did not markup/code the web page properly, or
• the browser or media player isn’t handling the markup properly, or
• the user’s AT isn’t handling the markup properly, or
• the user doesn’t know how to use the browser, media player, or AT’s keyboard access features, or
• the page is poorly designed and it is a general usability problem for all users, including those without disabilities

Combine User Evaluation with Standards

Involving users with disabilities in evaluation has many benefits. Yet that alone cannot determine if a website is accessible. Combine user involvement with evaluating conformance to WCAG to ensure that accessibility is provided to users with a range of disabilities and situations.

Drawing Conclusions and Reporting

Be careful drawing conclusions from limited evaluations or studies. Results from only a couple of people with disabilities cannot be generalized to apply to all people with similar disabilities or people with other disabilities. See the