Validating WCAG versions 1.0 and 2.0 through usability testing with disabled users

Abstract

The Web Content Accessibility Guidelines (WCAG) by the Web Accessibility Initiative (WAI) has become the de-facto standard for accessibility on the Web. WCAG version 1.0 has become significant both as a practical tool and as an academic set of principles and is presently the basis of Web accessibility evaluations and guidelines in many countries. WCAG version 2.0 was released in 2008. This paper reports on a study that empirically validated the usefulness of using WCAG as a heuristic for website accessibility. Through controlled usability tests of two websites with disabled users (N = 7) and a control group (N = 6), it was found that only 27% of the identified website accessibility problems could have been identified through the use of WCAG 1.0. A similar analysis of conformance to WCAG 2.0 showed a marginal 5% improvement concerning identified website accessibility problems. Compensating for the low number of test subjects with confidence tests gave results that were still low (42% for WCAG 1.0 and 49% for WCAG 2.0, with 95% confidence). It is concluded from this that the application of WAI accessibility guidelines is not sufficient to guarantee website accessibility. It is recommended that future versions of the accessibility guidelines should be based on empirical data and validated empirically and that WAI expand their definition of accessibility to include "usability for all" in accordance with ISO 9241-171:2008.