Markel Vigo
ABSTRACT
The use of web accessibility evaluation tools is a widespread practice. Evaluation tools are heavily employed as they help in reducing the burden of identifying accessibility barriers. However, an over-reliance on automated tests often leads to setting aside further testing that entails expert evaluation and user tests. In this paper we empirically show the capabilities of current automated evaluation tools. To do so, we investigate the effectiveness of 6 state-of-the-art tools by analysing their coverage, completeness and correctness with regard to WCAG 2.0 conformance. We corroborate that relying on automated tests alone has negative effects and can have undesirable consequences. Coverage is very narrow as, at most, 50% of the success criteria are covered. Similarly, completeness ranges between 14% and 38%; however, some of the tools that exhibit higher completeness scores produce lower correctness scores (66-71%) due to the fact that catching as many violations as possible can lead to an increase in false positives. Therefore, relying on just automated tests entails that 1 of 2 success criteria will not even be analysed and among those analysed, only 4 out of 10 will be caught at the further risk of generating false positives.
- S. Abou-Zahra. Evaluation and report language (earl) 1.0 schema -- w3c working draft, 2011.Google Scholar
- S. Abou-Zahra and M. Cooper. Wcag 2.0 test samples repository. In Universal Access in Human-Computer Interaction. Applications and Services, volume 5616 of LNCS, pages 619--627. 2009. Google ScholarDigital Library
- A. Aizpurua, M. Arrue, M. Vigo, and J. Abascal. Transition of accessibility evaluation tools to new standards. In Proceedings of the 2009 International Cross-Disciplinary Conference on Web Accessibililty (W4A), W4A '09, pages 36--44, 2009. Google ScholarDigital Library
- Australian Government. Web accessibility national transition strategy available at http://www.finance.gov.au/publications/wcag-2-implementation/docs/wcag-transition-strategy.pdf. 2010.Google Scholar
- G. Brajnik. Comparing accessibility evaluation tools: a method for tool effectiveness. Universal Access in the Information Society, 3:252--263, 2004. Google ScholarDigital Library
- G. Brajnik. Beyond conformance: The role of accessibility evaluation methods. In Proceedings of the 2008 international workshops on Web Information Systems Engineering, WISE '08, pages 63--80, 2008. Google ScholarDigital Library
- G. Brajnik. A comparative test of web accessibility evaluation methods. In Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility, Assets '08, pages 113--120, 2008. Google ScholarDigital Library
- G. Brajnik, A. Mulas, and C. Pitton. Effects of sampling methods on web accessibility evaluations. In Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility, Assets '07, pages 59--66, 2007. Google ScholarDigital Library
- G. Brajnik, Y. Yesilada, and S. Harper. The expertise effect on web accessibility evaluation methods. Human-Computer Interaction, 26(3):246--283, 2011.Google Scholar
- M. Cooper, Q. Limbourg, C. Mariage, and J. Vanderdonckt. Integrating universal design into a global approach for managing very large web sites. In Proceedings of the 5th ERCIM Workshop on User Interfaces for All, 1999.Google Scholar
- C. Farenc, V. Liberati, and M.-F. Barthet. Automatic ergonomic evaluation: What are the limits? In Proceedings of the Second International Workshop on Computer-Aided Design of User Interfaces, CADUI '96, pages 159--170, 1996.Google Scholar
- T. D. Gilbertson and C. H. C. Machin. Guidelines, icons and marketable skills: an accessibility evaluation of 100 web development company homepages. In Proceedings of the International Cross-Disciplinary Conference on Web Accessibility, W4A '12, pages 17:1--17:4, 2012. Google ScholarDigital Library
- B. Kessler, G. Numberg, and H. Schütze. Automatic detection of text genre. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics, ACL '98, pages 32--38. Association for Computational Linguistics, 1997. Google ScholarDigital Library
- M. King, J. W. Thatcher, P. M. Bronstad, and R. Easton. Managing usability for people with disabilities in a large web presence. IBM Systems Journal, 44(3):519--535, 2005. Google ScholarDigital Library
- J. Lazar, A. Dudley-Sponaugle, and K.-D. Greenidge. Improving web accessibility: a study of webmaster perceptions. Computers in Human Behavior, 20(2):269--288, 2004.Google ScholarCross Ref
- R. Lopes, D. Gomes, and L. Carriço. Web not for all: a large scale study of web accessibility. In Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A), W4A '10, pages 10:1--10:4, 2010. Google ScholarDigital Library
- J. Mankoff, H. Fait, and T. Tran. Is your web page accessible?: a comparative study of methods for assessing web page accessibility for the blind. In Proceedings of the SIGCHI conference on Human factors in computing systems, CHI '05, pages 41--50, 2005. Google ScholarDigital Library
- B. Martins and M. J. Silva. Language identification in web pages. In Proceedings of the 2005 ACM symposium on Applied computing, SAC '05, pages 764--768, 2005. Google ScholarDigital Library
- C. Power, A. Freire, H. Petrie, and D. Swallow. Guidelines are only half of the story: accessibility problems encountered by blind users on the web. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, CHI '12, pages 433--442, 2012. Google ScholarDigital Library
- D. Sato, H. Takagi, M. Kobayashi, S. Kawanaka, and C. Asakawa. Exploratory analysis of collaborative web accessibility improvement. ACM Transactions on Accessible Computing, 3(2):5:1--5:30, 2010. Google ScholarDigital Library
- C. C. Shelly and M. Barta. Application of traditional software testing methodologies to web accessibility. In Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A), W4A '10, pages 11:1--11:4, 2010. Google ScholarDigital Library
- D. Sloan, A. Heath, F. Hamilton, B. Kelly, H. Petrie, and L. Phipps. Contextual web accessibility - maximizing the benefit of accessibility guidelines. In Proceedings of the 2006 international cross-disciplinary workshop on Web accessibility (W4A), W4A '06, pages 121--131, 2006. Google ScholarDigital Library
- C. Strobbe, J. Koch, E. Vlachogiannis, R. Ruemer, C. Velasco, and J. Engelen. The bentoweb test case suites for the web content accessibility guidelines (wcag) 2.0. In Computers Helping People with Special Needs, volume 5105 of LNCS, pages 402--409. 2008. Google ScholarDigital Library
- H. Takagi, C. Asakawa, K. Fukuda, and J. Maeda. Accessibility designer: visualizing usability for the blind. In Proceedings of the 6th international ACM SIGACCESS conference on Computers and accessibility, Assets '04, pages 177--184, 2004. Google ScholarDigital Library
- M. Vigo and G. Brajnik. Automatic web accessibility metrics: Where we are and where we can go. Interacting with Computers, 23(2):137--155, 2011. Google ScholarDigital Library
- W3C-WAI. Conformance evaluation of web sites for accessibility. 2005.Google Scholar
- R. Walpole. Elementary statistical concepts. Macmillan, 1976.Google Scholar
Index Terms
- Benchmarking web accessibility evaluation tools: measuring the harm of sole reliance on automated tests
Recommendations
Interdependent components of web accessibility
W4A '05: Proceedings of the 2005 International Cross-Disciplinary Workshop on Web Accessibility (W4A)Increasingly, the Web is providing unprecedented access to information and interaction for people with disabilities. However, the Web will not be equally accessible, allowing people with disabilities to access and contribute to the Web, until:• ...
Web Accessibility Evaluation Tools: A Survey and Some Improvements
Web Content Accessibility Guidelines (WCAG) from W3C consist of a set of 65 checkpoints or specifications that Web pages should accomplish in order to be accessible to people with disabilities or using alternative browsers. Many of these 65 checkpoints ...
Evaluation of authoring tools under ATAG and WCAG recommendations
AbstractAuthoring tools are software programs that allow users to create learning content, lessons, and courses, usually to be used in virtual learning environments. These tools are a reasonable choice for novice users, as they use pre-programmed elements ...
Comments