skip to main content
10.1145/2899475.2899484acmotherconferencesArticle/Chapter ViewAbstractPublication Pagesw4aConference Proceedingsconference-collections
research-article

Measuring the impact of automated evaluation tools on alternative text quality: a web translation study

Published:11 April 2016Publication History

ABSTRACT

The number of Internet users has increased tenfold since the beginning of the century up to present, especially thanks to the improvements experienced in web accessibility and the growing number of languages which online content is available in. While translation professionals are making a considerable contribution to that digital information richness, little evidence exists regarding their involvement in the achievement of a more accessible web for all. In this paper, we present the main results of the first empirical study on web accessibility conceived around a translation task. The experiment sought to particularly investigate the quality of image text alternatives produced by French translators with the help of two evaluation tools: aDesigner and Acrolinx. The assessment of their alt text proposals, carried out by seven screen reader users, suggests that using both tools helps translators to create more appropriate text alternatives than when trying to do so with only one tool or without any automated support. A more in-depth analysis of the data gathered shows that Acrolinx offers better guidance than aDesigner for translators to render images accessible.

References

  1. von Ahn, L. and Dabbish, L. 2004. Labeling Images with a Computer Game. In Proceedings of CHI 2004, Vienna, Austria, April 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. von Ahn, L., Ginosar, S., Kedia, M., Liu, R. and Blum, M. 2006. Improving Accessibility of the Web with a Computer Game. In Proceedings of CHI 2006, Montréal, Québec, Canada, April 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Asakawa, C. 2005. What's the Web Like if You Can'T See It? In Proceedings of W4A 2005. Chiba, Japan, May 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Bigham, J., Kaminsky, R., Ladner, R., Danielsson, O. and Hempton, G. 2006. WebInSight: Making Images Accessible. In Proceedings of ASSETS 2006, Portland, Oregon, USA, October 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Bigham, J. 2007. Increasing Web Accessibility by Automatically Judging Alternative Text Quality. In Proceedings of IUI 2007. Honolulu, USA, January 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Brajnik, G. 2004. Comparing accessibility evaluation tools: a method for tool effectiveness. Universal Access in the Information Society. 3 (3-4), 252--263. Springer, Berlin. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Bredenkamp, A., Crysmann, B. and Petrea, M. 2000. Looking for Errors: A Declarative Formalism for Resource-Adaptive Language Checking. In Proceedings of LREC 2000, Athens, Greece, May 2000.Google ScholarGoogle Scholar
  8. Caldwell, B., Cooper, M., Reid, L. and Vanderheiden, G. (eds). 2008. Web Content Accessibility Guidelines 2.0. World Wide Web Consortium (W3C) Recommendation. http://www.w3.org/TR/WCAG20/.Google ScholarGoogle Scholar
  9. Craven, T. 2006. Some features of alt texts associated with images in Web pages. Information Research, 11(2).Google ScholarGoogle Scholar
  10. Debove, A., Furlan, S. and Depraetere, I. 2011. A Contrastive Analysis of Five Automated QA Tools. In Depraetere, I. (ed). Perspectives on Translation Quality, 161--92. Text, Translation, Computational Processing (TTCP) 9. De Gruyter Mouton, Germany.Google ScholarGoogle Scholar
  11. Faulkner, S. (ed). 2014. HTML5: Techniques for providing useful text alternatives. (W3C) Working Draft. http://www.w3.org/TR/html-alt-techniques/Google ScholarGoogle Scholar
  12. Fischer, D. and Wyatt, T. 2011. The case for a WCAG-based evaluation scheme with a graded rating scale. In Proceedings of the W3C WAI Symposium on Website Accessibility Metrics. Article 7. http://www.w3.org/WAI/RD/2011/metrics/paper7/.Google ScholarGoogle Scholar
  13. Folaron, D. 2012. Digitalizing translation. Translation Spaces. John Benjamins, 1 (2012), 5--31.Google ScholarGoogle ScholarCross RefCross Ref
  14. Harper, S. and Chen, A. 2012. Web Accessibility Guidelines: A Lesson from the Evolving Web. World Wide Web. 15 (1): 61--88. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Hu, J. and Bagga, A. 2003. Identifying Story and Preview Images in News Web Pages. In Proceedings of ICDAR 2003, Edinburgh, Scotland, UK, August 2003. IEEE Computer Society Washington, DC, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. ISO/EC TS 20071-11:2012. Guidance for alternative text for images. 2012. Switzerland: International Organization for Standardization (ISO) and International Electrotechnical Commission (IEC).Google ScholarGoogle Scholar
  17. Ivory, M. and Hearst, M. 2001. The State of the Art in Automating Usability Evaluation. ACM Computing Surveys, 33(4), December 2001, 470--516. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Keysers, D., Renn, M. and Breuel, T. 2007. Improving Accessibility of HTML Documents by Generating Image-Tags in a Proxy. In Proceedings of ASSETS 2007, Tempe, Arizona, USA, October 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Korpela, J. 2010. Guidelines on alt texts in IMG elements. https://www.cs.tut.fi/~jkorpela/html/alt.html.Google ScholarGoogle Scholar
  20. Lazar, J., Dudley-Sponaugle, A., and Greenidge, K. 2004. Improving Web Accessibility: A Study of Webmaster Perceptions. The Compass of Human-Computer Interaction 20 (2): 269--88.Google ScholarGoogle Scholar
  21. Nyberg, E., Mitamura, T. and Olaf-Huijsen, W. 2003. Controlled Language for Authoring and Translation. In Somers, H. (ed), Computers and Translation. A Translator's Guide, 245--81. John Benjamins.Google ScholarGoogle Scholar
  22. O'Brien, S. 2012. Translation as Human-Computer Interaction. Translation Spaces, John Benjamins, 1 (2012), 101--122.Google ScholarGoogle ScholarCross RefCross Ref
  23. O'Brien, S., and Roturier, J. 2007. How Portable Are Controlled Language Rules? A Comparison of Two Empirical MT Studies. In Proceedings of MT Summit XI, 345--52. Copenhagen, Denmark. September, 2007.Google ScholarGoogle Scholar
  24. Olsen, M., Snaprud, M. and Nietzio, A. 2010. Automatic Checking of Alternative Text on Web Pages. In Miesenberger et al. (eds), ICCHP 2010, Part I, 425--432. Springer, Berlin. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Petrie, H., Harrison, C. and Dev, S. 2005. Describing images on the Web: a survey of current practice and prospects for the future. In Proceedings of HCII 2005, Las Vegas, Nevada, USA, July 2005.Google ScholarGoogle Scholar
  26. Petrie, H., Power, C., Swallow, D., Velasco, C. A., Gallagher, B., Magennis, M., Murphy, E., Collin, S. and Down, K. 2011. The value chain for web accessibility: challenges and opportunities. In Proceedings of ADDW 2011, Sun SITE Central Europe.Google ScholarGoogle Scholar
  27. R Core Team. 2015. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/.Google ScholarGoogle Scholar
  28. Richards, J., Montague, K. and Hanson, V. 2012. Web Accessibility as a Side Effect. In Proceedings of ASSETS 2012, Boulder, Colorado, USA, October 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Rodríguez Vázquez, S. 2015. Unlocking the Potential of Web Localizers as Contributors to Image Accessibility: What Do Evaluation Tools Have to Offer? In Proceedings of W4A 2015, Florence, Italy, May 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Rodríguez Vázquez, S. 2015. Exploring Current Accessibility Challenges in the Multilingual Web for Visually-Impaired Users. In The 24th World Wide Web (WWW) Conference 2015 Companion Volume. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Rodríguez Vázquez, S. 2015. A controlled language-based evaluation approach to ensure image accessibility during web localisation. Translation Spaces. John Benjamins. 4 (2): 187--215.Google ScholarGoogle ScholarCross RefCross Ref
  32. Rodríguez Vázquez, S. and Bolfing, A. 2013. Multilingual Website Assessment for Accessibility: a Survey on Current Practices. In Proceedings of ASSETS 2013, Bellevue, WA, USA, October 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Rodríguez Vázquez, S., Bouillon, P. and Bolfing, A. 2014. Applying Accessibility-Oriented Controlled Language (CL) Rules to Improve Appropriateness of Text Alternatives for Images: An Exploratory Study. In Proceedings of LREC 2014, Reykjavik, Iceland, May 2014.Google ScholarGoogle Scholar
  34. Rodríguez Vázquez, S. and Lehmann, S. 2015. Acrolinx: a Controlled-Language Checker Turned into an Accessibility Evaluation Tool for Image Text Alternatives. In Proceedings of W4A 2015, Florence, Italy, May 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Rösener, C. 2010. Computational Linguistics in the Translator's Workflow--Combining Authoring Tools and Translation Memory Systems. In Proceedings of the NAACL HLT 2010 Workshop on Computational Linguistics and Writing, Los Angeles, California, June 2010. ACL, 1--6. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Roturier, J. 2006. An Investigation into the Impact of Controlled English Rules on the Comprehensibility, Usefulness and Acceptability of Machine-Translated Technical Documentation for French and German Users. PhD Thesis. Dublin City University, Ireland.Google ScholarGoogle Scholar
  37. Sandrini, P. 2008. Localization and Translation. MuTra Journal, 2 (2008), 167--191.Google ScholarGoogle Scholar
  38. Sullivan, T., and Matson, R. 2000. Barriers to use: usability and content accessibility on the Web's most popular sites. In Proceedings of the ACM Conference on Universal Usability (CUU'00). Arlington, USA, November 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Tang, L. 2012. Producing informative text alternatives for images. PhD thesis. University of Saskatchewan, Saskatoon. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Velasco, C. and Abou-Zahra, S. (eds). 2014. Developers' Guide to Features of Web Accessibility Evaluation Tools. W3C First Public Working Draft.Google ScholarGoogle Scholar
  41. Vigo, M. and Brajnik, G. 2011. Automatic web accessibility metrics: where we are and where we can go. Interacting with Computers. Elsevier, 23(2), 127--155. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Vigo, M., Brown, J. and Conway, V. 2013. Benchmarking Web Accessibility Evaluation Tools: Measuring the Harm of Sole Reliance on Automated Tests. In Proceedings of W4A 2013, Rio de Janeiro, Brazil, May 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Vinyals, O., Toshev, A., Bengio, S. and Ehran, D. 2014. Show and Tell: A Neural Image Caption Generator. CoRR, abs/1411.4555. http://arxiv.org/abs/1411.4555.Google ScholarGoogle Scholar
  44. Web Accessibility in Mind, WebAIM. 2013. Alternative Text. http://webaim.org/techniques/alttext/Google ScholarGoogle Scholar
  45. Yesilada, Y., Brajnik, G., Vigo, M. and Harper, S. 2015. Exploring perceptions of web accessibility: a survey approach. Behaviour & Information Technology. 34:2, 119--134. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Measuring the impact of automated evaluation tools on alternative text quality: a web translation study

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in
        • Published in

          cover image ACM Other conferences
          W4A '16: Proceedings of the 13th International Web for All Conference
          April 2016
          223 pages
          ISBN:9781450341387
          DOI:10.1145/2899475

          Copyright © 2016 ACM

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 11 April 2016

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article

          Acceptance Rates

          Overall Acceptance Rate171of371submissions,46%

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader