Christian Monrad Nielsen
ABSTRACT
The distinction between field and laboratory is classical in research methodology. In human-computer interaction, and in usability evaluation in particular, it has been a controversial topic for several years. The advent of mobile devices has revived this topic. Empirical studies that compare evaluations in the two settings are beginning to appear, but they provide very different results. This paper presents results from an experimental comparison of a field-based and a lab-based usability evaluation of a mobile system. The two evaluations were conducted in exactly the same way. The conclusion is that it is definitely worth the hassle to conduct usability evaluations in the field. In the field-based evaluation we identified significantly more usability problems and this setting revealed problems with interaction style and cognitive load that were not identified in the laboratory.
- Abowd, G. and Mynatt, E. (2000) Charting past, present and future research in ubiquitous computing. ACM Transactions on Computer-Human Interaction, 7(1):29--58. Google Scholar
Digital Library
- Baillie, L. (2003) Future Telecommunication: Exploring actual use, In Proceedings of IFIP TC13 International Conference on Human-Computer Interaction, (INTERACT '03). IOS Press.Google Scholar
- Bohnenberger, T., Jameson, A., Krüger, A., and Butz, A. (2002) Location-Aware Shopping Assistance: Evaluation of a Decision-Theoretic Approach. In Proceedings of Mobile HCI 2002. Springer-Verlag, LNCS. Google ScholarDigital Library
- Brewster S. (2002) Overcoming the Lack of Screen Space on Mobile Computers. Personal and Ubiquitous Computing, 6, 188--205 Google ScholarDigital Library
- Dix, A., Finlay, J., Abowd, G. and Beale, R. (1998) Human-Computer Interaction, Prentice Hall Europe, Second Edition. Google ScholarDigital Library
- Esbjörnsson M., Juhlin O. and Östergren M. (2003) Motorcyclists Using Hocman Field Trials on Mobile Interaction. In Proceedings of the 5th International Mobile HCI 2003 conference. Springer-Verlag, LNCS.Google ScholarCross Ref
- Frøkjær, E., Hertzum, M. and Hornbæk, K. (2000) Measuring Usability: Are Effectiveness, Efficiency, and Satisfaction Really Correlated? In Proceedings of the ACM CHI 2000 Conference on Human Factors in Computing Systems. ACM Press. Google ScholarDigital Library
- Gray, W. D. and Salzman, M. C. (1998) Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13(3):203--261. Google ScholarDigital Library
- Hart, S. G., and Staveland, L. E. (1988) Development of a multi-dimensional workload rating scale: Results of empirical and theoretical research. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload. Elsevier Science Publishers.Google Scholar
- Hertzum, M. (1999) User Testing in Industry: A Case Study of Laboratory, Workshop, and Field Tests. In Proceedings of the 5th ERCIM Workshop, pp. 59--72.Google Scholar
- Hertzum, M. and Jacobsen, N. E. (2001) The Evaluator Effect: A Chilling Fact about Usability Evaluation Methods. International Journal of Human-Computer Interaction, 13(4).Google Scholar
- ISO The international Organization for Standardization (1998) Ergonomic requirements for office work with visual display terminals (VDTs). Part 11: Guidance on usability (ISO 9241--11).Google Scholar
- Johnson P. (1998) Usability and Mobility; Interactions on the move. In Proceedings of the First Workshop on Human-Computer Interaction with Mobile Devices. GIST Technical Report G98--1.Google Scholar
- Kaikkonen, A., Kallio, T., Kekäläinen, A., Kankainen, A. and Cankar, M. (2005) Usability testing of mobile applications: A comparison between laboratory and field testing. Journal of Usability Studies, 1(1):4--16.Google ScholarDigital Library
- Karat, C., Campbell, R. and Fiegel, T. (1992) Comparison of Empirical Testing and Walkthrough Methods in User Interface Evaluation. In Proceedings of the SIGCHI conference on Human factors in computing systems 1992. ACM. Google ScholarDigital Library
- Kjeldskov, J. and Stage, J. (2004) New Techniques for Usability Evaluation of Mobile Systems. International Journal of Human-Computer Studies, 60(4--5):599--620.Google Scholar
- Kjeldskov, J. and Skov, M. B. (2003) Creating a Realistic Laboratory Setting: A Comparative Study of Three Think-Aloud Usability Evaluations of a Mobile System. In Proceedings of the 9th IFIP TC13 International Conference on Human Computer Interaction, Interact 2003. IOS Press.Google Scholar
- Kjeldskov, J., Skov, M. B., Als, B. S. and Høegh, R. T. (2004) Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. In Proceedings of the 6th International Mobile HCI 2004 conference. LNCS, Springer-Verlag.Google ScholarCross Ref
- Lai J., Cheng K., Green P. and Tsimhoni O. (2001) On the Road and on the Web? Comprehension of synthetic speech while driving. In Proceedings of CHI'2001, pp. 206--212. ACM. Google ScholarDigital Library
- Molich, R. (2000) Brugervenlige EDB-Systemer, 2nd edition. Ingeniøren|Bøger.Google Scholar
- Molich, R., Ede, M. R., Kaasgaard, K. and Karyukin, B. (2004) Comparative usability evaluation. Behaviour & Information Technology, 23(1). Google ScholarDigital Library
- Newman, W. H. and Lamming, M. G. (1995) Interactive System Design. Addison-Wesley. Google ScholarDigital Library
- Nielsen, C. M., Overgaard, M., Pedersen, M. B. and Stenild, S. (2004) The Development of a Mobile System for Communicating and Collaborating -- An Object-Oriented HCI Approach, Department of Computer Science, Aalborg University, 2004.Google Scholar
- Norman, D. (1990). The Design of Everyday Things, Doubleday and Company, 2002 Edition. Google ScholarDigital Library
- Nyyssönen, Roto and Kaikkonen (2002). Mini-Camera for Usability Tests and Demonstration. Presented in Demo Sessions at the 4th International Symposium on Human Computer Interaction with Mobile Devices, 2002, Nokia Research Center.Google Scholar
- Pedell, S., Graham C., Kjeldskov J. and Davies, J. (2003) Mobile Evaluation: What the Data and the Metadata Told Us. In Proceedings of OzCHI 2003, pp. 96--105.Google Scholar
- Pirhonen, A., Brewster, S. and Holguin, C. (2002) Gestural an Audio Metaphors as a Means of Control for Mobile Devices. In Proceedings of CHI'2002. ACM. Google ScholarDigital Library
- Rowley, D. E. (1994) Usability Testing in the Field: Bringing the Laboratory to the User. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM Press. Google ScholarDigital Library
- Rubin, Jeffrey (1994). Handbook of Usability Testing -- how to plan, design, and conduct effective tests, John Wiley & sons, Inc. Google ScholarDigital Library
- Salvucci D. D. (2001) Predicting the Effects of In-Car Interfaces on Driver Behaviour using a Cognitive Architecture. In Proceedings of CHI'2001, pp 120--127. ACM. Google ScholarDigital Library
- Sannella, M. J. (1994)Constraint Satisfaction and Debugging for Interactive User Interfaces. Ph.D. Thesis, University of Washington, Seattle, WA. Google ScholarDigital Library
Index Terms
- It's worth the hassle!: the added value of evaluating the usability of mobile systems in the field
Recommendations
A comparative study of two usability evaluation methods using a web-based e-learning application
SAICSIT '07: Proceedings of the 2007 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countriesUsability evaluation of e-learning applications is a maturing area, which addresses interfaces, usability and interaction from human-computer interaction (HCI) and pedagogy and learning from education. The selection of usability evaluation methods (UEMs)...
A framework for evaluating the usability of mobile phones based on multi-level, hierarchical model of usability factors
As a mobile phone has various advanced functionalities or features, usability issues are increasingly challenging. Due to the particular characteristics of a mobile phone, typical usability evaluation methods and heuristics, most of which are relevant ...
Comparing Low and High-Fidelity Prototypes in Mobile Phone Evaluation
This study compared usability testing results found with low-and high-fidelity prototypes for mobile phones. The main objective is to obtain deep understanding of usability problems found with different prototyping methods. Three mobile phones from ...
Comments