Ben Shneiderman
Catherine Plaisant
ABSTRACT
After an historical review of evaluation methods, we describe an emerging research method called Multi-dimensional In-depth Long-term Case studies (MILCs) which seems well adapted to study the creative activities that users of information visualization systems engage in. We propose that the efficacy of tools can be assessed by documenting 1) usage (observations, interviews, surveys, logging etc.) and 2) expert users' success in achieving their professional goals. We summarize lessons from related ethnography methods used in HCI and provide guidelines for conducting MILCs for information visualization. We suggest ways to refine the methods for MILCs in modest sized projects and then envision ambitious projects with 3-10 researchers working over 1-3 years to understand individual and organizational use of information visualization by domain experts working at the frontiers of knowledge in their fields.
- Amabile, T. M., Creativity in Context. Boulder, CO, Westview Press (1996).]]Google Scholar
- Argyris, C., Putnam, R., and Smith, D. M., Action Science. Jossey-Bass, San Francisco (1985).]]Google Scholar
- Bentley, R., Hughes, J., Randall, D., Rodden, T., Sawyer, P., Shapiro, D., and Sommerville, I., Ethnographically-informed systems design for air traffic control, Proc. CSCW '92, ACM Press, New York (1992), 123--129.]] Google ScholarDigital Library
- Beyer, H. and Holtzblatt, K., Contextual Design: Defining Customer-Centered Systems, Morgan Kaufmann Publishers, San Francisco, CA (1998).]] Google ScholarDigital Library
- Bodker, S., Scenarios in user-centered design --- setting the stage for reflection and action, Interacting with Computers 13, 1 (2000), 61--76.]]Google ScholarCross Ref
- Candy, L. & Edmonds, E. A., Supporting the creative user: A criteria based approach to interaction design. Design Studies 18 (1997), 185--194.]]Google ScholarCross Ref
- Candy, L. and Edmonds, E. A., Explorations in Art and Technology, Springer-Verlag, London (2002).]] Google ScholarDigital Library
- Dumas, J. and Redish, J., A Practical Guide to Usability Testing: Revised Ed., Intellect Books, Bristol, UK (1999).]] Google ScholarDigital Library
- Edmonds, E. A., Weakley, A. J., Candy, L., Fell, M. J., Knott, R. P. and Pauletto, S., The studio as laboratory: combining creative practice and digital technology research, Int'l Journal of Human Computer Studies 63, 4--5 (2005), Special Issue on Computer Support for Creativity, 452--481.]] Google ScholarDigital Library
- Erickson, T., The design and long-term use of a personal electronic notebook: A reflective analysis, Proc. ACM CHI 96 Human Factors in Computing Systems Conference, ACM Press, New York (1996), 11--18.]] Google ScholarDigital Library
- Fekete, J.-D., Grinstein, G., Plaisant, C., IEEE Info Vis 2004 Contest, the history of Info Vis, www.cs.umd.edu/hcil/iv04contest (2004)]]Google Scholar
- Fetterman, D. M., Ethnography: Step by Step: 2nd Edition, Sage, Thousand Oaks, CA (1998).]]Google Scholar
- Fjermestad, J. and Hiltz S. R. "Group Support Systems: A Descriptive Evaluation of Case and Field Studies," Journal of Management Information Systems 17, 3 (2000), 113--157.]]Google ScholarDigital Library
- Harper, R., The organization of ethnography, Proc. CSCW 2000, ACM Press, New York (2000), 239--264.]]Google Scholar
- Hewett, T. T., The role of iterative evaluation in designing systems for usability, In M. D. Harrison and A. F. Monk (Eds.), People and Computers: Designing for Usability Cambridge: Cambridge University Press. (1986), 196--214.]] Google ScholarDigital Library
- Hewett, T. T., Informing the design of computer based creativity support environments. International Journal of Human Computer Studies 63, 4--5 (2005), Special Issue on Computer Support for Creativity, 383--409.]] Google ScholarDigital Library
- Hughes, J., King, V., Rodden, T., and Andersen, H., The role of ethnography in interactive systems design, ACM interactions 2, 2 (April 1995), 56--65.]] Google ScholarDigital Library
- Hughes, J., O'Brien, J., Rodden, T., and Rouncefield, M., Design with ethnography: A presentation framework for design, Proc. of Design of Interaction Systems '97, ACM Press, New York (1997), 147--159.]] Google ScholarDigital Library
- Lewin, K. Field Theory in Social Science. Basic Books, New York (1951).]]Google Scholar
- Marchionini, G. and Crane, G. Evaluating Hypermedia and Learning: Methods and Results from the Perseus Project, ACM Trans. on Information Systems 12, 1 (1994), 5--34.]] Google ScholarDigital Library
- Millen, David, Rapid ethnography: time deepening strategies for HCI field, Proc. ACM Symposium on Designing Interactive Systems, ACM Press, New York (2000), 280--286.]] Google ScholarDigital Library
- Olson, G. and Olson, J., Groupware and computer supported cooperative work, In Jacko and Sears (Editors), The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, LEA Publishers, Mahwah, NJ (2002), 583--595.]] Google ScholarDigital Library
- Plaisant, C. The Challenge of Information Visualization Evaluation, Proc. of Conf. on Advanced Visual Interfaces, ACM, New York (2004), 109--116.]] Google ScholarDigital Library
- Rose, A., Plaisant, C., and Shneiderman, B., Using ethnographic methods in user interface re-engineering, Proc. DIS '95: Symposium on Designing Interactive Systems, ACM Press, New York (1995), 115--122.]] Google ScholarDigital Library
- Saraiya, P., North, C., and Duca, K., An evaluation of microarray visualization tools for biological insight, Proc. of IEEE Symposium on Information Visualization (2004), 1--8.]] Google ScholarDigital Library
- Seo, J. and Shneiderman, B., Knowledge discovery in high dimensional data: Case studies and a user survey for the rank-by-feature framework, IEEE Trans. on Visualization and Computer Graphics 12, 3 (May/June, 2006).]] Google ScholarDigital Library
- Shneiderman, B., Creating creativity: User interfaces for supporting innovation, ACM Transactions on Computer-Human Interaction 7, 1 (March 2000), 114--138.]] Google ScholarDigital Library
- Shneiderman, B., Fischer, G., Czerwinski, M., Resnick, M., Myers, B. and 13 others, Creativity Support Tools: Report from a U.S. National Science Foundation Sponsored Workshop, International Journal of Human---Computer Interaction 20, 2 (2006), 61--77.]]Google Scholar
- Shneiderman, B. and Plaisant, C., Designing the User Interface: Strategies for Effective Human-Computer Interaction: Fourth Edition, Addison-Wesley Publ. Co., Reading, MA (April 2004), 684 pages.]] Google ScholarDigital Library
- Sternberg, R. (Editor), Handbook of Creativity, Cambridge Univ. Press, Cambridge, UK (1999).]]Google Scholar
- Suchman, L. A., Plans and Situated Actions: The Problem of Human-Machine Communication, Cambridge University Press, UK (1987).]] Google ScholarDigital Library
- Trafton, J., Tsui, T., Miyamoto, R.; Ballas, J., Raymond, P., Turning pictures into numbers: extracting and generating information from complex visualizations. International Journal of Human Computer Studies 53, 5 (2000), 827--850.]] Google ScholarDigital Library
- VAST contest: www.cs.umd.edu/hcil/VASTcontest06]]Google Scholar
- Warmington, A. Action Research: Its Methods and its Implications. Journal of Applied Systems Analysis, 7 (1980), 23--39]]Google Scholar
- Yin, R. K., Case Study Research: Design and Methods, 3rd Edition, Sage Publications, Thousands Oaks, CA. (2003).]]Google Scholar
Index Terms
- Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies
Recommendations
Evaluating Climate Visualization: An Information Visualization Approach
IV '10: Proceedings of the 2010 14th International Conference Information VisualisationTo meet the growing demand of communicating climate science and policy research, the interdisciplinary field of climate visualization has increasingly extended its traditional use of 2D representations and techniques from the field of scientific ...
Casual Information Visualization: Depictions of Data in Everyday Life
Information visualization has often focused on providing deep insight for expert user populations and on techniques for amplifying cognition through complicated interactive visual models. This paper proposes a new subdomain for infovis research that ...
Comments