Computer interface evaluation using eye movements: methods and constructs

https://doi.org/10.1016/S0169-8141(98)00068-7Get rights and content

Abstract

Eye movement-based analysis can enhance traditional performance, protocol, and walk-through evaluations of computer interfaces. Despite a substantial history of eye movement data collection in tasks, there is still a great need for an organized definition and evaluation of appropriate measures. Several measures based upon eye movement locations and scanpaths were evaluated here, to assess their validity for assessment of interface quality. Good and poor interfaces for a drawing tool selection program were developed by manipulating the grouping of tool icons. These were subsequently evaluated by a collection of 50 interface designers and typical users. Twelve subjects used the interfaces while their eye movements were collected. Compared with a randomly organized set of component buttons, well-organized functional grouping resulted in shorter scanpaths, covering smaller areas. The poorer interface resulted in more, but similar duration, fixations than the better interface. Whereas the poor interface produced less efficient search behavior, the layout of component representations did not influence their interpretability. Overall, data obtained from eye movements can significantly enhance the observation of users' strategies while using computer interfaces, which can subsequently improve the precision of computer interface evaluations.


Relevance to industry

The software development industry requires improved methods for the objective analysis and design of software interfaces. This study provides a foundation for using eye movement analysis as part of an objective evaluation tool for many phases of interface analysis. The present approach is instructional in its definition of eye movement-based measures, and is evaluative with respect to the utility of these measures.

Introduction

The software development cycle requires frequent iterations of user testing and interface modification. These interface evaluations, whether at initial design or at later test and evaluation stages, should assess system functionality and the impact of the interface on the user. Earlier, design evaluation methods include cognitive walkthroughs, heuristic, review-based, and model-based evaluations. At more mature product phases, performance-based experiments, protocol/observation, and questionnaires are frequently used as a basis for evaluation (Dix et al., 1998). Performance-based studies assess errors and time to complete specified operations or scenarios (Wickens et al., 1998).

Interface evaluation and usability testing are expensive, time-intensive exercises, often done with poorly documented standards and objectives. They are frequently qualitative, with poor reliability and sensitivity. Provision of an improved tool for rapid and effective evaluation of graphical user interfaces was the motivating goal underlying the present work assessing eye movements as an indicator of interface usability.

While using a computer interface, one's eye movements usually indicate one's spatial focus of attention on a display. In order to foveate informative areas in a scene, the eyes naturally fixate upon areas that are surprising, salient, or important through experience (Loftus and Mackworth, 1978). Thus, current gazepoints on a display can approximate foci of attention over a time period. When considering short time intervals, however, one's attentional focus may lead or lag the gazepoint (Just and Carpenter, 1976). By choosing long enough sampling intervals for eye movements, temporal leads/lags should be averaged out.

Applied eye movement analysis has at least a 60 yr history in performance and usability assessments of spatial displays within information acquisition contexts such as aviation, driving, X-ray search, and advertising. Buswell (1935)measured fixation densities and serial scanpaths while individuals freely viewed artwork samples, noting that eyes follow the direction of principal lines in figures, and that more difficult processing produced longer fixation durations. Mackworth (1976)noted that higher display densities produced 50–100 ms longer fixation durations than lower density displays. Non-productive eye movements more than 20° from the horizontal scanning axis strongly increased as a percentage of all eye movements as the display width and density increased. Kolers et al. (1981)measured eye fixations (number, number per line, rate, duration, words per fixation) as a function of character and line spacing in a reading task. More fixations per line (and fewer fixations per word) were associated with more tightly-grouped, singled-spaced material. Fewer, yet longer fixations were made with smaller, more densely packed text characters. Yamamoto and Kuto (1992)found improved Japanese character reading performance associated with series of sequential rather than backtracking eye movements. Eye tracking has aided the assessment of whether the order of product versus filler displays in a television commercial influences one's attention to that product (Janiszewski and Warlop, 1993). Using eye movement analyses while scanning advertisements on telephone yellow pages, quarter-page ad displays were much more noticed than text listings, and color ads were perceived more quickly, more often, and longer than black and white ads (Lohse, 1997).

Prior eye movement-based interface and usability characterizations have relied heavily upon cumulative fixation time and areas of interest approaches, dividing an interface into predefined areas. Transitions into and from these areas, as well as time spent in each area, are tallied. While these approaches can signal areas where more or less attention is spent while using a display, few investigations have considered the complex nature of scanpaths, defined from a series of fixations and saccades on the interface. Scanpath complexity and regularity measures are needed to approach some of the subtler interface usability issues in screen design.

Eye tracking systems are now inexpensive, reliable, and precise enough to significantly enhance system evaluations. While the hardware technology is quite mature (Young and Sheena, 1975, for a general review), methods of evaluating data from eye tracking experiments are still somewhat immature and disorganized. The objective of the present paper is to provide an introduction and framework for eye movement data analysis techniques. These eye movement measures and algorithms are presented in light of results from an experiment presenting users with both “good” and “poor” interfaces.

Section snippets

Methods

Scanpaths were collected from 12 subjects while using both “good” and “poor” software interfaces. The resulting scanpaths were characterized using a number of quantitative measures, each designed to characterize different aspects of scanpath behaviors and relate to the cognitive behavior underlying visual search and information processing. A comparison of expected user search behavior using each interface with the results of scanpath measures were used to determine the relative effectiveness of

Classification of measures

Scanpaths are defined by a saccade–fixate–saccade sequence on a display. For information search tasks, the optimal scanpath is a straight line to a desired target, with relatively short fixation duration at the target. The derived scanpath measures discussed below attempt to quantitatively measure the divergence from this optimal scanpath in several ways. The measures each provide a single quantitative value, with some requiring no knowledge of the content of the computer interface. Table 1

Measures of search

Illustrated descriptions of each of the eye movement measures and algorithms are provided below. Results from the good versus poor interfaces and other factors are also presented here. The same hypothetical scanpath is used in all examples below, for easy comparison. All of these measures may be used for a given scanpath, with each offering a slightly different interpretation of the data. Scanpaths may also be viewed as directed or undirected graphs, allowing additional characterizations of

Measures of processing

Visual search is conducted to obtain information from an interface, where more extensive search allows more interface objects to be processed. This does not consider the depth of required processing, however. In the present study, as the same representations were used in both interfaces, the depth of processing required to distinguish and interpret a component was not expected to differ.

Discussion

Successful interaction with a computer clearly requires many elements, including good visibility, meaningfulness, transparency, and the requirement of simple motor skills. Eye movement-based evaluation of the interface, as espoused here, can only address a subset of critical interface issues that revolve around software object visibility, meaningfulness, and placement. Though not a panacea tool for design evaluation, characterization of eye movements can help by providing easily comparable

References (23)

  • M.A. Just et al.

    Eye fixations and cognitive processes

    Cognitive Psychology

    (1976)
  • V. Ponsoda et al.

    A probability vector and transition matrix analysis of eye movements during visual search

    Acta Psycholgica

    (1995)
  • J. Anliker

    On line measurements, analysis and control

  • Belofsky, M.S., Lyon, D.R., 1988. Modeling eye movement sequences using conceptual clustering techniques. Air Force...
  • G.T. Buswell

    How People Look at Pictures

    A Study of the Psychology of Perception in Art

    (1935)
  • A. Dix et al.

    Human–Computer Interaction

    (1988)
  • J.H. Goldberg et al.

    Eye-gaze determination of user intent at the computer interface

  • J.H. Goldberg et al.

    Eye-gaze contingent control of the computer interface: Methodology and example for zoom detection

    Behavior Research Methods, Instruments and Computers

    (1995)
  • Jones, R.E., Milton, J.L., Fitts, P.M., 1949. Eye fixations of aircraft pilots; IV: Frequency, duration and sequence of...
  • P.A. Kolers et al.

    Eye movement measurement of readability of CRT displays

    Human Factors

    (1981)
  • B.J.A. Krose et al.

    Spatial interactions in rapid pattern discrimination

    Spatial Vision

    (1989)
  • Cited by (771)

    • Usability Study of UUM Student Portal Using Eye Tracker

      2024, Communications in Computer and Information Science
    View all citing articles on Scopus
    1

    Present address: Lucent Technologies, Bell Laboratories, Holmdel, NY.

    View full text