Abstract
The speed-accuracy trade-off (SAT) is a ubiquitous phenomenon in experimental psychology. One popular strategy for controlling SAT is to use the response signal paradigm. This paradigm produces time-accuracy curves (or SAT functions), which can be compared across different experimental conditions. The typical approach to analyzing time-accuracy curves involves the comparison of goodness-of-fit measures (e.g., adjusted-R2), as well as interpretation of point estimates. In this article, we examine the implications of this approach and discuss a number of alternative methods that have been successfully applied in the cognitive modeling literature. These methods include model selection criteria (the Akaike information criterion and the Bayesian information criterion) and interval estimation procedures (bootstrap and Bayesian). We demonstrate the utility of these methods with a hypothetical data set.
Article PDF
Similar content being viewed by others
References
Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In B. N. Petrov & F. Caski (Eds.), Proceedings of the Second International Symposium on Information Theory (pp. 267–281). Budapest: Akademiai Kiado.
Anderson-Sprecher, R. (1994). Model comparisons and R2. American Statistician, 48, 113–117.
Batchelder, W. H., & Riefer, D. M. (1990). Multinomial processing models of source monitoring. Psychological Review, 97, 548–564.
Bozdogan, H. (2000). Akaike’s information criterion and recent developments in information complexity. Journal of Mathematical Psychology, 44, 62–91.
Brown, S., & Heathcote, A. (2003). Averaging learning curves across and within participants. Behavior Research Methods, Instruments, & Computers, 35, 11–21.
Burnham, K. P., & Anderson, D. R. (2002). Model selection and multimodel inference: A practical information-theoretic approach. New York: Springer.
Busemeyer, J. R., & Wang, Y.-M. (2000). Model comparisons and model selections based on generalization criterion methodology. Journal of Mathematical Psychology, 44, 171–189.
Carrasco, M., Giordano, A. M., & McElree, B. (2004). Temporal performance fields: Visual and attentional factors. Vision Research, 44, 1351–1365.
Carrasco, M., Giordano, A. M., & McElree, B. (2006). Attention speeds processing across eccentricity: Feature and conjunctive searches. Vision Research, 46, 2028–2040.
Carrasco, M., & McElree, B. (2001). Covert attention accelerates the rate of visual information processing. Proceedings of the National Academy of Sciences, 98, 5363–5367.
Chechile, R. A. (1977). Likelihood and posterior identification: Implications for mathematical psychology. British Journal of Mathematical & Statistical Psychology, 30, 177–184.
Davison, A. C., & Hinkley, D. V. (1997). Bootstrap methods and their applications. Cambridge: Cambridge University Press.
Dosher, B. A. (1979). Empirical approaches to information processing: Speed-accuracy trade-off functions or reaction time. Acta Psychologica, 43, 347–359.
Dosher, B. A. (1981). The effect of delay and interference: A speed-accuracy study. Cognitive Psychology, 13, 551–582.
Dosher, B. A. (1982). Sentence size, network distance and sentence retrieval. Journal of Experimental Psychology: Learning, Memory, & Cognition, 8, 173–207.
Dosher, B. A. (1984). Degree of learning and retrieval speed: Study time and multiple exposures. Journal of Experimental Psychology: Learning, Memory, & Cognition, 10, 541–574.
Dosher, B. A., Han, S. M., & Lu, Z. L. (2004). Parallel processing in visual search asymmetry. Journal of Experimental Psychology: Human Perception & Performance, 30, 3–27.
Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York: Chapman & Hall.
Estes, W. K., & Maddox, W. T. (2005). Risks of drawing inferences about cognitive processes from model fits to individual versus average performance. Psychonomic Bulletin & Review, 12, 403–408.
Gill, J. (2002). Bayesian methods: A social and behavioral sciences approach. Boca Raton, FL: Chapman & Hall/CRC.
Glover, S., & Dixon, P. (2004). Likelihood ratios: A simple and flexible statistic for empirical psychologists. Psychonomic Bulletin & Review, 11, 791–806.
Gourevitch, V., & Galanter, E. (1967). A significance test for one parameter isosensitivity functions. Psychometrika, 32, 25–33.
Gronlund, S. D., & Ratcliff, R. (1989). Time course of item and associative information: Implications for global memory models. Journal of Experimental Psychology: Learning, Memory, & Cognition, 15, 846–858.
Heathcote, A., Brown, S. D., & Mewhort, D. J. K. (2000). The power law repealed: The case for an exponential law of practice. Psychonomic Bulletin & Review, 7, 185–207.
Huber, D. E. (2006). Computer simulations of the ROUSE model: An analytic simulation technique and a comparison between the error variance-covariance and bootstrap methods for estimating parameter confidence. Behavior Research Methods, 38, 557–568.
Kass, R. E. (1993). Bayes factors in practice. Journal of the Royal Statistical Society, 42, 551–560.
Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795.
Kuha, J. (2004). AIC and BIC: Comparisons of assumptions and performance. Sociological Methods & Research, 33, 188–229.
Kuss, M., Jäkel, F., & Wichmann, F. A. (2005). Bayesian inference for psychometric functions. Journal of Vision, 5, 478–492.
Lee, M. D. (2004). A Bayesian analysis of retention functions. Journal of Mathematical Psychology, 48, 310–321.
Lee, M. D. (2008). Three case studies in the Bayesian analysis of cognitive models. Psychonomic Bulletin & Review, 15, 1–15.
Lee, M. D., & Wagenmakers, E.-J. (2005). Bayesian statistical inference in psychology: Comment on Trafimow (2003). Psychological Review, 112, 662–668.
Lee, M. D., & Webb, M. R. (2005). Modeling individual differences in cognition. Psychonomic Bulletin & Review, 12, 605–621.
Link, W. A. (1992). The wave theory of difference and similarity. Hillsdale, NJ: Erlbaum.
Link, W. A., & Barker, R. J. (2006). Model weights and the foundations of multimodel inference. Ecology, 87, 2626–2635.
Liu, C. C., & Aitkin, M. (2008). Bayes factors: Prior sensitivity and model generalizability. Journal of Mathematical Psychology, 52, 362–375.
Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. (2000). WinBUGS—a Bayesian modeling framework: Concepts, structure, and extensibility. Statistics & Computing, 10, 325–337.
McElree, B. (1996). Accessing short-term memory with semantic and phonological information: A time-course analysis. Memory & Cognition, 24, 173–187.
McElree, B. (1998). Attended and nonattended states in working memory: Accessing categorized structures. Journal of Memory & Language, 38, 225–252.
McElree, B. (2001). Working memory and focal attention. Journal of Experimental Psychology: Learning, Memory, & Cognition, 27, 817–835.
McElree, B., & Dosher, B. A. (1989). Serial position and set size in short-term memory: The time course of recognition. Journal of Experimental Psychology: General, 118, 346–373.
McElree, B., & Dosher, B. A. (1993). Serial retrieval processes in the recovery of order information. Journal of Experimental Psychology: General, 122, 291–315.
McElree, B., & Griffith, T. (1995). Syntactic and thematic processing in sentence comprehension: Evidence for a temporal dissociation. Journal of Experimental Psychology: Learning, Memory, & Cognition, 21, 134–157.
Miller, J. (1996). The sampling distribution of d’. Perception & Psychophysics, 58, 65–72.
Mulligan, N., & Hirshman, E. (1995). Speed-accuracy trade-offs and the dual process model of recognition memory. Journal of Memory & Language, 34, 1–18.
Myung, I. J. (2000). The importance of complexity in model selection. Journal of Mathematical Psychology, 44, 190–204.
Myung, I. J. (2003). Tutorial on maximum likelihood estimation. Journal of Mathematical Psychology, 47, 90–100.
Myung, I. J., Kim, C., & Pitt, M. A. (2000). Toward an explanation of the power law artifact: Insights from response surface analysis. Memory & Cognition, 28, 832–840.
Myung, I. J., & Pitt, M. A. (1997). Applying Occam’s razor in modeling cognition: A Bayesian approach. Psychonomic Bulletin & Review, 4, 79–95.
Nikolic, D., & Gronlund, S. D. (2002). A tandem random walk model of the SAT paradigm: Response times and accumulation of evidence. British Journal of Mathematical & Statistical Psychology, 55, 263–288.
Öztekin, I., & McElree, B. (2007). Proactive interference slows recognition by eliminating fast assessments of familiarity. Journal of Memory & Language, 57, 126–149.
Penny, W. D., Stephan, K. E., Mechelli, A., & Friston, K. J. (2004). Comparing dynamic causal models. NeuroImage, 22, 1157–1172.
Pitt, M. A., & Myung, I. J. (2002). When a good fit can be bad. Trends in Cognitive Sciences, 6, 421–425.
Pitt, M. A., Myung, I. J., & Zhang, S. (2002). Toward a method of selecting among computational models of cognition. Psychological Review, 109, 472–491.
Raftery, A. E. (1995). Bayesian model selection in social research. Sociological Methodology, 25, 111–163.
Ratcliff, R. (1978). A theory of memory retrieval. Psychological Review, 85, 59–108.
Ratcliff, R. (2006). Modeling response signal and response time data. Cognitive Psychology, 53, 195–237.
Ratcliff, R., & Smith, P. L. (2004). A comparison of sequential sampling models for two-choice reaction time. Psychological Review, 111, 333–367.
Reed, A. V. (1973). Speed-accuracy trade-off in recognition memory. Science, 181, 574–576.
Reed, A. V. (1976). List length and the time course of recognition in human memory. Memory & Cognition, 4, 16–30.
Rouder, J. N., & Lu, J. (2005). An introduction to Bayesian hierarchical models with an application in the theory of signal detection. Psychonomic Bulletin & Review, 12, 573–604.
Rouder, J. N., Lu, J., Speckman, P., Sun, D., & Jiang, Y. (2005). A hierarchical model for estimating response time distributions. Psychonomic Bulletin & Review, 12, 195–223.
Rubin, D. B., & Wenzel, A. (1996). One hundred years of forgetting: A quantitative description of retention. Psychological Review, 103, 734–760.
Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464.
Smith, B. J. (2005). Bayesian Output Analysis Program (BOA), Version 1.1.5 [Online]. Iowa City: University of Iowa. Available at www.public-health.uiowa.edu/boa.
Smith, P. L. (2000). Stochastic dynamic models of response time and accuracy: A foundational primer. Journal of Mathematical Psychology, 44, 408–463.
Smith, P. L., Ratcliff, R., & Wolfgang, B. J. (2004). Attention orienting and the time course of perceptual decisions: Response time distributions with masked and unmasked displays. Vision Research, 44, 1297–1320.
Smith, P. L., & Wolfgang, B. J. (2004). The attentional dynamics of masked detection. Journal of Experimental Psychology: Human Perception & Performance, 30, 119–136.
Smith, P. L., Wolfgang, B. J., & Sinclair, A. J. (2004). Mask-dependent attentional cuing effects in visual signal detection: The psychometric function for contrast. Perception & Psychophysics, 66, 1056–1075.
Snodgrass, J. G., & Corwin, J. (1988). Pragmatics of measuring recognition memory: Applications to dementia and amnesia. Journal of Experimental Psychology: General, 117, 34–50.
Stephan, K. E., & Penny, W. D. (2007). Dynamic causal models and Bayesian selection. In K. Friston, J. Ashburner, S. Kiebel, T. Nichols, & W. Penny (Eds.), Statistical parametric mapping: The analysis of functional brain images (pp. 577–585). Amsterdam: Elsevier.
Thomas, R. D. (2006). Processing time predictions of current models of perception in the classic additive factors paradigm. Journal of Mathematical Psychology, 50, 441–455.
Usher, M., & McClelland, J. L. (2001). The time course of perceptual choice: The leaky, competing accumulator model. Psychological Review, 108, 550–592.
Vandekerckhove, J., & Tuerlinckx, F. (2008). Diffusion model analysis with MATLAB: A DMAT primer. Behavior Research Methods, 40, 61–72.
Verguts, T., & Storms, G. (2004). Assessing the informational value of parameter estimates in cognitive models. Behavior Research Methods, Instruments, & Computers, 36, 1–10.
Voss, A., & Voss, J. (2007). Fast-dm: A free program for efficient diffusion model analysis. Behavior Research Methods, 39, 767–775.
Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804.
Wagenmakers, E.-J., & Farrell, S. (2004). AIC model selection using Akaike weights. Psychonomic Bulletin & Review, 11, 192–196.
Wagenmakers, E.-J., Ratcliff, R., Gomez, P., & Iverson, G. J. (2004). Assessing model mimicry using the parametric bootstrap. Journal of Mathematical Psychology, 48, 28–50.
Wasserman, L. (2000). Bayesian model selection and model averaging. Journal of Mathematical Psychology, 44, 92–107.
Weakliem, D. L. (1999). A critique of the Bayesian information criterion for model selection. Sociological Methods & Research, 27, 359–397.
Wichmann, F. A., & Hill, N. J. (2001). The psychometric function: II. Bootstrap-based confidence intervals and sampling. Perception & Psychophysics, 63, 1314–1329.
Wickelgren, W. A. (1977). Speed-accuracy trade-off and information processing dynamics. Acta Psychologica, 41, 67–85.
Wickelgren, W. A., & Corbett, A. T. (1977). Associate interference and retrieval dynamics in yes-no recall and recognition. Journal of Experimental Psychology: Human Learning & Memory, 3, 189–202.
Author information
Authors and Affiliations
Corresponding author
Additional information
This research was supported by a Melbourne Research Scholarship to C.C.L. and an Australian Research Council grant to P.L.S. Preparation of the manuscript was supported by a MUARC publication award.
Rights and permissions
About this article
Cite this article
Liu, C.C., Smith, P.L. Comparing time-accuracy curves: Beyond goodness-of-fit measures. Psychonomic Bulletin & Review 16, 190–203 (2009). https://doi.org/10.3758/PBR.16.1.190
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/PBR.16.1.190