Abstract
Mental workload is a concept central to a range of disciplines (including cognitive psychology and ergonomics) that has given rise to various theoretical and methodological debates. As a result, researchers have used a number of techniques for measuring mental workload. Traditionally, three categories of measurement technique have been recognized: performance measures (e.g., the dual-task paradigm), subjective measures (e.g., rating scales), and physiological measures (e.g., heart, respiration, and blink rates). Each technique has advantages and limitations; however, some limitations may prevent an accurate evaluation of the mental workload. In this article, we focus on the benefits of combining multiple measures of mental workload. However, because combining several techniques is a very complex process, we have developed the Tholos software in an attempt to reduce this complexity. This software package uses measures from each of the three categories: a dual-task paradigm with auditory signals; the National Aeronautics and Space Administration task load index (NASA-TLX) scale and its simplified version, the “raw” task load index (RTLX); and physiological (such as pupil-dilation) measurements with which our software can merge results from the dual-task paradigm. To illustrate the benefits of using Tholos, we describe a puzzle-solving experiment in which we combined multiple measures of mental workload. The results indicate the importance of combining multiple measures to build upon the theoretical and methodological foundations of mental workload.
Article PDF
Similar content being viewed by others
References
Ahlstrom, U., & Friedman-Berg, F. J. (2006). Using eye movement activity as a correlate of cognitive workload. International Journal of Industrial Ergonomics, 36, 623–636.
Backs, R. W., Navidzadeh, H. T., & Xu, X. (2000). Cardiorespiratory indices of mental workload during simulated air traffic control. Proceedings of the Human Factors and Ergonomics Society 44th Annual Meeting (pp. 3–89). Santa Monica, CA: Human Factors and Ergonomics Society.
Backs, R. W., & Seljos, K. A. (1994). Metabolic and cardiorespiratory measures of mental effort: The effects of level of difficulty in a working memory task. International Journal of Psychophysiology, 16, 57–68.
Bainbridge, L. (1974). Problems in the assessment of mental load. Le Travail Humain, 37, 279–302.
Beatty, J. (1982). Task evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91, 276–292.
Bi, S., & Salvendy, G. (1994). Analytical modeling and experimental study of human workload in scheduling of advanced manufacturing systems. International Journal of Human Factors in Manufacturing, 4, 205–234.
Bortolussi, M. R., Kantowitz, B. H., & Hart, S. G. (1986). Measuring pilot workload in a motion base trainer: A comparison of four techniques. Applied Ergonomics, 17, 278–283.
Brünken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in multimedia learning. Educational Psychologist, 38, 53–61.
Byers, J. C., Bittner, A. C., Jr., & Hill, S. G. (1989). Traditional and raw task load index (TLX) correlations: Are paired comparisons necessary? In A. Mital (Ed.), Advances in industrial ergonomics and safety (pp. 481–485). London: Taylor & Francis.
Casali, J. G., & Wierwille, W. W. (1983). A comparison of rating scale, secondary task, physiological, and primary-task workload estimation techniques in a simulated flight task emphasizing communications load. Human Factors, 25, 623–641.
Cegarra, J., & Hoc, J.-M. (2006). Cognitive styles as an explanation of experts’ individual differences: A case study in computer-assisted troubleshooting diagnosis. International Journal of Human-Computer Studies, 64, 123–136.
Chevalier, A., & Kicka, M. (2006). Web designers and Web users: Influence of the ergonomic quality of the Web site on the information search. International Journal of Human-Computer Studies, 64, 1031–1048.
Colle, H. A., & Reid, G. B. (1998). Context effects in subjective mental workload ratings. Human Factors, 40, 591–600.
Cooper, G. E., & Harper, R. P., Jr. (1969). The use of pilot rating in the evaluation of aircraft handling qualities (Tech. Rep. D-5153). Washington, DC: NASA.
DooWon, C., & Peom, P. (1997). Simulator-based cognitive load assessment of the in-vehicle navigation system driver using revision of NASA-TLX. IE-Interfaces, 10, 145–154.
Fisk, A. D., Derrick, W. L., & Schneider, W. (1987). A methodological assessment and evaluation of dual-task paradigms. Current Psychological Research & Reviews, 5, 315–327.
Granholm, E., Asarnow, R. F., Sarkin, A. J., & Dykes, K. L. (1996). Pupillary responses index cognitive resource limitations. Psychophysiology, 33, 457–461.
Hart, S. G., & Staveland, L. E. (1988). Development of NASATLX (task load index): Results of empirical and theoretical research. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 139–183). Amsterdam: North-Holland.
Hendy, K. C., Hamilton, K. M., & Landry, L. N. (1993). Measuring subjective workload: When is one scale better than many? Human Factors, 35, 579–601.
Hill, S. G., Iavecchia, H. P., Byers, J. C., Bittner, A. C., Zaklad, A. L., & Christ, R. E. (1992). Comparison of four subjective workload rating scales. Human Factors, 34, 429–439.
Isreal, J. B., Chesney, G. L., Wickens, C. D., & Donchin, E. (1980). P300 and tracking difficulty: Evidence for multiple resources in dualtask performance. Psychophysiology, 17, 259–273.
James, R. H., Elderfield, H., Palmer, M. R., & Connelly, C. S. (1995). Toward an understanding of DCS control operator workload. ISA Transactions, 34, 175–184.
Jex, H. R. (1988). Measuring mental workload: Problems, progress, and promises. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 5–39). Amsterdam: North-Holland.
Just, M. A., & Carpenter, P. A. (1993). The intensity dimension of thought: Pupillometric indices of sentence processing. Canadian Journal of Experimental Psychology, 47, 310–339.
Kahneman, D. (1973). Attention and effort. Englewood Cliffs, NJ: Prentice Hall.
Kahneman, D., & Beatty, J. (1966). Pupil diameter and load on memory. Science, 154, 1583–1585.
Kahneman, D., Peavler, W. S., & Onuska, L. (1968). Effects of verbalization and incentive on the pupil response to mental activity. Canadian Journal of Psychology, 22, 186–196.
Kellogg, R. T. (1990). Effectiveness of prewriting strategies as a function of task demands. American Journal of Psychology, 103, 327–342.
Kilmer, K. J., Knapp, R., Burdsal, C., Borresen, R., Bateman, R., & Malzahn, D. (1988). Techniques of subjective assessment: A comparison of the SWAT and modified Cooper-Harper scales. In Proceedings of the Human Factors Society 32nd Annual Meeting (pp. 155–159). Santa Monica, CA: Human Factors Society.
Kramer, A. F., Trejo, L. J., & Humphrey, D. (1995). Assessment of mental workload with task-irrelevant auditory probes. Biological Psychology, 40, 83–100.
Lee, D. H., & Park, K. S. (1990). Multivariate analysis of mental and physical load components in sinus arrhythmia scores. Ergonomics, 33, 35–47.
Levy, C. M., & Ransdell, S. (1995). Is writing as difficult as it seems? Memory & Cognition, 23, 767–779.
Luximon, A., & Goonetilleke, R. S. (2001). Simplified subjective workload assessment technique. Ergonomics, 44, 229–243.
Lysaght, R. J., Hill, S. G., Dick, A. O., Plamondon, B. D., Linton, P. M., Wierwille, W. W., et al. (1989). Operator workload: Comprehensive review and evaluation of operator workload methodologies (Tech. Rep. 851). Fort Bliss, TX: U.S. Army Research Institute, Field Unit.
Miyake, S. (2001). Multivariate workload evaluation combining physiological and subjective measures. International Journal of Psychophysiology, 40, 233–238.
Moroney, W. F., Biers, D. W., & Eggemeier, F. T. (1995). Some measurement and methodological considerations in the application of subjective workload measurement techniques. International Journal of Aviation Psychology, 5, 87–106.
Norman, D. A., & Bobrow, D. G. (1975). On data-limited and resource-limited processes. Cognitive Psychology, 7, 44–64.
O’Donnell, R. D., & Eggemeier, F. T. (1986). Workload assessment methodology. In K. R. Boff, L. Kaufman, & J. P. Thomas (Eds.), Handbook of perception and human performance (Vol. 2, pp. 1–49). New York: Wiley.
Paas, F., Tuovinen, J. E., van Merriënboer, J. J. G., & Darabi, A. A. (2005). A motivational perspective on the relation between mental effort and performance: Optimizing learner involvement in instruction. Journal of Educational Technology, Research, & Development, 53, 5–11.
Papanicolaou, A. C., & Johnstone, J. (1984). Probe evoked potentials: Theory, method and applications. International Journal of Neuroscience, 24, 107–131.
Partala, T., & Surakka, V. (2003). Pupil size variation as an indication of affective processing. International Journal of Human-Computer Studies, 59, 185–198.
Peavler, W. S. (1974). Pupil size, information overload, and performance differences. Psychophysiology, 11, 559–566.
Piolat, A., Olive, T., Roussey, J.-Y., Thunin, O., & Ziegler, J. C. (1999). SCRIPTKELL: A tool for measuring cognitive effort and time processing in writing and other complex cognitive activities. Behavior Research Methods, Instruments, & Computers, 31, 113–121.
Pomplun, M., & Sunkara, S. (2003). Pupil dilation as an indicator of cognitive workload in human-computer interaction. Proceedings of HCI International 2003 (Vol. 3, pp. 542–546). Mahwah, NJ: Erlbaum.
Reid, G. B., Potter, S. S., & Bressler, J. R. (1987). User’s guide for the Subjective Workload Assessment Technique (SWAT) (Tech. Rep. AAMRL-TR-87-023), Wright-Patterson AFB, OH: Armstrong Aerospace Medical Research Laboratory.
Rubio, S., Díaz, E., Martín, J., & Puente, J. M. (2004). Evaluation of subjective mental workload: A comparison of SWAT, NASA-TLX, and workload profile methods. Applied Psychology, 53, 61–86.
Ryu, K., & Myung, R. (2005). Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. International Journal of Industrial Ergonomics, 35, 991–1009.
Schvaneveldt, R. W., Reid, G. B., Gomez, R. L., & Rice, S. (1998). Modeling mental workload. Cognitive Technology, 3, 19–31.
Shi, B., Moloney, K., Emery, V., Jacko, J., Sainfort, F., & Vidakovic, B. (2003). Multifractal discrimination model of high-frequency pupil-diameter measurements. Working paper.
Ullsperger, P., Freude, G., & Erdmann, U. (2001). Auditory probe sensitivity to mental workload changes—An event-related potential study. International Journal of Psychophysiology, 40, 201–209.
Veltman, J. A., & Gaillard, A. W. K. (1998). Physiological indices of workload in a simulated flight task. Biological Psychology, 42, 323–342.
Wickens, C. D. (1984). Processing resources in attention. In R. Parasuraman & D. R. Davies (Eds.), Varieties of attention (pp. 63–102). San Diego, CA: Academic Press.
Wickens, C. D., & Hollands, J. (2000). Engineering psychology and human performance. London: Longmans.
Wierwille, W. W., & Casali, J. G. (1983). A validated rating scale for global mental workload measurement applications. In Proceedings of the 27th Annual Meeting of the Human Factors Society (pp. 129–133). Santa Monica, CA: Human Factors and Ergonomics Society.
Wilson, G. F., & O’Donnell, R. D. (1988). Measurement of operator workload with the neuropsychological workload test battery. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 63–100). Amsterdam: Elsevier.
Xie, B., & Salvendy, G. (2000). Prediction of mental workload in single and multiple tasks environments. International Journal of Cognitive Ergonomics, 4, 213–242.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Cegarra, J., Chevalier, A. The use of Tholos software for combining measures of mental workload: Toward theoretical and methodological improvements. Behavior Research Methods 40, 988–1000 (2008). https://doi.org/10.3758/BRM.40.4.988
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BRM.40.4.988