Abstract
Contrary to the predictions of established theory, Schutz and Lipscomb (2007) have shown that visual information can influence the perceived duration of concurrent sounds. In the present study, we deconstruct the visual component of their illusion, showing that (1) cross-modal influence depends on visible cues signaling an impact event (namely, a sudden change of direction concurrent with tone onset) and (2) the illusion is controlled primarily by the duration of post-impact motion. Other aspects of the post-impact motion—distance traveled, velocity, acceleration, and the rate of its change (i.e., its derivative, jerk)—play a minor role, if any. Together, these results demonstrate that visual event duration can influence the perception of auditory event duration, but only when stimulus cues are sufficient to give rise to the perception of a causal cross-modal relationship. This refined understanding of the illusion’s visual aspects is helpful in comprehending why it contrasts so markedly with previous research on cross-modal integration, demonstrating that vision does not appreciably influence auditory judgments of event duration (Walker & Scott, 1981).
Article PDF
Similar content being viewed by others
References
Alais, D., & Burr, D. (2004). The ventriloquist effect results from nearoptimal bimodal integration. Current Biology, 14, 257–262.
Arieh, Y., & Marks, L. E. (2008). Cross-modal interaction between vision and hearing: A speed-accuracy analysis. Perception & Psychophysics, 70, 412–421.
Arrighi, R., Alais, D., & Burr, D. (2006). Perceptual synchrony of audiovisual streams for natural and artificial motion sequences. Journal of Vision, 6, 260–268.
Baayen, R. H. (2008). Analyzing linguistic data: A practical introduction to statistics using R. Cambridge: Cambridge University Press.
Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008). Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory & Language, 59, 390–412.
Bates, D. M., & Sarkar, D. (2007). lme4: Linear mixed-effects models using S4 classes (R package Version 0.9975-13).
Battaglia, P. W., Jacobs, R. A., & Aslin, R. N. (2003). Bayesian integration of visual and auditory signals for spatial localization. Journal of the Optical Society of America, 20, 1391–1397.
Bedford, F. L. (2001a). Object identity theory and the nature of general laws: Commentary reply. Cahiers de Psychologie Cognitive/Current Psychology of Cognition, 20, 277–293.
Bedford, F. L. (2001b). Towards a general law of numerical/object identity. Cahiers de Psychologie Cognitive/Current Psychology of Cognition, 20, 113–175.
Bedford, F. L. (2004). Analysis of a constraint on perception, cognition, and development: One object, one place, one time. Journal of Experimental Psychology: Human Perception & Performance, 30, 907–912.
Bertelson, P., & Radeau, M. (1981). Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Perception & Psychophysics, 29, 578–584.
Bertelson, P., Vroomen, J., de Gelder, B., & Driver, J. (2000). The ventriloquist effect does not depend on the direction of deliberate visual attention. Perception & Psychophysics, 62, 321–332.
Broughton, M., & Stevens, C. (2009). Music, movement and marimba: An investigation of the role of movement and gesture in communicating musical expression to an audience. Psychology of Music, 37, 137–153.
Crawley, M. J. (2007). The R book. Chichester, U.K.: Wiley.
Dahl, S., & Friberg, A. (2007). Visual perception of expressiveness in musicians’ body movements. Music Perception, 24, 433–454.
Davidson, J. W. (1993). Visual perception of performance manner in the movements of solo musicians. Psychology of Music, 21, 101–113.
Davidson, J. W. (1994). Which areas of a pianist’s body convey information about expressive intention to an audience? Journal of Human Movement Studies, 26, 279–301.
Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433.
Fendrich, R., & Corballis, P. M. (2001). The temporal cross-capture of audition and vision. Perception & Psychophysics, 63, 719–725.
Gillespie, R. (1997). Ratings of violin and viola vibrato performance in audio-only and audiovisual presentations. Journal of Research in Music Education, 45, 212–220.
Hidalgo-Barnes, M., & Massaro, D. W. (2007). Read my lips: An animated face helps communicate musical lyrics. Psychomusicology, 19, 3–12
Ihaka, R., & Gentleman, R. (1996). R: A language for data analysis and graphics. Journal of Computational & Graphical Statistics, 5, 299–314.
Jack, C. E., & Thurlow, W. R. (1973). Effects of degree of visual association and angle of displacement on the “ventriloquism” effect. Perceptual & Motor Skills, 37, 967–979.
Jackson, C. (1953). Visual factors in auditory localization. Quarterly Journal of Experimental Psychology, 5, 52–65.
Johansson, G. (1973). Visual perception of biological motion and a model for its analysis. Perception & Psychophysics, 14, 201–211.
Körding, K. P., Beierholm, U., Ma, W. J., Quartz, S., Tenenbaum, J. B., & Shams, L. (2007). Causal inference in multisensory perception. PLoS ONE, 2, 1–10. doi:10.1371/journal.pone.0000943
Kreft, I. G. G., & de Leeuw, J. (1998). Introducing multilevel modeling. London: Sage.
Krumhansl, C., & Schenck, D. L. (1997). Can dance reflect the structural and expressive qualities of music? A perceptual experiment on Balanchine’s choreography of Mozart’s Divertimento No. 15. Musicae Scientiae, 1, 63–85.
Kubovy, M., & Schutz, M. (in press). Audio-visual objects. Review of Philosophy & Psychology.
Loftus, G. R. (2002). Analysis, interpretation, and visual presentation of experimental data. In H. Pashler (Series Ed.) & J. Wixted (Vol. Ed.), Stevens’ handbook of experimental psychology: Vol. 4. Methodology in experimental psychology (pp. 339–390). New York: Wiley.
Maxwell, S. E., & Delaney, H. D. (2004). Designing experiments and analyzing data: A model comparison perspective (2nd ed.). Mahwah, NJ: Erlbaum.
Petrini, K., Russell, M., & Pollick, F. (2009). When knowing can replace seeing in audiovisual integration of actions. Cognition, 110, 432–439.
Pollick, F. E., Lestou, V., Ryu, J., & Cho, S.-B. (2002). Estimating the efficiency of recognizing gender and affect from biological motion. Vision Research, 42, 2345–2355.
Pollick, F. E., Paterson, H. M., Bruderlin, A., & Sanford, A. J. (2001). Perceiving affect from arm movement. Cognition, 82, B51-B61.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. London: Sage.
Rosenblum, L. D., & Fowler, C. A. (1991). Audiovisual investigation of the loudness-effort effect for speech and nonspeech events. Journal of Experimental Psychology: Human Perception & Performance, 17, 976–985.
Saldaña, H. M., & Rosenblum, L. D. (1993). Visual influences on auditory pluck and bow judgments. Perception & Psychophysics, 54, 406–416.
Saygin, A. P., Driver, J., & de Sa, V. R. (2008). In the footsteps of biological motion and multisensory perception: Judgments of audiovisual temporal relations are enhanced for upright walkers. Psychological Science, 19, 469–475.
Schutz, M. (2008). Seeing music? What musicians need to know about vision. Empirical Musicology Review, 3, 83–108.
Schutz, M., & Kubovy, M. (2009). Deconstructing a musical illusion: Point-light representations capture salient properties of impact motions. Canadian Acoustics, 37, 23–28.
Schutz, M., & Kubovy, M. (in press). Causality and cross-modal integration. Journal of Experimental Psychology: Human Perception & Performance.
Schutz, M., & Lipscomb, S. (2007). Hearing gestures, seeing music: Vision influences perceived tone duration. Perception, 36, 888–897.
Sekuler, R., Sekuler, A. B., & Lau, R. (1997). Sound alters visual motion perception. Nature, 385, 308.
Shipley, T. (1964). Auditory flutter-driving of visual flicker. Science, 145, 1328–1330.
Snijders, T., & Bosker, R. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.
Spence, C. (2007). Audiovisual multisensory integration. Acoustic Science & Technology, 28, 61–70.
Thomas, G. J. (1941). Experimental study of the influence of vision on sound localization. Journal of Experimental Psychology, 28, 163–177.
Thompson, W. F., Graham, P., & Russo, F. A. (2005). Seeing music performance: Visual influences on perception and experience. Semiotica, 156, 203–227.
Thompson, W. F., & Russo, F. A. (2007). Facing the music. Psychological Science, 18, 756–757.
Thompson, W. F., Russo, F. A., & Quinto, L. (2008). Audio-visual integration of emotional cues in song. Cognition & Emotion, 22, 1457–1470.
Tremoulet, P. D., & Feldman, J. (2000). Perception of animacy from the motion of a single object. Perception, 29, 943–951.
Vatakis, A., & Spence, C. (2007). Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli. Perception & Psychophysics, 69, 744–756.
Vatakis, A., & Spence, C. (2008). Evaluating the influence of the “unity assumption” on the temporal perception of realistic audiovisual stimuli. Acta Psychologica, 127, 12–23.
Vines, B. W., Krumhansl, C. L., Wanderley, M. M., & Levitin, D. J. (2006). Cross-modal interactions in the perception of musical performances. Cognition, 101, 80–113.
Vroomen, J. (1999). Ventriloquism and the nature of the unity decision: Commentary on Welch. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.), Cognitive contributions to the perception of spatial and temporal events (pp. 389–393). Amsterdam: Elsevier.
Wada, Y., Kitagawa, N., & Noguchi, K. (2003). Audio-visual integration in temporal perception. International Journal of Psychophysiology, 50, 117–124.
Walker, J. T., & Scott, K. J. (1981). Auditory-visual conflicts in the perceived duration of lights, tones, and gaps. Journal of Experimental Psychology: Human Perception & Performance, 7, 1327–1339.
Wapnick, J., Darrow, A.-A., Kovacs, J., & Dalrymple, L. (1997). Effects of physical attractiveness on evaluation of vocal performance. Journal of Research in Music Education, 45, 470–479.
Wapnick, J., Mazza, J. K., & Darrow, A.-A. (1998). Effects of performer attractiveness, stage behavior, and dress on violin performance evaluation. Journal of Research in Music Education, 46, 510–521.
Welch, R. B. (1972). The effect of experienced limb identity upon adaptation to simulated displacement of the visual field. Perception & Psychophysics, 12, 453–456.
Welch, R. B. (1999). Meaning, attention, and the “unity assumption” in the intersensory bias of spatial and temporal perceptions. In G. Aschersleben, T. Bachmann, & J. Müsseler (Eds.), Cognitive contributions to the perception of spatial and temporal events (pp. 371–387). Amsterdam: Elsevier.
Welch, R. B., DuttonHurt, L. D., & Warren, D. H. (1986). Contributions of audition and vision to temporal rate perception. Perception & Psychophysics, 39, 294–300.
Welch, R. B., & Warren, D. H. (1980). Immediate perceptual response to intersensory discrepancy. Psychological Bulletin, 88, 638–667.
Wilkinson, L., & the Task Force on Statistical Inference, American Psychological Association (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604.
Witkin, H. A., Wapner, S., & Leventhal, T. (1952). Sound localization with conflicting visual and auditory cues. Journal of Experimental Psychology, 43, 58–67.
Author information
Authors and Affiliations
Corresponding authors
Additional information
Supported by NIDCD Grant R01 DC 005636 (M.K., principal investigator). The research was performed by J.A.A. for a thesis in the Psychology Department Distinguished Majors Program (M.K., advisor; William Epstein, reader).
This study was completed while M.S. was in the graduate program at the University of Virginia.
Rights and permissions
About this article
Cite this article
Armontrout, J.A., Schiutz, M. & Kubovy, M. Visual determinants of a cross-modal illusion. Attention, Perception, & Psychophysics 71, 1618–1627 (2009). https://doi.org/10.3758/APP.71.7.1618
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/APP.71.7.1618