Abstract
Research on multisensory interactions has shown that the perceived timing of a visual event can be captured by a temporally proximal sound. This effect has been termed ‘temporal ventriloquism effect.’ Using the Ternus display, we systematically investigated how auditory configurations modulate the visual apparent-motion percepts. The Ternus display involves a multielement stimulus that can induce either of two different percepts of apparent motion: ‘element motion’ or ‘group motion’. We found that two sounds presented in temporal proximity to, or synchronously with, the two visual frames, respectively, can shift the transitional threshold for visual apparent motion (Experiments 1 and 3). However, such effects were not evident with single-sound configurations (Experiment 2). A further experiment (Experiment 4) provided evidence that time interval information is an important factor for crossmodal interaction of audiovisual Ternus effect. The auditory interval was perceived as longer than the same physical visual interval in the sub-second range. Furthermore, the perceived audiovisual interval could be predicted by optimal integration of the visual and auditory intervals.
Similar content being viewed by others
References
Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14(3):257–262
Allen PG, Kolers PA (1981) Sensory specificity of apparent motion. J Exp Psychol Hum Percept Perform 7(6):1318–1328
Battaglia PW, Jacobs RA, Aslin RN (2003) Bayesian integration of visual and auditory signals for spatial localization. J Opt Soc Am A Opt Image Sci Vis 20(7):1391–1397
Bermant RI, Welch RB (1976) Effect of degree of separation of visual-auditory stimulus and eye position upon spatial interaction of vision and audition. Percept Mot Skills 42(43):487–493
Bertelson P, Radeau M (1981) Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept Psychophys 29(6):578–584
Brainard DH (1997) The psychophysics toolbox. Spat Vis 10(4):433–436
Bruns P, Getzmann S (2008) Audiovisual influences on the perception of visual apparent motion: exploring the effect of a single sound. Acta Psychol 129(2):273–283
Burr D, Banks MS, Morrone MC (2009) Auditory dominance over vision in the perception of interval duration. Exp Brain Res 198(1):49–57
Calvert G, Spence C, Stein BE (2004) The handbook of multisensory processes. MIT Press, Cambridge
Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721
Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433
Ernst MO, Bülthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8(4):162–169
Fendrich R, Corballis PM (2001) The temporal cross-capture of audition and vision. Percept Psychophy 63(4):719–725
Fraisse P (1984) Perception and estimation of time. Annu Rev Psychol 35:1–36
Freeman E, Driver J (2008) Direction of visual apparent motion driven by timing of a static sound. Curr Biol 18:1262–1266
Getzmann S (2007) The effect of brief auditory stimuli on visual apparent motion. Perception 36(7):1089–1103
Goldstone S, Goldfarb JL (1964a) Auditory and visual time judgment. J Gen Psychol 70:369–387
Goldstone S, Goldfarb JL (1964b) Direct comparison of auditory and visual durations. J Exp Psychol 67:483–485
Goldstone S, Lhamon WT (1974) Studies of auditory-visual differences in human time judgment. 1. Sounds are judged longer than lights. Percept Mot Skills 39(1):63–82
Harrar V, Harris LR (2007) Multimodal Ternus: visual, tactile, and multisensory grouping in apparent motion. Perception 36:1455–1464
Howard IP, Templeton WB (1966) Human spatial orientation. Wiley, New York
Jaekl PM, Harris LR (2007) Auditory-visual temporal integration measured by shifts in perceived temporal location. Neurosci Lett 417(3):219–224
Keetels M, Stekelenburg J, Vroomen J (2007) Auditory grouping occurs prior to intersensory pairing: evidence from temporal ventriloquism. Exp Brain Res 180(3):449–456
Kramer P, Yantis S (1997) Perceptual grouping in space and time: evidence from the Ternus display. Percept Psychophys 59(1):87–99
Levitin DJ, MacLean K, Mathews M, Chu L (2000) The perception of cross-modal simultaneity. Int J Comput Anticip Syst 323–329
Ley I, Haggard P, yarrow K (2009) Optimal integration of auditory and vibrotactile information for judgments of temporal order. J Exp Psychol Hum Percept Perform 35(4):1005–1019
Morein-Zamir S, Soto-Faraco S, Kingstone A (2003) Auditory capture of vision: examining temporal ventriloquism. Cogn Brain Res 17(1):154–163
Pantle AJ, Petersik JT (1980) Effects of spatial parameters on the perceptual organization of a bistable motion display. Percept Psychophys 27(4):307–312
Pantle AJ, Picciano L (1976) A multistable movement display: evidence for two separate motion systems in human vision. Science 193(4252):500–502
Pelli DG (1997) The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat Vis 10:437–442
Petersik JT, Rice CM (2006) The evolution of explanations of a perceptual phenomenon: a case history using the Ternus effect. Perception 35(6):807–821
Pick HL, Warren DH, Hay JC (1969) Sensory conflict in judgments of spatial direction. Percept Psychophys 6(4):203–205
Posner MI, Nissen MJ, Klein RM (1976) Visual dominance: an information-processing account of its origins and significance. Psychol Rev 83(2):157–171
Radeau M, Bertelson P (1987) Auditory-visual interaction, the timing of inputs. Thomas (1941) revisited. Psychol Res 49(1):17–22
Sanabria D, Soto-Faraco S, Chan JS, Spence C (2004) When does visual perceptual grouping affect multisensory integration? Cogn Affect Behav Neurosci 4(2):218–229
Scheier C, Nijhawan R, Shimojo S (1999) Sound alters visual temporal resolution. Invest Ophthalmol Vis Sci 40(Suppl. 4):4169
Shimojo S, Scheier C, Nijhawan R, Shams L, Kamitani Y, Watanabe K (2001) Beyond perceptual modality: auditory effects on visual perception. Acoust Sci Technol 22(2):61–67
Soto-Faraco S, Kingstone A (2004) Multisensory integration of dynamic information. In: Calvert G, Spence C, Stein BE (eds) The handbook of multisensory processes. MIT Press, Cambridge
Spence C (2007) Audiovisual multisensory integration. Acoust Sci Technol 28:61–70
Spence C, Sanabria D, Soto-Faraco S (2007) Intersensory Gestalten and crossmodal scene perception. In: Noguchi K (ed) Psychology of beauty and Kansei: new horizons of Gestalt perception. Fuzanbo International, Tokyo, pp 519–579
Stone JV, Hunkin NM, Porrill J, Wood R, Keeler V, Beanland M et al (2001) When is now? Perception of simultaneity. Proc R Soc Lond B Biol Sci 268(1462):31–38
Ternus J (1926) Experimentelle Untersuchungen über phänomenale Identität. Psychologische Forschung 7:81–136
Treutwein B, Strasburger H (1999) Fitting the psychometric function. Percept Psychophys 61(1):87–106
van Erp JBF, Werkhoven PJ (2004) Vibro-tactile and visual asynchronies: sensitivity and consistency. Perception 33:103–111
Vroomen J, de Gelder B (2004) Temporal ventriloquism: sound modulates the flash-lag effect. J Exp Psychol Hum Percept Perform 30:513–518
Vroomen J, Keetels M (2006) The spatial constraint in intersensory pairing: no role in temporal ventriloquism. J Exp Psychol Hum Percept Perform 32(4):1063–1071
Walker JT, Scott KJ (1981) Auditory-visual conflicts in the perceived duration of lights, tones and gaps. J Exp Psychol Hum Percept Perform 7(6):1327–1339
Wearden JH (2006) When do auditory/visual differences in duration judgments occur? Q J Exp Psychol 59(10):1709–1724
Wearden JH, Edwards H, Fakhri M, Percival A (1998) Why “sounds are judged longer than lights”: application of a model of the internal clock in humans. Q J Exp Psychol 51(2):97–120
Welch RB (1999) Meaning, attention, and the ‘unity assumption’ in the intersensory bias of spatial and temporal perceptions. In: Aschersleben G, Bachmann T, Müsseler J (eds) Cognitive contribution to the perception of spatial and temporal events. Elsevier, Amsterdam, pp 317–387
Welch RB, Warren DH (1980) Immediate perceptual response to intersensory discrepancy. Psychol Bull 88(3):638–667
Welch RB, Warren DH (1986) Intersensory interactions. In: Boff KR, Kaufman L, Thomas JP (eds) Handbook of perception and human performance: sensory processes and perception, vol 1. Wiley, New York, pp 1–36
Witten IB, Knudsen EI (2005) Why seeing is believing: merging auditory and visual worlds. Neuron 48(3):489–496
Acknowledgments
This research was supported in part by a grant from the German Research Foundation (DFG) within the Collaborative Research Centre SFB 453. We thank Hans Strasburger for stimulating comments and the reviewers for insightful suggestions.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Shi, Z., Chen, L. & Müller, H.J. Auditory temporal modulation of the visual Ternus effect: the influence of time interval. Exp Brain Res 203, 723–735 (2010). https://doi.org/10.1007/s00221-010-2286-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-010-2286-3