Abstract
Visual information provided by a talker’s mouth movements can influence the perception of certain speech features. Thus, the “McGurk effect” shows that when the syllable /bi/ is presented audibly, in synchrony with the syllable /gi/, as it is presented visually, a person perceives the talker as saying /di/. Moreover, studies have shown that interactions occur between place and voicing features in phonetic perception, when information is presented audibly. In our first experiment, we asked whether feature interactions occur when place information is specified by a combination of auditory and visual information. Members of an auditory continuum ranging from /ibi/ to /ipi/ were paired with a video display of a talker saying /igi/. The auditory tokens were heard as ranging from /ibi/ to /ipi/, but the auditory-visual tokens were perceived as ranging from /idi/ to /iti/. The results demonstrated that the voicing boundary for the auditory-visual tokens was located at a significantly longer VOT value than the voicing boundary for the auditory continuum presented without the visual information. These results demonstrate that place-voice interactions are not limited to situations in which place information is specified audibly. In three follow-up experiments, we show that (1) the voicing boundary is not shifted in the absence of a change in the global percept, even when discrepant auditory-visual information is presented; (2) the number of response alternatives provided for the subjects does not affect the categorization or the VOT boundary of the auditory-visual stimuli; and (3) the original effect of a VOT boundary shift is not replicated when subjects are forced by instruction to \ldrelabel\rd the /b-p/auditory stimuli as/d/or/t/. The subjects successfully relabeled the stimuli, but no shift in the VOT boundary was observed.
Article PDF
Similar content being viewed by others
References
Carden, G., Levevitt, A., Jusczyk, P. W., &Walley, A. (1981). Evidence for phonetic processing of cues to place of articulation: Perceived manner affects perceived place.Perception & Psychophysics,29, 26–36.
Eimas, P. D., &Corbit, J. D. (1973). Selective adaplation of linguistic feature detectors.Cognitive Psychology,4, 99–109.
Fitch, H. L., Halwes, T., Erickson, D. M., &Liberman, A. M. (1980). Perceptual equivalence of two acoustic cues for stop-consonant manner.Perception & Psychophysics,27, 343–350.
Foster, G. A. (1984). Where do features interact?Proceedings of the Institute of Acoustics,6, 393–400.
Ganong, W. F. (1980). Phonetic categorization in auditory word perception.Journal of Experimental Psychology: Human Perception & Performance,6, 110–125.
Green, K. P., &Kuhl, P. K. (1988). The interaction of visual place and auditory voicing information during the perception of speech.Journal of the Acoustical Society of America,113, S85.
Hillenbrand, J. (1984). Perception of sine-wave analogs of voice onset time stimuli.Journal of the Acoustical Society of America,75, 231–240.
Hirsh, I. J. (1959). Auditory perception of temporal order.Journal of the Acoustical Society of America,31, 759–767.
Jusczyk, P. (1985). On characterizing the development of speech perception. In J. Mehler & R. Fox (Eds.),Neonate cognition: Beyond the blooming buzzing confusion (pp. 199–229). Hillsdale, NJ: Edbaum.
Kuhl, P. K. (1982). Speech perception: An overview of current issues. In N. J. Lass, L. V. McReynolds, J. L. Northern, & D. E. Yoder (Eds.),Speech, language, and hearing: Vol. I. Normal processes (pp. 286–322). Philadelphia, PA: W. B. Saunders.
Kuhl, P. K. (1986). Theoretical contributions of tests on animals to the special-mechanisms debate in speech.Experimental Biology,45, 233–265.
Kuhl, P. K. (1987a). Perception of speech and sound in early infancy. In P. Salapatek & L. Cohen (Eds.),Handbook of infant perception: Vol. II. From perception to cognition (pp. 275–382). New York: Acadetmc Press.
Kuhl, P. K. (1987b). The special-mechanisms debate in speech research: Categorization tests on animals and infants. In S. Hamad (Ed.),Categorical perception: The groundwork of cognition (pp. 355–386). Cambridge, England: Cambridge University Press.
Kuhl P. K., Geen, K. P., &Meltzoff, A. N. (1988). Factors affecting the integration of auditory and visual information in speech: The level effect.Journal of the Acouaical Society of America,113, S86.
Kuhl, P. K., &Meltzoff, A. (1982). The bimodal perception of speech in infancy.Science,218, 1138–1141.
Kuhl, P. K., &Miller, J. D. (1978). Speech perception by the chinchilla: Identification functions for synthetic VOT stimuli.Journal of the Acoustical Society of America,63, 905–917.
Kuhl, P. K., &Padden, D. M. (1982). Enhanced discriminability at the phonetic boundaries for the voicing feature in macaques.Perception & Psychophysics,32, 542–550.
Liberman, A. M. (1982). On the finding that speech is special.Amencan Psychologist,37, 148–167.
Liberman, A. M., Delattre, P., &Cooper, F. S. (1958). Distinction between voiced and voiceless stops.Language & Speech,1, 153–167.
Lisker, L. (1975). Is It VOT or a first-formant detector?Journal of the Acoustical Society of America,57, 1547–1551.
Lisker, L., &Abramson, A. S. (1964). A cross-language study of voicing in initial stops: Acoustical measurements.Word,20, 384–422.
Lisker, L., &Abramson, A. S. (1970). The voicing dimension: Some experiments in comparative phonetics. InProceedings of the Sixth International Congress of Phonetic Sciences (pp. 563–567). Prague: Academia.
MacDonald, J., &McGurk, H. (1978). Visual influences on speech perception processes.Perception & Psychophysics,24, 253–257.
Manuel, S. Y., Repp, B. H., Liberman, A. M., &Studdert-Kennedy, M. (1983). Exploring the “McGurk Effect”.Journal of the Acoustical Society of America,74, S66.
Massaro, D., &Cohen, M. (1983). Evaluation and integration of visual and an&tory information in speech perception.Journal of Experimental Psychology: Human Perception & Performance,9, 753–771.
Massaro, D. W., &Oden, G. C. (1980). Evaluation and integration of acoustic features in speech perception.Journal of the Acoustical Society of America,67, 996–1013.
McGurk, H., &MacDonald, J. (1976). Hearing lips and seeing voices.Nature,264, 746–748.
Miller, J. D., Wier, C. C., Pastore, R. E., Kelly, W. J., &Dooling, R. J. (1976). Discrimination and labeling of noise-buzz sequences with varying noise-lead times: An example of categorical perception.Journal of the Acoustical Society of America,60, 410–417.
Miller, J. L. (1977). Nonindependence of feature processing in initial consonants.Journal of Speech & Hearing Research,20, 510–518.
Miller, J. L. (1981). Phonetic perception: Evidence for contextdependent and context-independent processing.Journal of the Acoustical Society of America,69, 822–831.
Miller, J. L., &Eimas, P. D. (1983). Studies on the categorization of speech by infants.Cognition,13, 135–165.
Mills, A. E., &Thiem, R. (1980). Auditory-visual fusions and illusions in speech perception.Linguistische Berichte,68, 85–109.
Parker, E. M. (1988). Auditory constraints on the perception of voiceonset time: The influence of lower tone frequency on judgments of tone-onset simu Raneity.Journal of the Acoustical Society of America,83, 1597–1607.
Pisoni, D. B. (1977). Identification and discrimination of the relative onset time of two component tones: Implications for voicing perception in stops.Journal of the Acoustical Soctet of America,61, 1352–1361.
Rein, B. H. (1983). Trading relations among acoustic cues in speech perception are largely a result of phonetic categorization.Speech Communication,2, 341–361.
Roberts, M., &Summerfield, Q. (1981). Audiovisual presentation demonstrates that selective adaptation in speech perception is purely auditory.Perception & Psychophysics,30, 309–314.
Sawusch, J. R., &Pisoni, D. B. (1974). On the identification of place and voicing features in synthetic stop consonants.Journal of Phonetics,2, 181–194.
Stevens, K. N., &Klatt, D. H. (1974). Role of formant transitions in the voiced-voiceless distinction for stops.Journal of the Acoustical Society of America,55, 653–659.
Summerfield, Q. (1979). Use of visual information for phonetic perception.Phonetica,36, 314–331.
Summerffield, Q. (1982). Differences between spectral dependencies in auditory and phonetic temporal processing: Relevance to the perception of voicing in imtial stops.Journal of the Acoustical Society of America,72, 51–61.
Summerfield, Q. (1986). Some preliminaries to a comprehensive account of audio-visual speech perception. In R. Campbell & B. Dodd (Eds.),Hearing by eye (pp. 3–51). London: Erlbaum.
Summerfield, Q., &Haggard, M. P. (1977). On the dissociation of spectral and temporal cues to the voicing distinction in initial stop consonants.Journal of the Acoustical Society of America,62, 435–448.
Author information
Authors and Affiliations
Additional information
This research was supported by NIH Grant HD-18286 to Patricia K. Kuhl. Kerry P. Green was supported by NIH Training Grant HD-07239 to the University of Washington.
Rights and permissions
About this article
Cite this article
Green, K.P., Kuhl, P.K. The role of visual information in the processing of. Perception & Psychophysics 45, 34–42 (1989). https://doi.org/10.3758/BF03208030
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03208030