Abstract
We describe audio-to-visual conversion techniques for efficient multimedia communications. The audio signals are automatically converted to visual images of mouth shape. The visual speech can be represented as a sequence of visemes, which are the generic face images corresponding to particular sounds. Visual images synchronized with audio signals can provide user-friendly interface for man machine interactions. Also, it can be used to help the people with impaired-hearing. We use HMMs (hidden Markov models) to convert audio signals to a sequence of visemes. In this paper, we compare two approaches in using HMMs. In the first approach, an HMM is trained for each viseme, and the audio signals are directly recognized as a sequence of visemes. In the second approach, each phoneme is modeled with an HMM, and a general phoneme recognizer is utilized to produce a phoneme sequence from the audio signals. The phoneme sequence is then converted to a viseme sequence. We implemented the two approaches and tested them on the TIMIT speech corpus. The viseme recognizer shows 33.9% error rate, and the phoneme-based approach exhibits 29.7% viseme recognition error rate. When similar viseme classes are merged, we have found that the error rates can be reduced to 20.5% and 13.9%, respectably.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
K. Choi, and J. Hwang, “Baum-Welch HMM inversion for audio-to-visual conversion”, IEEE International Workshop on Multimedia Signal Processing, pp. 175–180, 1999.
C. Fisher, “Confusions among visually perceived consonants”, Journal on Speech and Hearing Research, vol. 11, pp. 796–804, 1968.
K. Grant, B. Walden, and P. Seitz, “Auditory-visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory-visual integration”, Journal of Acoustic Society of America, vol. 103, pp. 2677–2690, 1998.
S. Morishima and H. Harashima, “A media conversion from speech to facial image for intelligent man-machine interface”, IEEE Journal on selected areas in communications, vol. 9, no. 4, pp. 594–600, 1991.
Nist Speech Disc 1-1.1, TIMIT Acoustic-phonetic continuous speech corpus, October 1990.
L. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition”, Proceedings of the IEEE, vol. 77, no.2, pp. 257–286, 1989.
R. Rao, T. Chen and R. Mersereau, “Audio-to-visual conversion for multimedia communication”, IEEE Transaction on Industrial Electronics, vol. 45, no. 1, pp. 15–22, 1998.
A. Rogozan and P. Delelise, “Adaptive fusion of acoustic and visual sources for automatic speech recognition”, Speech Communication, vol. 26, pp. 149–161, 1998.
S. Tamura and A. Waibel, “Noise reduction using connectionist models”, IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 553–556, 1988.
A. Viterbi, “Error bounds of convolutional codes and an asymmetrically optimum decoding algorithm”, IEEE Transactions on Information Theory, IT-13, pp. 260–267, 1967.
A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. Lang, “Phoneme recognition using time-delay neural networks”, IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 37, no. 3, pp. 328–339, 1989.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Lee, S., Yook, D. (2002). Audio-to-Visual Conversion Using Hidden Markov Models. In: Ishizuka, M., Sattar, A. (eds) PRICAI 2002: Trends in Artificial Intelligence. PRICAI 2002. Lecture Notes in Computer Science(), vol 2417. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45683-X_60
Download citation
DOI: https://doi.org/10.1007/3-540-45683-X_60
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-44038-3
Online ISBN: 978-3-540-45683-4
eBook Packages: Springer Book Archive