Abstract
In this paper we describe a text dependent audio-visual speaker identification approach that combines face recognition and audio-visual speech-based identification systems. The temporal sequence of audio and visual observations obtained from the acoustic speech and the shape of the mouth are modeled using a set of coupled hidden Markov models (CHMM), one for each phoneme-viseme pair and for each person in the database. The use of CHMM in our system is justified by the capability of this model to describe the natural audio and visual state asynchrony as well as their conditional dependence over time. Next, the likelihood obtained for each person in the database is combined with the face recognition likelihood obtained using an embedded hidden Markov model (EHMM). Experimental results on XM2VTS database show that our system improves the accuracy of the audio-only or video-only speaker identification at all levels of acoustic signal-to-noise ratio (SNR) from 5 to 30db.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
P.N. Belhumeur, J.P Hespanha, and D.J. Kriegman. Eigenfaces vs Fisherfaces: Recognition using class specific linear projection. In Proceedings of Fourth Europeean Conference on Computer Vision, ECCV’96, pages 45–58, April 1996.
M. Brand, N. Oliver, and A. Pentland. Coupled hidden Markov models for complex action recognition. In IEEE International Conference on Computer Vision and Pattern Recognition, pages 994–999, 1997.
C. Chibelushi, F. Deravi, and S.D. Mason. A review of speech-based bimodal recognition. IEEE Transactions on Multimedia, 4(1):23–37, March 2002.
S. Dupont and J. Luettin. Audio-visual speech modeling for continuous speech recognition. IEEE Transactions on Multimedia, 2:141–151, September 2000.
S. Kuo and O.E. Agazzi. Keyword spotting in poorly printed documents using pseudo 2-D Hidden Markov Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(8):842–848, August 1994.
A. Lawrence, C.L. Giles, A.C Tsoi, and A.D. Back. Face recognition: A convolutional neural network approach. IEEE Transactions on Neural Networks, 8(1):98–113, 1997.
Jia Lia, A. Najmi, and R.M. Gray. Image classification by a two-dimensional hidden markov model. IEEE Transactions on Signal Processing, 48(2):517–533, February 2000.
L. Liang, X. Liu, X. Pi, Y. Zhao, and A. V. Nefian. Speaker independent audiovisual continuous speech recognition. In International Conference on Multimedia and Expo, volume 2, pages 25–28, 2002.
R. Lienhart and J. Maydt. An extened set of Haar-like features for rapid objection detection. In IEEE International Conference on Image Processing, volume 1, pages 900–903, 2002.
X. Liu, L. Liang, Y. Zhao, X. Pi, and A. V. Nefian. Audio-visual continuous speech recognition using a coupled hidden Markov model. In International Conference on Spoken Language Processing, 2002.
J. Luettin and G. Maitre. Evaluation protocol for the XM2FDB database. In IDIAP-COM 98-05, 1998.
A. V. Nefian and M. H. Hayes. Face recognition using an embedded HMM. In Proceedings of the IEEE Conference on Audio and Video-based Biometric Person Authentication, pages 19–24, March 1999.
A. V. Nefian, L. Liang, X. Pi, X. Liu, and C. Mao. A coupled hiddenMarkov model for audio-visual speech recognition. In International Conference on Acoustics, Speech and Signal Processing, volume 2, pages 2013–2016, 2002.
Ara V. Nefian. Embedded Bayesian networks for face recognition. In IEEE International Conference on Multimedia and Expo, volume 2, pages 25–28, 2002.
Ara V. Nefian and Monson H. Hayes III. Maximum likelihood training of the embedded HMM for face detection and recognition. In IEEE International Conference on Image Processing, volume 1, pages 33–36, 2000.
C. Neti, G. Potamianos, J. Luettin, I. Matthews, D. Vergyri, J. Sison, A. Mashari, and J. Zhou. Audio visual speech recognition. In Final Workshop 2000 Report, 2000.
Jonathon Phillips. Matching pursuit filters applied to face identification. IEEE Transactions on Image Processing, 7(8):1150–1164, August 1998.
G. Potamianos, J. Luettin, and C. Neti. Asynchronous stream modeling for large vocabulary audio-visual speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing, volume 1, pages 169–172, 2001.
D.A. Reynolds, T.F. Quatieri, and R.B. Dunn. Speaker verification using an adapted gaussian mixture model. Digital Signal Processing, 10:19–41, 2000.
F. Samaria. Face Recognition Using Hidden Markov Models. PhD thesis, University of Cambridge, 1994.
L.K. Saul and M.L. Jordan. Boltzmann chains and hidden Markov models. In G. Tesauro, David S. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7. The MIT Press, 1995.
S.B. Yacouband, S. Luettin, J. Jonsson, K. Matas, and J. Kittler. Audio-visual person verification. In IEEE International Conference on Computer Vision and Pattern Recognition, pages 580–585, 1999.
M. Turk and A.P. Pentland. Face recognition using eigenfaces. In Proceedings of International Conference on Pattern Recognition, pages 586–591, 1991.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Nefian, A.V., Liang, L.H., Fu, T., Liu, X.X. (2003). A Bayesian Approach to Audio-Visual Speaker Identification. In: Kittler, J., Nixon, M.S. (eds) Audio- and Video-Based Biometric Person Authentication. AVBPA 2003. Lecture Notes in Computer Science, vol 2688. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44887-X_88
Download citation
DOI: https://doi.org/10.1007/3-540-44887-X_88
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40302-9
Online ISBN: 978-3-540-44887-7
eBook Packages: Springer Book Archive