Abstract
Emotions play a major role in human-to-human communication enabling people to express themselves beyond the verbal domain. In recent years, important advances have been made in unimodal speech and video emotion analysis where facial expression information and prosodic audio features are treated independently. The need however to combine the two modalities in a naturalistic context, where adaptation to specific human characteristics and expressivity is required, and where single modalities alone cannot provide satisfactory evidence, is clear. Appropriate neural network classifiers are proposed for multimodal emotion analysis in this paper, in an adaptive framework, which is able to activate retraining of each modality, whenever deterioration of the respective performance is detected. Results are presented based on the IST HUMAINE NoE naturalistic database; both facial expression information and prosodic audio features are extracted from the same data and feature-based emotion analysis is performed through the proposed adaptive neural network methodology.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Young, J.W.: Head and face anthropometry of adult U.S. civilians, FAA Civil Aeromedical Institute (1993)
Doulamis, A., Doulamis, N., Kollias, S.: On-line Retrainable Neural Networks: Improving the Performance of Neural Networks in Image Analysis Problems. IEEE Transactions on Neural Networks 11(1), 137–157 (2000)
Krog, A., Vedelsby, J.: Neural network ensembles, cross validation and active learning. In: Tesauro, G., Touretzky, D., Leen, T. (eds.) Advances in neural information processing systems 7, pp. 231–238. MIT Press, Cambridge (1995)
Ioannou, S., Raouzaiou, A., Tzouvaras, V., Mailis, T., Karpouzis, K., Kollias, S.: Emotion recognition through facial expression analysis based on a neurofuzzy network. Special Issue on Emotion: Understanding & Recognition, Neural Networks 18(4), 423–435 (2005)
HUMAINE, Human-Machine Interaction Network on Emotion IST-2002-2.3.1.6, http://emotion-research.net/
Fransens, R., De Prins, J.: SVM-based Nonparametric Discriminant Analysis. In: An Application to Face Detection, October 13 - 16, 2003. Ninth IEEE International Conference on Computer Vision, vol. 2 (2003)
Kollias, S., Anastassiou, D.: An adaptive least squares algorithm for the efficient training of artificial neural networks. IEEE Transactions on Circuits and Systems 36(8), 1092–1101 (1989)
Hagan, M.T., Menhaj, M.: Training feedforward networks with the Marquardt algorithm. IEEE Transactions on Neural Networks 5(6), 989–993 (1994)
Raouzaiou, A., Tsapatsoulis, N., Karpouzis, K., Kollias, S.: Parameterized facial expression synthesis based on MPEG-4. EURASIP Journal on Applied Signal Processing 2002(10) (October 2002)
Baldonado, M., Chang, C.-C.K., Gravano, L., Paepcke, A.: The Stanford Digital Library Metadata Architecture. Int. J. Digit. Libr. 1, 108–121 (1997)
Mertens, P.: The Prosogram: Semi-Automatic Transcription of Prosody based on a Tonal Perception Model. In: Bel, B., Marlien, I. (eds.) Proceedings of Speech Prosody, Nara (Japan), March 23-26 (2004), ISBN 2-9518233-1-2
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., Taylor, J.: Emotion Recognition in Human-Computer Interaction. IEEE Signal Processing Magazine (2001)
Picard, R.W.: Affective Computing. MIT Press, Cambridge (2000)
Cowie, R., Douglas-Cowie, E.: Automatic statistical analysis of the signal and prosodic signs of emotion in speech. In: Proceedings of the 4th International Conference of Spoken Language Processing, Philadelphia, USA, pp. 1989–1992 (1996)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ioannou, S., Kessous, L., Caridakis, G., Karpouzis, K., Aharonson, V., Kollias, S. (2006). Adaptive On-Line Neural Network Retraining for Real Life Multimodal Emotion Recognition. In: Kollias, S.D., Stafylopatis, A., Duch, W., Oja, E. (eds) Artificial Neural Networks – ICANN 2006. ICANN 2006. Lecture Notes in Computer Science, vol 4131. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11840817_9
Download citation
DOI: https://doi.org/10.1007/11840817_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-38625-4
Online ISBN: 978-3-540-38627-8
eBook Packages: Computer ScienceComputer Science (R0)