Abstract
The work presented in this paper introduces a new model for emotion recognition from videos, Tandem Modelling (TM). The core of the proposed system consists of a hybrid neural network model that joins two feed-forward neural net models with a bottle-neck connection layer (BNL). Specifically, appearance and motion of each video sequence are encoded using a hand-crafted spatio-temporal descriptor. The obtained features are propagated through a not fully-connected neural net (NFCN) and a new tandem features are generated from the BNL. In a second level, a fully connected network (FCN) is trained with the so-extracted features to encode one of the six basic emotional states (anger, disgust, fear, happiness, sadness and surprise) with the neutral state. The classification results reached by the proposed TM show superiority over state-of-the-art approaches.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, J., Chen, Z., Chi, Z., Fu, H.: Facial expression recognition based on multiple feature fusion in video. In: ICCPR 2018 (2018)
Zhao, J., Mao, X.: Learning deep facial expression features from image and optical ow sequences using 3D CNN. Visual Comput. 34(10), 1461–1475 (2018)
Fan, Y., et al.: Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In: Proceedings of the 18th ACM ICMI 2016, Tokyo, Japan, pp. 445–450 (2016)
Kim, D.H., Lee, M.K., Choi, D.Y., Song, B.C.: Multi-modal emotion recognition using semi-supervised learning and multiple temporal models. In: Proceedings of the 19th ACM ICMI 2017, Glasgow, UK, pp. 569–576 (2017)
Bargal, S.A., Barsoum, E., Ferrer, C.C., Zhang, C.: Emotion recognition in the wild from videos using images. In: Proceedings of the 18th ACM ICMI 2016, Tokyo, Japan, pp. 433–436 (2016)
Liu, C., et al.: Multiple spatio-temporal feature learning for video-based emotion recognition in the wild. In: Proceedings of the 20th ACM ICMI 2018, Boulder, CO, USA, pp. 646–652 (2018)
Yao, A., Cai, D., Hu, P., Wang, S., Sha, L., Chen, Y.: HoloNet: towards robust emotion recognition in the wild. In: Proceedings of the 18th ACM ICMI 2016, Tokyo, Japan, pp. 472–478 (2016)
Hu, P., Cai, D., Wang, S., Yao, A., Chen, Y.: Learning supervised scoring ensemble for emotion recognition in the wild. In: Proceedings of the 19th ACM ICMI17, Glasgow, UK, pp. 553–560 (2017)
Fan, Y., Lam, J.C.K., Li, V.O.K.: Networks, video-based emotion recognition using deeply-supervised neural. In: Proceedings of the 20th ACM ICMI18, Boulder, CO, USA, pp. 584–588 (2018)
Yan, J., et al.: Multi-clue fusion for emotion recognition in the wild. In: Proceedings of the 18th ACM ICMI16, Tokyo, Japan, pp. 458–463 (2016)
Liu, C., Tang, T., Lv, K., Wang, M.: Multi-features based emotion recognition for video clips. In: Proceedings of the 20th ACM ICMI18, Boulder, CO, USA, pp. 630–634 (2018)
Hermansky, H., Ellis, D.P.W., Sharma, S.: Tandem connectionist feature extraction for conventional HMM systems. In: Proceedings of ICASSP, Istanbul, Turkey, pp. 1635–1638 (2000)
Grezl, F., Karafiat, M., Kontav, S., Cernocky, J.: Probabilistic and bottle-neck features for LVCSR of meetings. In: Proceedings of ICASSP, Honolulu, HI, USA, pp. 757–760 (2007)
Kasraoui, S., Lachiri, Z., Madani, K.: Probabilistic and bottleneck tandem features for emotion recognition from videos. In: Proceedings of the International Conferences Interfaces and Human Computer Interaction, Game and Entertainment Technologies and Computer Graphics, pp. 387–391. IADIS Press, Madrid (2018)
Haq, S., Jackson, P.J.B., Edge, J.: Speaker-dependent audio-visual emotion recognition. In: AVSP (2009)
Dhall, A., Gedeon, L.T., Frampton, T.D., et al.: Acted Facial Expressions in the Wild Database, Technical report (2011)
Dhall, A., et al.: Emotiw 2016: video and group-level emotion recognition challenges. In: Proceedings of the 18th ACM ICMI16, Tokyo, Japan, pp. 427–432 (2016)
Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. PAMI 29(6), 915–928 (2007)
Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Kasraoui, S., Lachiri, Z., Madani, K. (2019). Tandem Modelling Based Emotion Recognition in Videos. In: Rojas, I., Joya, G., Catala, A. (eds) Advances in Computational Intelligence. IWANN 2019. Lecture Notes in Computer Science(), vol 11507. Springer, Cham. https://doi.org/10.1007/978-3-030-20518-8_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-20518-8_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20517-1
Online ISBN: 978-3-030-20518-8
eBook Packages: Computer ScienceComputer Science (R0)