Abstract
Several research methodologies and human face image databases have been developed based on deliberately produced facial expressions of prototypical emotions. However, real-time and spontaneous facial expression recognition cannot be adequately handled by those existing methods and datasets. To address this problem, research efforts have been made to create spontaneous facial expression image datasets as well as to develop algorithms that can process naturally induced affective behavior. This paper introduces these advances and focuses on a small and specific area of spontaneous facial expression recognition. In this paper, we are concentrating on non-posed image acquisition protocols, which strongly influence the subjects for evoking expressions as natural as possible. We categorize the acquisition protocols into four different parts: image acquisition while playing video games, watching emotional videos, during interviews and from other sources. The taxonomy of facial expression acquisition protocols tells about the typical conditions responsible for producing specific facial expressions in that condition. We also address some important design issues related to spontaneous facial expression recognition systems and list the facial expression databases, which are strictly not acted and non-posed. We also put light on the applications of spontaneously evoked facial expression acquisition and recognition because they have potential medical significance. Moreover, we provide a comprehensive analysis and summary of spontaneous facial expression recognition methods by revealing their pros and cons for future researchers.
Similar content being viewed by others
References
Aghevli MA, Blanchard JJ, Horan WP (2003) The expression and experience of emotion in schizophrenia: a study of social interactions. Psychiatry Res 119(3):261–270
Aina S, Zhou M, Chambers JA, Phan RC-W (2014) A new spontaneous expression database and a study of classification-based expression analysis methods, Proc. 22nd European Conf. Signal Processing, pp. 2505–2509
Alves NT (2013) Recognition of static and dynamic facial expressions: a study review. Estud Psicol 18(1):125–130
Antonin G, Popovici V, Thiran JP (2003) Independent component analysis and support vector machine for face feature extraction, in Proc. 4th International Conference on Audio- and Video-Based Biometric Person Authentication, pp. 111–118
Ashraf AB, Lucey S, Cohn JF, Chen T, Ambadar Z, Prkachin KM, Solomon PE (2009) The painful face-pain expression recognition using active appearance models. Img and vis Comput 27(12):1788–1796
Bänziger T, Mortillaro M, Scherer KR (2012) Introducing the Geneva Multimodal Expression corpus for experimental research on emotion perception. Emotion 12(5):1161–1179
Baron-Cohen S, Wheelwright S, Jolliffe T (1997) Is there a “language of the eyes”? Evidence from normal adults, and adults with autism or Asperger Syndrome. Vis Cogn 4(3):311–331
Bartlett MS, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2005) Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior, IEEE Conf. Computer Vision and Pattern Recognition, pp. 568–573
Berenbaum H (1992) Posed facial expressions of emotion in schizophrenia and depression. Psychol Med 22(4):929–937
Berenbaum H, Oltmanns TF (1992) Emotional experience and expression in schizophrenia and depression. J Abnorm Psychol 101(1):37–44
Bettadapura V (2012) Facial Expression Recognition and Analysis: The State of the Art, CoRR, abs/1203.6722
Biel J-I, Teijeiro-Mosquera L, Gatica-Perez D (2012) “FaceTube: predicting personality from facial expressions of emotion in online conversational video,” in Proc. ACM ICMI
Blom PM, Bakkes S, Tan CT, Whiteson S, Roijers D, Valenti R, Gevers T (2014) Towards Personalised Gaming via Facial Expression Recognition, Proc. Tenth Annual AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pp. 30–36
Cerezo E, Hupont I (2006) Emotional facial expression classification for multimodal user interfaces, Proc. 4th Int. Conf. AMDO, Port d’Andratx, Mallorca, Spain, pp. 405–413
Cho S-Y, Teoh T-T, Nguwi Y-Y (2009) Development of an Intelligent Facial Expression Recognizer for Mobile Applications, New Advances in Intelligent Decision Technologies, Studies in Computational Intelligence, pp. 21–29
Cohen I, Garg A, Huang TS (2000) Emotion Recognition from Facial Expressions using Multilevel HMM, presented at the Neural Inf. Process. Syst. (NIPS) Workshop Affective Comput., Colorado
Cohen L, Sebe N, Garg A, Chen L, Huang T (2003) Facial expression recognition from video sequences: Temporal and static modeling. Comp Vis and Img Understanding 91(1–2):160–1878
Cohn JF, Tronick EZ (1983) Three-month-old infants’ reaction to simulated maternal depression. Child Dev 54:185–193
Cohn JF, Kruez TS, Matthews I, Yang Y, Nguyen MH, Padilla MT, Zhou F, De la Torre F (2009) Detecting depression from facial actions and vocal prosody, Proc. 3rd Int. Conf. on Affective Computing and Intelligent Interaction, Amsterdam, Netherlands
Colbry D, Stockman G, Jain AK (2005) Detection of anchor points for 3d face verification. Proc. IEEE Workshop on Computer Vision and Pattern Recognition, San Diego, CA, USA
Cowing R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JG (2001) Emotion recognition in human-computer interaction, IEEE Signal Processing Magazine, pp. 33–80
De la Torre F and Cohn JF (2011) Facial Expression Analysis, in V. K. T. B. Moeslund, A. Hilton, and L. Sigal,(Eds.) Guide to Visual Analysis of Humans: Looking at People, New York: Springer-Verlag
Dhall A, Goecke R, Lucey S, Gedeon T (2012) Collecting large, richly annotated facial-expression databases from movies. IEEE Multimedia 19(3):34–41
Diaz RL, Wong U, Hodgins DC, Chiu CG, Goghari VM (2015) Violent Video Game Players and Non-Players Differ on Facial Emotion Recognition. Aggress Behav 9999:1–13
Douglas-Cowie E, Campbell N, Cowie R, Roach P (2003) Emotional speech: Towards a new generation of databases. Speech Comm 40:33–60
G. J. Edwards, T. F. Cootes, and C.J. Taylor, ªFace Recognition Using Active Appearance Models,” Proc European Conf Computer Vision, vol. 2, pp. 581–695, 1998.
Ekman P (2003) Darwin, deception, and facial expression. Ann N Y Acad Sci 1000(1):205–221
Ekman P, Friesen W (1982) Felt, false, and miserable smiles. J Nonverbal Behav 6(4):238–252
Ekman P, Rosenberg E (1997) What the Face Reveals: Basic and Applied Studies of Spontaneous Expression using the Facial Action Coding System (FACS). Oxford University Press, New York
Ekman P, Roper G, Hager JC (1980) Deliberate facial movement. Child Dev 51:886–891
Ekman P, Hager J, Friesen WV (1981) The symmetry of emotional and deliberate facial actions. Psychophysiology 18(2):101–106
Ekman P, Friesen WV, Ellsworth P (1982) What are the rela-tive contributions of facial behavior and contextual information to the judgment of emotion? In P. Ekman (Ed.), Emotion in the human face, Cambridge: Cambridge University Press, pp. 111–127
Eysenc MW (2004) Psychology: An International Perspective. Psychology Press, UK
Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36:259–275
Fernandez-Dols JM, Sierra B, Ruiz-Belda MA (1993) On the clarity of expressive and contextual information in the recognition of emotions: A methodological critique. European J. of Soc. Psychol. 23:195–202
Fridlund A (1994) Human facial expression: an evolutionary view. Academic Press, New York
Gaebel W, Wolwer W (2004) Facial expressivity in the course of schizophrenia and depression. Eur Arch Psychiatry Clin Neurosci 254(5):335–342
Gajsek R, Struc V, Mihelic F, Podlesek A, Komidar L, Socan G, Bajec B (2009) Multi-modal emotional database: AvID. Informatica 33:101–106
R. Gajšek, V. Štruc, B. Vesnicer, A. Podlesek, L. Komidar, and F. Mihelič (2009) Analysis and assessment of AvID: multi-modal emotional database, Proc. Int. Conf. Text, Speech and Dialogue, Pilsen, Czech Republic, pp. 266–273
Gan Q, Wu C, Wang S (2015) Posed and Spontaneous Facial Expression Differentiation Using Deep Boltzmann Machines, Int. Conf. Affective Computing and Intelligent Interaction, vol. 643–648
Gilbert DT, Krull DS (1988) Seeing less and knowing more: The benefits of perceptual ignorance. J Pers Soc Psychol 54:193–201
Grafsgaard JF, Wiggins JB, Boyer KE, Wiebe EN, Lester JC (2013) Automatically recognizing facial expression: predicting engagement and frustration, Proc. Int. Conf. Educational Data Mining
Grimm M, Kroschel K, Narayanan S (2008) The Vera am Mittag German audio-visual emotional speech database, IEEE Int. Conf. Multimedia and Expo, Hannover, pp. 865–868
Gritti T, Shan C, Jeanne V, Braspenning R (2008) Local Features based Facial Expression Recognition with Face Registration Errors, in Proc. 8th IEEE Int. Conf. Auto. Face Gesture Recog., Amsterdam, pp. 1–8
Hager JC, Ekman P (1983) The inner and outer meanings of facial expressions, in: J. T. Cacioppo and R. E. petty (Eds.), Soc. Psychophysiology, the Guilford Press, Newyork, pp. 287–307
Hall JA, Bernieri FJ, Carney DR (2005) Nonverbal behavior and interpersonal sensitivity, In J. A. Harrigan, R. Rosenthal, & K. R. Scherer (Eds.), The new handbook of methods in nonverbal behavior research, Oxford: Oxford University Press, pp. 237–281
Hammal Z, Couvreur L, Caplier A Rombaut M (2005) Facial Expressions Recognition Based on the Belief Theory: Comparison with Diferent Classifiers, Proc. 13th Int. Conf. Image Analysis and Processing, Italy
Hammal Z, Kunz M, Arguin M, Gosselin F (2008) Spontaneous Pain Expression Recognition in Video Sequences, Proc. Int. Conf. Visions of Computer Science, pp. 191–210
Happy SL, Patnaik P, Routray A, Guha R (2017) The Indian Spontaneous Expression Database for Emotion Recognition. IEEE Trans. Affective Computing 8(1):131–142
He M, Wang S, Liu Z, Chen X (2013) Analyses of the Differences between Posed and Spontaneous Facial Expressions, Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), Geneva, Switzerland, pp. 79–8
Hess U, Kleck RE (1990) Differentiating emotion elicited and deliberate emotional facial expressions. European J of Soc Psychol 20:369–385
Hess U, Kleck RE (2005) Differentiating emotion elicited and deliberate emotional facial expressions, in P. Ekman & E. L. Rosenberg (Eds.), What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (FACS) (2nd ed.) New York: Oxford University Press, pp. 271–286
M. E. Hoque, M. Courgeon, J.-C. Martin, B. Mutlu, and R. W. Picard (2013) MACH: My automated conversation coacH, Proc. ACM Int. Jnt. Conf. Pervasive and ubiquitous computing, pp. 697–706
Hussain A, Khan MS, Nazir M, Iqbal MA (2012) Survey of various feature extraction and classification techniques for facial expression recognition, Proc.11th WSEAS international conference on Electronics, Hardware, Wireless and Optical Communications, and proceedings of the 11th WSEAS international conference on Signal Processing, Robotics and Automation, and proceedings of the 4th WSEAS international conference on Nanotechnology, Stevens Point, Wisconsin, USA, pp. 138–142
Iwase M, Ouchi Y, Okada H, Yokoyama C, Nobezawa S, Yoshikawa E et al (2002) Neural substrates of human facial expression of pleasant emotion induced by comic films: a PET Study. Neuroimage 17(2):758–768
Jakobs E, Manstead ASR, Fischer AH (1999) Social motives and emotional feelings as determinants of facial displays: the case of smiling. Personal Soc Psychol Bull 25:424–435
Joho H, Jose JM, Valenti R, Sebe N (2009) Exploiting facial expressions for affective video summarisation, in Proc. ACM Int. Conf. Image and Video Retrieval, New York, USA
Jones V (2001) Robust real time object detection, Proc. 2nd Int. Workshop Statistical and Computational Theories of Vision
el Kaliouby R, Teeters A (2007) Eliciting, Capturing and Tagging Spontaneous Facial Affect in Autism Spectrum Disorder, Proc. 9th Int. Conf. Multimodal interfaces, NY, USA, pp. 46–53
Kaltwang S, Rudovic O, Pantic M (2012) Continuous pain intensity estimation from facial expressions. Advances in Visual Computing:368–377
Kirsh SJ, Mounts JR (2007) Violent video game play impacts facial emotion recognition. Aggress Behav 33(4):353–358
Kleck RE, Vaughen RC, Cartwright-Smith J, Vaughan KB, Colby CZ, Lanzetta JT (1976) Effects of being observed on expressive, subjective and physiological responses in painful stimuli. J Pers Soc Psychol 34:1211–1218
Knapp ML, Hall J (2005) Nonverbal communication in human interaction. Holt, Rinehart and Winston, New York
Ko BC (2018) A Brief Review of Facial Emotion Recognition Based on Visual Information. Sensors 18:401
Kohlera CG, Martina EA, Milonovaa M, Wangb P, Verma R, Brensingera CM, Bilkera W, Gura RE, Gura RC (2008) Dynamic evoked facial expressions of emotions in schizophrenia. Schizophr Res 105(1–3):30–39
Korb S, Grandjean D, Scherer K (2008) Investigating the production of emotional facial expressions: a combined electroencephalographic (EEG) and electromyographic (EMG) approach, IEEE Int. Conf. Automatic Face & Gesture Recognition, Amsterdam, pp. 1–6
Kring AM, Neale JM (1996) Do schizophrenic patients show a disjunctive relationship among expressive, experiential, and psychophysiological components of emotion? J Abnorm Psychol 105(2):249–257
Kring AM, Kerr SL, Smith DA, Neale JM (1993) Flat affect in schizophrenia does not reflect diminished subjective experience of emotion. J Abnorm Psychology 102(4):507–517
Krumhuber EG, Skora L, Küster D, Fou L (2017) A Review of Dynamic Datasets for Facial Expression Research. Emot Rev 9(3):280–292
Kumar S (2015) Facial expression recognition: A review, Proc. Nat. Conf. Cloud Computing and Big Data, Shanghai, China, pp. 159–162
Kunz M, Scharmann S, Hemmeter U, Schepelmann K, Lautenbacher S (2007) The facial expression of pain in patients with dementia. Pain 133(1–3):221–228
Lee KK, Xu Y (2003) Real-time Estimation of Facial Expression Intensity Proc. IEEE Int. Conf. Robotics & Automation, pp. 2567–2572
Lehr VT, Zeskind PS, Ofenstein JP, Cepeda E, Warrier I, Aranda JV (2007) Neonatal facial coding system scores and spectral characteristics of infant crying during newborn circumcision. Clin J Pain 23(5):417–424
Li Y, Mavadati SM, Mahoor MH, Zhao Y, Ji Q (2015) Measuring the intensity of spontaneous facial action units with dynamic Bayesian network. Pattern Recogn 48:3417–3427
Littlewort G, Whitehill J, Wu T, Fasel I, Frank M, Movellan J, Bartlett M, (2011)“The computer expression recognition toolbox (CERT),” Int. Conf. Automatic Face & Gesture Recognition and Workshops, Available: https://doi.org/10.1109/fg.2011.5771414
Loconsole C, Chiaradia D, Bevilacqua V, A. Frisoli (2014) Real-Time Emotion Recognition: An Improved Hybrid Approach for Classification Performance, Proc. 10th Inter. Conf. ICIC, Taiyuan, China, pp. 320–331
Lucey P, Cohn JF, Prkachin KM, Solomon PE, Matthews I (2011) Painful data: The unbc-mcmaster shoulder pain expression archive database. Proc IEEE Int Conf Automatic Face & Gesture Recognition and Workshops:57–64
Mahlke S, Minge M, Thüring M (2006) Measuring multiple components of emotions in interactive contexts, in Proc. Extended Abstracts on Human Factors in Computing, pp. 1061–1066
Malatesta C, Izard C (1984) The facial expression of emotion in young, middle-aged, and older adults. In: Malatesta C, Izard C (eds) Emotion in Adult Development. Sage, Beverly Hills, CA
Mandal MK, Pandey R, Prasad AB (1998) Facial Expressions of Emotions and Schizophrenia: A Review. Schizophr Bull 24(1):399–412
Marrero-Fernández P, Montoya-Padrón A, Jaume-i-Capó A, Rubio JMB (2014) Evaluating the Research in Automatic Emotion Recognition. IETE Tech Rev 31(3):220–232
Martin CC, Borod JC, Alpert M, Brozgold A, Welkowitz J (1990) Spontaneous Expression of Facial Emotion in Schizophrenic and Right brain-Damaged Patients. J Commun Disord 23:287–301
Martin C, Werner U, Gross H-M (2008) A Real-time Facial Expression Recognition System based on Active Appearance Models using Gray Images and Edge Images, Proc. 8th IEEE Int. Conf. Automatic Face & Gesture Recognition, Amsterdam, pp. 1–6
Mavadati SM, Mahoor MH, Bartlett K, Trinh P, Cohn JF (2013) DISFA: A Spontaneous Facial Action Intensity Database. IEEE Trans. Affective Computing 4(2):151–160
McDuff D, El Kaliouby R, Senechal T, Amr M, Cohn JF, Picard R (2013) Affectiva-MIT Facial Expression Dataset (AM-FED): Naturalistic and Spontaneous Facial Expressions Collected “In-the-Wild”, IEEE Conf. Computer Vision and Pattern Recognition Workshops, pp. 881–888
McKeown G, Valstar M, Cowie R, Pantic M, Schröder M The SEMAINE Database: Annotated Multimodal Records of Emotionally Colored Conversations between a Person and a Limited Agent. IEEE Trans Affect Comput 3(1):5–17
Mehrabian A (1968) Communication without words. Psychol Today 2(4):53–56
Michel P and Kaliouby R (2003) Real time facial expression recognition in video using support vector machines, in Proc. 5th Int. Conf. Multimodal interfaces, Vancouver, BC, Canada, pp. 258–264
Moridis CN, Economides AA (2012) Affective Learning: Empathetic Agents with Emotional Facial and Tone of Voice Expressions. IEEE Trans on Affective Computing 3(3):260–272
Murthy GRS, Jadon RS (2009) Effectiveness of Eigenspaces for Facial Expressions Recognition. Int J Computer Theory and Engineering 1(5):1793–8201
Naim I, Tanveer MI, Gildea D, Hoque ME (2015) Automated Prediction and Analysis of Job Interview Performance: The Role of What You Say and How You Say It, Proc. IEEE Int. Conf. and Workshops on Automatic Face and Gesture Recognition (FG), Ljubljana, Slovenia
K. Nosu and T. Kurokawa, Facial Tracking for an Emotion-Diagnosis Robot to Support e-Learning,” Proc. Int. Conf. Mach. Learning and Cyber, pp. 3811–3816, 2006.
Nosu K, Kurokawa T, Horita H, Ohhazama Y, Takeda H (2007) Real Time Emotion-Diagnosis of Video Game Players from their Facial Expressions and its Applications to Voice Feed-Backing to Game Players, Int. Conf. Mach. Learn. and Cyber., Hong Kong, pp. 2208–2212
O’Toole AJ, Harms J, Snow SL, Hurst DR, Pappas MR, Ayyad JH, Abdi H (2005) A video database of moving faces and people. IEEE Trans Pattern Anal Mach Intell 27(5):812–816
Ouellet S (2014) Real-time emotion recognition for gaming using deep convolutional network features, CoRR abs/1408.3750
Pantic M, Rothkrantz LJM (2000) Automatic Analysis of Facial Expressions: The State of the Art. IEEE Trans Pattern Anal Mach Intell 22(12):1424–1445
Pantic M, Rothkrantz LJM (2000) Expert System for Automatic Analysis of Facial Expression. J Image and Vision Computing 18(11):881–905
Petridis S, Martinez B, Pantic M (2013) The MAHNOB Laughter database. Image Vis Comput 31:186–202
F. Ringeval, A. Sonderegger, J. Sauer and D. Lalanne (2013) Introducing the RECOLA Multimodal Corpus of Remote Collaborative and Affective Interactions, Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition (FG)
Ryu YS, Oh SY (2002) Automatic extraction of eye and mouth fields from a face image using eigenfeatures and ensemble networks. Appl Intell 17:171–185
Sariyanidi E, Gunes H, Cavallaro A (2015) Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition. IEEE Trans on Pattern Analysis And Mach Intel 37(6):1113–1133
Scherer KR, Ceschi G (1997) Lost Luggage: A Field Study of Emotion-Antecedent Appraisal. Motiv Emot 21(3):211–235
Sebe N, Lew MS, Cohen I, Sun Y, Gevers T, Huang TS (2004) Authentic Facial Expression Analysis, Proc. IEEE Int. Conf. Automatic Face and Gesture Recognition, pp. 517–522
Seckington M (2011) Using Dynamic Bayesian Networks for Posed versus Spontaneous Facial Expression Recognition Master Thesis, Department of Computer Science, Delft University of Technology
Sikka K, Dhall A, Bartlett M (2015) Exemplar Hidden Markov Models for Classification of Facial Expressions in Videos, IEEE Conf. Computer Vision and Pattern Recognition Workshops, Boston, MA, pp. 18–25
Sneddon I, McRorie M, McKeown G, Hanratty J (2012) The Belfast Induced Natural Emotion Database. IEEE Trans. on Affective Computing 3(1):32–41
Suk M, Prabhakaran B (2014) Real-time Mobile Facial Expression Recognition System – A Case Study, Proc. Computer Vision and Pattern Recognition, Columbus, OH, pp. 132–137
Sun Y, Sebe N, Lew MS, Gevers T (2004) Authentic Emotion Detection in Real-time Video, in: Proc. Computer Vision in Human-Computer Interaction, ECCV 2004 Workshop on HCI, Prague, Czech Republic
Sun Y, Akansu AN, Cicon JE (2014) The power of fear: Facial emotion analysis of CEOs to forecast firm performance, Proc. IEEE 15th International Conf. Information Reuse and Integration, Redwood City, CA, pp. 695–702
Sung J, Kim D (2009) Real-time facial expression using STAAM and layered GDA classifier. Image and Vis Comput 27(9):1313–1325
Tan CT, Rosser D, Bakkes S, Pisan Y (2012) A feasibility study in using facial expressions analysis to evaluate player experiences, in Proc. 8th Australasian Conf. Interactive Entertainment: Playing the System, 5
Tcherkassof A, Dupre D, Meillon B, Mandran N, Dubois M, Adam J (2013) DynEmo: A video database of natural facial expressions of emotions. Int J Multimedia & Its Applications 5(5):61–80
Teijeiro-Mosquera L, Biel J-I, Alba-Castro JL, Gatica-Perez D (2015) What Your Face Vlogs About: Expressions of Emotion and Big-Five Traits Impressions in YouTube. IEEE Trans On Affective Computing 6(2):193–205
Tian Y, Cohn JF, Kanade T (2005) Facial expression recognition, in Handbook of face recognition, S. Z. Li, and A. K. Jain (Eds.), New York: Springer, pp. 247–276
Turan C, Lam K-M, He X (2015) Facial expression recognition with emotion-based feature fusion, Proc. Annual Summit and Conf. on Signal and Information Processing Association, Hong Kong, China
Tzirakis P, Trigeorgis G, Nicolaou MA, Schuller BW, Zafeiriou S (2017) End-to-End Multimodal Emotion Recognition Using Deep Neural Networks. IEEE Journal of Selected Topics in Signal Processing 11(8):1301–1309
Valstar MF, Pantic M, Ambadar Z, Cohn JF (2006) Spontaneous vs. posed facial behavior: automatic analysis of brow actions, Proc. 8th Int. Conf. Multimodal interfaces, pp. 162–170
Wallhoff F, Schuller B, Hawellek M, Rigoll G (2006) Efficient recognition of authentic dynamic facial expressions on the feedtum database, IEEE Int. Conf. Multimedia and Expo, Toronto, Ont., pp. 493–496
Wang S, Liu Z, Wang Z, Wu G, Shen P, He S, Wang X (2013) Analyses of a multi-modal spontaneous facial expression database. IEEE Trans Affective Computing 4(1):34–46
Wang S, Wu C, He M, Wang J, Ji Q (2015) Posed and spontaneous expression recognition through modeling their spatial patterns. Mach Vis Appl 26:219–231
Wanlnlhoff F (2006) Facial Expressions and Emotion Database, Available at: http://www.mmk.ei.tum.de/~waf/fgnet/feedtum.html, Technische Universität München
Wild B, Rodden FA, Rapp A, Erb M, Grodd W, Ruch W (2006) Humor and smiling: cortical regions selective for cognitive, affective, and volitional components. Neurology 66(6):887–893
Wu C, Wang S (2016) Posed and Spontaneous Expression Recognition through Restricted Boltzmann Machine, Proc. 22nd Int. Conf. MultiMedia Modeling, Miami, USA, pp. 127–137
Xue Z, Li SZ, Teoh EK (2003) Bayesian shape model for facial feature extraction and recognition. Pattern Recogn 36:2819–2833
Yannakakis GN, Hallam J (2005) A scheme for creating digital entertainment with substance, Proc. Reasoning, Representation, and Learning in Computer Games, Int. Jnt. Conf. Artificial Intelligence, pp. 119–124
Yeasin M, Bullot B, Sharma R (2006) Recognition of Facial Expressions and Measurement of Levels of Interest from Video. IEEE Trans Multimedia 8(3):500–508
Zaman B, Shrimpton-Smith T (2006) The face reader: measuring instant fun of use, in Proc. 4th Nordic conf. Human-computer interaction: changing roles, pp. 457–460
Zeng Z, Hu Y, Roisman GI, Wen Z, Fu Y, Huang TS (2007) Audio-Visual Spontaneous Emotion Recognition. Artificial Intelligence for Human Computing, Lecture Notes in Computer Science:72–90
Zeng Z, Pantic M, Roisman GI, Huang TS (2009) A Survey of Affect Recognition Methods: Audio,Visual, and Spontaneous Expressions. IEEE Trans on Pattern Anal And Mach Intel 31(1):39–58
Zhalehpour S, Onder O, Akhtar Z, Erdem CE (2017) BAUM-1: A Spontaneous Audio-Visual Face Database of Affective and Mental States. IEEE Trans Affect Comput 8(03):300–313
Zhan C, Li W, Ogunbona P, Safaei F (2008) A real-time facial expression recognition system for online games. Int J Computer Games Techn 10:1–7
Zhang Z (1998) Comparison between Geometry-Based and Gabor-wavelet-based Facial Expression Recognition Using Multi-layer Perception, Proc. Int. Conf. Auto. Face Gesture Recog., Nara, pp. 454–459
Zhang Y, Ji Q (May 2005) Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Trans Pattern Analysis and Machine Intelligence 27(5):699–714
Zhang Z, Lyons M, Schuster M, Akamatsu S (1998) Comparison between Geometry-based and Gabor wavelets-based facial expression recognition using multilayer perception, Proc. Int. Conf. Automatic Face and Gesture Recognition, pp. 454–459
Zhang L, Tjondronegoro D, Chandran V (2012) Discovering the Best Feature Extraction and Selection Algorithms For Spontaneous Facial Expression Recognition, Proc. IEEE Int. Conf. Multimedia and Expo (ICME), Melbourne, VIC, pp. 1027–1032
Zhang W, Zhang Y, Mab L, Guan J, Gong S (2015) Multimodal learning for facial expression recognition. Pattern Recogn 48:3191–3202
Zhao X, Zhang S (2016) A Review on Facial Expression Recognition: Feature Extraction and Classification. IETE Tech Rev 33(5):505–517
Zheng D, Zhao Y, Wang J (2004) Features extraction using a Gabor filter family, Proc. Sixth IASTED International Conference Signal and Image Processing, Hawaii, USA
Zhou X, Huang X, Wang Y (2004) Real-time Facial Expression Recognition in the Interactive Game Based on Embedded Hidden Markov Model, Proc. Int. Conf. Comp. Graphics, Imaging and Visualization, pp. 144–148
Acknowledgments
The first author is grateful to Department of Science and Technology (DST), Government of India for providing her Junior Research Fellowship (JRF) under DST-INSPIRE fellowship program (No. IF131067).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Saha, P., Bhattacharjee, D., De, B.K. et al. A Survey on Image Acquisition Protocols for Non-posed Facial Expression Recognition Systems. Multimed Tools Appl 78, 23329–23368 (2019). https://doi.org/10.1007/s11042-019-7596-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-019-7596-2