Skip to main content
SpringerLink
Log in

A multi-scale and rotation-invariant phase pattern (MRIPP) and a stack of restricted Boltzmann machine (RBM) with preprocessing for facial expression classification

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

In facial expression recognition applications, the classification accuracy decreases because of the blur, illumination and localization problems in images. Therefore, a robust emotion recognition technique is needed. In this work, a Multi-scale and Rotation-Invariant Phase Pattern (MRIPP) is proposed. The MRIPP extracts the features from facial images, and the extracted patterns are blur-insensitive, rotation-invariant and robust. The performance of classification algorithms like Fisher faces, Support Vector Machine (SVM), Extreme Learning Machine (ELM), Convolutional Neural Network (CNN) and Deep Neural Network (DNN) are analyzed. In order to reduce the time for classification, an OPTICS-based pre-processing of the features is proposed that creates a non-redundant and compressed training set to classify the test set. Ten-fold cross validation is used in experimental analysis and the performance metric classification accuracy is used. The proposed approach has been evaluated with six datasets Japanese Female Facial Expression (JAFFE), Cohn Kanade (CK +), Multi- media Understanding Group (MUG), Static Facial Expressions in the Wild (SFEW), Oulu-Chinese Academy of Science, Institute of Automation (Oulu-CASIA) and Man–Machine Interaction (MMI) datasets to meet a classification accuracy of 98.2%, 97.5%, 95.6%, 35.5%, 87.7% and 82.4% for seven class emotion detection using a stack of Restricted Boltzmann Machines(RBM), which is high when compared to other latest methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  • Agarwal S, Santra B, Mukherjee DP (2016) Anubhav: recognizing emotions through facial expression. Vis Comput. https://doi.org/10.1007/s00371-016-1323-z

    Article  Google Scholar 

  • Ahmed F, Kabir MH (2012) Directional ternary pattern (DTP) for facial expression recognition. In: IEEE international conference on consumer electronics, pp 265–266

  • Aifanti, N, Papachristou C, Delopoulos A (2010) The MUG facial expression database. In: Proc. 11th Int. Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), Desenzano, Italy, April 12–14

  • Alphonse AS, Dharma D (2017a) A novel Monogenic Directional Pattern (MDP) and pseudo-Voigt kernel for facilitating the identification of facial emotions. J Vis Commun Image Represent 49:459–470

    Article  Google Scholar 

  • Alphonse AS, Dharma D (2017b) Enhanced Gabor (E-Gabor), Hypersphere-based normalization and Pearson General Kernel-based discriminant analysis for dimension reduction and classification of facial emotions. Expert Syst Appl 90:127–145

    Article  Google Scholar 

  • Alphonse AS, Dharma D (2018) Novel directional patterns and a generalized supervised dimension reduction system (GSDRS) for facial emotion recognition. Multimedia Tools Appl 77(8):9455–9488

    Article  Google Scholar 

  • Anisetti M, Bellandi V (2009) Emotional state inference using face related features. New directions in intelligent interactive multimedia systems and services-2. Springer, Berlin Heidelberg, pp 401–411

    Chapter  Google Scholar 

  • Ankerst M, Breunig MM, Kriegel HP, Sander J (1999) OPTICS: ordering points to identify the clustering structure. ACM Sigmod record 28(2):49–60

    Article  Google Scholar 

  • Asthana A, Zafeiriou S, Cheng S, Pantic M (2014) Incremental face alignment in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1859–1866.

  • Athisayamani S, Singh AR, Athithan T (2020) Recognition of ancient Tamil palm leaf vowel characters in historical documents using B-spline curve recognition. Proc Comput Sci 171:2302–2309

  • Berretti S, Amor BB, Daoudi M, Del Bimbo A (2011) 3D facial expression recognition using SIFT descriptors of automatically detected key points. Vis Comput 27(11):1021–1036

    Article  Google Scholar 

  • Bousmalis K, Mehu M, Pantic M (2013) Towards the automatic detection of spontaneous agreement and disagreement based on nonverbal behaviour: a survey of related cues, databases, and tools. Image Vis Comput 31(2):203–221

    Article  Google Scholar 

  • Burkert P, Trier F, Afzal MZ, Dengel A, Liwicki M (2015) Dexpression: deep convolutional neural network for expression recognitionarXiv preprint arXiv:1509.05371. https://arxiv.org/abs/1509.05371

  • Chan CH (2008) Multi-scale Local Binary Pattern Histogram for Face Recognition, PhD thesis, UNIS.

  • Chan CH, Kittler J, Poh N, Ahonen T, Pietikäinen M (2009) (Multiscale) local phase quantisation histogram discriminant analysis with score normalisation for robust face recognition. In: Computer vision workshops (ICCV Workshops), IEEE 12th international conference on 2009 Sep 27 pp.633–640.

  • Chang CC, Lin CJ (2011) LIBSVM : a library for support vector machines. ACM Trans Intell Syst Technol 2(3):27

    Article  Google Scholar 

  • Dalal N, Triggs (2005) Histograms of oriented gradients for human detection, In: Computer vision and pattern recognition, CVPR 2005, IEEE Computer Society Conference pp. 1886–893.

  • Dhall A, Goecke R, Joshi J, Sikka K, Gedeon T (2014) Emotion recognition in the wild challenge 2014:baseline, data and protocol, ACM ICMI 2014

  • Dhall A, Goecke R, Lucey S, Gedeon T (2012) Collecting large, richly annotated facial expression databases from movies. IEEE Multimedia 19:34–41

    Article  Google Scholar 

  • Eddy SR (1998) Profile hidden Markov models. Bioinformatics (Oxford, England) 14(9):755–763

    Article  Google Scholar 

  • Ghimire D, Lee J, Li ZN, Jeong S (2016) Recognition of facial expressions based on salient geometric features and support vector machines. Multimedia Tools Appl 15:1–26

    Google Scholar 

  • Haghighat M, Zonouz S, Abdel-Mottaleb M (2015) CloudID: trustworthy cloud-based and cross-enterprise biometric identification. Expert Syst Appl 42(21):7905–7916

    Article  Google Scholar 

  • Hao XL, Tian M (2017) Deep Belief Network based on double weber local descriptor in micro-expression recognition, In: Advanced multimedia and ubiquitous engineering. Pp 419–425.

  • Huang GB, Zhu QY, Siew CK (2004) Extreme learning machine: a new learning scheme of feed forward neural networks. Neural networks 2:985–990

    Google Scholar 

  • Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. Part B: IEEE Tran Syst Man, Cybernet 42(2):513–529

    Google Scholar 

  • Iosifidis A, Tefas A, Pitas I (2017) Approximate kernel extreme learning machine for large scale data classification. Neurocomputing 219:210–220

    Article  Google Scholar 

  • Jabid T, Kabir MH, Chae O (2010) Robust facial expression recognition based on local directional pattern. ETRI journal 32(5):784–794

    Article  Google Scholar 

  • Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings of fourth IEEE international conference in automatic face and gesture recognition, pp. 46–53.

  • Keyvanrad MA, Homayounpour MM (2014) A brief survey on deep belief networks and introducing a new object oriented toolbox (DeeBNet) arXiv: 1408.3264 [cs], Aug. 2014.

  • Khan SA, Hussain A, Usman (2017) Reliable facial expression recognition for multi-scale images using weber local binary image based cosine transform features, Multimedia Tools and Applications, pp. 1–33.

  • Kim Y, Yoo B, Kwak Y, Choi C, Kim J (2017) Deep generative-contrastive networks for facial expression recognition. arXiv preprint arXiv: 1703.07140.

  • Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks, In Advances in neural information processing systems pp. 1097–1105

  • LeCun Y, Bengio Y, Hinton G, Deep learning (2015).

  • Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 94–101.

  • Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Third IEEE international conference on automatic face and gesture recognition, pp 200–205

  • Nguyen HT, Caplier A (2014) Patch based local phase quantization of monogenic components for face recognition, In: 2014 IEEE International conference on image processing (ICIP) pp. 229–233

  • Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  Google Scholar 

  • Ojansivu V, Rahtu E, Heikkila J (2008) Rotation invariant local phase quantization for blur insensitive texture analysis, In 2008 19th International Conference on Pattern Recognition pp.1–4.

  • Ojansivu V, Heikkilä J(2008) Blur insensitive texture classification using local phase quantization, In International conference on image and signal processing pp. 236–243.

  • Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In Multimedia and Expo, IEEE International Conference, pp. 5.

  • Rahtu E, Heikkilä J, Ojansivu J, Ahonen T (2012) Local phase quantization for blur-insensitive image analysis. Image Vis Comput 30(8):501–512

    Article  Google Scholar 

  • Ramirez Rivera A, Rojas Castillo J, Chae O (2013) Local directional number pattern for face analysis: face and expression recognition. IEEE Trans Image Process 22(5):1740–1752

    Article  MathSciNet  Google Scholar 

  • Rivera AR, Castillo JR, Chae O (2015) Local directional texture pattern image descriptor. Pattern Recogn Lett 51:94–100

    Article  Google Scholar 

  • Shinohara Y, Otsuf N (2004) Facial expression recognition using fisher weight maps. Sixth IEEE Int Conf Auto Face Gesture Recognition Proceedings 17:499–504

    Article  Google Scholar 

  • Shufu X, Shiguang S, Xilin C, Jie C (2010) Fusing local patterns of gabor magnitude and phase for face recognition. Image Processing IEEE Trans 19(5):1349–1361

    Article  MathSciNet  Google Scholar 

  • Siddiqi MH, Lee S, Lee YK, Khan AM, Truc P (2013) Hierarchical recognition scheme for human facial expression recognition systems. Sensors 13:16682–16713

    Article  Google Scholar 

  • Song I, Kim HJ, Jeon PB (2014) Deep learning for real-time robust facial expression recognition on a smartphone, In: 2014 international conference on consumer electronics (ICCE), pp 564–567

  • Suykens JA, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300

    Article  Google Scholar 

  • Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19(6):1635–1650

    Article  MathSciNet  Google Scholar 

  • Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the MMI facial expression database. In: Proc. 3rd intern. workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect, pp 65.

  • Venkateswara H, Eusebio J, Chakraborty S, Panchanathan S (2017) Deep hashing network for unsupervised domain adaptation, In: Proceedings of the IEEE conference on computer vision and pattern recognition pp.5018–5027.

  • Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vision 57(2):137–154

    Article  Google Scholar 

  • Vipparthi SK, Nagar SK (2014) Expert image retrieval system using directional local motif XoR patterns. Expert Syst Appl 41(17):8016–8026

    Article  Google Scholar 

  • Wen G, Hou Z, Li H, Li D, Jiang L, Xun E (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition, Cognitive Computation, pp.1–4.

  • Zhang B, Shan S, Chen X, Gao W (2006) Histogram of gabor phase patterns (HGPP): a Novel object representation approach for face recognition. IEEE TIP 16(1):57–68

    MathSciNet  Google Scholar 

  • Zhang K, Huang Y, Du Y (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. In: IEEE Transactions on Image Processing, pp 4193–4203

  • Zhang W, Shan S, Gao W, Chen X, Zhang H (2005) Local gabor binary pattern histogram sequence (LGBPHS): a novel non-statistical model for face representation and and recognition, In: Proc. ICCV, 786–791

  • Zhao G, Huang X, Taini M, Li SZ, Pietikäinen M (2011) Facial expression recognition from near-infrared videos. Image Vision Comput 29(9):607–619

    Article  Google Scholar 

  • Zhao L, Wang Z, Zhang G (2017) Facial expression recognition from video sequences based on spatial-temporal motion local binary pattern and gabor multi-orientation fusion histogram. Math Prob Eng 2017

Download references

Author information

Authors and Affiliations

  1. Department of Information Technology, Ponjesly College of Engineering, Nagercoil, India

    A. Sherly Alphonse

  2. Department of Computer Applications, Alagappa University, Karaikudi, India

    K. Shankar

  3. School of Computing, Sastra Deemed To Be University, Thanjavur, India

    M. J. Jeyasheela Rakkini, S. Ananthakrishnan & Suganya Athisayamani

  4. School of Computing, Kalasalingam Academy of Research and Education, Anand Nagar, India

    A. Robert Singh

  5. Department of Computer Science, Christ University, Bengalore, India

    R. Gobi

Authors
  1. A. Sherly Alphonse
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. K. Shankar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M. J. Jeyasheela Rakkini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. S. Ananthakrishnan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Suganya Athisayamani
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. A. Robert Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. R. Gobi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to A. Robert Singh.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alphonse, A.S., Shankar, K., Jeyasheela Rakkini, M.J. et al. A multi-scale and rotation-invariant phase pattern (MRIPP) and a stack of restricted Boltzmann machine (RBM) with preprocessing for facial expression classification. J Ambient Intell Human Comput 12, 3447–3463 (2021). https://doi.org/10.1007/s12652-020-02517-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-020-02517-7

Keyword

Search

Navigation