Skip to main content
SpringerLink
Log in

Image-based face beauty analysis via graph-based semi-supervised learning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Automatic facial beauty analysis has become an emerging research topic. Despite some achieved advances, current methods and systems suffer from at least two limitations. Firstly, many developed systems rely on the use of ad-hoc hand-crafted features that were designed for generic pattern recognition problems. Secondly, while Deep Convolutional Neural Nets (DCNN) have been recently demonstrated to be a promising area of research in statistical machine learning, their use for automatic face beauty analysis may not guarantee optimal performances due to the use of a limited amount of face images with beauty scores. In this paper, we attempt to overcome these two main limitations by jointly exploiting two tricks. First, instead of using hand-crafted face features we use deep features of a pre-trained DCNN able to generate a high-level representation of a face image. Second, we exploit manifold learning theory and deploy three graph-based semi-supervised learning methods in order to enrich model learning without the need of additional labeled face images. These schemes perform graph-based score propagation. The proposed schemes were tested on three public datasets for beauty analysis: SCUT-FBP, M2B, and SCUT-FBP5500. These experiments, as well as many comparisons with supervised schemes, show that the scheme coined Kernel Flexible Manifold Embedding compares favorably with many supervised schemes. They also show that its performances in terms of error prediction and Pearson Correlation are better than those reported for the used datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. The network can be downloaded from http://www.vlfeat.org/matconvnet/pretrained/.

References

  1. Aarabi P, Hughes D, Mohajer K, Emami M (2001) The automatic measurement of facial beauty. In: 2001 IEEE international conference on systems, man, and cybernetics, vol 4. IEEE, pp 2644–2647

  2. Bainbridge W, Isola P, Oliva A (2013) The intrinsic memorability of face images. J Exp Psychol Gen 142(4):1323–1334

    Article  Google Scholar 

  3. Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput 15(6):1373–1396

    Article  Google Scholar 

  4. Bi J, Bennett KP (2003) Regression error characteristic curves. In: Proceedings of the 20th international conference on machine learning (ICML-03), pp 43–50

  5. Bottino A, Laurentini A (2010) The analysis of facial beauty: an emerging area of research in pattern analysis. In: ICIAR. Part I, LNCS (6111), pp 425–435

    Google Scholar 

  6. Cheng B, Yang J, Yan S, Fu Y, Huang T (2010) Learning with l1-graph for image analysis. Trans Img Proc 19(4):858–866

    Article  Google Scholar 

  7. datatang. http://datatang.com/

  8. De Bacco C, Larremore DB, Moore C (2018) A physical model for efficient ranking in networks. arXiv:1709.09002

  9. Dornaika F, Bosaghzadeh A (2015) Adaptive graph construction using data self-representativeness for pattern classification. Inf Sci 325:118–139

    Article  MathSciNet  Google Scholar 

  10. Dornaika F, Dhabi R, Bosaghzadeh A, Ruichek Y (2016) Dynamic adaptive graph construction: application to graph-based multi-observation classification. In: IEEE international conference on pattern recognition

  11. Dornaika F, El Traboulsi Y (2016) Learning flexible graph-based semi-supervised embedding. IEEE Trans Cybern 46(1):206–218

    Article  Google Scholar 

  12. Dornaika F, Traboulsi YE, Assoum A (2013) Adaptive two phase sparse representation classifier for face recognition. In: LNCS 8192. Advanced concepts for intelligent vision systems

    Chapter  Google Scholar 

  13. Eisenthal Y, Dror G, Ruppin E (2006) Facial attractiveness: beauty and the machine. Neural Comput 18(1):119–142

    Article  Google Scholar 

  14. Gan J, Li L, Zhai Y, Liu Y (2014) Deep self-taught learning for facial beauty prediction. Neurocomputing 144:295–303

    Article  Google Scholar 

  15. Gan J, Wang B, Xu Y (2015) A novel method for predicting facial beauty under unconstrained condition. Springer, Cham, pp 350–360

    Google Scholar 

  16. Gray D, Yu K, Xu W, Gong Y (2010) Predicting facial beauty without landmarks. Comput Vision–ECCV 2010, 434–447

  17. He R, Zheng W-S, Hu B-G, Kong X-W (2011) Nonnegative sparse coding for discriminative semi-supervised learning. In: 2011 IEEE conference on computer vision and pattern recognition (CVPR), pp 2849–2856

  18. Kagian A, Dror G, Leyvand T, Cohen-Or D, Ruppin E (2007) A humanlike predictor of facial attractiveness, pp 649–656

  19. Kagian A, Dror G, Leyvand T, Meilijson I, Cohen-Or D, Ruppin E (2008) A machine learning predictor of facial attractiveness revealing human-like psychophysical biases. Vision Res 48(2):235–243

    Article  Google Scholar 

  20. Laurentini A, Bottino A (2014) Computer analysis of face beauty a survey. Computer Vision and Image Understanding 125(Supplement C):184–199

    Article  Google Scholar 

  21. Liang L, Lin L, Jin L, Xie D, Li M (2018) SCUT-FBP5500: a diverse benchmark dataset for multi-paradigm facial beauty prediction. arXiv:1801.06345v1[cs.CV] 19 Jan 2018

  22. Liu S, Fan Y-Y, Samal A, Guo Z (2016) Advances in computational facial attractiveness methods. Multimed Tools Appl 75(23):16633–16663

    Article  Google Scholar 

  23. Mu Y (2013) Computational facial attractiveness prediction by aesthetics-aware features. Neurocomputing 99:59–64

    Article  Google Scholar 

  24. Nguyen TV, Liu S, Ni B, Tan J, Rui Y, Yan S (2013) Towards decrypting attractiveness via multi-modality cues. ACM Trans Multimed Comput, Commun, Appl (TOMM) 9(4):28

    Google Scholar 

  25. Nie F, Wang X, Jordan MI, Huang H (2016) The constrained laplacian rank algorithm for graph-based clustering. In: AAAI conference on artificial intelligence

  26. Nie F, Xu D, Tsang IW-H, Zhang C (2010) Flexible manifold embedding: a framework for semi-supervised and unsupervised dimension reduction. IEEE Trans Image Process 19(7):1921–1932

    Article  MathSciNet  Google Scholar 

  27. Nie L, Song X, Chua T-S (2016) Learning from multiple social networks. Morgan and Claypool Publishers, San Rafael

    Book  Google Scholar 

  28. Nie L, Wang M, Zha Z-J, Chua T-S (2012) Oracle in image search: a content-based approach to performance prediction. ACM Trans Inf Syst 30:13:1–13:23

    Article  Google Scholar 

  29. Parkhi OM, Vedaldi A, Zisserman A, et al. (2015) Deep face recognition. In: BMVC, vol 1, p 6

  30. Raducanu B, Dornaika F (2014) Embedding new observations via sparse-coding for non-linear manifold learning. Pattern Recogn 47(1):480–492

    Article  Google Scholar 

  31. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Fei-Fei L (2018) Imagenet large scale visual recognition challenge. arXiv:1409.0575, 2014

  32. Schmid K, Marx D, Samal A (2008) Computation of a face attractiveness index based on neoclassical canons, symmetry, and golden ratios. Pattern Recogn 41 (8):2710–2717

    Article  Google Scholar 

  33. Sharif Razavian A, Azizpour H, Sullivan J, Carlsson S (2014) Cnn features off-the-shelf: an astounding baseline for recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 806–813

  34. Sutic D, Rreskovic I, Huic R, Jukic I (2010) Automatic evaluation of facial attractiveness. In: Proceedings of the 33rd international convention, pp 1339–1342

  35. Traboulsi YE, Dornaika F, Assoum A (2015) Kernel flexible manifold embedding for pattern classification. Neurocomputing 167:517–527

    Article  Google Scholar 

  36. Wang S, Shao M, Fu Y (2014) Attractive or not?: beauty prediction with attractiveness-aware encoders and robust late fusion. In: Proceedings of the 22nd ACM international conference on multimedia. ACM, pp 805–808

  37. Whitehill J, Movellan JR (2008) Personalized facial attractiveness prediction. In: IEEE international conference on automatic face gesture recognition, pp 1–7

  38. Xie D, Liang L, Jin L, Xu J, Li M (2015) Scut-fbp: a benchmark dataset for facial beauty perception. In: 2015 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, pp 1821–1826

  39. Xu J, Jin L, Liang L, Feng Z, Xie D (2015) A new humanlike facial attractiveness predictor with cascaded fine-tuning deep learning model. arXiv:1511.02465

  40. Yan M, Duan Y, Deng S, Zhu W, Wu X (2016) Facial beauty assessment under unconstrained conditions. In: 2016 8th international conference on electronics, computers and artificial intelligence (ECAI), pp 1–6

  41. Zhang D, Chen F, Xu Y (2016) Computer models for facial beauty analysis. Springer, Berlin

    Book  Google Scholar 

  42. Zhang D, Zhao Q, Chen F (2011) Quantitative analysis of human facial beauty using geometric features. Pattern Recogn 44:940–950

    Article  Google Scholar 

  43. Zhou D, Bousquet O, Lal TN, Weston J, Schölkopf B (2004) Learning with local and global consistency. In: Advances in neural information processing systems, pp 321–328

Download references

Author information

Authors and Affiliations

  1. University of the Basque Country UPV/EHU, San Sebastian, Spain

    F. Dornaika, A. Elorza, K. Wang & I. Arganda-Carreras

  2. IKERBASQUE, Basque Foundation for Science, Bilbao, Spain

    F. Dornaika & I. Arganda-Carreras

  3. Northwestern Polytechnic University, Xian, China

    K. Wang

  4. Donostia International Physics Center (DIPC), San Sebastian, Spain

    I. Arganda-Carreras

Authors
  1. F. Dornaika
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. Elorza
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. K. Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. I. Arganda-Carreras
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to F. Dornaika.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dornaika, F., Elorza, A., Wang, K. et al. Image-based face beauty analysis via graph-based semi-supervised learning. Multimed Tools Appl 79, 3005–3030 (2020). https://doi.org/10.1007/s11042-019-08206-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-08206-8

Keywords

Search

Navigation