Skip to main content
SpringerLink
Log in

Video-Based Performance Driven Facial Animation

  • Reference work entry
  • First Online:
Handbook of Human Motion

  • 591 Accesses

Abstract

Video-based performance driven facial animation is appealing as it offers the lowest cost, a simplified setup, and the potential use of legacy sources and uncontrolled videos. It is also difficult as it is ill-posed due to the loss of depth. This chapter introduces techniques in video-based facial reconstruction in three levels. Given the input video, the first level is to reconstruct 3D head poses and large-scale facial deformation at each frame. Representations of the facial deformation as well as the related 2D feature detection/tracking and 3D shape parameters optimization methods are introduced. Next, we discuss methods on recovering the fine-scale surface details such as emerging and disappearing wrinkles and folds. Finally, we briefly introduce the advanced applications based on the reconstructed facial performance, such as video editing and facial component enhancement.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

References

  • BaltruÅ¡aitis T, Robinson P, Morency LP (2012) 3D constrained local model for rigid and non-rigid facial tracking. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2610–2617

    Google Scholar 

  • Basri R, Jacobs DW (2003) Lambertian reflectance and linear subspaces. IEEE Trans Pattern Anal Mach Intell 25(2):218–233

    Article  Google Scholar 

  • Beeler T, Bickel B, Beardsley P, Sumner B, Gross M (2010) High-quality single-shot capture of facial geometry. ACM Trans Graph 29(4):40:1–40:9

    Article  Google Scholar 

  • Beeler T, Hahn F, Bradley D, Bickel B, Beardsley P, Gotsman C, Sumner RW, Gross M (2011) High-quality passive facial performance capture using anchor frames. ACM Trans Graph 30(4):75:1–75:10

    Article  Google Scholar 

  • Bickel B, Botsch M, Angst R, Matusik W, Otaduy M, Pfister H, Gross M (2007) Multi-scale capture of facial geometry and motion. ACM Trans Graph 26(3):33:1–33:10

    Article  Google Scholar 

  • Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing, New York, pp 187–194

    Google Scholar 

  • Bouaziz S, Wang Y, Pauly M (2013) Online modeling for real-time facial animation. ACM Trans Graph 32(4):40:1–40:10. https://doi.org/10.1145/2461912.2461976

    Article  MATH  Google Scholar 

  • Bradley D, Heidrich W, Popa T, Sheffer A (2010) High resolution passive facial performance capture. ACM Trans Graph 29(4):41:1–41:10

    Article  Google Scholar 

  • Cao X, Wei Y, Wen F, Sun J (2012) Face alignment by explicit shape regression. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 2887–2894

    Google Scholar 

  • Cao C, Weng Y, Lin S, Zhou K (2013) 3D shape regression for real-time facial animation. ACM Trans Graph 32(4):41:1–41:10. https://doi.org/10.1145/2461912.2462012

    Article  MATH  Google Scholar 

  • Cao C, Hou Q, Zhou K (2014a) Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions on Graphics (TOG) 33(4):43

    Google Scholar 

  • Cao C, Weng Y, Zhou S, Tong Y, Zhou K (2014b) Facewarehouse: a 3D facial expression database for visual computing. IEEE Trans Vis Comput Graph 20(3):413–425

    Article  Google Scholar 

  • Cao C, Bradley D, Zhou K, Beeler T (2015) Real-time high-fidelity facial performance capture. ACM Transactions on Graphics (TOG) 34(4):46

    Article  Google Scholar 

  • Chai J, Xiao J, Hodgins J (2003) Vision-based control of 3D facial animation. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on computer animation, pp 193–206

    Google Scholar 

  • Chen YL, Wu Ht, Shi F, Tong X, Chai J (2013) Accurate and robust 3D facial capture using a single rgbd camera. In: IEEE international conference on computer vision (ICCV), pp 3615–3622

    Google Scholar 

  • Garrido P, Valgaerts L, Wu C, Theobalt C (2013) Reconstructing detailed dynamic face geometry from monocular video. ACM Trans Graph 32(6):158

    Article  Google Scholar 

  • Garrido P, Zollhöfer M, Casas D, Valgaerts L, Varanasi K, Pérez P, Theobalt C (2016) Reconstruction of personalized 3D face rigs from monocular video. ACM Trans Graph (TOG) 35(3):28

    Article  Google Scholar 

  • Horn BK, Brooks MJ (1989) Shape from shading. MIT Press, Cambridge, MA

    MATH  Google Scholar 

  • Hsieh PL, Ma C, Yu J, Li H (2015) Unconstrained real-time facial performance capture. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1675–1683

    Google Scholar 

  • Huang H, Chai J, Tong X, Wu HT (2011) Leveraging motion capture and 3D scanning for high-fidelity facial performance acquisition. ACM Trans Graph 30(4):74:1–74:10

    Article  Google Scholar 

  • Ichim AE, Bouaziz S, Pauly M (2015) Dynamic 3D avatar creation from handheld video input. ACM Trans Graph (TOG) 34(4):45

    Article  Google Scholar 

  • Kemelmacher-Shlizerman I, Basri R (2011) 3D face reconstruction from a single image using a single reference face shape. IEEE Trans Pattern Anal Mach Intell 33(2):394–405

    Article  Google Scholar 

  • Li H, Yu J, Ye Y, Bregler C (2013) Real-time facial animation with on-the-fly correctives. ACM Trans Graph 32(4):42:1–42:10. https://doi.org/10.1145/2461912.2462019

    MATH  Google Scholar 

  • Li H, Trutoiu L, Olszewski K, Wei L, Trutna T, Hsieh PL, Nicholls A, Ma C (2015) Facial performance sensing head-mounted display. ACM Trans Graph (TOG) 34(4):47

    Google Scholar 

  • Liu Y, Xu F, Chai J, Tong X, Wang L, Huo Q (2015) Videoaudio driven real-time facial animation. ACM Trans Graph 34(6):182:1–182:10. https://doi.org/10.1145/2816795.2818122

    Google Scholar 

  • Ma WC, Jones A, Chiang JY, Hawkins T, Frederiksen S, Peers P, Vukovic M, Ouhyoung M, Debevec P (2008) Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans Graph 27(5):121:1–121:10

    Article  Google Scholar 

  • Matthews I, Baker S (2004) Active appearance models revisited. Int J Comp Vision 60(2):135–164

    Article  Google Scholar 

  • Ren S, Cao X, Wei Y, Sun J (2014) Face alignment at 3000 fps via regressing local binary features. In: IEEE conference on computer vision and pattern recognition (CVPR), IEEE, pp 1685–1692

    Google Scholar 

  • Saragih JM, Lucey S, Cohn JF (2011) Real-time avatar animation from a single image. In: IEEE international conference on automatic face & gesture recognition and workshops (FG 2011), IEEE, pp 117–124

    Google Scholar 

  • Shi F, Wu HT, Tong X, Chai J (2014) Automatic acquisition of high-fidelity facial performances using monocular videos. ACM Transactions on Graphics (TOG) 33(6):222

    Article  Google Scholar 

  • Sumner RW, Popović J (2004) Deformation transfer for triangle meshes. ACM Transactions on Graphics (TOG) 23(3):399–405

    Article  Google Scholar 

  • Suwajanakorn S, Kemelmacher-Shlizerman I, Seitz SM (2014) Total moving face reconstruction. In: European conference on computer vision, Springer, pp 796–812

    Google Scholar 

  • Suwajanakorn S, Seitz SM, Kemelmacher-Shlizerman I (2015) What makes tom hanks look like tom hanks. In: Proceedings of the IEEE international conference on computer vision, pp 3952–3960

    Google Scholar 

  • Thies J, Zollhöfer M, Stamminger M, Theobalt C, Nießner M (2016) Face2face: real-time face capture and reenactment of RGB videos. In: Proceedings of computer vision and pattern recognition (CVPR), IEEE 1

    Google Scholar 

  • Valgaerts L, Wu C, Bruhn A, Seidel HP, Theobalt C (2012) Lightweight binocular facial performance capture under uncontrolled lighting. ACM Trans Graph 31(6):187:1–187:11. https://doi.org/10.1145/2366145.2366206

    Article  Google Scholar 

  • Vlasic D, Brand M, Pfister H, Popović J (2005) Face transfer with multilinear models. ACM Trans Graph (TOG) 24:426–433

    Article  Google Scholar 

  • Wang C, Shi F, Xia S, Chai J (2016) Real-time 3D eye gaze animation using a single RGB camera. ACM Trans Graph (TOG) 35:1

    Google Scholar 

  • Weise T, Li H, Van Gool L, Pauly M (2009) Face/off: live facial puppetry. In: Symposium on computer animation, pp 7–16. https://doi.org/10.1145/1599470.1599472

  • Weise T, Bouaziz S, Li H, Pauly M (2011) Real-time performance-based facial animation. ACM Trans Graph 30(4):77:1–77:10

    Article  Google Scholar 

  • Xiong X, De la Torre F (2013) Supervised descent method and its applications to face alignment. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 532–539

    Google Scholar 

  • Zhang L, Snavely N, Curless B, Seitz S (2004) Spacetime faces: high resolution capture for modeling and animation. ACM Transactions on Graphics 23(3):548–558

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Texas A&M University, College Station, TX, USA

    Fuhao Shi

Authors
  1. Fuhao Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fuhao Shi .

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Shi, F. (2018). Video-Based Performance Driven Facial Animation. In: Handbook of Human Motion. Springer, Cham. https://doi.org/10.1007/978-3-319-14418-4_189

Download citation

Publish with us

Policies and ethics

Search

Navigation