Wan-Chun Ma
Paul Debevec
Abstract
We present a novel method for acquisition, modeling, compression, and synthesis of realistic facial deformations using polynomial displacement maps. Our method consists of an analysis phase where the relationship between motion capture markers and detailed facial geometry is inferred, and a synthesis phase where novel detailed animated facial geometry is driven solely by a sparse set of motion capture markers. For analysis, we record the actor wearing facial markers while performing a set of training expression clips. We capture real-time high-resolution facial deformations, including dynamic wrinkle and pore detail, using interleaved structured light 3D scanning and photometric stereo. Next, we compute displacements between a neutral mesh driven by the motion capture markers and the high-resolution captured expressions. These geometric displacements are stored in a polynomial displacement map which is parameterized according to the local deformations of the motion capture dots. For synthesis, we drive the polynomial displacement map with new motion capture data. This allows the recreation of large-scale muscle deformation, medium and fine wrinkles, and dynamic skin pore detail. Applications include the compression of existing performance data and the synthesis of new performances. Our technique is independent of the underlying geometry capture system and can be used to automatically generate high-frequency wrinkle and pore details on top of many existing facial animation systems.
Supplemental Material
- Bickel, B., Botsch, M., Angst, R., Matusik, W., Otaduy, M., Pfister, H., and Gross, M. 2007. Multi-scale capture of facial geometry and motion. ACM Transactions on Graphics 26, 3 (July), 33: 1--33: 10. Google Scholar
Digital Library
- Bickel, B., Lang, M., Botsch, M., Otaduy, M., and Gross, M. 2008. Pose-space animation and transfer of facial details. In 2008 ACM SIGGRAPH / Eurographics Symposium on Computer Animation, ACM, New York, NY, USA, 57--66. Google ScholarDigital Library
- Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Proc. of the 8th European Conference on Computer Vision, Springer-Verlag, Prague, Czech Republic.Google Scholar
- Cerda, E., and Mahadevan, L. 2003. Geometry and physics of wrinkling. Physical Review Letters 90, 7 (February), 074302+.Google ScholarCross Ref
- Davis, J., Nehab, D., Ramamoorthi, R., and Rusinkiewicz, S. 2005. Spacetime stereo: A unifying framework for depth from triangulation. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 2, 296--302. Google ScholarDigital Library
- Golovinskiy, A., Matusik, W., Pfister, H., Rusinkiewicz, S., and Funkhouser, T. 2006. A statistical model for synthesis of detailed facial geometry. ACM Transactions on Graphics 25, 3 (July), 1025--1034. Google ScholarDigital Library
- Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In Proceedings of SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, 55--66. Google Scholar
- Haro, A., Guenter, B., and Essa, I. 2001. Real-time, photorealistic, physically based rendering of fine scale human skin structure. In Rendering Techniques 2001: 12th Eurographics Workshop on Rendering, 53--62. Google Scholar
- Hawkins, T., Wenger, A., Tchou, C., Gardner, A., Göransson, F., and Debevec, P. 2004. Animatable facial reflectance fields. In Rendering Techniques 2004: 15th Eurographics Workshop on Rendering, 309--320. Google Scholar
- Jones, A., Gardner, A., Bolas, M., McDowall, I., and Debevec, P. 2006. Performance geometry capture for spatially varying relighting. In CVMP 2006.Google Scholar
- Joshi, P., Tien, W. C., Desbrun, M., and Pighin, F. 2003. Learning controls for blend shape based realistic facial animation. In 2003 ACM SIGGRAPH / Eurographics Symposium on Computer Animation, 187--192. Google ScholarDigital Library
- Lee, Y., Terzopoulos, D., and Waters, K. 1995. Realistic modeling for facial animation. Proceedings of SIGGRAPH 95, 55--62. Google Scholar
- Ma, W.-C., Hawkins, T., Peers, P., Chabert, C.-F., Weiss, M., and Debevec, P. 2007. Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In Rendering Techniques, 183--194. Google Scholar
- Malzbender, T., Gelb, D., and Wolters, H. 2001. Polynomial texture maps. In Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, 519--528. Google Scholar
- Malzbender, T., Wilburn, B., Gelb, D., and Ambrisco, B. 2006. Surface enhancement using real-time photometric stereo and reflectance transformation. In Rendering Techniques 2006: 17th Eurographics Workshop on Rendering, 245--250. Google Scholar
- Nehab, D., Rusinkiewicz, S., Davis, J., and Ramamoorthi, R. 2005. Efficiently combining positions and normals for precise 3d geometry. ACM Transactions on Graphics 24, 3 (Aug.), 536--543. Google ScholarDigital Library
- Oat, C. 2007. Animated wrinkle maps. In SIGGRAPH 2007: ACM SIGGRAPH 2007 courses, ACM, New York, NY, USA, 33--37. Google Scholar
- Parke, F. I. 1972. Computer generated animation of faces. In ACM'72: Proceedings of the ACM annual conference, ACM, New York, NY, USA, 451--457. Google Scholar
- Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D. H. 1998. Synthesizing realistic facial expressions from photographs. In Proceedings of SIGGRAPH 98, Computer Graphics Proceedings, Annual Conference Series, 75--84. Google Scholar
- Rusinkiewicz, S., Hall-Holt, O., and Levoy, M. 2002. Real-time 3d model acquisition. ACM Transactions on Graphics 21, 3 (July), 438--446. Google ScholarDigital Library
- Sánchez, M. A. 2006. Techniques for performance-based, realtime facial animation. PhD thesis, University of Sheffield.Google Scholar
- Sifakis, E., Neverov, I., and Fedkiw, R. 2005. Automatic determination of facial muscle activations from sparse motion capture marker data. ACM Transactions on Graphics 24, 3 (Aug.), 417--425. Google ScholarDigital Library
- Terzopoulos, D., and Waters, K. 1990. Physically-based facial modelling, analysis, and animation. Journal of Visualization and Computer Animation 1, 2, 73--80.Google ScholarCross Ref
- Vlasic, D., Brand, M., Pfister, H., and Popović, J. 2005. Face transfer with multilinear models. ACM Transactions on Graphics 24, 3 (Aug.), 426--433. Google ScholarDigital Library
- Wenger, A., Gardner, A., Tchou, C., Unger, J., Hawkins, T., and Debevec, P. 2005. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Transactions on Graphics 24, 3 (Aug.), 756--764. Google ScholarDigital Library
- Williams, L. 1990. Performance-driven facial animation. In Computer Graphics (Proceedings of SIGGRAPH 90), 235--242. Google Scholar
- Zhang, S., and Huang, P. 2006. High-resolution, real-time three-dimensional shape measurement. Optical Engineering 45, 12.Google Scholar
- Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Transactions on Graphics 23, 3 (Aug.), 548--558. Google ScholarDigital Library
Index Terms
- Facial performance synthesis using deformation-driven polynomial displacement maps
Recommendations
Facial performance synthesis using deformation-driven polynomial displacement maps
SIGGRAPH Asia '08: ACM SIGGRAPH Asia 2008 papersWe present a novel method for acquisition, modeling, compression, and synthesis of realistic facial deformations using polynomial displacement maps. Our method consists of an analysis phase where the relationship between motion capture markers and ...
Data-driven facial expression synthesis via Laplacian deformation
Realistic talking heads have important use in interactive multimedia applications. This paper presents a novel framework to synthesize realistic facial animations driven by motion capture data using Laplacian deformation. We first capture the facial ...
Comments