Thibaut Weise
Mark Pauly
ABSTRACT
We present a complete integrated system for live facial puppetry that enables high-resolution real-time facial expression tracking with transfer to another person's face. The system utilizes a real-time structured light scanner that provides dense 3D data and texture. A generic template mesh, fitted to a rigid reconstruction of the actor's face, is tracked offline in a training stage through a set of expression sequences. These sequences are used to build a person-specific linear face model that is subsequently used for online face tracking and expression transfer. Even with just a single rigid pose of the target face, convincing real-time facial animations are achievable. The actor becomes a puppeteer with complete and accurate control over a digital face.
- {ACP03} Allen B., Curless B., Popović Z.: The space of human body shapes: reconstruction and parameterization from range scans. ACM Trans. Graph. 22 (2003), 587--594. Google Scholar
Digital Library
- {ARV07} Amberg B., Romdhani S., Vetter T.: Optimal step nonrigid icp algorithms for surface registration. In CVPR'07 (2007).Google ScholarCross Ref
- {BBPV03} Blanz V., Basso C., Poggio T., Vetter T.: Reanimating faces in images and video. In EUROGRAPHICS (2003).Google Scholar
- {BHB00} Bregler C., Hertzmann A., Biermann H.: Recovering non-rigid 3d shape from image streams. CVPR'02 2 (2000), 2690.Google ScholarCross Ref
- {BLB*08} Bickel B., Lang M., Botsch M., Otaduy M. A., Gross M.: Pose-space animation and transfer of facial details. In Proc. of SCA'08 (2008). Google ScholarDigital Library
- {BS08} Botsch M., Sorkine O.: On linear variational surface deformation methods. IEEE Trans. on Visualization and Computer Graphics 14 (2008), 213--230. Google ScholarDigital Library
- {BV99} Blanz V., Vetter T.: A morphable model for the synthesis of 3d faces. In SIGGRAPH '99 (1999). Google ScholarDigital Library
- {CB05} Chuang E., Bregler C.: Mood swings: expressive speech animation. ACM Trans. Graph. 24, 2 (2005), 331--347. Google ScholarDigital Library
- {Chu04} Chuang E.: Analysis, Synthesis, and Retargeting of Facial Expressions. PhD thesis, Stanford University, 2004. Google ScholarDigital Library
- {CM93} Cohen M. M., Massaro D. W.: Modeling coarticulation in synthetic visual speech. In Models and Techniques in Computer Animation (1993).Google Scholar
- {CXH03} Chai J. X., Xiao J., Hodgins J.: Vision-based control of 3d facial animation. In Proc. of SCA '03 (2003). Google ScholarDigital Library
- {DCFN06} Deng Z., Chiang P.-Y., Fox P., Neumann U.: Animating blendshape faces by cross-mapping motion capture data. In I3D '06: Proc. of the Symp. on Interactive 3D graphics and games (2006). Google ScholarDigital Library
- {DM00} Decarlo D., Metaxas D.: Optical flow constraints on deformable models with applications to face tracking. Int. J. Comput. Vision 38 (2000), 99--127. Google ScholarDigital Library
- {DN07} Deng Z., Neumann U.: Computer Facial Animation: A Survey. Springer London, 2007.Google ScholarCross Ref
- {Fis36} Fisher R. A.: The use of multiple measurements in taxonomic problems. Annals Eugen. 7 (1936), 179--188.Google ScholarCross Ref
- {FKY08} Feng W.-W., Kim B.-U., Yu Y.: Real-time data driven deformation using kernel canonical correlation analysis. ACM Trans. Graph. 27 (2008), 1--9. Google ScholarDigital Library
- {Hav06} Havaldar P.: Sony pictures imageworks. In SIGGRAPH '06: Courses (2006). Google ScholarDigital Library
- {HIWZ05} Hyneman W., Itokazu H., Williams L., Zhao X.: Human face project. In SIGGRAPH '05: Courses (2005). Google ScholarDigital Library
- {HMN03} Hiwada K., Maki A., Nakashima A.: Mimicking video: real-time morphable 3d model fitting. In VRST '03: Proc. of the Symp. on Virtual Reality Software and Technology (2003). Google ScholarDigital Library
- {HS81} Horn B. K. P., Schunk B. G.: Determining optical flow. Artificial Intelligence 17 (1981), 185--203.Google ScholarDigital Library
- {JTDP03} Joshi P., Tien W. C., Desbrun M., Pighin F.: Learning controls for blend shape based realistic facial animation. In Proc. of SCA '03 (2003). Google ScholarDigital Library
- {KMG04} Kalberer G. A., Mueller P., Gool L. V.: Animation pipeline: Realistic speech based on observed 3d face dynamics. In 1st Europ. Conf. on Visual Media Prod. (2004).Google Scholar
- {LCXS07} Lau M., Chai J., Xu Y.-Q., Shum H.-Y.: Face poser: interactive modeling of 3d facial expressions using model priors. In Proc. of SCA '07 (2007). Google ScholarDigital Library
- {LSP08} Li H., Sumner R. W., Pauly M.: Global correspondence optimization for non-rigid registration of depth scans. SGP'08 27 (2008). Google ScholarDigital Library
- {MGPG04} Mitra N. J., Gelfand N., Pottmann H., Guibas L.: Registration of point cloud data from a geometric optimization perspective. In SGP '04 (2004). Google ScholarDigital Library
- {MJC*08} Ma W.-C., Jones A., Chiang J.-Y., Hawkins T., Frederiksen S., Peers P., Vukovic M., Ouhyoung M., Debevec P.: Facial performance synthesis using deformation-driven polynomial displacement maps. ACM Trans. Graph. 27 (2008), 1--10. Google ScholarDigital Library
- {NN01} Noh J.-Y., Neumann U.: Expression cloning. In SIGGRAPH '01 (2001). Google ScholarDigital Library
- {Par72} Parke F. I.: Computer generated animation of faces. In ACM'72: Proceedings of the ACM annual conference (1972). Google ScholarDigital Library
- {Par82} Parke F.: Parameterized models for facial animation. Computer Graphics and Applications, IEEE 2 (1982), 61--68. Google ScholarDigital Library
- {PB81} Platt S. M., Badler N. I.: Animating facial expressions. SIGGRAPH Comput. Graph. 15, 3 (1981), 245--252. Google ScholarDigital Library
- {PSS99} Pighin F., Szeliski R., Salesin D.: Resynthesizing facial animation through 3d model-based tracking. In Proc. 7th IEEE Int. Conf. on Computer Vision 1 (1999), 143--150 vol. 1.Google ScholarCross Ref
- {PW96} Parke F. I., Waters K.: Computer facial animation. A. K. Peters, Ltd., 1996. Google ScholarDigital Library
- {RHHL02} Rusinkiewicz S., Hall-Holt O., Levoy M.: Real-time 3d model acquisition. ACM Trans. Graph. 21 (2002). Google ScholarDigital Library
- {RL01} Rusinkiewicz S., Levoy M.: Efficient variants of the ICP algorithm. In 3DIM'01 (2001).Google Scholar
- {SG04} Schenk O., Gärtner K.: Solving unsymmetric sparse systems of linear equations with pardiso. Future Gener. Comput. Syst. 20 (2004), 475--487. Google ScholarDigital Library
- {Sir87} Sirovich L.; Kirby M.: Low-dimensional precedure for the characterization of human faces. Journal of the Optical Society of America A 4 (1987), 519--524.Google ScholarCross Ref
- {SP04} Sumner R. W., Popović J.: Deformation transfer for triangle meshes. In SIGGRAPH '04 (2004). Google ScholarDigital Library
- {SSRMF06} Sifakis E., Selle A., Robinson-Mosher A., Fedkiw R.: Simulating speech with a physics-based facial muscle model. In Proc. of SCA '06 (2006). Google ScholarDigital Library
- {TD90} Terzopoulos D. W. K.: Physically-based facial modeling, analysis and anmiation. Journal of Visualization and Computer Animation 1 (1990), 73--80.Google ScholarCross Ref
- {VBPP05} Vlasic D., Brand M., Pfister H., Popović J.: Face transfer with multilinear models. ACM Trans. Graph. 24 (2005), 426--433. Google ScholarDigital Library
- {Wil90} Williams L.: Performance-driven facial animation. In SIGGRAPH '90 (1990). Google ScholarDigital Library
- {WLG07} Weise T., Leibe B., Gool L. V.: Fast 3d scanning with automatic motion compensation. In CVPR'07 (2007).Google ScholarCross Ref
- {ZLG*06} Zhang Q., Liu Z., Guo B., Terzopoulos D., Shum H.-Y.: Geometry-driven photorealistic facial expression synthesis. IEEE Trans. on Vis. and Comp. Graph. 12, 1 (2006), 48--60. Google ScholarDigital Library
- {ZSCS04} Zhang L., Snavely N., Curless B., Seitz S. M.: Spacetime faces: High-resolution capture for modeling and animation. In ACM Annual Conf. on Comp. Graphics (2004). Google ScholarDigital Library
Index Terms
- Face/Off: live facial puppetry
Recommendations
Age-Invariant Face Recognition
One of the challenges in automatic face recognition is to achieve temporal invariance. In other words, the goal is to come up with a representation and matching scheme that is robust to changes due to facial aging. Facial aging is a complex process that ...
Lighting-aware face frontalization for unconstrained face recognition
Provide both lighting-recovered and lighting-normalized frontalized images.Basic frontalization with a generic 3D face model by the alignment of only five landmarks.Lighting recovered and normalized image filling by the symmetry of quotient image.LRFF ...
Fully automatic face normalization and single sample face recognition in unconstrained environments
We present a fully automatic face normalization and recognition system.It normalizes the face images for both in-plane and out-of-plane pose variations.The performance of AAM fitting is improved using a novel initialization technique.HOG and Gabor ...
Comments