Skip to main content
SpringerLink
Log in

Optical Flow Constraints on Deformable Models with Applications to Face Tracking

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Optical flow provides a constraint on the motion of a deformable model. We derive and solve a dynamic system incorporating flow as a hard constraint, producing a model-based least-squares optical flow solution. Our solution also ensures the constraint remains satisfied when combined with edge information, which helps combat tracking error accumulation. Constraint enforcement can be relaxed using a Kalman filter, which permits controlled constraint violations based on the noise present in the optical flow information, and enables optical flow and edge information to be combined more robustly and efficiently. We apply this framework to the estimation of face shape and motion using a 3D deformable face model. This model uses a small number of parameters to describe a rich variety of face shapes and facial expressions. We present experiments in extracting the shape and motion of a face from image sequences which validate the accuracy of the method. They also demonstrate that our treatment of optical flow as a hard constraint, as well as our use of a Kalman filter to reconcile these constraints with the uncertainty in the optical flow, are vital for improving the performance of our system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Adiv, G. 1985. Determining 3-d motion and structure from optical flow generated by several moving objects. IEEE Pattern Analysis and Machine Intelligence, 7(4):384–401.

    Google Scholar 

  • Ayache, N.J. and Faugeras, O.D. 1988. Building, registering, and fusing noisy visual maps. IJRR, 7(6):45–65.

    Google Scholar 

  • Bar-Shalom, Y. and Fortmann, T. 1988. Tracking and Data Association. Academic Press.

  • Basu, S., Essa, I., and Pentland, A. 1996. Motion regularization for model-based head tracking. In Proceedings ICPR’ 96, p. C8A.3.

  • Bergen, J., Anandan, P., Hanna, K., and Hingorani, R. 1992. Hierarchical model-based motion estimation. In Proceedings ECCV’ 92, pp. 237–252.

  • Black, M. and Yacoob, Y. 1995. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In Proceedings ICCV’ 95, pp. 374–381.

  • Broida, T.J. and Chellappa, R. 1996. Estimation of object motion parameters from noisy images. IEEE Pattern Analysis and Machine Intelligence, 8(1):90–99.

    Google Scholar 

  • Bryson, A. and Ho, Y. 1975. Applied Optimal Control. Halsted Press.

  • Choi, C., Aizawa, K., Harashima, H., and Takebe, T. 1994. Analysis and synthesis of facial image sequences in model-based image coding. IEEE Circuits and Systems for Video Technology, 4(3):257–275.

    Google Scholar 

  • Cootes, T., Edwards, G., and Taylor, C. 1998. Active appearance models. In ECCV98, pp. II:484–498.

    Google Scholar 

  • DeCarlo, D. and Metaxas, D. 1996. The integration of optical flow and deformable models with applications to human face shape and motion estimation. In Proceedings CVPR’ 96, pp. 231–238.

  • Durrant-Whyte, H.F. 1987. Consistent integration and propagation of disparate sensor observations. IJRR, 6(3):3–24.

    Google Scholar 

  • Ekman, P. and Friesen, W. 1978. The Facial Action Coding System. Consulting Psychologist Press, Inc.

  • Essa, I.A. and Pentland, A.P. 1997. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Pattern Analysis and Machine Intelligence, 19(7):757–763.

    Google Scholar 

  • Farkas, L. 1987. Anthropometric Facial Proportions in Medicine. Thomas Books.

  • Farkas, L. 1994. Anthropometry of the Head and Face. Raven Press.

  • Fua, P. and Brechbuhler, C. 1997. Imposing hard constraints on deformable models through optimization in orthogonal subspaces. Computer Vision and Image Understanding, 65(2):148–162.

    Google Scholar 

  • Fua, P. and Leclerc, Y.G. 1995. Object-centered surface reconstruction: Combining multi-image stereo and shading. International Journal of Computer Vision, 16(1):35–56.

    Google Scholar 

  • Gelb, A. 1974. Applied Optimal Estimation. MIT Press.

  • Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In Proceedings SIGGRAPH’ 98, pp. 55–66.

  • Hel-Or, Y. and Werman, M. 1996. Constraint fusion for recognition and localization of articulated objects. International Journal of Computer Vision, 19(1):5–28.

    Google Scholar 

  • Horn, B. 1986. Robot Vision. McGraw-Hill.

  • Horn, B. and Weldon, E. 1988. Direct methods for recovering motion. International Journal of Computer Vision, 2(1):51–76.

    Google Scholar 

  • Hrdlicka, A. 1972. Practical anthropometry. AMS Press.

  • Kaucic, A. and Blake, A. 1998. Accurate, real-time, unadorned lip tracking. In ICCV98, pp. 370–375.

  • Kriegman, D.J., Triendl, E., and Binford, T.O. 1989. Stereo vision and navigation in buildings for mobile robots. RA, 5(6):792–803.

    Google Scholar 

  • Lanitis, A., Taylor, C.J., and Cootes, T.F. 1997. Automatic interpretation and coding of face images using flexible models. IEEE Pattern Analysis and Machine Intelligence, 19(7):743–756.

    Google Scholar 

  • Li, H., Roivainen, P., and Forchheimer, R. 1993. 3-D motion estimation in model-based facial image coding. IEEE Pattern Analysis and Machine Intelligence, 15(6):545–555.

    Google Scholar 

  • Lowe, D.G. 1991. Fitting parameterized three-dimensional models to images. IEEE Pattern Analysis and Machine Intelligence, 13(5):441–450.

    Google Scholar 

  • Maybeck, P. 1979. Stochastic Models, Estimation and Control, Volume 1. Academic Press.

  • Maybeck, P. 1982. Stochastic Models, Estimation and Control, Volume 2. Academic Press.

  • Metaxas, D. 1996. Physics-Based Deformable Models: Applications to Computer Vision, Graphics, and Medical Imaging. Kluwer Academic Publishers.

  • Metaxas, D. and Terzopoulos, D. 1993. Shape and nonrigid motion estimation through physics-based synthesis. IEEE Pattern Analysis and Machine Intelligence, 15(6):580–591.

    Google Scholar 

  • Moses, Y., Reynard, D., and Blake, A. 1995. Robust real time tracking and classificiation of facial expressions. In Proceedings ICCV’ 95, pp. 296–301.

  • Negahdaripour, S. and Horn, B. 1987. Direct passive navigation. IEEE Pattern Analysis and Machine Intelligence, 9(1):168–176.

    Google Scholar 

  • Netravali, A. and Salz, J. 1985. Algorithms for estimation of threedimensional motion. AT&T Technical Journal, 64:335–346.

    Google Scholar 

  • Pentland, A. and Horowitz, B. 1991. Recovery of nonrigid motion and structure. IEEE Pattern Analysis and Machine Intelligence, 13(7):730–742.

    Google Scholar 

  • Pentland, A. and Sclaroff, S. 1991. Closed-form solutions for physically based shape modeling and recognition. IEEE Pattern Analysis and Machine Intelligence, 13(7):715–729.

    Google Scholar 

  • Press, W., Teukolsky, S., Vetterling, W., and Flannery, B. 1992. Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press.

  • Reynard, D., Wildenberg, A., Blake, A., and Marchant, J. 1996. Learning dynamics of complex motions from image sequences. In Proceedings ECCV’ 96, pp. I:357–368.

    Google Scholar 

  • Shabana, A. 1989. Dynamics of Multibody Systems. Wiley.

  • Sharma, R., Azoz, Y., and Devi, L. 1998. Reliable tracking of human arm dynamics by multiple cue integration and constraint fusion. In Proceedings CVPR’ 98.

  • Shi, J. and Tomasi, C. 1994. Good features to track. In Proceedings CVPR’ 94, pp. 593–600.

  • Simoncelli, E., Adelson, E., and Heeger, D. 1991. Probability distributions of optical flow. In Proceedings CVPR’ 91, pp. 310–315.

  • Strang, G. 1988. Linear algebra and its applications. Harcourt, Brace, Jovanovich.

    Google Scholar 

  • Terzopoulos, D. 1993. Physically-based fusion of visual data over space, time and scale. In Multisensor Fusion for Computer Vision, J. Aggarwal (Ed.). Springer-Verlag, pp. 63–69.

  • Terzopoulos, D. and Waters, K. 1993. Analysis and synthesis of facial image sequences using physical and anatomical models. IEEE Pattern Analysis and Machine Intelligence, 15(6):569–579.

    Google Scholar 

  • Terzopoulos, D., Witkin, A., and Kass, M. 1988. Constraints on deformable models: Recovering 3D shape and nonrigid motion. Artificial Intelligence, 36(1):91–123.

    Google Scholar 

  • Yacoob, Y. and Davis, L.S. 1994. Computing spatio-temporal representations of human faces. In Proceedings CVPR’ 94, pp. 70–75.

  • Yuille, A.L., Cohen, D.S., and Halliman, P. 1992. Feature extraction from faces using deformable templates. International Journal of Computer Vision, 8:104–109.

    Google Scholar 

  • Zhang, G.H. and Wallace, A. 1993. Physical modeling and combination of range and intensity edge data. Computer Vision, Graphics, and Image Processing, 58(2):191–220.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, and Center for Cognitive Science, Rutgers University, Piscataway, NJ, 08854-8019, USA

    Douglas DeCarlo

  2. Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, 19104-6389, USA

    Dimitris Metaxas

Authors
  1. Douglas DeCarlo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dimitris Metaxas
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

DeCarlo, D., Metaxas, D. Optical Flow Constraints on Deformable Models with Applications to Face Tracking. International Journal of Computer Vision 38, 99–127 (2000). https://doi.org/10.1023/A:1008122917811

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1008122917811

Keywords

Search

Navigation