Skip to main content
SpringerLink
Log in

Optic Flow in Harmony

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Most variational optic flow approaches just consist of three constituents: a data term, a smoothness term and a smoothness weight. In this paper, we present an approach that harmonises these three components. We start by developing an advanced data term that is robust under outliers and varying illumination conditions. This is achieved by using constraint normalisation, and an HSV colour representation with higher order constancy assumptions and a separate robust penalisation. Our novel anisotropic smoothness is designed to work complementary to the data term. To this end, it incorporates directional information from the data constraints to enable a filling-in of information solely in the direction where the data term gives no information, yielding an optimal complementary smoothing behaviour. This strategy is applied in the spatial as well as in the spatio-temporal domain. Finally, we propose a simple method for automatically determining the optimal smoothness weight. This method bases on a novel concept that we call “optimal prediction principle” (OPP). It states that the flow field obtained with the optimal smoothness weight allows for the best prediction of the next frames in the image sequence. The benefits of our “optic flow in harmony” (OFH) approach are demonstrated by an extensive experimental validation and by a competitive performance at the widely used Middlebury optic flow benchmark.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alvarez, L., Esclarín, J., Lefébure, M., & Sánchez, J. (1999). A PDE model for computing the optical flow. In Proc. XVI congreso de ecuaciones diferenciales y aplicaciones (pp. 1349–1356). Las Palmas de Gran Canaria, Spain.

    Google Scholar 

  • Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2009). A database and evaluation methodology for optical flow. Tech. Rep. MSR-TR-2009-179, Microsoft Research, Redmond, WA.

  • Barron, J. L., Fleet, D. J., & Beauchemin, S. S. (1994). Performance of optical flow techniques. International Journal of Computer Vision, 12(1), 43–77.

    Article  Google Scholar 

  • Bertero, M., Poggio, TA, & Torre, V. (1988). Ill-posed problems in early vision. Proceedings of the IEEE, 76(8), 869–889.

    Article  Google Scholar 

  • Bigün, J., Granlund, G. H., & Wiklund, J. (1991). Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(8), 775–790.

    Article  Google Scholar 

  • Black, M. J., & Anandan, P. (1996). The robust estimation of multiple motions: parametric and piecewise smooth flow fields. Computer Vision and Image Understanding, 63(1), 75–104.

    Article  Google Scholar 

  • Blake, A., & Zisserman, A. (1987). Visual reconstruction. Cambridge: MIT Press.

    Google Scholar 

  • Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In T. Pajdla & J. Matas (Eds.), Computer vision—ECCV 2004, Part IV. Lecture notes in computer science (Vol. 3024, pp. 25–36). Berlin: Springer.

    Chapter  Google Scholar 

  • Bruhn, A., & Weickert, J. (2005). Towards ultimate motion estimation: combining highest accuracy with real-time performance. In Proc. tenth international conference on computer vision (Vol. 1, pp. 749–755). Beijing: IEEE Computer Society Press.

    Google Scholar 

  • Bruhn, A., Weickert, J., & Schnörr, C. (2005). Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods. International Journal of Computer Vision, 61(3), 211–231.

    Article  Google Scholar 

  • Bruhn, A., Weickert, J., Kohlberger, T., & Schnörr, C. (2006). A multigrid platform for real-time motion computation with discontinuity-preserving variational methods. International Journal of Computer Vision, 70(3), 257–277.

    Article  Google Scholar 

  • Charbonnier, P., Blanc-Féraud, L., Aubert, G., & Barlaud, M. (1994). Two deterministic half-quadratic regularization algorithms for computed imaging. In Proc. IEEE international conference on image processing (Vol. 2, pp. 168–172). Austin: IEEE Computer Society Press.

    Chapter  Google Scholar 

  • Cohen, I. (1993). Nonlinear variational method for optical flow computation. In Proc. eighth Scandinavian conference on image analysis (Vol. 1, pp. 523–530). Norway, Tromsø.

    Google Scholar 

  • Coons, S. A. (1967). Surfaces for computer aided design of space forms (Tech. Rep. MIT/LCS/TR-41). Massachusetts Institute of Technology, Cambridge.

  • Elsgolc, L. E. (1962). Calculus of variations. London: Pergamon.

    Google Scholar 

  • Fleet, D. J., & Jepson, A. D. (1990). Computation of component image velocity from local phase information. International Journal of Computer Vision, 5(1), 77–104.

    Article  Google Scholar 

  • Förstner, W., & Gülch, E. (1987). A fast operator for detection and precise location of distinct points, corners and centres of circular features. In Proc. ISPRS intercommission conference on fast processing of photogrammetric data (pp. 281–305). Interlaken, Switzerland.

    Google Scholar 

  • Golland, P., & Bruckstein, A. M. (1997). Motion from color. Computer Vision and Image Understanding, 68(3), 346–362.

    Article  Google Scholar 

  • Grossauer, H., & Thoman, P. (2008). GPU-based multigrid: real-time performance in high resolution nonlinear image processing. In A. Gasteratos, M. Vincze, & J. K. Tsotsos (Eds.), Lecture notes in computer science: Vol. 5008. Computer vision systems (pp. 141–150). Berlin: Springer.

    Chapter  Google Scholar 

  • Gwosdek, P., Zimmer, H., Grewenig, S., Bruhn, A., & Weickert, J. (2010). A highly efficient GPU implementation for variational optic flow based on the Euler-Lagrange framework. In Proc. 2010 ECCV workshop on computer vision with GPUs, Heraklion, Greece.

    Google Scholar 

  • Horn, B., & Schunck, B. (1981). Determining optical flow. Artificial Intelligence, 17, 185–203.

    Article  Google Scholar 

  • Krajsek, K., & Mester, R. (2007). Bayesian model selection for optical flow estimation. In F. A. Hamprecht, C. Schnörr, & B. Jähne (Eds.), Pattern recognition. Lecture notes in computer science (pp. 142–151). Berlin: Springer.

    Chapter  Google Scholar 

  • Lai, S. H., & Vemuri, B. C. (1998). Reliable and efficient computation of optical flow. International Journal of Computer Vision, 29(2), 87–105.

    Article  Google Scholar 

  • Lei, C., & Yang, Y. H. (2009). Optical flow estimation on coarse-to-fine region-trees using discrete optimization. In Proc. 2009 IEEE international conference on computer vision. Kyoto: IEEE Computer Society Press.

    Google Scholar 

  • Lucas, B., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. In Proc. seventh international joint conference on artificial intelligence (pp. 674–679). Vancouver, Canada.

    Google Scholar 

  • Mileva, Y., Bruhn, A., & Weickert, J. (2007). Illumination-robust variational optical flow with photometric invariants. In F. A. Hamprecht, C. Schnörr, & B. Jähne (Eds.), Pattern recognition. Lecture notes in computer science (pp. 152–162). Berlin: Springer.

    Chapter  Google Scholar 

  • Murray, D. W., & Buxton, B. F. (1987). Scene segmentation from visual motion using global optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(2), 220–228.

    Article  Google Scholar 

  • Nagel, H. H. (1990). Extending the ‘oriented smoothness constraint’ into the temporal domain and the estimation of derivatives of optical flow. In O. Faugeras (Ed.), Lecture notes in computer science: Vol. 427. Computer vision—ECCV ’90 (pp. 139–148). Berlin: Springer.

    Google Scholar 

  • Nagel, H. H., & Enkelmann, W. (1986). An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8, 565–593.

    Article  Google Scholar 

  • Ng, L., & Solo, V. (1997). A data-driven method for choosing smoothing parameters in optical flow problems. In Proc. 1997 IEEE international conference on image processing (Vol. 3, pp. 360–363). Los Alamitos: IEEE Computer Society.

    Chapter  Google Scholar 

  • Perona, P., & Malik, J. (1990). Scale space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12, 629–639.

    Article  Google Scholar 

  • Rudin, L. I., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D, 60, 259–268.

    Article  MATH  Google Scholar 

  • Schnörr, C. (1993). On functionals with greyvalue-controlled smoothness terms for determining optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(10), 1074–1079.

    Article  Google Scholar 

  • Schnörr, C. (1994). Segmentation of visual motion by minimizing convex non-quadratic functionals. In Proc. twelfth international conference on pattern recognition (Vol. A, pp. 661–663). Jerusalem: IEEE Computer Society Press.

    Google Scholar 

  • Schoenemann, T., & Cremers, D. (2006). Near real-time motion segmentation using graph cuts. In K. Franke, K. R. Müller, B. Nickolay, & R. Schäfer (Eds.), Pattern recognition, Lecture notes in computer science (pp. 455–464). Berlin: Springer.

    Chapter  Google Scholar 

  • Shulman, D., & Hervé, J. (1989). Regularization of discontinuous flow fields. In Proc. workshop on visual motion (pp. 81–86). Irvine: IEEE Computer Society Press.

    Chapter  Google Scholar 

  • Simoncelli, E. P., Adelson, E. H., & Heeger, D. J. (1991). Probability distributions of optical flow. In Proc. 1991 IEEE computer society conference on computer vision and pattern recognition (pp. 310–315). Maui: IEEE Computer Society Press.

    Chapter  Google Scholar 

  • Sun, D., Roth, S., Lewis, J. P., & Black, M. J. (2008). Learning optical flow. In D. Forsyth, P. Torr, & A. Zisserman (Eds.), Computer vision—ECCV 2008, Part III, Lecture notes in computer science (pp. 83–97). Berlin: Springer.

    Chapter  Google Scholar 

  • Sun, D., Roth, S., & Black, M. J. (2010). Secrets of optical flow estimation and their principles. In Proc. 2010 IEEE computer society conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society Press.

    Google Scholar 

  • Sundaram, N., Brox, T., & Keutzer, K. (2010). Dense point trajectories by GPU-accelerated large displacement optical flow. In Lecture notes in computer science: Vol. 6311. Computer vision—ECCV 2010 (pp. 438–451). Berlin: Springer.

    Chapter  Google Scholar 

  • Tretiak, O., & Pastor, L. (1984). Velocity estimation from image sequences with second order differential operators. In Proc. seventh international conference on pattern recognition (pp. 16–19). Montreal, Canada.

    Google Scholar 

  • van de Weijer, J., & Gevers, T. (2004). Robust optical flow from photometric invariants. In Proc. 2004 IEEE international conference on image processing (Vol. 3, pp. 1835–1838). Singapore: IEEE Signal Processing Society.

    Google Scholar 

  • Wedel, A., Pock, T., Zach, C., Bischof, H., & Cremers, D. (2008). An improved algorithm for TV-L 1 optical flow computation. In D. Cremers, B. Rosenhahn, A. L. Yuille, & F. R. Schmidt (Eds.), Lecture notes in computer science: Vol. 5604. Statistical and geometrical approaches to visual motion analysis (pp. 23–45). Berlin: Springer.

    Google Scholar 

  • Wedel, A., Cremers, D., Pock, T., & Bischof, H. (2009). Structure- and motion-adaptive regularization for high accuracy optic flow. In Proc. 2009 IEEE international conference on computer vision. Kyoto: IEEE Computer Society Press.

    Google Scholar 

  • Weickert, J. (1996). Theoretical foundations of anisotropic diffusion in image processing. Computing Supplement, 11, 221–236.

    Google Scholar 

  • Weickert, J., & Schnörr, C. (2001a). A theoretical framework for convex regularizers in PDE-based computation of image motion. International Journal of Computer Vision, 45(3), 245–264.

    Article  MATH  Google Scholar 

  • Weickert, J., & Schnörr, C. (2001b). Variational optic flow computation with a spatio-temporal smoothness constraint. Journal of Mathematical Imaging and Vision, 14(3), 245–255.

    Article  MATH  Google Scholar 

  • Werlberger, M., Pock, T., & Bischof, H. (2010). Motion estimation with non-local total variation regularization. In Proc. 2010 IEEE computer society conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society Press.

    Google Scholar 

  • Xiao, J., Cheng, H., Sawhney, H., Rao, C., & Isnardi, M. (2006). Bilateral filtering-based optical flow estimation with occlusion detection. In A. Leonardis, H. Bischof, & A. Pinz (Eds.), Lecture notes in computer science: Vol. 3951. Computer vision—ECCV 2006, Part I (pp. 211–224). Berlin: Springer.

    Google Scholar 

  • Xu, L., Jia, J., & Matsushita, Y. (2010). Motion detail preserving optical flow estimation. In Proc. 2010 IEEE computer society conference on computer vision and pattern recognition. San Francisco: IEEE Computer Society Press.

    Google Scholar 

  • Yaroslavsky, L. P. (1985). Digital picture processing: an introduction. Berlin: Springer.

    MATH  Google Scholar 

  • Yoon, K. J., & Kweon, I. S. (2006). Adaptive support-weight approach for correspondence search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(4), 650–656.

    Article  Google Scholar 

  • Zach, C., Pock, T., & Bischof, H. (2007). A duality based approach for realtime TV-L 1 optical flow. In F. Hamprecht, C. Schnörr, & B. Jähne (Eds.), Lecture notes in computer science: Vol. 4713. Pattern recognition (pp. 214–223). Berlin: Springer.

    Chapter  Google Scholar 

  • Zimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., & Seidel, H. P. (2009). Complementary optic flow. In D. Cremers, Y. Boykov, A. Blake, & F. R. Schmidt (Eds.), Lecture notes in computer science: Vol. 5681. Energy minimization methods in computer vision and pattern recognition (EMMCVPR) (pp. 207–220). Berlin: Springer.

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Mathematical Image Analysis Group, Faculty of Mathematics and Computer Science, Saarland University, Campus E1.1, 66041, Saarbrücken, Germany

    Henning Zimmer & Joachim Weickert

  2. Vision and Image Processing Group, Cluster of Excellence Multimodal Computing and Interaction, Saarland University, Campus E1.1, 66041, Saarbrücken, Germany

    Andrés Bruhn

Authors
  1. Henning Zimmer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrés Bruhn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joachim Weickert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Henning Zimmer.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zimmer, H., Bruhn, A. & Weickert, J. Optic Flow in Harmony. Int J Comput Vis 93, 368–388 (2011). https://doi.org/10.1007/s11263-011-0422-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-011-0422-6

Keywords

Search

Navigation