Abstract
When a rigid scene is imaged by a moving camera, the set of all displacements of all points across multiple frames often resides in a low-dimensional linear subspace. Linear subspace constraints have been used successfully in the past for recovering 3D structure and 3D motion information from multiple frames (e.g., by using the factorization method of Tomasi and Kanade (1992, International Journal of Computer Vision, 9:137–154)). These methods assume that the 2D correspondences have been precomputed. However, correspondence estimation is a fundamental problem in motion analysis. In this paper we show how the multi-frame subspace constraints can be used for constraining the 2D correspondence estimation process itself.
We show that the multi-frame subspace constraints are valid not only for affine cameras, but also for a variety of imaging models, scene models, and motion models. The multi-frame subspace constraints are first translated from constraints on correspondences to constraints directly on image measurements (e.g., image brightness quantities). These brightness-based subspace constraints are then used for estimating the correspondences, by requiring that all corresponding points across all video frames reside in the appropriate low-dimensional linear subspace.
The multi-frame subspace constraints are geometrically meaningful, and are {not} violated at depth discontinuities, nor when the camera-motion changes abruptly. These constraints can therefore replace {heuristic} constraints commonly used in optical-flow estimation, such as spatial or temporal smoothness.
Similar content being viewed by others
References
Adiv, G. 1985. Determining three-dimensional motion and structure from optical flow generated by several moving objects. IEEE Trans. on Pattern Analysis and Machine Intelligence, 7(4):384-401.
Anandan, P. 1989. A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision, 2:283–310.
Anandan, P. and Avidan, S. 2000. Integrating local affine into global perspective images in the joint image space. In European Conference on Computer Vision, Dublin, pp. 907–921.
Barron, J.L., Fleet, D.J., Beauchemin, S.S., and Burkitt, T.A. 1992. Performance of optical flow techniques. In IEEE Conference on Computer Vision and Pattern Recognition, Champaign, pp. 236-242.
Bergen, J.R., Anandan, P., Hanna, K.J., and Hingorani, R. 1992. Hierarchical model-based motion estimation. In European Conference on Computer Vision, Santa Margarita Ligure, pp. 237–252.
Bergen, J.R., Burt, P.J., Hingorani, R., and Peleg, S. 1992. A three-frame algorithm for estimating two-component image motion. IEEE Trans. on Pattern Analysis and Machine Intelligence, 14:886–895.
Black, M.J. and Anandan, P. 1991. Robust dynamic motion estimation over time. In IEEE Conference on Computer Vision and Pattern Recognition, Maui, pp. 296–302.
Black, M.J. and Anandan, P. 1996. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding, 63:75–104.
Criminisi, A., Reid, I., and Zisserman, A. 1998. Duality, rigidity and planar parallax. In European Conference on Computer Vision, Freiburg.
Golub, G.H. and Van Loan, C.F. 1996. Matrix Computations. The John Hopkins University Press: Baltimore, MD.
Hanna, K. 1991. Direct multi-resolution estimation of ego-motion and structure from motion. In IEEE Workshop on Visual Motion, Princeton, NJ, pp. 156–162.
Hanna, K.J. and Okamoto, N.E. 1993. Combining stereo and motion for direct estimation of scene structure. In International Conference on Computer Vision, Berlin, pp. 357–365.
Heeger, D.J. and Jepson, A.D. 1992. Subspace methods for recovering rigid motion i: Algorithm and implementation. International Journal of Computer Vision, 7:95–117.
Horn, B.K.P. and Schunck, B.G. 1981. Determining optical flow. Artificial Intelligence, 17(1-3):185–203.
Irani, M. 1999. Multi-frame optical flow estimation using subspace constraints. In International Conference on Computer Vision, Corfu, pp. 626–633.
Irani, M. and Anandan, P. 1996. Parallax geometry of pairs of points for 3D scene analysis. In European Conference on Computer Vision, Cambridge, UK, pp. 17–30.
Irani, M. and Anandan, P. 1998. A unified approach to moving object detection in 2D and 3D scenes. In IEEE Trans. on Pattern Analysis and Machine Intelligence, 20:577–589.
Irani, M. and Anandan, P. 2000. Factorization with uncertainty. In European Conference on Computer Vision, Dublin, pp. 539–553.
Irani, M., Anandan, P., and Cohen, M. 1999. Direct recovery of planar-parallax from multiple frames. In Vision Algorithms 99, Corfu.
Irani, M., Anandan, P., and Weinshall, D. 1998. From reference frames to reference planes: Multi-view parallax geometry and applications. In European Conference on Computer Vision, Freiburg.
Irani, M., Rousso, B., and Peleg, S. 1994. Computing occluding and transparent motions. International Journal of Computer Vision, 12:5–16.
Irani, M., Rousso, B., and Peleg, P. 1997. Recovery of ego-motion using region alignment. IEEE Trans. on Pattern Analysis and Machine Intelligence, 19(3):268–272.
Kumar, R., Anandan, P., and Hanna, K. 1994. Direct recovery of shape from multiple views: A parallax based approach. In Proc 12th ICPR, pp. 685–688.
Longuet-Higgins, H.C. and Prazdny, K. 1980. The interpretation of a moving retinal image. Proceedings of The Royal Society of London B, 208:385–397.
Lucas, B.D. and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Image Understanding Workshop, pp. 121–130.
Sawhney, H. 1994. 3D geometry from planar parallax. In IEEE Conference on Computer Vision and Pattern Recognition.
Shapiro, L.S. 1995. Affine Analysis of Image Sequences. Cambridge University Press: Cambridge, UK.
Shashua, A. and Navab, N. 1994. Relative affine structure: Theory and application to 3D reconstruction from perspective views. In IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, pp. 483–489.
Stein, G.P. and Shashua, A. 1997. Model-based brightness constraints: On direct estimation of structure and motion. In IEEE Conference on Computer Vision and Pattern Recognition, San-Juan, pp. 400–406.
Szeliski, R. 1990. Bayesian modeling of uncertainty in low-level vision. International Journal of Computer Vision, 5:271–301.
Szeliski, R. and Kang, S.B. 1995. Direct methods for visual scene reconstruction. In Workshop on Representations of Visual Scenes.
Tomasi, C. and Kanade, T. 1992. Shape and motion from image streams under orthography: A factorization method. International Journal of Computer Vision, 9:137–154.
Torr, P.H.S. 1998. Geometric motion segmentation and model selection. Proceedings of The Royal Society of London A, 356:1321-1340.
Zelnik-Manor, L. and Irani, M. 2000. Multi-frame estimation of pla-nar motion. In IEEE Trans. on Pattern Analysis and Machine Intelligence, 22:1105–1116.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Irani, M. Multi-Frame Correspondence Estimation Using Subspace Constraints. International Journal of Computer Vision 48, 173–194 (2002). https://doi.org/10.1023/A:1016372015744
Issue Date:
DOI: https://doi.org/10.1023/A:1016372015744