Skip to main content
Log in

Epipolar-plane image analysis: An approach to determining structure from motion

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

We present a technique for building a three-dimensional description of a static scene from a dense sequence of images. These images are taken in such rapid succession that they form a solid block of data in which the temporal continuity from image to image is approximately equal to the spatial continuity in an individual image. The technique utilizes knowledge of the camera motion to form and analyze slices of this solid. These slices directly encode not only the three-dimensional positions of objects, but also such spatiotemporal events as the occlusion of one object by another. For straight-line camera motions, these slices have a simple linear structure that makes them easier to analyze. The analysis computes the three-dimensional positions of object features, marks occlusion boundaries on the objects, and builds a three-dimensional map of “free space.” In our article, we first describe the application of this technique to a simple camera motion, and then show how projective duality is used to extend the analysis to a wider class of camera motions and object types that include curved and moving objects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Bibliography

  1. E.H. Adelson and J.R. Bergen, “Spatiotemporal energy models for the perception of motion,” Journal of the Optical Society of America A 2, pp. 284–299, 1985.

    Google Scholar 

  2. G. Adiv, “Determining three-dimensional motion and structure from optical flow generated by several moving objects,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-7, pp. 384–401, 1985.

    Google Scholar 

  3. J. Aloimonos and I. Rigoutsos, “Determining the 3-D motion of a rigid planar patch without correspondence, under perspective projection,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 167–174.

  4. T. Atherton, Report of private discussion with B. Steer of the University of Warwick, 1986.

  5. L. Auslander and R.E. MacKenzie, Introduction to Differentiable Manifolds, New York: Dover, 1977.

    Google Scholar 

  6. H.H. Baker, R.C. Bolles, and D.H. Marimont, “A new technique for obtaining depth information from a moving sensor,” in Proceedings of the ISPRS Commission II Symposium on Photogrammetric and Remote Sensing Systems for Data Processing and Analysis, Baltimore, 1986.

  7. S.T. Barnard and W.B. Thompson, “Disparity analysis of images,” IEEE Transactions on Pattern Analysis and Machine Intelligence 2, 1980, pp. 333–340.

    Google Scholar 

  8. J. Barron, “A Survey of Approaches for Determining Optic Flow, Environmental Layout and Egomotion,” Department of Computer Science, University of Toronto, Report No. RBCV-TR-84-5, 1984.

  9. J.V. Beck and K.J. Arnold, “Parameter Estimation in Engineerign and Science, New York: John Wiley and Sons, 1977.

    Google Scholar 

  10. W.H. Beyer (ed.), CRC Standard Mathematical Tables, 26th edn., Boca Raton, FL: CRC Press, 1981.

    Google Scholar 

  11. A.P. Blicher and S.M. Omohundro, “Unique recovery of motion and optic flow via lie algebras,” in Proceedings of the Ninth International Joint Conference on Artificial Intelligence (IJCAI-85), Los Angeles, 1985, pp. 889–891.

  12. R.C. Bolles, and H.H. Baker, “Epipolar-plane image analysis: A technique for analyzing motion sequences,” in Proceedings of the Third Workshop on Computer Vision: Representation and Control, Bellaire, MI, 1985, pp. 168–178.

  13. N.J. Bridwell and T.S. Huang, “A discrete spatial representation for lateral motion stereo,” Computer Vision, Graphics, and Image Processing 21, pp. 33–57, 1983.

    Google Scholar 

  14. T.J. Broida and R. Chellappa, “Estimation of object motion parameters from noisy images,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-8, pp. 90–99, 1986.

    Google Scholar 

  15. T.J., Broida and R. Chellappa, “Kinematics and structure of a rigid object from a sequence of noisy images,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 95–100.

  16. A.R. Bruss and B.K.P. Horn, “Passive navigation,” Computer Vision, Graphics, and Image Processing, 21, pp. 3–20, 1983.

    Google Scholar 

  17. B.F. Buxton and H. Buxton, “Monocular depth perception from optical flow by space time signal processing,” Proceedings of the Royal Society of London B, 218, pp. 27–47, 1983.

    Google Scholar 

  18. B.F. Buxton and H. Buxton, “Computation of optic flow from the motion of edge features in image sequences,” Image and Vision Computing, 2, 1984, pp. 59–75.

    Google Scholar 

  19. R. Chatila and J.-P. Laumond, “Position referencing and consistent world modeling for mobile robots,” in Proceedings of the 1985 IEEE International Conference on Robotics and Automation, St. Louis, 1985, pp. 138–145.

  20. H.S.M. Coxeter, Projective Geometry, New York: Blaisdell, 1964.

    Google Scholar 

  21. L.S. Dreschler and H.-H. Nagel, “Volumetric model and 3-D trajectory of a moving car derived from nomocular TV-frame sequences of a street scene,” in Proceedings of the Seventh International Joint Conference on Artificial Intelligence (IJCAI-81), Vancouver, 1981, pp. 692–697.

  22. L.S. Dreschler and H.-H. Nagel, “On the selection of critical points and local curvature extrema of region boundaries for interframe matching,” in Image sequence Processing and Dynamic Scene Analysis, T.S. Huang (ed.), Berlin: Spring-Verlag, 1983, pp. 457–470.

    Google Scholar 

  23. J.-Q. Fang and T.S. Huang, “Solving three-dimensional small-rotation motion equations: Uniqueness, algorithms, and numerical results, Computer Vision, Graphics, and Image Processing 26, pp. 183–206, 1984.

    Google Scholar 

  24. J.-Q. Fang and T.S. Huang, “Some experiments on estimating the 3-D motion parameters of a rigid body from two consecutive image frames,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-6, pp. 545–554, 1984.

    Google Scholar 

  25. C.L. Fennema and W.B. Thompson, “Velocity determination in scenes containing several moving objects,” Computer Graphics and Image Processing 9, pp. 301–315, 1979.

    Google Scholar 

  26. D.B. Gennery, “Tracking known three-dimensional objects,” in Proceedings of the Second National Conference on Artificial Intelligence (AAAI-82), Pittsburgh, 1982, pp. 13–17.

  27. M.J. Hannah, “Bootstrap stereo,” in Proceedings of the Image Understanding Workshop, College Park, MD, 1980, pp. 201–208.

  28. S.M. Haynes and R. Jain, “Detection of moving edges,” Computer Vision, Graphics, and Image Processing, 21, 1983, pp. 345–367.

    Google Scholar 

  29. S.M. Haynes and R. Jain, “Low level motion events: trajectory discontinuities,”.in Proceedings of the First Conference on AI Applications, Denver, 1984, pp. 251–256.

  30. D.J. Heeger, “Depth and flow from motion energy,” in Proceedings of the Fifth National Conference on Artificial Intelligence (AAAI-86), Philadelphia, 1986, pp. 657–663.

  31. E.C. Hildreth, “Computations underlying the measurement of visual motion,” Artificial Intelligence, 23, pp. 309–354, 1984.

    Google Scholar 

  32. E.C. Hildreth and N.M. Grzywacz, “The incremental recovery of structure from motion: Position vs. velocity based formulations,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 137–143.

  33. D.D. Hoffman, “Inferring local surface orientation from motion fields,” Journal of the Optical Society of America 72, 888–892, 1982.

    Google Scholar 

  34. D.D. Hoffman and B.E. Flinchbaugh, “The interpretation of biological motion,” Biological Cybernetics 42, pp. 195–204, 1982.

    Google Scholar 

  35. B.K.P. Horn and B.G. Schunk, “Determining optical flow,” Artificial Intelligence 17, pp. 185–203, 1981.

    Google Scholar 

  36. B.K.P. Horn and E.J. Weldon, Jr., “Robust direct methods for recovering motion,” unpublished manuscript, February 1986.

  37. T.S. Huang (ed.), Image sequence Analysis, Berlin: Springer-Verlag, 1981.

    Google Scholar 

  38. D.A. Huffman, “Impossible objects as nonsense sentences,” Machine Intelligence 6, pp. 295–324, 1971.

    Google Scholar 

  39. D.A. Huffman, “A duality concept for the analysis of polyhedral scenes,” Machine Intelligence 8, pp. 475–492, 1977.

    Google Scholar 

  40. R.Jain, “Direct computation of the focus of expansion,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-5, pp. 58–63, 1983.

  41. R. Jain, “Detection on moving edges” in Proceedings of the First Conference on AI Applications, Denver, 1984, pp. 142–148.

  42. K. Kanatani, “Tracing planar surface motion from projection without knowing correspondence,” Computer Vision, Graphics, and Image Processing 29, pp. 1–12, 1985.

    Google Scholar 

  43. K. Kanatani, “Detecting the motion of a planar surface by line and surface integrals,” Computer Vision, Graphics, and Image Processing 29, pp. 13–22, 1985.

    Google Scholar 

  44. K. Kanatani, “Structure from motion without correspondence: General principle,” in Proceedings of the Ninth International Joint Conference on Artificial Intelligence (IJCAI-85), Los Angeles, 1985, pp. 886–888.

  45. K. Kanatani, “Transformation of optical flow by camera rotation,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 113–118.

  46. J.J. Koenderink and A.J.van Doorn, “Invariant properties of the motion parallax field due to the movement of rigid bodies relative to an observer,” Optica Acta 22, pp. 773–791, 1975.

    Google Scholar 

  47. J.J. Koenderink and A.J.van Doorn, “Local structure of movement parallax of the plane,” Journal of the Optical Society of America 66, pp. 717–723, 1976.

    Google Scholar 

  48. J.J. Koenderink and A.J.van Doorn, “The singularities of the visual mapping,” Biological Cybernetics 24, pp. 51–59, 1976.

    Google Scholar 

  49. J.J. Koenderink and A.J. van Doorn, “How an ambulant observer can construct a model of the environment from the geometrical structure of the visual inflow,” in Kybernetik 77, Hauske and Butenandt (eds.), 1977, pp. 224–227.

  50. J.J. Koenderink and A.J.van Doorn, “Exterospecific component of the motion parallax field,” Journal of the Optical Society of America 71, pp. 953–957, 1981.

    Google Scholar 

  51. D.T. Lawton, “Processing translational motion sequences,” Computer Vision, Graphics, and Image Processing 22, pp. 116–144, 1983.

    Google Scholar 

  52. Y. Liu and T.S. Huang, “Estimation of rigid body motion using straight line correspondences,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 47–52.

  53. H.D. Longuet-Higgins, “A computer algorithm for reconstructing a scene from two projections,” Nature 293, pp. 133–135, 1981.

    Google Scholar 

  54. H.D. Longuet-Higgins, “The visual ambiguity of a moving plane,” Proceedings of the Royal Society of London B 223, pp. 165–175, 1984.

    Google Scholar 

  55. H.D. Longuet-Higgins and K. Prazdny, “The interpretation of a moving retinal image,” Proceedings of the Royal Society of London B 208, pp. 385–397, 1980.

    Google Scholar 

  56. A.K. Mackworth, “Interpreting pictures of polyhedral scenes,” Artificial Intelligence 4, pp. 121–137, 1977.

    Google Scholar 

  57. D.H. Marimont, “Inferring spatial structure from feature correspondences,” PhD dissertation, Stanford University, 1986.

  58. D.H. Marimont, “Projective duality and the analysis of image sequences,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 7–14.

  59. D. Marr and E. Hildreth, “Theory of edge detection,” Proceedings of the Royal Society of London B 207, pp. 187–217, 1980.

    Google Scholar 

  60. E.A. Maxwell, The Methods of Plane Projective Geometry Based on the Use of General Homogeneous Coordinates, Cambridge: Cambridge University Press, 1946.

    Google Scholar 

  61. E.A. Maxwell, General Homogeneous Coordinates in Space of Three Dimensions, Cambridge: Cambridge University Press, 1959.

    Google Scholar 

  62. J.L. Melsa and D.L. Cohn, Decision and Estimation Theory, New York: McGraw-Hill, 1978.

    Google Scholar 

  63. A. Mitiche, S. Seida, and J.K. Aggarwal, “Line-based computation of structure and motion using angular invariance,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 175–180.

  64. H.P. Moravec, “Visual mapping by a robot rover,” in Proceedings of the International Joint Conference on Artificial Intelligence, Tokyo, 1979, pp. 598–600.

  65. H.P. Moravec and A. Elfes, “High resolution maps from wide angle sonar,” in Proceedings of the 1985 IEEE International Conference on Robotics and Automation, St. Louis, 1985, pp. 116–120.

  66. H.-H. Nagel, “Displacement vectors derived from second-order intensity variations in image sequences,” Computer Vision, Graphics, and Image Processing 21, pp. 85–117, 1983.

    Google Scholar 

  67. S. Negahdaripour and B.K.P. Horn, “Direct passive navigation,” MIT Artif. Intell. Lab., Massachusetts Inst. Technol., AI Memo 821, February 1985.

  68. S. Negahdaripour, “Direct passive navigation: analytical solution for planes,” MIT Artif. Intell. Lab., Massachusetts Inst. Technol., AI Memo 863, August 1985.

  69. R. Nevatia, “Depth measurement from motion stereo,” Computer Graphics and Image Processing, 5, 1976, pp. 203–214.

    Google Scholar 

  70. A.P. Pentland, “Perceptual organization and the representation of natural form,” Artificial Intelligence 28, pp. 293–331, 1986.

    Google Scholar 

  71. K. Prazdny, “Motion and structure from optical flow,” in Proceedings of the Sixth International Joint Conference on Artificial Intelligence (IJCAI-79), Tokyo, 1979, pp. 702–704.

  72. K. Prazdny, “Egomotion and relative depth map from optical flow,” Biological Cybernetics, 36, pp. 87–102, 1980.

    Google Scholar 

  73. K. Prazdny, “Determining the instantaneous direction of motion from optical flow generated by a curvilinearly moving observer,” Computer and Image Processing 17, pp. 238–248, 1981.

    Google Scholar 

  74. K. Prazdny, “On the information in optical flows,” Computer Vision, Graphics, and Image Processing 22, pp. 239–259, 1983.

    Google Scholar 

  75. K. Ramer, “An iterative procedure for the polygonal approximation of plane curves,” Computer Graphics and Image Processing 1, 1972, pp. 224–256.

    Google Scholar 

  76. J.W. Roach and J.K. Aggarwal, “Determining the movement of objects from a sequence of images,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-2, pp. 554–562, 1980.

    Google Scholar 

  77. J.W. Roach and J.S. Wright, “Spherical dual images: A 3D representation method for solid objects that combines dual space and Gaussian spheres,” in Proceedings of the IEEE International Conference on Robotics and Automation, San Francisco, 1986, pp. 1087–1092.

  78. M. Subbarao, “Interpretation of image motion fields: A spatio-temporal approach,” in Proceedings: Workshop on Motion: Representation and Analysis, Kiawah Island, 1986, pp. 157–165.

  79. M. Subbarao and A.M. Waxman, “On the uniqueness of image flow solutions for planar surfaces in motion,” in Proceedings of the Third Workshop on Computer Vision: Representation and Control, Bellaire, MI, 1985, pp. 129–140.

  80. E.H. Thompson, “A rational algebraic formulation of the problem of relative orientation,” Photogrammetric Record 3, pp. 152–159, 1959.

    Google Scholar 

  81. W.B. Thompson and S.T. Barnard, “Lower-level estimation and interpretation of visual motion,” Computer 14, 1981, pp. 20–28.

    Google Scholar 

  82. W.B. Thompson, K.M. Mutch, and V.A. Berzins, “Dynamic occlusion analysis in optical flow fields,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-7, pp. 374–383, 1985.

    Google Scholar 

  83. R.Y. Tsai, “Estimating 3-D motion parameters and object surface structures from the image motion of conic arcs. I: Theoretical basis,” in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Boston, 1983, pp. 122–125.

  84. R.Y. Tsai, “Estimating 3-D motion parameters and object surface structures from the image motion of curved edges,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, 1983, pp. 259–266.

  85. R.Y. Tsai and T.S. Huang, “Estimating three-dimensional motion parameters of a rigid planar patch,” IEEE Transactions on Acoustics, Speech, and Signal Processing 29, pp. 1147–1152, 1981.

    Google Scholar 

  86. R.Y. Tsai and T.S. Huang, “Estimating three-dimensional motion parameters of a rigid planar patch. II: Singular value decomposition,” IEEE Transactions on Acoustics, Speech, and Signal Processing 30, pp. 525–533, 1982.

    Google Scholar 

  87. R.Y. Tsai and T.S. Huang, “Uniqueness and estimation of three-dimensional motion parameters of rigid objects with curved surfaces,” IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-6, pp. 13–27, 1984.

    Google Scholar 

  88. S. Ullman, The Interpretation of Visual Motion, Cambridge, MA: MIT Press, 1979.

    Google Scholar 

  89. S. Ullman, “Recent computational studies in the interpretation of structure from motion,” in Human and Machine Vision, J. Beck, B. Hope, and A. Rosenfeld (eds.), Orlando, FL: Academic Press, 1983, pp. 459–480.

    Google Scholar 

  90. S. Ullman, “Maximizing rigidity: The incremental recovery of 3-D structure from rigid and rubbery motion,” Perception 13, pp. 255–274, 1984.

    Google Scholar 

  91. A.M. Waxman, “An image flow paradigm,” in Proceedings of the Second IEEE Workshop on Computer Vision: Representation and Control, Annapolis, 1984, pp. 49–57.

  92. A.M. Waxman, and S. Ullman, “Surface structure and 3-D motion from image flow: A kinematic analysis,” Center for Automation Research, University of Maryland, CAR Tech. Report 24, October 1983.

  93. A.M. Waxman, and S. Ullman, “Surface structure and three-dimensional motion from image flow kinematics,” International Journal of Robotics Research 4, pp. 72–94, 1985.

    Google Scholar 

  94. A.M. Waxman and K. Wohn, “Contour evolution, neighborhood deformation, and global image flow: Planar surfaces in motion,” Center for Automation Research, University of Maryland, CAR Tech. Report 58, April 1984.

  95. A.M. Waxman and K. Wohn, “Contour evolution, neighborhood deformation, and global image flow: Planar surfaces in motion,” International Journal of Robotics Research 4, pp. 95–108, 1985.

    Google Scholar 

  96. J.A. Webb and J.K. Aggarwal, “Visually interpreting the motion of objects in space,” Computer 14, pp. 40–46, 1981.

    Google Scholar 

  97. K.W. Wong (author-editor), “Basic mathematics of photogrammetry,” in Manual of Photogrammetry, 4th edn., Falls Church, VA: American Society of Photogrammetry, 1980.

    Google Scholar 

  98. C.R. WylieJr., Introduction to Projective Geometry, New York: McGraw-Hill, 1970.

    Google Scholar 

  99. M. Yamamoto, “Motion analysis using the visualized locus method,” untranslated Japanese articles, 1981.

  100. B.L. Yen and T.S. Huang, “Determining 3-D motion and structure of a rigid body using the spherical projection,” Computer Graphics and Image Processing, 21, pp. 21–32, 1983.

    Google Scholar 

  101. B.L. Yen and T.S. Huang, “Determining the 3-D motion and structure of a rigid body using straight line correspondences,” Image sequence Processing and Dynamic Scene Analysis, T.S. Huang (ed.), Berlin: Spring-Verlag, pp. 365–394, 1983.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Additional information

This research was supported by DARPA contracts MDA 903-83-C-0027 and DACA 76-85-C-0004.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bolles, R.C., Baker, H.H. & Marimont, D.H. Epipolar-plane image analysis: An approach to determining structure from motion. Int J Comput Vision 1, 7–55 (1987). https://doi.org/10.1007/BF00128525

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00128525

Keywords

Navigation