Skip to main content
SpringerLink
Log in

Depth Estimation Through a Generative Model of Light Field Synthesis

  • Conference paper
  • First Online:
Pattern Recognition (GCPR 2016)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 9796))

Included in the following conference series:

Abstract

Light field photography captures rich structural information that may facilitate a number of traditional image processing and computer vision tasks. A crucial ingredient in such endeavors is accurate depth recovery. We present a novel framework that allows the recovery of a high quality continuous depth map from light field data. To this end we propose a generative model of a light field that is fully parametrized by its corresponding depth map. The model allows for the integration of powerful regularization techniques such as a non-local means prior, facilitating accurate depth map estimation. Comparisons with previous methods show that we are able to recover faithful depth maps with much finer details. In a number of challenging real-world examples we demonstrate both the effectiveness and robustness of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    We discuss the implication of the no-occlusion assumption in more detail in the supplemental material.

  2. 2.

    We use the given default values \(\sigma =1.0\) and \(\tau =0.5\).

  3. 3.

    http://openmp.org

  4. 4.

    http://webdav.tue.mpg.de/pixel/lightfield_depth_estimation/

  5. 5.

    The images have neither been gamma compressed nor rectified.

  6. 6.

    Buddha, buddha2, horses, medieval, monasRoom, papillon, stillLife.

References

  1. The (new) stanford light field archive (2008). http://lightfield.stanford.edu. Accessed 07 Apr 2016

  2. Adelson, E.H., Wang, J.Y.A.: Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 14(2), 99–106 (1992)

    Article  Google Scholar 

  3. Bishop, T.E., Favaro, P.: The light field camera: extended depth of field, aliasing, and superresolution. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 34(5), 972–986 (2012)

    Article  Google Scholar 

  4. Bolles, R.C., Baker, H.H., Marimont, D.H.: Epipolar-plane image analysis: an approach to determining structure from motion. Int. J. Comput. Vis. 1(1), 7–55 (1987)

    Article  Google Scholar 

  5. Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: CVPR (2005)

    Google Scholar 

  6. Carbonetto, P.: A programming interface for L-BFGS-B in MATLAB (2014). https://github.com/pcarbo/lbfgsb-matlab. Accessed 15 Apr 2015

  7. Chai, J.X., Tong, X., Chan, S.C., Shum, H.Y.: Plenoptic sampling. In: ACM SIGGRAPH (2000)

    Google Scholar 

  8. Cho, D., Kim, S., Tai, Y.-W.: Consistent matting for light field images. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part IV. LNCS, vol. 8692, pp. 90–104. Springer, Heidelberg (2014)

    Google Scholar 

  9. Dansereau, D.G., Bongiorno, D.L., Pizarro, O., Williams, S.B.: Light field image denoising using a linear 4D frequency-hyperfan all-in-focus filter. In: IS&T/SPIE Electronic Imaging (2013)

    Google Scholar 

  10. Dansereau, D.G., Mahon, I., Pizarro, O., Williams, S.B.: Plenoptic flow: closed-form visual odometry for light field cameras. In: IROS (2011)

    Google Scholar 

  11. Dansereau, D.G., Pizarro, O., Williams, S.B.: Decoding, calibration and rectification for lenselet-based plenoptic cameras. In: CVPR (2013)

    Google Scholar 

  12. Diebold, M., Goldlücke, B.: Epipolar plane image refocusing for improved depth estimation and occlusion handling. In: Annual Workshop on Vision, Modeling and Visualization: VMV (2013)

    Google Scholar 

  13. Favaro, P.: Recovering thin structures via nonlocal-means regularization with application to depth from defocus. In: CVPR (2010)

    Google Scholar 

  14. Ferstl, D., Reinbacher, C., Ranftl, R., Rüther, M., Bischof, H.: Image guided depth upsampling using anisotropic total generalized variation. In: ICCV (2013)

    Google Scholar 

  15. Goldluecke, B., Wanner, S.: The variational structure of disparity and regularization of 4D light fields. In: CVPR (2013)

    Google Scholar 

  16. Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: The lumigraph. In: ACM SIGGRAPH (1996)

    Google Scholar 

  17. Heber, S., Pock, T.: Shape from light field meets robust PCA. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part VI. LNCS, vol. 8694, pp. 751–767. Springer, Heidelberg (2014)

    Google Scholar 

  18. Heber, S., Ranftl, R., Pock, T.: Variational shape from light field. In: Heyden, A., Kahl, F., Olsson, C., Oskarsson, M., Tai, X.-C. (eds.) EMMCVPR 2013. LNCS, vol. 8081, pp. 66–79. Springer, Heidelberg (2013)

    Chapter  Google Scholar 

  19. Isaksen, A., McMillan, L., Gortler, S.J.: Dynamically reparameterized light fields. In: ACM SIGGRAPH. ACM (1996)

    Google Scholar 

  20. Kim, C., Zimmer, H., Pritch, Y., Sorkine-Hornung, A., Gross, M.H.: Scene reconstruction from high spatio-angular resolution light fields. ACM SIGGRAPH (2013)

    Google Scholar 

  21. Levoy, M., Hanrahan, P.: Light field rendering. In: ACM SIGGRAPH. ACM (1996)

    Google Scholar 

  22. Li, N., Ye, J., Ji, Y., Ling, H., Yu, J.: Saliency detection on light field. In: CVPR (2014)

    Google Scholar 

  23. Liang, C.K., Lin, T.H., Wong, B.Y., Liu, C., Chen, H.H.: Programmable aperture photography: multiplexed light field acquisition. ACM SIGGRAPH (2008)

    Google Scholar 

  24. Lin, H., Chen, C., Bing Kang, S., Yu, J.: Depth recovery from light field using focal stack symmetry. In: ICCV (2015)

    Google Scholar 

  25. Ng, R.: Digital light field photography. Ph.D. thesis, stanford university (2006). Ren Ng founded Lytro

    Google Scholar 

  26. Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., Hanrahan, P.: Light field photography with a hand-held plenoptic camera. Computer Science Technical Report CSTR 2(11) (2005)

    Google Scholar 

  27. Park, J., Kim, H., Tai, Y.W., Brown, M.S., Kweon, I.: High quality depth map upsampling for 3D-TOF cameras. In: ICCV (2011)

    Google Scholar 

  28. Perwass, C., Wietzke, L.: The next generation of photography (2010). https://github.com/pcarbo/lbfgsb-matlab. Accessed 15 Apr 2015, Perwass and Wietzke founded Raytrix

  29. Perwass, C., Wietzke, L.: Single lens 3D-camera with extended depth-of-field. In: IS&T/SPIE Electronic Imaging (2012)

    Google Scholar 

  30. Sebe, I.O., Ramanathan, P., Girod, B.: Multi-view geometry estimation for light field compression. In: Annual Workshop on Vision, Modeling and Visualization: VMV (2002)

    Google Scholar 

  31. Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: CVPR (2010)

    Google Scholar 

  32. Tao, M.W., Hadap, S., Malik, J., Ramamoorthi, R.: Depth from combining defocus and correspondence using light-field cameras. In: ICCV (2013)

    Google Scholar 

  33. Tosic, I., Berkner, K.: Light field scale-depth space transform for dense depth estimation. In: CVPR Workshops (2014)

    Google Scholar 

  34. Vaish, V., Wilburn, B., Joshi, N., Levoy, M.: Using plane + parallax for calibrating dense camera arrays. In: CVPR (2004)

    Google Scholar 

  35. Wang, T.C., Efros, A.A., Ramamoorthi, R.: Occlusion-aware depth estimation using light-field cameras. In: ICCV (2015)

    Google Scholar 

  36. Wanner, S., Goldluecke, B.: Globally consistent depth labeling of 4D light fields. In: CVPR (2012)

    Google Scholar 

  37. Wanner, S., Goldluecke, B.: Spatial and angular variational super-resolution of 4D light fields. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 608–621. Springer, Heidelberg (2012)

    Google Scholar 

  38. Wanner, S., Goldluecke, B.: Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 36(3), 606–619 (2014)

    Article  Google Scholar 

  39. Wanner, S., Meister, S., Goldluecke, B.: Datasets and benchmarks for densely sampled 4D light fields. In: Annual Workshop on Vision, Modeling and Visualization: VMV (2013)

    Google Scholar 

  40. Wanner, S., Straehle, C., Goldluecke, B.: Globally consistent multi-label assignment on the ray space of 4D light fields. In: CVPR (2013)

    Google Scholar 

  41. Zhang, Z., Liu, Y., Dai, Q.: Light field from micro-baseline image pair. In: CVPR (2015)

    Google Scholar 

  42. Zhu, C., Byrd, R.H., Lu, P., Nocedal, J.: Algorithm 778: L-BFGS-B: fortran subroutines for large-scale bound-constrained optimization. ACM TOMS 23(4), 550–560 (1997)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Max-Planck-Institute for Intelligent Systems, Spemannstraße 38, 72076, Tübingen, Germany

    Mehdi S. M. Sajjadi, Rolf Köhler, Bernhard Schölkopf & Michael Hirsch

Authors
  1. Mehdi S. M. Sajjadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rolf Köhler
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bernhard Schölkopf
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Hirsch
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Hirsch .

Editor information

Editors and Affiliations

  1. University of Hannover, Hannover, Germany

    Bodo Rosenhahn

  2. Max Planck Institute for Informatics, Saarbrücken, Germany

    Bjoern Andres

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Sajjadi, M.S.M., Köhler, R., Schölkopf, B., Hirsch, M. (2016). Depth Estimation Through a Generative Model of Light Field Synthesis. In: Rosenhahn, B., Andres, B. (eds) Pattern Recognition. GCPR 2016. Lecture Notes in Computer Science(), vol 9796. Springer, Cham. https://doi.org/10.1007/978-3-319-45886-1_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-45886-1_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-45885-4

  • Online ISBN: 978-3-319-45886-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation