Skip to main content
SpringerLink
Log in

What Is Learned in Deep Uncalibrated Photometric Stereo?

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12359))

Included in the following conference series:

Abstract

This paper targets at discovering what a deep uncalibrated photometric stereo network learns to resolve the problem’s inherent ambiguity, and designing an effective network architecture based on the new insight to improve the performance. The recently proposed deep uncalibrated photometric stereo method achieved promising results in estimating directional lightings. However, what specifically inside the network contributes to its success remains a mystery. In this paper, we analyze the features learned by this method and find that they strikingly resemble attached shadows, shadings, and specular highlights, which are known to provide useful clues in resolving the generalized bas-relief (GBR) ambiguity. Based on this insight, we propose a guided calibration network, named GCNet, that explicitly leverages object shape and shading information for improved lighting estimation. Experiments on synthetic and real datasets show that GCNet achieves improved results in lighting estimation for photometric stereo, which echoes the findings of our analysis. We further demonstrate that GCNet can be directly integrated with existing calibrated methods to achieve improved results on surface normal estimation. Our code and model can be found at https://guanyingc.github.io/UPS-GCNet.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Please refer to our supplemental material for more visualizations.

References

  1. Ackermann, J., Fuhrmann, S., Goesele, M.: Geometric point light source calibration. In: Vision, Modeling & Visualization (2013)

    Google Scholar 

  2. Ackermann, J., Goesele, M.: A survey of photometric stereo techniques. Foundations and Trends in Computer Graphics and Vision (2015)

    Google Scholar 

  3. Agarwal, S., et al.: Building Rome in a day. Commun. ACM 54, 105–112 (2011)

    Google Scholar 

  4. Alldrin, N.G., Mallick, S.P., Kriegman, D.J.: Resolving the generalized bas-relief ambiguity by entropy minimization. In: CVPR (2007)

    Google Scholar 

  5. Basri, R., Jacobs, D., Kemelmacher, I.: Photometric stereo with general, unknown lighting. IJCV 5, 105–113 (2007)

    Google Scholar 

  6. Belhumeur, P.N., Kriegman, D.J., Yuille, A.L.: The bas-relief ambiguity. IJCV 35, 33–44 (1999)

    Google Scholar 

  7. Chandraker, M.K., Kahl, F., Kriegman, D.J.: Reflections on the generalized bas-relief ambiguity. In: CVPR (2005)

    Google Scholar 

  8. Chen, G., Han, K., Shi, B., Matsushita, Y., Wong, K.Y.K.: Self-calibrating deep photometric stereo networks. In: CVPR (2019)

    Google Scholar 

  9. Chen, G., Han, K., Shi, B., Matsushita, Y., Wong, K.Y.K.: Deep photometric stereo for non-Lambertian surfaces. TPAMI (2020)

    Google Scholar 

  10. Chen, G., Han, K., Wong, K.-Y.K.: PS-FCN: a flexible learning framework for photometric stereo. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11213, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01240-3_1

    Chapter  Google Scholar 

  11. Cho, D., Matsushita, Y., Tai, Y.W., Kweon, I.S.: Semi-calibrated photometric stereo. TPAMI (2018)

    Google Scholar 

  12. Drbohlav, O., Chantler, M.: Can two specular pixels calibrate photometric stereo? In: ICCV (2005)

    Google Scholar 

  13. Drbohlav, O., Šára, R.: Specularities reduce ambiguity of uncalibrated photometric stereo. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 46–60. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-47967-8_4

    Chapter  Google Scholar 

  14. Einarsson, P., et al.: Relighting human locomotion with flowed reflectance fields. In: EGSR (2006)

    Google Scholar 

  15. Epstein, R., Yuille, A.L., Belhumeur, P.N.: Learning object representations from lighting variations. In: International Workshop on Object Representation in Computer Vision (1996)

    Google Scholar 

  16. Georghiades, A.S.: Incorporating the torrance and sparrow model of reflectance in uncalibrated photometric stereo. In: ICCV (2003)

    Google Scholar 

  17. Goldman, D.B., Curless, B., Hertzmann, A., Seitz, S.M.: Shape and spatially-varying BRDFs from photometric stereo. TPAMI (2010)

    Google Scholar 

  18. Haefner, B., Ye, Z., Gao, M., Wu, T., Quéau, Y., Cremers, D.: Variational uncalibrated photometric stereo under general lighting. In: ICCV (2019)

    Google Scholar 

  19. Hayakawa, H.: Photometric stereo under a light source with arbitrary motion. JOSA A 11, 3079–3089 (1994)

    Google Scholar 

  20. Herbort, S., Wöhler, C.: An introduction to image-based 3D surface reconstruction and a survey of photometric stereo methods. 3D Research (2011)

    Google Scholar 

  21. Hertzmann, A., Seitz, S.M.: Example-based photometric stereo: shape reconstruction with general, varying BRDFs. TPAMI 27, 1254–1264 (2005)

    Google Scholar 

  22. Ikehata, S.: CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11219, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01267-0_1

    Chapter  Google Scholar 

  23. Ikehata, S., Aizawa, K.: Photometric stereo using constrained bivariate regression for general isotropic surfaces. In: CVPR (2014)

    Google Scholar 

  24. Jakob, W.: Mitsuba renderer (2010)

    Google Scholar 

  25. Kingma, D., Ba, J.: ADAM: a method for stochastic optimization. In: ICLR (2015)

    Google Scholar 

  26. Kriegman, D.J., Belhumeur, P.N.: What shadows reveal about object structure. JOSA A 18, 1804–1813 (2001)

    Google Scholar 

  27. Li, J., Robles-Kelly, A., You, S., Matsushita, Y.: Learning to minify photometric stereo. In: CVPR (2019)

    Google Scholar 

  28. Lu, F., Chen, X., Sato, I., Sato, Y.: SymPS: BRDF symmetry guided photometric stereo for shape and light source estimation. TPAMI 40, 221–234 (2018)

    Google Scholar 

  29. Lu, F., Matsushita, Y., Sato, I., Okabe, T., Sato, Y.: Uncalibrated photometric stereo for unknown isotropic reflectances. In: CVPR (2013)

    Google Scholar 

  30. Lu, F., Matsushita, Y., Sato, I., Okabe, T., Sato, Y.: From intensity profile to surface normal: Photometric stereo for unknown light sources and isotropic reflectances. TPAMI 37, 1999–2012 (2015)

    Google Scholar 

  31. Matusik, W., Pfister, H., Brand, M., McMillan, L.: A data-driven reflectance model. In: SIGGRAPH (2003)

    Google Scholar 

  32. Midorikawa, K., Yamasaki, T., Aizawa, K.: Uncalibrated photometric stereo by stepwise optimization using principal components of isotropic BRDFs. In: CVPR (2016)

    Google Scholar 

  33. Mo, Z., Shi, B., Lu, F., Yeung, S.K., Matsushita, Y.: Uncalibrated photometric stereo under natural illumination. In: CVPR (2018)

    Google Scholar 

  34. Okabe, T., Sato, I., Sato, Y.: Attached shadow coding: estimating surface normals from shadows under unknown reflectance and lighting conditions. In: ICCV (2009)

    Google Scholar 

  35. Papadhimitri, T., Favaro, P.: A closed-form, consistent and robust solution to uncalibrated photometric stereo via local diffuse reflectance maxima. IJCV (2014)

    Google Scholar 

  36. Paszke, A., Gross, S., Chintala, S., Chanan, G.: PyTorch: tensors and dynamic neural networks in Python with strong GPU acceleration (2017)

    Google Scholar 

  37. Quéau, Y., Wu, T., Lauze, F., Durou, J.D., Cremers, D.: A non-convex variational approach to photometric stereo under inaccurate lighting. In: CVPR (2017)

    Google Scholar 

  38. Santo, H., Samejima, M., Sugano, Y., Shi, B., Matsushita, Y.: Deep photometric stereo network. In: ICCV Workshops (2017)

    Google Scholar 

  39. Santo, H., Waechter, M., Samejima, M., Sugano, Y., Matsushita, Y.: Light structure from pin motion: simple and accurate point light calibration for physics-based modeling. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_1

    Chapter  Google Scholar 

  40. Sato, I., Okabe, T., Yu, Q., Sato, Y.: Shape reconstruction based on similarity in radiance changes under varying illumination. In: ICCV (2007)

    Google Scholar 

  41. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: CVPR (2006)

    Google Scholar 

  42. Shen, H.L., Cheng, Y.: Calibrating light sources by using a planar mirror. J. Electr. Imaging 20(1), 013002-1–013002-6 (2011)

    Google Scholar 

  43. Shi, B., Inose, K., Matsushita, Y., Tan, P., Yeung, S.K., Ikeuchi, K.: Photometric stereo using internet images. In: 3DV (2014)

    Google Scholar 

  44. Shi, B., Matsushita, Y., Wei, Y., Xu, C., Tan, P.: Self-calibrating photometric stereo. In: CVPR (2010)

    Google Scholar 

  45. Shi, B., Mo, Z., Wu, Z., Duan, D., Yeung, S.K., Tan, P.: A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. TPAMI 41, 271–284 (2019)

    Google Scholar 

  46. Shi, B., Tan, P., Matsushita, Y., Ikeuchi, K.: Bi-polynomial modeling of low-frequency reflectances. TPAMI 36, 1078–1091 (2014)

    Google Scholar 

  47. Silver, W.M.: Determining shape and reflectance using multiple images. Ph.d. thesis, Massachusetts Institute of Technology (1980)

    Google Scholar 

  48. Tan, P., Mallick, S.P., Quan, L., Kriegman, D.J., Zickler, T.: Isotropy, reciprocity and the generalized bas-relief ambiguity. In: CVPR (2007)

    Google Scholar 

  49. Taniai, T., Maehara, T.: Neural inverse rendering for general reflectance photometric stereo. In: ICML (2018)

    Google Scholar 

  50. Woodham, R.J.: Photometric method for determining surface orientation frommultiple images. Opt. Eng. (1980)

    Google Scholar 

  51. Wu, Z., Tan, P.: Calibrating photometric stereo by holistic reflectance symmetry analysis. In: CVPR (2013)

    Google Scholar 

  52. Yuille, A.L., Snow, D., Epstein, R., Belhumeur, P.N.: Determining generative models of objects under varying illumination: shape and albedo from multiple images using SVD and integrability. IJCV 35, 203–222 (1999)

    Google Scholar 

  53. Zheng, Q., Jia, Y., Shi, B., Jiang, X., Duan, L.Y., Kot, A.C.: SPLINE-Net: sparse photometric stereo through lighting interpolation and normal estimation networks. In: ICCV (2019)

    Google Scholar 

  54. Zhou, Z., Tan, P.: Ring-light photometric stereo. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6312, pp. 265–279. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15552-9_20

    Chapter  Google Scholar 

Download references

Acknowledgments

Michael Waechter was supported through a JSPS Postdoctoral Fellowship (JP17F17350). Boxin Shi is supported by the National Natural Science Foundation of China under Grant No. 61872012, National Key R&D Program of China (2019YFF0302902), and Beijing Academy of Artificial Intelligence (BAAI). Kwan-Yee K. Wong is supported by the Research Grant Council of Hong Kong (SAR), China, under the project HKU 17203119. Yasuyuki Matsushita is supported by JSPS KAKENHI Grant Number JP19H01123.

Author information

Authors and Affiliations

  1. The University of Hong Kong, Hong Kong, China

    Guanying Chen & Kwan-Yee K. Wong

  2. Osaka University, Suita, Japan

    Michael Waechter & Yasuyuki Matsushita

  3. Peking University, Beijing, China

    Boxin Shi

  4. Peng Cheng Laboratory, Shenzhen, China

    Boxin Shi

Authors
  1. Guanying Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Waechter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Boxin Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kwan-Yee K. Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yasuyuki Matsushita
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guanying Chen .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 22238 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, G., Waechter, M., Shi, B., Wong, KY.K., Matsushita, Y. (2020). What Is Learned in Deep Uncalibrated Photometric Stereo?. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12359. Springer, Cham. https://doi.org/10.1007/978-3-030-58568-6_44

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58568-6_44

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58567-9

  • Online ISBN: 978-3-030-58568-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation