Skip to main content
SpringerLink
Log in

NormAttention-PSN: A High-frequency Region Enhanced Photometric Stereo Network with Normalized Attention

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Photometric stereo aims to recover the surface normals of a 3D object from various shading cues, establishing the relationship between two-dimensional images and the object geometry. Traditional methods usually adopt simplified reflectance models to approximate the non-Lambertian surface properties, while recently, photometric stereo based on deep learning has been widely used to deal with non-Lambertian surfaces. However, previous studies are limited in dealing with high-frequency surface regions, i.e., regions with rapid shape variations, such as crinkles, edges, etc., resulted in blurry reconstructions. To alleviate this problem, we present a normalized attention-weighted photometric stereo network, namely NormAttention-PSN, to improve surface orientation prediction, especially for those complicated structures. In order to address these challenges, in this paper, we (1) present an attention-weighted loss to produce better surface reconstructions, which applies a higher weight to the detail-preserving gradient loss in high-frequency areas, (2) adopt a double-gate normalization method for non-Lambertian surfaces, to explicitly distinguish whether the high-frequency representation is stimulated by surface structure or spatially varying reflectance, and (3) adopt a parallel high-resolution structure to generate deep features that can maintain the high-resolution details of surface normals. Extensive experiments on public benchmark data sets show that the proposed NormAttention-PSN significantly outperforms traditional calibrated photometric stereo algorithms and state-of-the-art deep learning-based methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. We thank Dr. Zhipeng Mo for helping us test the results on the DiLiGenT test data set.

References

  • Ackermann, J., Goesele, M., et al. (2015). A survey of photometric stereo techniques. Foundations and Trends ® in Computer Graphics and Vision, 9(3), 149–254.

    Article  Google Scholar 

  • Alldrin, N. Zickler, T. & Kriegman, D. (2008). Photometric stereo with non-parametric and spatially-varying reflectance. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1–8). IEEE.

  • Alldrin, N. G., & Kriegman, D. J. (2007). Toward reconstructing surfaces with arbitrary isotropic reflectance: A stratified photometric stereo approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, (pp. 1–8). IEEE.

  • Barsky, S., & Petrou, M. (2003). The 4-source photometric stereo technique for three-dimensional surfaces in the presence of highlights and shadows. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(10), 1239–1252.

    Article  Google Scholar 

  • Basri, R., & Jacobs, D. W. (2003). Lambertian reflectance and linear subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(2), 218–233.

    Article  Google Scholar 

  • Blau, Y. & Michaeli, T. (2018). The perception-distortion tradeoff. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 6228–6237).

  • Chandraker, M. Agarwal, S. & Kriegman, D. (2007). Shadowcuts: Photometric stereo with shadows. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1–8). IEEE.

  • Chandraker, M., Bai, J., & Ramamoorthi, R. (2012). On differential photometric reconstruction for unknown, isotropic brdfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12), 2941–2955.

    Article  Google Scholar 

  • Chen, G. Han, K. & Wong, K. Y. K. (2018). Ps-fcn: A flexible learning framework for photometric stereo. In Proceedings of the European conference on computer vision (pp. 3–18).

  • Chen, G. Han, K. Shi, B. Matsushita, Y. & Wong, K. Y. K. (2019) Self-calibrating deep photometric stereo networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8739–8747).

  • Chen, G., Han, K., Shi, B., Matsushita, Y., & Wong, K. Y. K. (2020). Deep photometric stereo for non-lambertian surfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1), 129–142.

    Article  Google Scholar 

  • Chen, G. Waechter, M. Shi, B. Wong, K. Y. K. & Matsushita, Y. (2020b). What is learned in deep uncalibrated photometric stereo? In Proceedings of the European conference on computer vision (pp. 745–762). Springer.

  • Cheng, WC. (2006). Neural-network-based photometric stereo for 3d surface reconstruction. In The 2006 IEEE International joint conference on neural network proceedings (pp. 404–410) IEEE.

  • Chung, H. S. & Jia, J. (2008). Efficient photometric stereo on glossy surfaces with wide specular lobes. In Proceedings of the IEEE conference on computer vision and pattern recognition. (pp. 1–8). IEEE.

  • Einarsson, P. Chabert, C. F. Jones, A. Ma, W. C. Lamond, B. Hawkins, T. Bolas, M. Sylwan, S. & Debevec, P. (2006). Relighting human locomotion with flowed reflectance fields. In proceedings of the eurographics conference on rendering techniques (pp. 183–194).

  • Georghiades A. S. (2003). Incorporating the torrance and sparrow model of reflectance in uncalibrated photometric stereo. In: Proceedings of the IEEE international conference on computer vision (p. 816).

  • Goldman, D. B., Curless, B., Hertzmann, A., & Seitz, S. M. (2010). Shape and spatially-varying brdfs from photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(6), 1060–1071.

    Article  Google Scholar 

  • Hartmann, W. Galliani, S. Havlena, M. Van Gool, L. & Schindler, K. (2017). Learned multi-patch similarity. In Proceedings of the IEEE international conference on computer vision. (pp. 1586–1594).

  • He, K. Zhang, X. Ren, S. & Sun, J. (2016) Deep residual learning for image recognition. In Proceedings of the IEEE international conference on computer vision (pp. 770–778).

  • Herbort, S. & Wöhler, C. (2011). An introduction to image-based 3d surface reconstruction and a survey of photometric stereo methods. 3D Research, 2(3):4 .

  • Higo, T. Matsushita, Y. & Ikeuchi, K .(2010). Consensus photometric stereo. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1157–1164) IEEE

  • Holroyd, M., Lawrence, J., Humphreys, G., & Zickler, T. (2008). A photometric approach for estimating normals and tangents. ACM Transactions on Graphics, 27(5), 1–9.

    Article  Google Scholar 

  • Honzátko, D. Türetken, E. Fua, P. Dunbar, L. A. (2021). Leveraging spatial and photometric context for calibrated non-lambertian photometric stereo. In Proceedings of the international conference on 3D vision (pp. 394–402).

  • Hui, Z., & Sankaranarayanan, A. C. (2016). Shape and spatially-varying reflectance estimation from virtual exemplars. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(10), 2060–2073.

    Article  Google Scholar 

  • Ikehata, S. (2018). Cnn-ps: Cnn-based photometric stereo for general non-convex surfaces. In Proceedings of the European conference on computer vision (pp. 3–18).

  • Ikehata, & S. Aizawa, K. (2014). Photometric stereo using constrained bivariate regression for general isotropic surfaces. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2179–2186).

  • Ikehata, S. Wipf, D. Matsushita, Y. & Aizawa, K. (2012). Robust photometric stereo using sparse regression. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 318–325), IEEE

  • Isola, P. Zhu, J. Y. Zhou, T. & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1125–1134).

  • Iwahori, Y. Woodham, R. J. Tanaka, H., & Ishii, N. (1993). Neural network to reconstruct specular surface shape from its three shading images. In Proceedings of international conference on neural networks 2, (pp.1181–1184) IEEE.

  • Jian, M., Dong, J., Gong, M., Yu, H., Nie, L., Yin, Y., & Lam, K. M. (2019). Learning the traditional art of chinese calligraphy via three-dimensional reconstruction and assessment. IEEE Transactions on Multimedia, 22(4), 970–979.

    Article  Google Scholar 

  • Johnson, M. K. & Adelson, E. H. (2011). Shape estimation in natural illumination. In Proceedings of the IEEE international conference on computer vision, (pp. 2553–2560). IEEE.

  • Ju, Y. Jian, M. Dong, J. & Lam, K. M. (2020a). Learning photometric stereo via manifold-based mapping. In: Proceedings of the IEEE international conference on visual communications and image processing (VCIP), (pp. 411–414). IEEE.

  • Ju, Y. Lam, K. M. Chen, Y. Qi, L. & Dong, J. (2020b). Pay attention to devils: A photometric stereo network for better details. In: Proceedings of the twenty-ninth international joint conference on artificial intelligence (pp. 694–700).

  • Ju, Y., Dong, J., & Chen, S. (2021). Recovering surface normal and arbitrary images: A dual regression network for photometric stereo. IEEE Transactions on Image Processing, 30, 3676–3690.

    Article  Google Scholar 

  • Ju, Y., Peng, Y., Jian, M., Gao, F., & Dong, J. (2022). Learning conditional photometric stereo with high-resolution features. Computational Visual Media, 8(1), 105–118.

  • Li, J. Robles-Kelly, A. You, S. & Matsushita, Y. (2019). Learning to minify photometric stereo. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7568–7576).

  • Logothetis, F. Budvytis, I. Mecca, R. & Cipolla, R. (2021). Px-net: Simple and efficient pixel-wise training of photometric stereo networks. In Proceedings of the IEEE international conference on computer vision (pp. 12757–12766).

  • Matusik, W., Pfister, H., Brand, M., & McMillan, L. (2003). A data-driven reflectance model. ACM Transactions on Graphics, 22(3), 759–769.

    Article  Google Scholar 

  • McAuley, S. Hill, S. Hoffman, N. Gotanda, Y. Smits, B. Burley, B. & Martinez, A. (2012) Practical physically-based shading in film and game production. In ACM SIGGRAPH 2012 Courses (pp. 1–7).

  • Miyazaki, D., Hara, K., & Ikeuchi, K. (2010). Median photometric stereo as applied to the segonko tumulus and museum objects. International Journal of Computer Vision, 86(2–3), 229–242.

    Article  Google Scholar 

  • Mukaigawa, Y., Ishii, Y., & Shakunaga, T. (2007). Analysis of photometric factors based on photometric linearization. JOSA A, 24(10), 3326–3334.

    Article  Google Scholar 

  • Nayar, S. K., Ikeuchi, K., & Kanade, T. (1991). Shape from interreflections. International Journal of Computer Vision, 6(3), 173–195.

    Article  Google Scholar 

  • Santo, H. Samejima, M. Sugano, Y. Shi, B. Matsushita, Y. (2017). Deep photometric stereo network. In Proceedings of the IEEE international conference on computer vision workshops (pp. 501–509)

  • Santo, H. Samejima, M. Sugano, Y. Shi, B. Matsushita, Y. (2020). Deep photometric stereo networks for determining surface normal and reflectances. IEEE Transactions on Pattern Analysis and Machine Intelligence p early access.

  • Shi, B. Tan, P. Matsushita, Y. Ikeuchi, K. (2012). Elevation angle from reflectance monotonicity: Photometric stereo for general isotropic reflectances. In: Proceedings of the european conference on computer vision (pp. 455–468). Springer.

  • Shi, B., Tan, P., Matsushita, Y., & Ikeuchi, K. (2014). Bi-polynomial modeling of low-frequency reflectances. IEEE transactions on pattern analysis and machine intelligence, 36(6), 1078–1091.

    Article  Google Scholar 

  • Shi, B., Mo, Z., Wu, Z., Duan, D., Yeung, S., & Tan, P. (2019). A benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 271–284.

    Article  Google Scholar 

  • Simchony, T., Chellappa, R., & Shao, M. (1990). Direct analytical methods for solving poisson equations in computer vision problems. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(5), 435–446.

    Article  Google Scholar 

  • Solomon, F., & Ikeuchi, K. (1996). Extracting the shape and roughness of specular lobe objects using four light photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(4), 449–454.

    Article  Google Scholar 

  • Sun, K. Xiao, B. Liu, D. & Wang, J. (2019). Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5693–5703).

  • Taniai, T. & Maehara, T. (2018). Neural inverse rendering for general reflectance photometric stereo. In Proceedings of the international conference on machine learning (pp. 4857–4866).

  • Tozza, S., Mecca, R., Duocastella, M., & Del Bue, A. (2016). Direct differential photometric stereo shape recovery of diffuse and specular surfaces. Journal of Mathematical Imaging and Vision, 56(1), 57–76.

  • Ummenhofer, B. Zhou, H. Uhrig, J. Mayer, N. Ilg, E. Dosovitskiy, A. Brox, T. (2017) Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5038–5047)

  • Verbiest, F. & Van Gool, L. (2008). Photometric stereo with coherent outlier handling and confidence estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1–8).

  • Wang, X., Jian, Z., & Ren, M. (2020). Non-lambertian photometric stereo network based on inverse reflectance model with collocated light. IEEE Transactions on Image Processing, 29, 6032–6042.

    Article  Google Scholar 

  • Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE transactions on image processing, 13(4), 600–612.

    Article  Google Scholar 

  • Wiles, O. & Zisserman, A. (2017). Silnet: Single-and multi-view reconstruction by learning from silhouettes. In Proceedings of the British machine vision conference.

  • Woodham, R. J. (1980). Photometric method for determining surface orientation from multiple images. Optical Engineering, 19(1), 139–144.

    Article  Google Scholar 

  • Wu, L. Ganesh, A. Shi, B. Matsushita, Y. Wang, Y. & Ma, Y.(2010) Robust photometric stereo via low-rank matrix completion and recovery. In Proceedings of the asian conference on computer vision (pp. 703–717). Springer.

  • Wu, S. Rupprecht, C. & Vedaldi, A. (2020). Unsupervised learning of probably symmetric deformable 3d objects from images in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1–10).

  • Yao, Z. Li, K. Fu, Y. Hu, H. & Shi, B. (2020). Gps-net: Graph-based photometric stereo network. In Proceedings of the advances in neural information processing systems

  • Yeung, S. K., Wu, T. P., Tang, C. K., Chan, T. F., & Osher, S. J. (2015). Normal estimation of a transparent object using a video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(4), 890–897.

    Article  Google Scholar 

  • Yu, C. Seo, Y. Lee, & S. W. (2010). Photometric stereo from maximum feasible lambertian reflections. In: Proceedings of the European conference on computer vision (pp. 115–126) Springer.

  • Zheng, Q. Jia, Y. Shi, B. Jiang, X. Duan, L. Y. & Kot, A.C. (2019) Spline-net: Sparse photometric stereo through lighting interpolation and normal estimation networks. In Proceedings of the IEEE international conference on computer vision (pp. 8549–8558).

  • Zheng, Q., Shi, B., & Pan, G. (2020). Summary study of data-driven photometric stereo methods. Virtual Reality & Intelligent Hardware, 2(3), 213–221.

    Article  Google Scholar 

Download references

Acknowledgements

The work was supported by the Key-Area Research and Development Program of Guangdong Province (2020B090928001), the Project of Strategic Importance Fund from The Hong Kong Polytechnic University (No. ZE1X), the National Key R &D Program of China under Grant (2018AAA0100602), the National Key Scientific Instrument and Equipment Development Projects of China (41927805), and the National Natural Science Foundation of China (61872012, 62136001, 61976123, 61601427), the Key Development Program for Basic Research of Shandong Province (ZR2020ZD44), and the Taishan Young Scholars Program of Shandong Province.

Author information

Authors and Affiliations

  1. Department of Computer Science and Technology, Ocean University of China, Qingdao, China

    Yakun Ju, Lin Qi & Junyu Dong

  2. Department of Electronic and Information Engineering, The Hong Kong Polytechnic University, Hung Hom, Hong Kong, China

    Yakun Ju & Kin-Man Lam

  3. National Engineering Research Center of Visual Technology, School of Computer Science, and Institute for Artificial Intelligence, Peking University, Beijing, China

    Boxin Shi

  4. Peng Cheng Laboratory, Shenzhen, China

    Boxin Shi

  5. School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan, China

    Muwei Jian

  6. School of Information Science and Engineering, Linyi University, Linyi, China

    Muwei Jian

  7. Centre for Advances in Reliability and Safety, Tai Po, Hong Kong, China

    Kin-Man Lam

Authors
  1. Yakun Ju
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Boxin Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Muwei Jian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lin Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Junyu Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Kin-Man Lam
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Junyu Dong or Kin-Man Lam.

Additional information

Communicated by Kwan-Yee Kenneth Wong.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ju, Y., Shi, B., Jian, M. et al. NormAttention-PSN: A High-frequency Region Enhanced Photometric Stereo Network with Normalized Attention. Int J Comput Vis 130, 3014–3034 (2022). https://doi.org/10.1007/s11263-022-01684-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-022-01684-8

Keywords

Search

Navigation