Michael Zollhöfer
Angela Dai
Skip Abstract Section
Skip Supplemental Material SectionAbstract
We present a novel method to obtain fine-scale detail in 3D reconstructions generated with low-budget RGB-D cameras or other commodity scanning devices. As the depth data of these sensors is noisy, truncated signed distance fields are typically used to regularize out the noise, which unfortunately leads to over-smoothed results. In our approach, we leverage RGB data to refine these reconstructions through shading cues, as color input is typically of much higher resolution than the depth data. As a result, we obtain reconstructions with high geometric detail, far beyond the depth resolution of the camera itself. Our core contribution is shading-based refinement directly on the implicit surface representation, which is generated from globally-aligned RGB-D images. We formulate the inverse shading problem on the volumetric distance field, and present a novel objective function which jointly optimizes for fine-scale surface geometry and spatially-varying surface reflectance. In order to enable the efficient reconstruction of sub-millimeter detail, we store and process our surface using a sparse voxel hashing scheme which we augment by introducing a grid hierarchy. A tailored GPU-based Gauss-Newton solver enables us to refine large shape models to previously unseen resolution within only a few seconds.
Supplemental Material
Available for Download
zip
a96-zollhofer.zip (183.5 MB)
Supplemental files
- Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S. M., and Szeliski, R. 2011. Building rome in a day. Communications of the ACM 54, 10, 105--112. Google Scholar
Digital Library
- AgiSoft, L. 2014. Agisoft photoscan. Professional Edition.Google Scholar
- Beeler, T., Bradley, D., Zimmer, H., and Gross, M. 2012. Improved reconstruction of deforming surfaces by cancelling ambient occlusion. In Proc. ECCV, 30--43. Google ScholarDigital Library
- Bylow, E., Sturm, J., Kerl, C., Kahl, F., and Cremers, D. 2013. Real-time camera tracking and 3d reconstruction using signed distance functions. In Robotics: Science and Systems (RSS) Conference 2013, vol. 9.Google Scholar
- Carr, J. C., Beatson, R. K., Cherrie, J. B., Mitchell, T. J., Fright, W. R., McCallum, B. C., and Evans, T. R. 2001. Reconstruction and representation of 3d objects with radial basis functions. In Proc. SIGGRAPH, ACM, 67--76. Google ScholarDigital Library
- Chan, D., Buisman, H., Theobalt, C., Thrun, S., et al. 2008. A noise-aware filter for real-time depth upsampling. In Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications-M2SFA2 2008.Google Scholar
- Chen, Q., and Koltun, V. 2013. A simple model for intrinsic image decomposition with depth cues. In The IEEE International Conference on Computer Vision (ICCV). Google ScholarDigital Library
- Chen, J., Bautembach, D., and Izadi, S. 2013. Scalable real-time volumetric surface reconstruction. ACM TOG 32, 4, 113. Google ScholarDigital Library
- Cui, Y., Schuon, S., Chan, D., Thrun, S., and Theobalt, C. 2010. 3d shape scanning with a time-of-flight camera. In Proc. CVPR, 1173--1180.Google Scholar
- Curless, B., and Levoy, M. 1996. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM, 303--312. Google ScholarDigital Library
- Debevec, P. 2012. The light stages and their applications to photoreal digital actors. In SIGGRAPH Asia Technical Briefs.Google Scholar
- Diebel, J., and Thrun, S. 2006. An application of Markov Random Fields to range sensing. 291--298.Google Scholar
- Dolson, J., Baek, J., Plagemann, C., and Thrun, S. 2010. Upsampling range data in dynamic environments. In Proc. CVPR, IEEE, 1141--1148.Google Scholar
- Fuhrmann, S., and Goesele, M. 2014. Floating scale surface reconstruction. ACM TOG 33, 4, 46. Google ScholarDigital Library
- Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., and Debevec, P. 2011. Multiview face capture using polarized spherical gradient illumination. ACM TOG 30. Google ScholarDigital Library
- Goldluecke, B., Aubry, M., Kolev, K., and Cremers, D. 2014. A super-resolution framework for high-accuracy multiview reconstruction. ijcv 106, 2 (jan), 172--191. Google ScholarDigital Library
- Haber, T., Fuchs, C., Bekaer, P., Seidel, H.-P., Goesele, M., and Lensch, H. 2009. Relighting objects from image collections. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, 627--634.Google Scholar
- Han, Y., Lee, J.-Y., and Kweon, I. S. 2013. High quality shape from a single rgb-d image under uncalibrated natural illumination. In Proc. ICCV. Google ScholarDigital Library
- Hasinoff, S., Levin, A., Goode, P., and Freeman, W. 2011. Diffuse reflectance imaging with astronomical applications. In Proc. ICCV, 185--192. Google ScholarDigital Library
- Henry, P., Krainin, M., Herbst, E., Ren, X., and Fox, D. 2012. RGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments. Int. J. Robotics Research 31 (apr), 647--663. Google ScholarDigital Library
- Hernández, C., Vogiatzis, G., and Cipolla, R. 2008. Multiview photometric stereo. IEEE PAMI 30, 3, 548--554. Google ScholarDigital Library
- Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., and Stuetzle, W. 1992. Surface reconstruction from unorganized points. In Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH '92, 71--78. Google ScholarDigital Library
- Horn, B. K. 1975. Obtaining shape from shading information. The psychology of computer vision, 115--155.Google Scholar
- Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A., et al. 2011. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In Proc. UIST, ACM, 559--568. Google ScholarDigital Library
- Kazhdan, M., Bolitho, M., and Hoppe, H. 2006. Poisson surface reconstruction. In Proc. SGP. Google ScholarDigital Library
- Kehl, W., Navab, N., and Ilic, S. 2014. Coloured signed distance fields for full 3d object reconstruction. In Proc. BMVC.Google Scholar
- Keller, M., Lefloch, D., Lambers, M., Izadi, S., Weyrich, T., and Kolb, A. 2013. Real-time 3d reconstruction in dynamic scenes using point-based fusion. In Proc. 3DV, 1--8. Google ScholarDigital Library
- Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. 2007. Joint bilateral upsampling. ACM TOG 26, 3. Google ScholarDigital Library
- Lee, K. J., Zhao, Q., Tong, X., Gong, M., Izadi, S., Lee, S. U., Tan, P., and Lin, S. 2012. Estimation of intrinsic image sequences from image+depth video. In Proc. ECCV, 327--340. Google ScholarDigital Library
- Levoy, M., Pulli, K., Curless, B., Rusinkiewicz, S., Koller, D., Pereira, L., Ginzton, M., Anderson, S., Davis, J., Ginsberg, J., Shade, J., and Fulk, D. 2000. The digital michelangelo project: 3d scanning of large statues. In Proc. SIGGRAPH, 131--144. Google ScholarDigital Library
- Lindner, M., Kolb, A., and Hartmann, K. 2007. Data-fusion of PMD-based distance-information and high-resolution RGB-images. In Proc. ISSCS, 121--124.Google Scholar
- Lowe, D. G. 2004. Distinctive image features from scale-invariant keypoints. IJCV 60, 2, 91--110. Google ScholarDigital Library
- Mulligan, J., and Brolly, X. 2004. Surface determination by photometric ranging. In Proc. CVPR Workshops. Google ScholarDigital Library
- Nair, R., Ruhl, K., Lenzen, F., Meister, S., Schäfer, H., Garbe, C. S., Eisemann, M., Magnor, M., and Kondermann, D. 2013. A survey on time-of-flight stereo fusion. In Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications. Springer, 105--127.Google Scholar
- Nehab, D., Rusinkiewicz, S., Davis, J., and Ramamoorthi, R. 2005. Efficiently combining positions and normals for precise 3D geometry. Proc. SIGGRAPH 24, 3. Google ScholarDigital Library
- Newcombe, R. A., and Davison, A. J. 2010. Live dense reconstruction with a single moving camera. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, IEEE, 1498--1505.Google Scholar
- Newcombe, R. A., Davison, A. J., Izadi, S., Kohli, P., Hilliges, O., Shotton, J., Molyneaux, D., Hodges, S., Kim, D., and Fitzgibbon, A. 2011. Kinectfusion: Real-time dense surface mapping and tracking. In Proc. ISMAR, IEEE, 127--136. Google ScholarDigital Library
- Niessner, M., Zollhöfer, M., Izadi, S., and Stamminger, M. 2013. Real-time 3d reconstruction at scale using voxel hashing. ACM TOG 32, 6, 169. Google ScholarDigital Library
- Park, J., Kim, H., Tai, Y.-W., Brown, M. S., and Kweon, I.-S. 2011. High quality depth map upsampling for 3d-tof cameras. In Proc. ICCV, IEEE, 1623--1630. Google ScholarDigital Library
- Pradeep, V., Rhemann, C., Izadi, S., Zach, C., Bleyer, M., and Bathiche, S. 2013. Monofusion: Real-time 3d reconstruction of small scenes with a single web camera. In Proc. ISMAR.Google Scholar
- Prados, E., and Faugeras, O. 2005. Shape from shading: a well-posed problem? In Proc. CVPR. Google ScholarDigital Library
- Ramamoorthi, R., and Hanrahan, P. 2001. A signal-processing framework for inverse rendering. In Proc. SIGGRAPH, ACM, 117--128. Google ScholarDigital Library
- Richardt, C., Stoll, C., Dodgson, N. A., Seidel, H.-P., and Theobalt, C. 2012. Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos. CGF (Proceedings of Eurographics) 31, 2 (May). Google ScholarDigital Library
- Roth, H., and Vona, M. 2012. Moving volume kinectfusion. In BMVC, 1--11.Google Scholar
- Rusinkiewicz, S., Hall-Holt, O., and Levoy, M. 2002. Real-time 3D model acquisition. ACM TOG 21, 3, 438--446. Google ScholarDigital Library
- Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nešić, N., Wang, X., and Westling, P. 2014. High-resolution stereo datasets with subpixel-accurate ground truth. In Pattern Recognition. Springer, 31--42.Google Scholar
- Schuon, S., Theobalt, C., Davis, J., and Thrun, S. 2009. Lidarboost: Depth superresolution for tof 3d shape scanning. In Proc. CVPR, IEEE, 343--350.Google Scholar
- Seitz, S., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In Proc. CVPR, vol. 1, 519--528. Google ScholarDigital Library
- Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: exploring photo collections in 3d. ACM TOG 25, 3, 835--846. Google ScholarDigital Library
- Triggs, B., Mclauchlan, P. F., Hartley, R. I., and Fitzgibbon, A. W. 2000. Bundle adjustment--a modern synthesis. In Vision algorithms: theory and practice. Springer, 298--372. Google ScholarDigital Library
- Weber, D., Bender, J., Schnoes, M., Stork, A., and Fellner, D. 2013. Efficient gpu data structures and methods to solve sparse linear systems in dynamics applications. In CGF, vol. 32, Wiley Online Library, 16--26.Google Scholar
- Weise, T., Wismer, T., Leibe, B., and Van Gool, L. 2009. In-hand scanning with online loop closure. In ICCV Workshops, 1630--1637.Google ScholarCross Ref
- Whelan, T., Kaess, M., Fallon, M., Johannsson, H., Leonard, J., and McDonald, J. 2012. Kintinuous: Spatially extended kinectfusion.Google Scholar
- Wu, C., Varanasi, K., Liu, Y., Seidel, H.-P., and Theobalt, C. 2011. Shading-based dynamic shape refinement from multi-view video under general illumination. In Proc. ICCV, IEEE, 1108--1115. Google ScholarDigital Library
- Wu, C., Stoll, C., Valgaerts, L., and Theobalt, C. 2013. On-set performance capture of multiple actors with a stereo camera. ACM TOG (Proc. SIGGRAPh Asia) 32, 6, 161. Google ScholarDigital Library
- Wu, C., Zollhöfer, M., Niessner, M., Stamminger, M., Izadi, S., and Theobalt, C. 2014. Real-time shading-based refinement for consumer depth cameras. ACM TOG (Proc. SIGGRAPH Asia) 33, 6, 200:1--200:10. Google ScholarDigital Library
- Yu, L.-F., Yeung, S.-K., Tai, Y.-W., and Lin, S. 2013. Shading-based shape refinement of rgb-d images. In Proc. CVPR. Google ScholarDigital Library
- Zhang, Z., Tsa, P.-S., Cryer, J. E., and Shah, M. 1999. Shape from shading: A survey. IEEE PAMI 21, 8, 690--706. Google ScholarDigital Library
- Zhou, Q.-Y., and Koltun, V. 2014. Color map optimization for 3d reconstruction with consumer depth cameras. ACM Transactions on Graphics (TOG) 33, 4, 155. Google ScholarDigital Library
- Zollhöfer, M., Niessner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., et al. 2014. Real-time non-rigid reconstruction using an rgb-d camera. ACM TOG (Proc. SIGGRAPH) 4. Google ScholarDigital Library
Index Terms
- Shading-based refinement on volumetric signed distance functions
Recommendations
Real-time shading-based refinement for consumer depth cameras
We present the first real-time method for refinement of depth data using shape-from-shading in general uncontrolled scenes. Per frame, our real-time algorithm takes raw noisy depth data and an aligned RGB image as input, and approximates the time-...
Truncated Signed Distance Function: Experiments on Voxel Size
Image Analysis and RecognitionAbstractReal-time 3D reconstruction is a hot topic in current research. Several popular approaches are based on the truncated signed distance function (TSDF), a volumetric scene representation that allows for integration of multiple depth images taken ...
Volumetric reconstruction and interactive rendering of trees from photographs
SIGGRAPH '04: ACM SIGGRAPH 2004 PapersReconstructing and rendering trees is a challenging problem due to the geometric complexity involved, and the inherent difficulties of capture. In this paper we propose a volumetric approach to capture and render trees with relatively sparse foliage. ...
Comments