Abstract
Time-of-flight (ToF) imaging has become a widespread technique for depth estimation, allowing affordable off-the-shelf cameras to provide depth maps in real time. However, multipath interference (MPI) resulting from indirect illumination significantly degrades the captured depth. Most previous works have tried to solve this problem by means of complex hardware modifications or costly computations. In this work, we avoid these approaches and propose a new technique to correct errors in depth caused by MPI, which requires no camera modifications and takes just 10 milliseconds per frame. Our observations about the nature of MPI suggest that most of its information is available in image space; this allows us to formulate the depth imaging process as a spatially-varying convolution and use a convolutional neural network to correct MPI errors. Since the input and output data present similar structure, we base our network on an autoencoder, which we train in two stages. First, we use the encoder (convolution filters) to learn a suitable basis to represent MPI-corrupted depth images; then, we train the decoder (deconvolution filters) to correct depth from synthetic scenes, generated by using a physically-based, time-resolved renderer. This approach allows us to tackle a key problem in ToF, the lack of ground-truth data, by using a large-scale captured training set with MPI-corrupted depth to train the encoder, and a smaller synthetic training set with ground truth depth to train the decoder stage of the network. We demonstrate and validate our method on both synthetic and real complex scenarios, using an off-the-shelf ToF camera, and with only the captured, incorrect depth as input.
- Ayush Bhandari, Achuta Kadambi, Refael Whyte, Christopher Barsi, Micha Feigin, Adrian Dorrington, and Ramesh Raskar. 2014. Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization. Opt. Lett. 39, 6 (2014), 1705--1708.Google ScholarCross Ref
- Jean-Yves Bouguet. 2004. Camera calibration toolbox for Matlab. (2004).Google Scholar
- Inchang Choi, Daniel S. Jeon, Giljoo Nam, Diego Gutierrez, and Min H. Kim. 2017. High-Quality Hyperspectral Reconstruction Using a Spectral Prior. ACM Transactions on Graphics (SIGGRAPH Asia 2017) 36, 6 (2017). Google ScholarDigital Library
- A. A. Dorrington, J. P. Godbaz, M. J. Cree, A. D. Payne, and L. V. Streeter. 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras. In Proceedings of SPIE, Vol. 7864. 786404-786404-10.Google Scholar
- B. Du, W. Xiong, J. Wu, L. Zhang, L. Zhang, and D. Tao. 2016. Stacked Convolutional Denoising Auto-Encoders for Feature Representation. IEEE Trans. Cybernetics 99 (2016), 1--11.Google Scholar
- David Eigen and Rob Fergus. 2015. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In Proceedings of ICCV. Google ScholarDigital Library
- David Eigen, Christian Puhrsch, and Rob Fergus. 2014. Depth map prediction from a single image using a multi-scale deep network. In Proceedings of NIPS. Google ScholarDigital Library
- Dominik Maria Endres and Johannes E Schindelin. 2003. A new metric for probability distributions. IEEE Transactions on Information theory 49, 7 (2003), 1858--1860. Google ScholarDigital Library
- M. Feigin, A. Bhandari, S. Izadi, C. Rhemann, M. Schmidt, and R. Raskar. 2016. Resolving Multipath Interference in Kinect: An Inverse Problem Approach. IEEE Sensors Journal 16, 10 (May 2016), 3419--3427.Google ScholarCross Ref
- Daniel Freedman, Yoni Smolin, Eyal Krupka, Ido Leichter, and Mirko Schmidt. 2014. SRA: Fast removal of general multipath for ToF sensors. In Proceedings of ECCV. Springer, 234--249.Google Scholar
- Stefan Fuchs. 2010. Multipath Interference Compensation in Time-of-Flight Camera Images. In Proceedings of the International Conference on Pattern Recognition. 3583--3586. Google ScholarDigital Library
- Stefan Fuchs, Michael Suppa, and Olaf Hellwich. 2013. Compensation for Multipath in ToF Camera Measurements Supported by Photometric Calibration and Environment Integration. In Proceedings of the International Conference on Computer Vision Systems (ICVS'13). Springer-Verlag, Berlin, Heidelberg, 31--41. Google ScholarDigital Library
- John P. Godbaz, Michael J. Cree, and Adrian A. Dorrington. 2012. Closed-form inverses for the mixed pixel/multipath interference problem in AMCW lidar. In Proceedings of SPIE, Vol. 8296. 829618-829618-15.Google Scholar
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org. Google ScholarDigital Library
- Mohit Gupta, Shree K. Nayar, Matthias B. Hullin, and Jaime Martin. 2015. Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging. ACM Trans. Graph. 34, 5, Article 156 (Nov. 2015), 18 pages. Google ScholarDigital Library
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR. 770--778.Google ScholarCross Ref
- Felix Heide, Matthias B. Hullin, James Gregson, and Wolfgang Heidrich. 2013. Low-budget Transient Imaging Using Photonic Mixer Devices. ACM Trans. Graph. 32, 4, Article 45 (July 2013), 10 pages. Google ScholarDigital Library
- Jinggang Huang, Ann B Lee, and David Mumford. 2000. Statistics of range images. In Proceedings of CVPR, Vol. 1. IEEE, 324--331.Google Scholar
- Jinggang Huang and David Mumford. 1999. Statistics of natural images and models. In Proceedings of CVPR, Vol. 1. IEEE, 541--547.Google Scholar
- Adrian Jarabo, Julio Marco, Adolfo Muhoz, Raul Buisan, Wojciech Jarosz, and Diego Gutierrez. 2014. A Framework for Transient Rendering. ACM Trans. Graph. 33, 6, Article 177 (2014). Google ScholarDigital Library
- Adrian Jarabo, Belen Masia, Julio Marco, and Diego Gutierrez. 2017. Recent Advances in Transient Imaging: A Computer Graphics and Vision Perspective. Visual Informatics 1, 1 (2017).Google Scholar
- David Jiménez, Daniel Pizarro, Manuel Mazo, and Sira Palazuelos. 2014. Modeling and correction of multipath interference in time-of-flight cameras. Image and Vision Computing 32, 1 (2014), 1--13. Google ScholarDigital Library
- Achuta Kadambi, Refael Whyte, Ayush Bhandari, Lee Streeter, Christopher Barsi, Adrian Dorrington, and Ramesh Raskar. 2013. Coded Time of Flight Cameras: Sparse Deconvolution to Address Multipath Interference and Recover Time Profiles. ACM Trans. Graph. 32, 6, Article 167 (Nov. 2013), 10 pages. Google ScholarDigital Library
- Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based View Synthesis for Light Field Cameras. ACM Trans. Graph. 35, 6, Article 193 (Nov. 2016), 10 pages. Google ScholarDigital Library
- S Karayev, Y Jia, J Barron, M Fritz, K Saenko, and T Darrell. 2011. A category-level 3-D object dataset: putting the Kinect to work. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 1167--1174.Google Scholar
- Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv.1412.6980 (2014).Google Scholar
- Ahmed Kirmani, Arrigo Benedetti, and Philip A Chou. 2013. Spumic: Simultaneous phase unwrapping and multipath interference cancellation in time-of-flight cameras using spectral methods. In Proceedings of IEEE International Conference on Multimedia and Expo. 1--6.Google ScholarCross Ref
- Nick Knighton and Bruce Bugbee. 2005. A mixture of barium sulfate and white paint is a low-cost substitute reflectance standard for Spectralon®. (2005).Google Scholar
- Ann B Lee, JG Huang, and DB Mumford. 2000. Random collage model for natural images. Int. J. of Computer Vision (2000).Google Scholar
- Bo Li, Chunhua Shen, Yuchao Dai, A. van den Hengel, and Mingyi He. 2015. Depth and surface normal estimation from monocular images using regression on deep features and hierarchical CRFs. In Proceedings of CVPR.Google Scholar
- Jianhua Lin. 1991. Divergence measures based on the Shannon entropy. IEEE Transactions on Information theory 37, 1 (1991), 145--151. Google ScholarDigital Library
- Marvin Lindner, Ingo Schiller, Andreas Kolb, and Reinhard Koch. 2010. Time-of-flight sensor calibration for accurate range sensing. Computer Vision and Image Understanding 114, 12 (2010), 1318--1328. Google ScholarDigital Library
- Fayao Liu, Chunhua Shen, and Guosheng Lin. 2015. Deep Convolutional Neural Fields for Depth Estimation from a Single Image. In Proceedings of CVPR.Google ScholarCross Ref
- Jonathan Masci, Ueli Meier, Dan Cireşan, and Jürgen Schmidhuber. 2011. Stacked convolutional auto-encoders for hierarchical feature extraction. In Proceedings of Int. Conf. Artificial Neural Networks. 52--59. Google ScholarDigital Library
- Nikhil Naik, Achuta Kadambi, Christoph Rhemann, Shahram Izadi, Ramesh Raskar, and Sing Bing Kang. 2015. A light transport model for mitigating multipath interference in time-of-flight sensors. In Proceedings of CVPR. 73--81.Google ScholarCross Ref
- Ren Ng, Ravi Ramamoorthi, and Pat Hanrahan. 2003. All-frequency shadows using non-linear wavelet lighting approximation. ACM Trans. Graph. 22, 3 (2003). Google ScholarDigital Library
- Matthew O'Toole, Felix Heide, Lei Xiao, Matthias B. Hullin, Wolfgang Heidrich, and Kiriakos N. Kutulakos. 2014. Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport. ACM Trans. Graph. 33, 4, Article 87 (July 2014), 11 pages. Google ScholarDigital Library
- EM Patterson, CE Shelden, and BH Stockton. 1977. Kubelka-Munk optical properties of a barium sulfate white reflectance standard. Applied Optics 16, 3 (1977), 729--732.Google ScholarCross Ref
- Ofir Pele and Michael Werman. 2010. The quadratic-chi histogram distance family. In Proceedings of ECCV. Springer, 749--762. Google ScholarDigital Library
- Christoph Peters, Jonathan Klein, Matthias B. Hullin, and Reinhard Klein. 2015. Solving Trigonometric Moment Problems for Fast Transient Imaging. ACM Trans. Graph. 34, 6 (Nov. 2015). Google ScholarDigital Library
- Hui Qiao, Jingyu Lin, Yebin Liu, Matthias B Hullin, and Qionghai Dai. 2015. Resolving transient time profile in ToF imaging via log-sum sparse regularization. Opt. Lett. 40, 6 (2015).Google ScholarCross Ref
- Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. 2012. Indoor segmentation and support inference from RGBD images. Computer Vision---ECCV 2012 (2012), 746--760. Google ScholarDigital Library
- Hao Su, Haoqiang Fan, and Leonidas Guibas. 2017. A Point Set Generation Network for 3D Object Reconstruction from a Single Image. In Proceedings of CVPR.Google Scholar
- Andreas Velten, Di Wu, Adrian Jarabo, Belen Masia, Christopher Barsi, Chinmaya Joshi, Everett Lawson, Moungi G. Bawendi, Diego Gutierrez, and Ramesh Raskar. 2013. Femto-Photography: Capturing and Visualizing the Propagation of Light. ACM Transactions on Graphics (SIGGRAPH 2013) 32, 4 (2013). Google ScholarDigital Library
- Peng Wang, Xiaohui Shen, Zhe Lin, S. Cohen, B. Price, and A. Yuille. 2015. Towards unified depth and semantic prediction from a single image. In Proceedings of CVPR.Google Scholar
- Di Wu, Andreas Velten, Matthew O'Toole, Belen Masia, Amit Agrawal, Qionghai Dai, and Ramesh Raskar. 2014. Decomposing Global Light Transport Using Time of Flight Imaging. International Journal of Computer Vision 107, 2 (April 2014), 123 -- 138. Google ScholarDigital Library
- Jianxiong Xiao, Andrew Owens, and Antonio Torralba. 2013. Sun3d: A database of big spaces reconstructed using sfm and object labels. In Proceedings of ICCV. 1625--1632. Google ScholarDigital Library
- Dan Xu, Elisa Ricci, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. 2017. Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation. In Proceedings of CVPR.Google ScholarCross Ref
- Jianchao Yang, John Wright, Thomas S. Huang, and Yi Ma. 2010. Image Super-Resolution Via Sparse Representation. IEEE TIP 19, 11 (2010). Google ScholarDigital Library
- Jure Žbontar and Yann LeCun. 2015. Computing the Stereo Matching Cost with a Convolutional Neural Network. In Proceedings of CVPR.Google ScholarCross Ref
Index Terms
- DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging
Recommendations
Doppler time-of-flight imaging
Over the last few years, depth cameras have become increasingly popular for a range of applications, including human-computer interaction and gaming, augmented reality, machine vision, and medical imaging. Many of the commercially-available devices use ...
Epipolar time-of-flight imaging
Consumer time-of-flight depth cameras like Kinect and PMD are cheap, compact and produce video-rate depth maps in short-range applications. In this paper we apply energy-efficient epipolar imaging to the ToF domain to significantly expand the ...
Time-of-Flight Camera Calibration for Improved 3D Reconstruction of Indoor Scenes
ISCID '14: Proceedings of the 2014 Seventh International Symposium on Computational Intelligence and Design - Volume 02Calibration for Time-of-Flight (ToF) cameras are different from traditional color cameras, in that their calibration involves additional error compensation steps for the depth camera caused by complicated factors. Without this process, there may be ...
Comments