ABSTRACT
KinectFusion enables a user holding and moving a standard Kinect camera to rapidly create detailed 3D reconstructions of an indoor scene. Only the depth data from Kinect is used to track the 3D pose of the sensor and reconstruct, geometrically precise, 3D models of the physical scene in real-time. The capabilities of KinectFusion, as well as the novel GPU-based pipeline are described in full. Uses of the core system for low-cost handheld scanning, and geometry-aware augmented reality and physics-based interactions are shown. Novel extensions to the core GPU pipeline demonstrate object segmentation and user interaction directly in front of the sensor, without degrading camera tracking or reconstruction. These extensions are used to enable real-time multi-touch interactions anywhere, allowing any planar or non-planar reconstructed physical surface to be appropriated for touch.
- P. J. Besl and N. D. McKay. A method for registration of 3D shapes. IEEE Trans. Pattern Anal. Mach. Intell., 14:239--256, February 1992. Google ScholarDigital Library
- X. Cao and R. Balakrishnan. Interacting with dynamically defined information spaces using a handheld projector and a pen. In UIST, pages 225--234, 2006. Google ScholarDigital Library
- Y. Chen and G. Medioni. Object modeling by registration of multiple range images. Image and Vision Computing (IVC), 10(3):145--155, 1992. Google ScholarDigital Library
- Y. Cui et al. 3d shape scanning with a time-of-flight camera. In Computer Vision and Pattern Recognition (CVPR), pages 1173 --1180, June 2010.Google ScholarCross Ref
- B. Curless and M. Levoy. A volumetric method for building complex models from range images. ACM Trans. Graph., 1996.Google ScholarDigital Library
- S. Farsiu et al. Fast and robust multiframe super resolution. IEEE Transactions on Image Processing, 13(10):1327--1344, 2004. Google ScholarDigital Library
- J. Frahm et al. Building Rome on a cloudless day. In Proc. Europ. Conf. on Computer Vision (ECCV), 2010. Google ScholarDigital Library
- B. Freedman, A. Shpunt, M. Machline, and Y. Arieli. Depth Mapping Using Projected Patterns. Patent Application, 10 2008. WO 2008/120217 A2.Google Scholar
- S. L. Grand. Broad-phase collision detection with CUDA. In GPU Gems 3. Addison-Wesley, 2007.Google Scholar
- T. Harada. Real-time rigid body simulation on gpus. In GPU Gems 3. Addison-Wesley Professional, 2007.Google Scholar
- R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, second edition, 2004. Google ScholarDigital Library
- P. Henry et al. RGB-D mapping: Using depth cameras for dense 3D modeling of indoor environments. In Proc. of the Int. Symposium on Experimental Robotics (ISER), 2010.Google Scholar
- B. Huhle et al. Fusion of range and color images for denoising and resolution enhancement with a non-local filter. Computer Vision and Image Understanding, 114(12):1336--1345, 2010. Google ScholarDigital Library
- M. Kazhdan, M. Bolitho, and H. Hoppe. Poisson surface reconstruction. In Proc. of the Eurographics Symposium on Geometry Processing, 2006. Google ScholarDigital Library
- G. Klein and D. W. Murray. Parallel tracking and mapping for small AR workspaces. In ISMAR, 2007. Google ScholarDigital Library
- M. Levoy et al. The digital Michelangelo Project: 3D scanning of large statues. ACM Trans. Graph., 2000.Google ScholarDigital Library
- K. Low. Linear least-squares optimization for point-to-plane icp surface registration. Technical report, TR04-004, University of North Carolina, 2004.Google Scholar
- P. Merrell et al. Real-time visibility-based fusion of depth maps. In Proc. of the Int. Conf. on Computer Vision (ICCV), 2007.Google ScholarCross Ref
- R. A. Newcombe and A. J. Davison. Live dense reconstruction with a single moving camera. In Proc. of the IEEE (CVPR), 2010.Google ScholarCross Ref
- R. A. Newcombe, S. Lovegrove, and A. J. Davison. Dense tracking and mapping in real-time. In Proc. of the Int. Conf. on Computer Vision (ICCV), 2011. Google ScholarDigital Library
- R. A. Newcombe et al. Real-Time Dense Surface Mapping and Tracking with Kinect. In ISMAR, 2011. Google ScholarDigital Library
- S. Osher and R. Fedkiw. Level Set Methods and Dynamic Implicit Surfaces. Springer, 2002.Google Scholar
- S. Rusinkiewicz, O. Hall-Holt, and M. Levoy. Real-time 3D model acquisition. ACM Trans. Graph., 2002. Google ScholarDigital Library
- S. Rusinkiewicz and M. Levoy. Efficient variants of the ICP algorithm. 3D Digital Imaging and Modeling, Int. Conf. on, 0:145, 2001.Google Scholar
- S. Thrun. Robotic mapping: A survey. In Exploring Artificial Intelligence in the New Millenium. 2002. Google ScholarDigital Library
- D. Vlasic et al. Dynamic shape capture using multi-view photometric stereo. ACM Trans. Graph., 28(5), 2009. Google ScholarDigital Library
- D. Wagner, T. Langlotz, and D. Schmalstieg. Robust and unobtrusive marker tracking on mobile phones. In ISMAR, pages 121--124, 2008. Google ScholarDigital Library
- T. Weise, T. Wismer, B. Leibe, and L. V. Gool. In-hand scanning with online loop closure. In IEEE Int. Workshop on 3-D Digital Imaging and Modeling, 2009.Google ScholarCross Ref
- K. Zhou, M. Gong, X. Huang, and B. Guo. Data-parallel octrees for surface reconstruction. IEEE Trans. on Visualization and Computer Graphics, 17, 2011. Google ScholarDigital Library
Index Terms
- KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera
Recommendations
KinectFusion: real-time dynamic 3D surface reconstruction and interaction
SIGGRAPH '11: ACM SIGGRAPH 2011 TalksWe present KinectFusion, a system that takes live depth data from a moving Kinect camera and in real-time creates high-quality, geometrically accurate, 3D models. Our system allows a user holding a Kinect camera to move quickly within any indoor space, ...
Heterogeneous Sensor Data Fusion: How Many Cameras are Needed for an Accurate 3D Reconstruction of Indoor Scene?
ISCID '14: Proceedings of the 2014 Seventh International Symposium on Computational Intelligence and Design - Volume 02Vision systems, consisting of conventional color cameras and depth cameras have proved capable of solving the problem of indoor scene reconstruction with sufficient detail and satisfactory accuracy. However, there are few guidelines for how many cameras ...
Real-Time High Resolution Fusion of Depth Maps on GPU
CADGRAPHICS '13: Proceedings of the 2013 International Conference on Computer-Aided Design and Computer GraphicsA system for live high quality surface reconstruction using a single moving depth camera on a commodity hardware is presented. High accuracy and real-time frame rate is achieved by utilizing graphics hardware computing capabilities via OpenCLTM and by ...
Comments