Jonathan Tompson
Yann Lecun
Skip Abstract Section
Skip Supplemental Material SectionAbstract
We present a novel method for real-time continuous pose recovery of markerless complex articulable objects from a single depth image. Our method consists of the following stages: a randomized decision forest classifier for image segmentation, a robust method for labeled dataset generation, a convolutional network for dense feature extraction, and finally an inverse kinematics stage for stable real-time pose recovery. As one possible application of this pipeline, we show state-of-the-art results for real-time puppeteering of a skinned hand-model.
Supplemental Material
- 3GEAR. 2014. 3gear sytems hand-tracking development platform. http://www.threegear.com/.Google Scholar
- B. Allen, B. Curless, and Z. Popovic. 2003. The space of human body shapes: Reconstruction and parameterization from range scans. ACM Trans. Graph. 22, 3, 587--594. Google ScholarDigital Library
- L. Ballan, A. Taneja, J. Gall, L. Van Gool, and M. Pollefeys. 2012. Motion capture of hands in action using discriminative salient points. In Proceedings of the 12th European Conference on Computer Vision. 640--653. Google ScholarDigital Library
- Y. Boykov, O. Veksler, and R. Zabih. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23, 11, 1222--1239. Google ScholarDigital Library
- D. A. Butler, S. Izadi, O. Hilliges, D. Molyneaux, S. Hodges, and D. Kim. 2012. Shake'n'sense: Reducing interference for overlapping structured light depth cameras. In Proceedings of the ACM Annual Conference on Human Factors in Computing Systems. 1933--1936. Google ScholarDigital Library
- R. Collobert, K. Kavukcuoglu, and C. Farabet. 2011. Torch7: A matlab-like environment for machine learning. http://ronan.collobert.com/pub/matos/2011_torch7_nipsw.pdf.Google Scholar
- C. Couprie, C. Farabet, L. Najman, and Y. Lecun. 2013. Indoor semantic segmentation using depth information. In Proceedings of the International Conference on Learning Representations.Google Scholar
- A. Erol, G. Bebis, M. Nicolescu, R. D. Boyle, and X. Twombly. 2007. Vision-based hand pose estimation: A review. Comput. Vis. Image Understand. 108, 1--2, 52--73. Google ScholarDigital Library
- C. Farabet, C. Couprie, L. Najman, and Y. Lecun. 2013. Learning hierarchical features for scene labeling. IEEE Trans. Pattern Anal. Mach. Intell. 35, 8, 1915--1929. Google ScholarDigital Library
- B. K. P. Horn. 1987. Closed-form solution of absolute orientation using unit quaternions. J. Opt. Soc. Amer. 4, 4, 629--642.Google ScholarCross Ref
- K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. Lecun. 2009. What is the best multi-stage architecture for object recognition? In Proceedings of the 12th IEEE International Conference on Computer Vision. 2146--2153.Google Scholar
- M. Jiu, C. Wolf, G. W. Taylor, and A. Baskurt. 2013. Human body part estimation from depth images via spatially-constrained deep learning. Pattern Recogn. Lett. (to appear).Google Scholar
- C. Keskin, F. Kirac, Y. Kara, and L. Akarun. 2011. Real time hand pose estimation using depth sensors. In Proceedings of the IEEE International Computer Vision Workshops. 1228--1234.Google Scholar
- C. Keskin, F. Kirac, Y. E. Kara, and L. Akarun. 2012. Hand pose estimation and hand shape classification using multi-layered randomized decision forests. In Proceedings of the 12th European Conference on Computer Vision. Vol. 6. Springer, 852--863. Google ScholarDigital Library
- A. Krizhevsky, I. Sutskever, and G. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of the Neural Information Processing Systems Conference. P. Bartlett, F. Pereira, C. Burges, L. Bottou, and K. Weinberger, Eds., 1106--1114.Google Scholar
- Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11, 2278--2324.Google ScholarCross Ref
- Y. Lecun, F. J. Huang, and L. Bottou. 2004. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Vol. 2. 97--104. Google ScholarDigital Library
- H. Li, R. W. Sumner, and M. Pauly. 2008. Global correspondence optimization for non-rigid registration of depth scans. Comput. Graph. Forum 27, 5, 1421--1430. Google ScholarDigital Library
- H. Li, J. Yu, Y. Ye, and C. Bregler. 2013. Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. 32, 4. Google ScholarDigital Library
- S. Melax, L. Keselman, and S. Orsten. 2013. Dynamics based 3D skeletal hand tracking. In Proceedings of the ACM Symposium on Interactive 3D Graphics and Games. Google ScholarDigital Library
- J. Nagi, F. Ducatelle, G. Di Caro, D. Ciresan, U. Meier, A. Giusti, F. Nagi, J. Schmidhuber, and L. Gambardella. 2011. Max-pooling convolutional neural networks for vision-based hand gesture recognition. In Proceedings of the IEEE International Conference on Signal and Image Processing Applications. 342--347.Google Scholar
- S. J. Nowlan and J. C. Platt. 1995. A convolutional neural network hand tracker. In Proceedings of the Neural Information Processing Systems Conference. 901--908.Google Scholar
- I. Oikonomidis, N. Kyriazis, and A. Argyros. 2011. Efficient model-based 3D tracking of hand articulations using kinect. In Proceedings of the British Machine Vision Conference.Google Scholar
- M. Osadchy, Y. Lecun, M. L. Miller, and P. Perona. 2005. Synergistic face detection and pose estimation with energy-based model. In Proceedings of the Neural Information Processing Systems Conference. 1017--1024.Google Scholar
- J. M. Rehg and T. Kanade. 1994. Visual tracking of high dof articulated structures: An application to human hand tracking. In Proceedings of the 3rd European Conference on Computer Vision. 35--46. Google ScholarDigital Library
- M. Saric. 2011. Libhand: A library for hand articulation. http://www.libhand. org/.Google Scholar
- J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake. 2011. Real-time human pose recognition in parts from single depth images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1297--1304. Google ScholarDigital Library
- M. Stein, J. Tompson, X. Xiao, C. Hendeee, H. Ishii, and K. Perlin. 2012. Arcade: A system for augmenting gesture-based computer graphic presentations. In Proceedings of the ACM SIGGRAPH Computer Animation Festival. 77--77. Google ScholarDigital Library
- G. W. Taylor, I. Spiro, C. Bregler, and R. Fergus. 2011. Learning invariance through imitation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2729--2736. Google ScholarDigital Library
- P. Tseng. 1995. Fortified-descent simplicial search method: A general approach. SIAM J. Optim. 10, 1, 269--288. Google ScholarDigital Library
- R. Wang, S. Paris, and J. Popovic. 2011. 6d hands: Markerless tracking for computer aided design. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. 549--558. Google ScholarDigital Library
- R. Y. Wang. and J. Popovic. 2009. Real-time hand-tracking with a color glove. ACM Trans. Graph. 28, 3. Google ScholarDigital Library
- T. Weise, H. Li, L. Van Gool, and M. Pauly. 2009. Face/off: Live facial puppetry. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Google ScholarDigital Library
- T. Yasuda, K. Ohkura, and Y. Matsumura. 2010. Extended PSO with partial randomization for large scale multimodal problems. In Proceedings of the World Automation Congress. 1--6.Google Scholar
- W. Zhao, J. Chai, and Y.-Q. Xu. 2012. Combining marker-based MOCAP and RGB-d camera for acquiring high-fidelity hand motion data. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 33--42. Google ScholarDigital Library
Index Terms
- Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks
Recommendations
Accurate, Robust, and Flexible Real-time Hand Tracking
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing SystemsWe present a new real-time hand tracking system based on a single depth camera. The system can accurately reconstruct complex hand poses across a variety of subjects. It also allows for robust tracking, rapidly recovering from any temporary failures. ...
Tracking of deformable human hand in real time as continuous input for gesture-based interaction
IUI '07: Proceedings of the 12th international conference on Intelligent user interfacesGesture input is a natural and effective interactive model. The tracking of deformable hand gesture is a very important task in gesture-based interaction. A novel real-time tracking approach is proposed to capture hand motion with single camera. It ...
Continuous recovery for real time pan tilt zoom localization and mapping
AVSS '11: Proceedings of the 2011 8th IEEE International Conference on Advanced Video and Signal Based SurveillanceWe propose a method for real time recovering from tracking failure in monocular localization and mapping with a Pan Tilt Zoom camera (PTZ). The method automatically detects and seamlessly recovers from tracking failure while preserving map integrity. By ...
Comments