Otmar Hilliges
David Kim
Shahram Izadi
ABSTRACT
HoloDesk is an interactive system combining an optical see through display and Kinect camera to create the illusion that users are directly interacting with 3D graphics. A virtual image of a 3D scene is rendered through a half silvered mirror and spatially aligned with the real-world for the viewer. Users easily reach into an interaction volume displaying the virtual image. This allows the user to literally get their hands into the virtual display and to directly interact with an spatially aligned 3D virtual world, without the need for any specialized head-worn hardware or input device. We introduce a new technique for interpreting raw Kinect data to approximate and track rigid (e.g., books, cups) and non-rigid (e.g., hands, paper) physical objects and support a variety of physics-inspired interactions between virtual and real. In particular the algorithm models natural human grasping of virtual objects with more fidelity than previously demonstrated. A qualitative study highlights rich emergent 3D interactions, using hands and real-world objects. The implementation of HoloDesk is described in full, and example application scenarios explored. Finally, HoloDesk is quantitatively evaluated in a 3D target acquisition task, comparing the system with indirect and glasses-based variants.
Supplemental Material
- Agrawala, M., et al. The two-user responsive workbench: support for collaboration through individual views of a shared space. In Proc. ACM SIGGRAPH (1997), 327--332. Google Scholar
Digital Library
- Bimber, O., et al. The extended virtual table: An optical extension for table-like projection systems. Presence: Teleoper. Virtual Environ. 10 (December 2001), 613--631. Google ScholarDigital Library
- Bimber, O., and Raskar, R. Spatial Augmented Reality: Merging Real and Virtual Worlds. A. K. Peters, Ltd., Natick, MA, USA, 2005. Google ScholarDigital Library
- Brox, T., et al. High accuracy optical flow estimation based on a theory for warping. Computer 4, May (2004).Google Scholar
- Butler, A., et al. Vermeer: Direct Interaction with a 360° Viewable 3D Display. In Proc. ACM UIST (2011). Google ScholarDigital Library
- Czernuszenko, M., et al. The immersadesk and infinity wall projection-based virtual reality displays. SIGGRAPH Comput. Graph. 31 (May 1997), 46--49. Google ScholarDigital Library
- Feng, Z., et al. Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR. In Proc. IEEE ISMAR (2008), 193--202. Google ScholarDigital Library
- Graham, E. D., and MacKenzie, C. L. Physical versus virtual pointing. In Proc. ACM CHI (1996), 292--299. Google ScholarDigital Library
- Grossman, T., and Wigdor, D. Going Deeper: a Taxonomy of 3D on the Tabletop. In Proc. ACM ITS, IEEE (Oct. 2007), 137--144.Google ScholarCross Ref
- Hachet, M., et al. Toucheo: Multitouch and Stereo Combined in a Seamless Workspace. In Proc. ACM UIST (Oct. 2011). Google ScholarDigital Library
- Hancock, M., Carpendale, S., and Cockburn, A. Shallow-Depth 3D Interaction: design and evaluation of one-, two- and three-touch techniques. In Proc. ACM CHI (2007), 1147--1156. Google ScholarDigital Library
- Hilliges, O., et al. Interactions in the Air: Adding Further Depth to Interactive Tabletops. In Proc. ACM UIST (2009), 139--148. Google ScholarDigital Library
- Hoffman, D., et al. Vergence accommodation conflicts hinder visual performance and cause visual fatigue. IJVR 8, 3 (2008), 33.Google Scholar
- Izadi, S., et al. C-Slate: A Multi-Touch and Object Recognition System for Remote Collaboration using Horizontal Surfaces. In IEEE Tabletop, vol. 0, IEEE (2007), 3--10.Google ScholarCross Ref
- Izadi, S., et al. Going beyond the display. In Proc. ACM UIST, ACM (2008), 269. Google ScholarDigital Library
- Izadi, S., et al. KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera. In Proc. ACM UIST, ACM (2011). Google ScholarDigital Library
- Johnson, A., Scharver, C., and Leigh, J. Designing Cranial Implants in a Haptic Augmented Reality Environment. Communications of the ACM 47, 8 (2004), 32--38. Google ScholarDigital Library
- Kakehi, Y., et al. Tablescape Plus: Interactive Small-sized Vertical Displays on a Horizontal Tabletop Display. In Proc. ACM ITS, IEEE (2007), 155--162.Google Scholar
- Kim, S. W., Treskunov, A., and Marti, S. DRIVE: Directly Reaching Into Virtual Environment with bare hand manipulation behind mobile display. In Proc. IEEE 3D UI (2011), 107--108. Google ScholarDigital Library
- Knödel, S., and Hachet, M. Multi-touch RST in 2D and 3D Spaces: Studying the Impact of Directness on User Performance. In Proc. IEEE 3D UI (2011). Google ScholarDigital Library
- Lee, T., and Hollerer, T. Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertip Tracking. In Proc. ISWC, IEEE (2007), 1--8. Google ScholarDigital Library
- Malik, S., and Laszlo, J. Visual touchpad: a two-handed gestural input device. In Proc. ACM ICMI (2004), 289. Google ScholarDigital Library
- Martinet, A., et al. The design and evaluation of 3d positioning techniques for multi-touch displays. In Proc. IEEE 3D UI (2010), 115--118. Google ScholarDigital Library
- McKenna, M. Interactive viewpoint control and three-dimensional operations. In Proc SI3D, vol. 25, ACM (1992), 53--56. Google ScholarDigital Library
- Mulder, J. D., and Liere, R. V. The personal space station: Bringing interaction within reach. In Proc. VRIC (2002).Google Scholar
- Nakashima, B. K., et al. A 2D-3D Integrated Tabletop Environment for Multi-user Collaboration. Computer Animation and Virtual Worlds 18, 1 (November 2007), 39--56. Google ScholarDigital Library
- Olwal, A., et al. ASTOR: An Autostereoscopic Optical See-through Augmented Reality System. In Proc. IEEE ISMAR, IEEE (2005), 24--27. Google ScholarDigital Library
- Papagiannakis, G., et al. A survey of mobile and wireless technologies for augmented reality systems. Comput. Animat. Virtual Worlds 19 (February 2008), 3--22. Google ScholarDigital Library
- Paris, S., and Durand, F. A Fast Approximation of the Bilateral Filter Using a Signal Processing Approach. IJCV 81, 1 (Dec. 2007), 24--52. Google ScholarDigital Library
- Poston, T., and Serra, L. The virtual workbench: dextrous VR. In Proc. ACM VRST (1994), 111--121. Google ScholarDigital Library
- Prachyabrued, M., and Borst, C. W. Dropping the ball: Releasing a virtual grasp. In Proc. ACM VRST (2011), 59--66. Google ScholarDigital Library
- Reisman, J. L., et al. A screen-space formulation for 2D and 3D direct manipulation. In Proc. ACM UIST (2009), 69--78. Google ScholarDigital Library
- Stavness, I., Lam, B., and Fels, S. pCubee : A Perspective-Corrected Handheld Cubic Display. In Proc. ACM CHI (2010), 1381--1390. Google ScholarDigital Library
- Ware, C., Arthur, K., and Booth, K. S. Fish Tank Virtual Reality. In Proc. ACM CHI, ACM (1993), 37--42. Google ScholarDigital Library
- Wilson, A. D. Depth-Sensing Video Cameras for 3D Tangible Tabletop Interaction. In Proc. ACM ITS, IEEE (2007), 201--204.Google ScholarCross Ref
- Wilson, A. D. Simulating Grasping Behavior on an Imaging Interactive Surface. In Proc. ACM ITS, ACM (2007), 125--132. Google ScholarDigital Library
- Wilson, A. D., and Benko, H. Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In Proc. ACM UIST (2010), 273--282. Google ScholarDigital Library
- Wilson, A. D., et al. Bringing Physics to the Surface. In Proc. ACM UIST (2008), 67--76. Google ScholarDigital Library
- Yoshida, T., et al. Repro3d: full-parallax 3d display using retro-reflective projection technology. In Proc. ACM SIGGRAPH, ACM (2010). Google ScholarDigital Library
- Zhang, Z. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 11 (2000), 1330--1334. Google ScholarDigital Library
Index Terms
- HoloDesk: direct 3d interactions with a situated see-through display
Recommendations
Touchless gestural interaction with small displays: a case study
CHItaly '13: Proceedings of the Biannual Conference of the Italian Chapter of SIGCHITouchless gestural interaction enables users to interact with digital devices using body movements and gestures, and without the burden of a physical contact with technology (e.g., data gloves, body markers, or remote controllers). Most gesture-based ...
Comparing two head-mounted displays in ultrasound scanning
HCI'07: Proceedings of the 12th international conference on Human-computer interaction: interaction platforms and techniquesHead-mounted displays have been tested in various medical fields. According to some results, using a head-mounted display makes medical operations faster, more effective and accurate than using a conventional table display. In this study we aimed to ...
Immersive Multiplayer Games With Tangible and Physical Interaction
In this paper, we present a new immersive multiplayer game system developed for two different environments, namely, virtual reality (VR) and augmented reality (AR). To evaluate our system, we developed three game applications-a first-person-shooter game ...
Comments