Pedro Lopes
Sijing You
Alexandra Ion
Patrick Baudisch
ABSTRACT
We present a mobile system that enhances mixed reality experiences and games with force feedback by means of electrical muscle stimulation (EMS). The benefit of our approach is that it adds physical forces while keeping the users' hands free to interact unencumbered-not only with virtual objects, but also with physical objects, such as props and appliances. We demonstrate how this supports three classes of applications along the mixed-reality continuum: (1) entirely virtual objects, such as furniture with EMS friction when pushed or an EMS-based catapult game. (2) Virtual objects augmented via passive props with EMS-constraints, such as a light control panel made tangible by means of a physical cup or a balance-the-marble game with an actuated tray. (3) Augmented appliances with virtual behaviors, such as a physical thermostat dial with EMS-detents or an escape-room that repurposes lamps as levers with detents. We present a user-study in which participants rated the EMS-feedback as significantly more realistic than a no-EMS baseline.
Supplemental Material
- Hideyuki Ando, Takeshi Miki, Masahiko Inami, and Taro Maeda. 2002. The nail-mounted tactile display for the behavior modeling. In Proceedings of ACM SIGGRAPH 2002 conference abstracts and applications, 264--264. Google Scholar
Digital Library
- Hideyuki Ando, Eisuke Kusachi, and Junji Watanabe. 2007. Nail-mounted tactile display for boundary/texture augmentation. In Proceedings of the international conference on Advances in computer entertainment technology, 292--293. Google ScholarDigital Library
- Mohamed Benali-Khoudja, Moustapha Hafez, JeanMarc Alexandre, and Abderrahmane Kheddar. 2004. Tactile interfaces: a state-of-the-art survey. In Int. Symposium on Robotics, vol. 31, pp. 23--26.Google Scholar
- Matteo Bianchi, Edoardo Battaglia, Mattia Poggiani, Simone Ciotti, and Antonio Bicchi. 2016. A Wearable Fabric-based display for haptic multi-cue delivery. In Proceedings of Haptics Symposium (HAPTICS 2016), 277--283.Google ScholarCross Ref
- James Biggs and Mandayam A. Srinivasan. 2002. Tangential Versus Normal Displacements of Skin: Relative Effectiveness for Producing Tactile Sensations. In Proceedings of the 10th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS 2002), 121--128. Google ScholarDigital Library
- Christoph W. Borst and Richard A Volz. 2005. Evaluation of a Haptic Mixed Reality System for Interactions with a Virtual Control Panel. Presence, 14(6), 677--696. Google ScholarDigital Library
- Grigore C. Burdea. 2000. Haptic Issues in Virtual Environments. In Proceedings of Computer Graphics International (GCI '00). Google ScholarDigital Library
- Tom Carter, Sue Ann Seah, Benjamin Long, Bruce Drinkwater, and Sriram Subramanian. 2013. Ultrahaptics: Multi-Point Mid-Air Haptic Feedback for Touch Surfaces. In Proceedings of the 26th annual ACM symposium on User interface software and technology (UIST '13), 505--514. Google ScholarDigital Library
- Daniel K. Y. Chen, Iain A. Anderson, Cameron G. Walker and Thor F. Besier. 2016. Lower Extremity Lateral Skin Stretch Perception for Haptic Feedback. Haptics IEEE Transactions, 9, 62--68, 2016. Google ScholarDigital Library
- Ulrich Eck and Christian Sandor. 2013. HARP: A framework for visuo-haptic augmented reality. In IEEE Virtual Reality.Google Scholar
- Heather Culbertson, Juan José López Delgado, and Katherine J. Kuchenbecker. 2014. One hundred datadriven haptic texture models and open-source methods for rendering on 3D objects. In Proceedings of Haptics Symposium (HAPTICS 2014), IEEE, 319--325.Google Scholar
- Ayaka Ebisu, Satoshi Hashizume, Kenta Suzuki, Akira Ishii, Mose Sakashita, and Yoichi Ochiai. 2016. Stimulated percussions: techniques for controlling human as percussive musical instrument by using electrical muscle stimulation. In SIGGRAPH ASIA 2016 Posters (SA '16), Article 37. Google ScholarDigital Library
- Arthur Elsenaar and Remko Scha, Electric Body Manipulation as Performance Art: A Historical Perspective, in Leonardo Music Journal, December 2002, Vol. 12, Pages: 17--28.Google Scholar
- Escape Room, https://en.wikipedia.org/wiki/Escape_room, last accessed on 3/10/2017.Google Scholar
- Farzam Farbiz, Zhou Hao Yu, Corey Manders, and Waqas Ahmad. 2007. An electrical muscle stimulation haptic feedback for mixed reality tennis game. In Proceedings of ACM SIGGRAPH 2007 posters, p. 140. Google ScholarDigital Library
- Steven Feiner. 2002. Augmented reality: A new way of seeing. Scientific American, 286(4), 2002, 34--41.Google Scholar
- Xiaochi Gu, Yifei Zhang, Weize Sun, Yuanzhe Bian, Dao Zhou, and Per Ola Kristensson. 2016. Dexmo: An Inexpensive and Lightweight Mechanical Exoskeleton for Motion Capture and Force Feedback in VR. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 1991--1995. Google ScholarDigital Library
- Blake Hannaford and Allison M. Okamura, Haptics (chapter), Handbook of Robotics, 2008, Oussama Khatib, Bruno Siciliano(eds.), Springer, pp 719--739.Google Scholar
- Steven J. Henderson and Steven Feiner. 2008. Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality. In Proceedings of the 2008 ACM symposium on Virtual reality software and technology (VRST '08), 211--218. Google ScholarDigital Library
- Steven J. Henderson and Steven Feiner. 2009. Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret. In Proceedings of the 2009 8th IEEE International Symposium on Mixed and Augmented Reality (ISMAR '09), 135--144. Google ScholarDigital Library
- Ken Hinckley, Randy Pausch, J.C. Goble and N.F. Kassell. 1994. Passive Real-World Interface Props for Neurosurgical Visualization, in Proceedings of CHI '94, 452--458. Google ScholarDigital Library
- Anuruddha Hettiarachchi and Daniel Wigdor. 2016. Annexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1957--1967. Google ScholarDigital Library
- John M. Hollerbach, and Stephen C. Jacobsen. 1995. Haptic interfaces for teleoperation and virtual environments. In Proceedings of the First Workshop on Simulation and Interaction in Virtual Environments.Google Scholar
- HoloLens, https://www.microsoft.com/microsofthololens/en-us/developersGoogle Scholar
- John N. Howell, Robert L. Williams, Robert R. Conatser, Janet M. Burns, and David C. Eland. 2005. The Virtual Haptic Back (VHB): a virtual reality simulation of the human back for palpatory diagnostic training. SAE Technical Paper, No. 2005-01--2679.Google Scholar
- Wolfgang Hürst, Nina Rosa, and Jean-Paul van Bommel. 2016. Vibrotactile experiences for augmented reality. In Proceedings of the 2016 ACM on Multimedia Conference, 744--745. Google ScholarDigital Library
- Brent Edward Insko. 2001. Passive Haptics Significantly Enhances Virtual Environments. Ph.D. Dissertation. The University of North Carolina at Chapel Hill. Advisor(s) Frederick P. Brooks, Jr.. AAI3007820.Google Scholar
- Hiroshi Ishii and Brygg Ullmer. 1997. Tangible bits: towards seamless interfaces between people, bits and atoms. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems (CHI '97), 234241. Google ScholarDigital Library
- Seungzoo Jeong, Naoki Hashimoto, and Sato Makoto. 2004. A novel interaction system with force feedback between real-and virtual human: an entertainment system: virtual catch ball. In Proceedings of the 2004 ACM SIGCHI International Conference on Advances in computer entertainment technology, 61--66. Google ScholarDigital Library
- Hwan Kim, Minhwan Kim, and Woohun Lee. 2016. HapThimble: A Wearable Haptic Device towards Usable Virtual Touch Screen. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 3694--3705. Google ScholarDigital Library
- Benjamin Knoerlein, Gábor Székely, and Matthias Harders. 2007. Visuo-haptic collaborative augmented reality ping-pong. In Proceedings of the international conference on Advances in computer entertainment technology (ACE '07), 91--94. Google ScholarDigital Library
- Ernst Kruijff, Dieter Schmalstieg, and Steffi Beckhaus. 2006. Using neuromuscular electrical stimulation for pseudo-haptic feedback. In Proceedings of the ACM symposium on Virtual reality software and technology (VRST '06), 316--319. Google ScholarDigital Library
- Joshua Lifton and Joseph A. Paradiso, Dual Reality: Merging the Real and Virtual, 2017, International Conference on Facets of Virtual Environments FaVE 2009: pp. 12--28.Google Scholar
- Robert W., Lindeman, John L. Sibert and James K. Hahn. 1999. Hand-Held Windows: Towards Effective 2D Interaction in Immersive Virtual Environments. In Proceedings of the IEEE Virtual Reality (VR '99), 205. Google ScholarDigital Library
- Robert W. Lindeman, Yasuyuki Yanagida, Haruo Noma, and Kenichi Hosaka. 2006. Wearable vibrotactile systems for virtual contact and information display. Virtual Reality 9, no. 2--3 (2006): 203--213.Google Scholar
- Pedro Lopes and Patrick Baudisch. 2013. Musclepropelled force feedback: bringing force feedback to mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13), 2577--2580. Google ScholarDigital Library
- Pedro Lopes, Alexandra Ion, and Patrick Baudisch. 2015. Impacto: Simulating Physical Impact by Combining Tactile Stimulation with Electrical Muscle Stimulation. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15), 11--19. Google ScholarDigital Library
- Pedro Lopes, Alexandra Ion, Willi Mueller, Daniel Hoffmann, Patrik Jonell, and Patrick Baudisch. 2015. Proprioceptive Interaction. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 939--948. Google ScholarDigital Library
- Pedro Lopes, Patrik Jonell, and Patrick Baudisch. 2015. Affordance++: Allowing Objects to Communicate Dynamic Use. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 2515--2524. Google ScholarDigital Library
- Pedro Lopes, Doğa Yüksel, François Guimbretière and Patrick Baudisch,.2016. Muscle plotter: An Interactive System based on Electrical Muscle Stimulation that Produces Spatial Output. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16), 207--217. Google ScholarDigital Library
- Pedro Lopes, Sijing Young, Lung-pan Cheng, Sebastian Marwecki, and Patrick Baudisch, Providing Haptics to Walls and Other Heavy Objects in Virtual Reality by Means of Electrical Muscle Stimulation. In Proceedings of the 35th Annual ACM Conference on Human Factors in Computing Systems (CHI '17). Google ScholarDigital Library
- Cornelio Martinez, Patricia Ivette, Silvana De Pirro, Chi Thanh Vi, and Sriram Subramanian. 2017. Agency in mid-air interfaces. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). Google ScholarDigital Library
- Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino. 1995. Augmented reality: A class of displays on the reality-virtuality continuum. In Proceedings of SPIE 2351, Telemanipulator and Telepresence Technologies, 282--292.Google Scholar
- Paul Milgram and Fumio Kishino. 1994. A Taxonomyof Mixed Reality Visual Displays. IEICE Transactions on Information and Systems E77-D,1321--1329.Google Scholar
- Jun Murayama, Laroussi Bougrila, YanLin Luo, Katsuhito Akahane, Shoichi Hasegawa, Béat Hirsbrunner, and Makoto Sato. 2004. SPIDAR G&G: a two-handed haptic interface for bimanual VR interaction. In Proceedings of EuroHaptics, vol. 2004, 138--146.Google Scholar
- Kazuki Nagai, Soma Tanoue, Katsuhito Akahane, and Makoto Sato. 2015. Wearable 6-DoF wrist haptic device SPIDAR-W. In Proceedings of SIGGRAPH Asia 2015 Haptic Media And Contents Design, p. 19. Google ScholarDigital Library
- Geoff Norman. 2010. Likert scales, levels of measurement and the "laws" of statistics. Advances in health sciences education 15, no. 5 (2010): 625--632.Google Scholar
- Ohan Oda and Steven Feiner. 2010. Rolling and shooting: two augmented reality games. CHI EA '10, 30413044. Google ScholarDigital Library
- Ohan Oda, Carmine Elvezio, Mengu Sukan, Steven Feiner, and Barbara Tversky. 2015. Virtual Replicas for Remote Assistance in Virtual and Augmented Reality. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15), 405415. Google ScholarDigital Library
- Max Pfeiffer, Tim Dünte, Stefan Schneegass, Florian Alt, and Michael Rohs. 2015. Cruise Control for Pedestrians: Controlling Walking Direction using Electrical Muscle Stimulation. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15), 2505--2514. Google ScholarDigital Library
- Max Pfeiffer, Stefan Schneegass, Florian Alt, and Michael Rohs. 2014. Let me grab this: a comparison of EMS and vibration for haptic feedback in free-hand interaction. In Proceedings of the 5th Augmented Human International Conference (AH '14). ACM, New York, NY, USA,, Article 48, 8 pages. Google ScholarDigital Library
- Raf Ramakers, Fraser Anderson, Tovi Grossman, and George Fitzmaurice. 2016. RetroFab: A Design Tool for Retrofitting Physical Interfaces using Actuators, Sensors and 3D Printing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16), 409--419. Google ScholarDigital Library
- Rehastim by Hasomed, https://www.hasomed.de/en/products.htmlGoogle Scholar
- Joan Sol Roo, Martin Hachet, One Reality: Augmenting How the Physical World is Experienced by combining Multiple Mixed Reality Modalities, to appear, UIST 2017. Google ScholarDigital Library
- Antti Sand, Ismo Rakkolainen, Poika Isokoski, Jari Kangas, Roope Raisamo, and Karri Palovuori. 2015. Head-mounted display with mid-air tactile feedback. In Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology (VRST '15), 51--58. Google ScholarDigital Library
- Stefano Scheggi, Gionata Salvietti, and Domenico Prattichizzo. 2010. Shape and weight rendering for haptic augmented reality. In Proceedings of the 19th International Symposium in Robot and Human Interactive Communication, 44--49.Google ScholarCross Ref
- William R. Sherman, and Alan B. Craig. 2002. Understanding virtual reality: Interface, application, and design. Elsevier, 2002. Google ScholarDigital Library
- Rajinder Sodhi, Ivan Poupyrev, Matthew Glisson, and Ali Israr. 2013. AIREAL: interactive tactile experiences in free air. ACM Transactoions on Graphics. 32, 4, Article 134 (July 2013), 10 pages. Google ScholarDigital Library
- Primož Strojnik, Alojz Kralj, and I. Ursic. 1979. Programmed six-channel electrical stimulator for complex stimulation of leg muscles during walking. IEEE Transactions on Biomedical Engineering 2 (1979): 112116.Google Scholar
- Emi Tamaki, Takashi Miyaki, and Jun Rekimoto. 2011. PossessedHand: techniques for controlling human hands using electrical muscles stimuli. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 543--552. Google ScholarDigital Library
- Dzmitry Tsetserukou, Katsunari Sato, and Susumu Tachi. 2010. ExoInterfaces: novel exosceleton haptic interfaces for virtual reality, augmented sport and rehabilitation. In Proceedings of the 1st Augmented Human International Conference (AH '10), 1--6. Google ScholarDigital Library
- Vuforia, https://developer.vuforia.comGoogle Scholar
- Sean White and Steven Feiner. 2009. SiteLens: situated visualization techniques for urban site visits. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09), 1117--1120. Google ScholarDigital Library
- Yasuyoshi Yokokohji, Ralph L. Hollis, and Takeo Kanade. 1996. What you can see is what you can feeldevelopment of a visual/haptic interface to virtual environment." In Proceedings of the IEEE 1996 Virtual Reality Annual International Symposium, 46--53. Google ScholarDigital Library
Index Terms
- Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation
Recommendations
Journalism in virtual reality: opportunities and future research challenges
AcademicMindtrek '16: Proceedings of the 20th International Academic Mindtrek ConferenceThis paper presents a state-of-the-art overview of journalism and its opportunities and challenges in virtual reality (VR). First, we examine what kind of real-life journalistic experiments have been made in this field thus far, and then we analyze the ...
HoloLens-based Mixed Reality Experiences for Subjects with Alzheimer's Disease
CHItaly '17: Proceedings of the 12th Biannual Conference on Italian SIGCHI ChapterHoloLens technology enables mixed reality experiences that integrate holographic objects into the real world in which the mixed reality head-mounted device is used. Since HoloLens' launch in March 2016, some mixed-reality applications of this technology ...
Exploring Mixed Reality in Specialized Surgical Environments
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing SystemsRecent technology advances in both Virtual Reality and Augmented Reality are creating an opportunity for a paradigm shift in the design of human-computer interaction systems. Delving into the Reality-Virtuality Continuum, we find Mixed Reality - systems ...
Comments