Wenjie Ruan
Quan Z. Sheng
ABSTRACT
Hand gesture is becoming an increasingly popular means of interacting with consumer electronic devices, such as mobile phones, tablets and laptops. In this paper, we present AudioGest, a device-free gesture recognition system that can accurately sense the hand in-air movement around user's devices. Compared to the state-of-the-art, AudioGest is superior in using only one pair of built-in speaker and microphone, without any extra hardware or infrastructure support and with no training, to achieve fine-grained hand detection. Our system is able to accurately recognize various hand gestures, estimate the hand in-air time, as well as average moving speed and waving range. We achieve this by transforming the device into an active sonar system that transmits inaudible audio signal and decodes the echoes of hand at its microphone. We address various challenges including cleaning the noisy reflected sound signal, interpreting the echo spectrogram into hand gestures, decoding the Doppler frequency shifts into the hand waving speed and range, as well as being robust to the environmental motion and signal drifting. We implement the proof-of-concept prototype in three different electronic devices and extensively evaluate the system in four real-world scenarios using 3,900 hand gestures that collected by five users for more than two weeks. Our results show that AudioGest can detect six hand gestures with an accuracy up to 96%, and by distinguishing the gesture attributions, it can provide up to 162 control commands for various applications.
- Leap Motion, Inc. Leap Motion: Mac PC Gesture Controller for Game, Design and More. https://www.leapmotion.com/, 2013.Google Scholar
- Nintendo. Wii console. http://www.nintendo.com/wii.Google Scholar
- RoboRealm. Microsoft Kinect, http://www.roborealm.com/help/Microsoft Kinect.php, 2013.Google Scholar
- Heba Abdelnasser, Moustafa Youssef, and Khaled A Harras. 2015. Wigest: A ubiquitous wifi-based gesture recognition system. In Computer Communications (INFOCOM), 2015 IEEE Conference on. IEEE, 1472--1480.Google ScholarCross Ref
- Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and Frédo Durand. 2015. Capturing the human figure through a wall. ACM Transactions on Graphics (TOG) 34, 6 (2015), 219. Google ScholarDigital Library
- Fadel Adib and Dina Katabi. 2013. See Through Walls with WiFi!. In Proceedings of the ACM SIGCOMM 2013 Conference (SIGCOMM '13). 75--86. Google ScholarDigital Library
- Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Romit Roy Choudhury, Kevin Caves, and Frank DeRuyter. 2011. Using mobile phones to write in air. In Proceedings of the 9th international conference on Mobile systems, applications, and services. ACM, 15--28. Google ScholarDigital Library
- Parvin Asadzadeh, Lars Kulik, and Egemen Tanin. 2012. Gesture recognition using RFID technology. Personal and Ubiquitous Computing 16, 3 (2012), 225--234. Google ScholarDigital Library
- Gabe Cohn, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Humantenna: using the body as an antenna for real-time whole-body interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1901--1910. Google ScholarDigital Library
- Nasser H Dardas and Nicolas D Georganas. 2011. Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. Instrumentation and Measurement, IEEE Transactions on 60, 11 (2011), 3592--3607.Google Scholar
- G Deng and LW Cahill. 1993. An adaptive Gaussian filter for noise reduction and edge detection. In Nuclear Science Symposium and Medical Imaging Conference, 1993., 1993 IEEE Conference Record. IEEE, 1615--1619.Google ScholarCross Ref
- Han Ding, Longfei Shangguan, Zheng Yang, Jinsong Han, and others. 2015. FEMO: A Platform for Free-weight Exercise Monitoring with RFIDs. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems (SenSys '15). 141--154. Google ScholarDigital Library
- Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Soundwave: using the doppler effect to sense gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1911--1914. Google ScholarDigital Library
- Kaustubh Kalgaonkar and Bhiksha Raj. 2009. One-handed gesture recognition using ultrasonic Doppler sonar. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 1889--1892. Google ScholarDigital Library
- Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing gesture recognition to all devices. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14). 303--316. Google ScholarDigital Library
- Hamed Ketabdar, Peyman Moghadam, Babak Naderi, and Mehran Roshandel. 2012. Magnetic signatures in air for mobile devices. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services companion. ACM, 185--188. Google ScholarDigital Library
- David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 167--176. Google ScholarDigital Library
- Pedro Melgarejo, Xinyu Zhang, Parameswaran Ramanathan, and David Chu. 2014. Leveraging directional antenna capabilities for fine-grained gesture recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 541--551. Google ScholarDigital Library
- Pavlo Molchanov, Shalini Gupta, Kihwan Kim, and Kari Pulli. 2015a. Multi-sensor system for driver's hand-gesture recognition. In Automatic Face and Gesture Recognition (FG), 2015 11th IEEE International Conference and Workshops on, Vol. 1. IEEE, 1--8.Google ScholarCross Ref
- Pavlo Molchanov, Shalini Gupta, Kihwan Kim, and Kari Pulli. 2015b. Short-range FMCW monopulse radar for hand-gesture sensing. In Radar Conference (RadarCon), 2015 IEEE. IEEE, 1491--1496.Google ScholarCross Ref
- May Moussa and Moustafa Youssef. 2009. Smart devices for smart environments: Device-free passive detection in real environments. In Pervasive Computing and Communications, 2009. PerCom 2009. IEEE International Conference on. IEEE, 1--6. Google ScholarDigital Library
- Rajalakshmi Nandakumar, Shyamnath Gollakota, and Nathaniel Watson. 2015. Contactless sleep apnea detection on smartphones. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 45--57. Google ScholarDigital Library
- Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). 1515--1525. Google ScholarDigital Library
- Taiwoo Park, Jinwon Lee, Inseok Hwang, Chungkuk Yoo, Lama Nachman, and Junehwa Song. 2011. E-gesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices. In Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems. ACM, 260--273. Google ScholarDigital Library
- Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th annual international conference on Mobile computing & networking. ACM, 27--38. Google ScholarDigital Library
- Siddharth S Rautaray and Anupam Agrawal. 2015. Vision based hand gesture recognition for human computer interaction: a survey. Artificial Intelligence Review 43, 1 (2015), 1--54. Google ScholarDigital Library
- W. Ruan. 2016. Unobtrusive human localization and activity recognition for supporting independent living of the elderly. In Proceedings of 2016 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops). 1--3.Google ScholarCross Ref
- Wenjie Ruan, Lina Yao, Quan Z. Sheng, and others. 2015. TagFall: Towards Unobstructive Fine-Grained Fall Detection based on UHF Passive RFID Tags. In Proceedings of the 12th International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MOBIQUITOUS '15). 140--149. Google ScholarDigital Library
- Thad Starner and Alex Pentland. 1997. Real-time american sign language recognition from video using hidden markov models. In Motion-Based Recognition. Springer, 227--243.Google Scholar
- Stephen P Tarzia, Robert P Dick, Peter A Dinda, and Gokhan Memik. 2009. Sonar-based measurement of user presence and attention. In Proceedings of the 11th international conference on Ubiquitous computing. ACM, 89--92. Google ScholarDigital Library
- Juan Pablo Wachs, Mathias Kölsch, Helman Stern, and Yael Edan. 2011. Vision-based Hand-gesture Applications. Commun. ACM 54, 2 (Feb. 2011), 60--71. DOI:http://dx.doi.org/10.1145/1897816.1897838 Google ScholarDigital Library
- Sy Bor Wang, Ariadna Quattoni, Louis-Philippe Morency, David Demirdjian, and Trevor Darrell. 2006. Hidden conditional random fields for gesture recognition. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, Vol. 2. IEEE, 1521--1527. Google ScholarDigital Library
- Jiahui Wu, Gang Pan, Daqing Zhang, Guande Qi, and Shijian Li. 2009. Gesture recognition with a 3-d accelerometer. In Ubiquitous intelligence and computing. Springer, 25--38. Google ScholarDigital Library
- Lina Yao, Quan Z. Sheng, Wenjie Ruan, Tao Gu, Xue Li, Nick Falkner, and Zhi Yang. 2015a. RF-Care: Device-Free Posture Recognition for Elderly People Using A Passive RFID Tag Array. In Proceedings of the 12th International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MOBIQUITOUS '15). 120--129. Google ScholarDigital Library
- L. Yao, Q. Z. Sheng, W. Ruan, X. Li, S. Wang, and Z. Yang. 2015b. Unobtrusive Posture Recognition via Online Learning of Multi-dimensional RFID Received Signal Strength. In Proceedings of IEEE 21st International Conference on Parallel and Distributed Systems (ICPADS'15). 116--123. Google ScholarDigital Library
- Chen Zhao, Ke-Yu Chen, Md Tanvir Islam Aumi, Shwetak Patel, and Matthew S Reynolds. 2014. SideSwipe: detecting in-air gestures around mobile devices using actual GSM signal. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 527--534. Google ScholarDigital Library
Index Terms
- AudioGest: enabling fine-grained hand gesture detection by decoding echo signal
Recommendations
Evaluating the placement of arm-worn devices for recognizing variations of dynamic hand gestures
Dynamic hand gestures have become increasingly popular as touch-free input modality for interactive systems. There exists a variety of arm-worn devices for the recognition of hand gestures, which differ not only in their capabilities, but also in their ...
Comparing the Placement of Two Arm-Worn Devices for Recognizing Dynamic Hand Gestures
MoMM '16: Proceedings of the 14th International Conference on Advances in Mobile Computing and Multi MediaDynamic hand gestures have become increasingly popular as an input modality for interactive systems. There exists a variety of arm-worn devices for the recognition of hand gestures, which differ not only in their capabilities, but also in the arm ...
Comparing hand gestures and a gamepad interface for locomotion in virtual environments
Highlights- Motivated by previous work, we designed and implemented three hand gesture interfaces for virtual locomotion.
AbstractHand gesture is a new and promising interface for locomotion in virtual environments. While several previous studies have proposed different hand gestures for virtual locomotion, little is known about their differences in terms of ...
Comments