Abstract
This paper explores the possibility of extending the input and interactions beyond the small screen of the mobile device onto ad hoc adjacent surfaces, e.g., a wooden tabletop with acoustic signals. While the existing finger tracking approaches employ the active acoustic signal with a fixed frequency, our proposed system Ipanel employs the acoustic signals generated by sliding of fingers on the table for tracking. Different from active signal tracking, the frequency of the finger-table generated acoustic signals keeps changing, making accurate tracking much more challenging than the traditional approaches with fix frequency signal from the speaker. Unique features are extracted by exploiting the spatio-temporal and frequency domain properties of the generated acoustic signals. The features are transformed into images and then we employ the convolutional neural network (CNN) to recognize the finger movement on the table. Ipanel is able to support not only commonly used gesture (click, flip, scroll, zoom, etc.) recognition, but also handwriting (10 numbers and 26 alphabets) recognition at high accuracies. We implement Ipanel on smartphones, and conduct extensive real environment experiments to evaluate its performance. The results validate the robustness of Ipanel, and show that it maintains high accuracies across different users with varying input behaviours (e.g., input strength, speed and region). Further, Ipanel's performance is robust against different levels of ambient noise and varying surface materials.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, Your Table Can Be an Input Panel: Acoustic-based Device-Free Interaction Recognition
- 2015. AN580: Ifrared gesture recognition by Silicon Labs. https://www.silabs.com/Support%20Documents/TechnicalDocs/AN580.pdfGoogle Scholar
- 2016. Google Project Soli. https://atap.google.com/soli/Google Scholar
- 2017. Kinect for Xbox One. https://www.xbox.com/en-US/xbox-one/accessories/kinect-for-xbox-oneGoogle Scholar
- 2017. Leap Motion. https://www.leapmotion.com/Google Scholar
- Haikal El Abed and Volker Mirgner. 2011. ICDAR 2009-Arabic handwriting recognition competition. In International Journal on Document Analysis and Recognition. 3--13. Google ScholarDigital Library
- Fadel Adib and Dina Katabi. 2013. See through walls with WiFi!. In Proceedings of the 2013 ACM conference on SIGCOMM. ACM. Google ScholarDigital Library
- Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Romit Roy Choudhury Kevin Caves, and Frank DeRuyter. 2011. Using mobile phones to write in air. In Proceedings of the 9th international conference on Mobile systems, applications, and services (MobiSys). ACM, 15--28. Google ScholarDigital Library
- Adnan Akay. 2002. Acoustics of friction. The Journal of the Acoustical Society of America 111, 4 (2002), 1525--1548.Google ScholarCross Ref
- J AlmazÍćn, A Gordo, A FornÍęs, and E Valveny. 2014. Word Spotting and Recognition with Embedded Attributes. In Pattern Analysis and Machine Intelligence IEEE Transactions on. 2552--2566.Google Scholar
- Sherif Abdel Azeem and Hany Ahmed. 2013. Effective technique for the recognition of offline Arabic handwritten words using hidden Markov models. In International Journal on Document Analysis and Recognition. 399--412. Google ScholarDigital Library
- Anne Laure Biannebernard. 2012. The A2iA French handwriting recognition system at the Rimes-ICDAR2011 competition. In Document Recognition and Retrieval XIX. 51.Google Scholar
- ThÍęodore Bluche, Hermann Ney, and Christopher Kermorvant. 2014. A Comparison of Sequence-Trained Deep Neural Networks and Recurrent Neural Networks Optical Modeling for Handwriting Recognition. In International Conference on Statistical Language and Speech Processing. 199--210.Google ScholarCross Ref
- Andreas Braun, Stefan Krepp, and Arjan Kuijper. 2015. Acoustic tracking of hand activities on surfaces. In Proceedings of the 2nd international Workshop on Sensor-based Activity Recognition and Interaction. ACM, 9. Google ScholarDigital Library
- Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: private and subtle interaction using fingertips. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 255--260. Google ScholarDigital Library
- Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013. uTrack: 3D input using two magnetic sensors. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 237--244. Google ScholarDigital Library
- Ke-Yu Chen, Shwetak Patel, and Sean Keller 2016. Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1504--1514. Google ScholarDigital Library
- Tiffany Yu-Han Chen, Lenin Ravindranath, Shuo Deng, Paramvir Bahl, and Hari Balakrishnan. 2015. Glimpse: Continuous, Real-Time Object Recognition on Mobile Devices. In Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems (SenSys). ACM, 155--168. Google ScholarDigital Library
- Patrick Doetsch, Michal Kozielski, and Hermann Ney. 2014. Fast and Robust Training of Recurrent Neural Networks for Offline Handwriting Recognition. In International Conference on Frontiers in Handwriting Recognition. 279--284.Google Scholar
- J. G Fiscus. 1997. A post-processing system to yield reduced word error rates: Recognizer Output Voting Error Reduction (ROVER). In IEEE Workshop on Automatic Speech Recognition and Understanding. 347--354.Google ScholarCross Ref
- Luo Gan, Chen Mingshi, Yang Panlong, and Li. Ping. 2017. SoundWrite II: Ambient Acoustic Sensing for Noise Tolerant Device-Free Gesture Recognition. In ICPAD.Google Scholar
- Mayank Goel, Brendan Lee, Md Tanvir Islam Aumi, Shwetak Patel, Gaetano Borriello, Stacie Hibino, and Bo Begole. 2014. SurfaceLink: using inertial and acoustic sensing to enable multi-device interaction on a surface. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 1387--1396. Google ScholarDigital Library
- Jeremy Gummeson, Bodhi Priyantha, and Jie Liu. 2014. An energy harvesting wearable ring platform for gesture input on surfaces. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services (MobiSys). ACM, 162--175. Google ScholarDigital Library
- Du Haishi, Yang Panlong, luo Gan, and Li. Ping. 2017. WordRecorder: Accurate Acoustic-based Handwriting Recognition Using Deep Learning. In Infocom.Google Scholar
- Chris Harrison and Scott E Hudson. 2008. Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces. In Proceedings of the 21st annual ACM symposium on User interface software and technology (UIST). ACM, 205--208. Google ScholarDigital Library
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition. 770--778.Google Scholar
- Wenchao Huang, Yan Xiong, Xiang-Yang Li, Hao Lin, Xufei Mao, Panlong Yang, and Yunhao Liu. 2014. Shake and walk: acoustic direction finding and fine-grained indoor localization using smartphones. In INFOCOM, 2014 Proceedings IEEE. IEEE, 370--378.Google ScholarCross Ref
- CEI IEC. 1985. Integrating-averaging sound level meters. (1985).Google Scholar
- Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing gesture recognition to all devices. In Usenix NSDI, Vol. 14. Google ScholarDigital Library
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In International Conference on Neural Information Processing Systems. 1097--1105. Google ScholarDigital Library
- Patrick Lazik and Anthony Rowe. 2012. Indoor pseudo-ranging of mobile devices using ultrasonic chirps. In Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems (SenSys). ACM, 99--112. Google ScholarDigital Library
- Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. In Proceedings of the IEEE. 2278--2324.Google Scholar
- Youngki Lee, Chulhong Min, Chanyou Hwang, Jaeung Lee, Inseok Hwang, Younghyun Ju, Chungkuk Yoo, Miri Moon, Uichin Lee, and Junehwa Song. 2013. Sociophone: Everyday face-to-face interaction monitoring platform using multi-phone sensor fusion. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services (MobiSys). ACM, 375--388. Google ScholarDigital Library
- Tianxing Li, Chuankai An, Zhao Tian, Andrew T Campbell, and Xia Zhou. 2015. Human sensing using visible light communication. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom). ACM, 331--344. Google ScholarDigital Library
- Jian Liu, Yan Wang, Gorkem Kar, Yingying Chen, Jie Yang, and Marco Gruteser. 2015. Snooping Keystrokes with mm-level Audio Ranging on a Single Phone. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom). ACM, 142--154. Google ScholarDigital Library
- Pedro Lopes, Ricardo Jota, and Joaquim A Jorge. 2011. Augmenting touch interaction through acoustic sensing. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces. ACM, 53--56. Google ScholarDigital Library
- Wenguang Mao, Jian He, and Lili Qiu. 2016. CAT: high-precision acoustic motion tracking. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 69--81. Google ScholarDigital Library
- Raafat Salih Muhammad and Mohammed Issam Younis. 2017. The Limitation of Pre-processing Techniques to Enhance the Face Recognition System Based on LBP. Iraqi Journal of Science 58, 581B (2017), 355--363.Google Scholar
- Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1515--1525. Google ScholarDigital Library
- Shahriar Nirjon, Robert F Dickerson, Qiang Li, Philip Asare, John A Stankovic, Dezhi Hong, Ben Zhang, Xiaofan Jiang, Guobin Shen, and Feng Zhao. 2012. Musicalheart: A hearty way of listening to music. In Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems (SenSys). ACM, 43--56. Google ScholarDigital Library
- Shahriar Nirjon, Jeremy Gummeson, Dan Gelb, and Kyu-Han Kim. 2015. TypingRing: A Wearable Ring Platform for Text Input. In Proceedings of the 9th international conference on Mobile systems, applications, and services (MobiSys). ACM. Google ScholarDigital Library
- Joseph A Paradiso, Che King Leo, Nisha Checka, and Kaijen Hsiao. 2002. Passive acoustic sensing for tracking knocks atop large interactive displays. In Sensors, 2002. Proceedings of IEEE, Vol. 1. IEEE, 521--527.Google ScholarCross Ref
- Taiwoo Park, Jinwon Lee, Inseok Hwang, Chungkuk Yoo, Lama Nachman, and Junehwa Song. 2011. E-gesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices. In Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems (SenSys). ACM, 260--273. Google ScholarDigital Library
- Arik Poznanski and Lior Wolf. 2016. CNN-N-Gram for HandwritingWord Recognition. In Computer Vision and Pattern Recognition. 2305--2314.Google Scholar
- Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th annual international conference on Mobile computing & networking (MobiCom). ACM, 27--38. Google ScholarDigital Library
- James M Rehg and Takeo Kanade. 1994. Visual tracking of high dof articulated structures: an application to human hand tracking. In Computer VisionąłECCV'94. Springer, 35--46. Google ScholarDigital Library
- Wenjie Ruan, Quan Z Sheng, Lei Yang, Tao Gu, Peipei Xu, and Longfei Shangguan. 2016. AudioGest: enabling fine-grained hand gesture detection by decoding echo signal. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 474--485. Google ScholarDigital Library
- James She, Alvin Chin, Feng Xia, and Jon Crowcroft. 2015. Introduction to: Special Issue on Smartphone-Based Interactive Technologies, Systems, and Applications. ACM Transactions on Multimedia Computing, Communications, and Applications 12, 1s (2015), 11. Google ScholarDigital Library
- Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Computer Science.Google Scholar
- Zheng Sun, Aveek Purohit, Raja Bose, and Pei Zhang. 2013. Spartacus: spatially-aware interaction for mobile devices through energy-efficient audio sensing. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services (MobiSys). ACM, 263--276. Google ScholarDigital Library
- Zheng Sun, Aveek Purohit, Kaifei Chen, Shijia Pan, Trevor Pering, and Pei Zhang. 2011. PANDAA: physical arrangement detection of networked devices through ambient-sound awareness. In Proceedings of the 13th international conference on Ubiquitous computing (Ubicomp). ACM, 425--434. Google ScholarDigital Library
- Sanjib Sur, Teng Wei, and Xinyu Zhang. 2014. Autodirective audio capturing through a synchronized smartphone array. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services (MobiSys). ACM, 28--41. Google ScholarDigital Library
- Yu-Chih Tung and Kang G Shin. 2015. EchoTag: Accurate Infrastructure-Free Indoor Location Tagging with Smartphones. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom). ACM, 525--536. Google ScholarDigital Library
- Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RF-IDraw: virtual touch screen in the air using RF signals. In Proceedings of the 2014 ACM conference on SIGCOMM. ACM, 235--246. Google ScholarDigital Library
- Junjue Wang, Kaichen Zhao, Xinyu Zhang, and Chunyi Peng. 2014. Ubiquitous keyboard for small mobile devices: harnessing multipath fading for fine-grained keystroke localization. In Proceedings of the 12th annual international conference on Mobile systems, applications, and services (MobiSys). ACM, 14--27. Google ScholarDigital Library
- Wei Wang, Alex X Liu, and Ke Sun. 2016. Device-free gesture tracking using acoustic signals. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 82--94. Google ScholarDigital Library
- Robert Xiao, Greg Lew, James Marsanico, Divya Hariharan, Scott Hudson, and Chris Harrison. 2014. Toffee: enabling ad hoc, around-device interaction with acoustic time-of-arrival correlation. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services. ACM, 67--76. Google ScholarDigital Library
- Jie Xiong and Kyle Jamieson. 2013. ArrayTrack: a fine-grained indoor location system. In USENIX NSDI. Google ScholarDigital Library
- Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning a Mobile Device into a Mouse in the Air. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys). ACM, 15--29. Google ScholarDigital Library
- Cheng Zhang, Anhong Guo, Dingtian Zhang, Yang Li, Caleb Southern, Rosa I Arriaga, and Gregory D Abowd. 2016. Beyond the Touchscreen: An Exploration of Extending Interactions on Commodity Smartphones. ACM Transactions on Interactive Intelligent Systems (TiiS) 6, 2 (2016), 16. Google ScholarDigital Library
- Zengbin Zhang, David Chu, Xiaomeng Chen, and Thomas Moscibroda. 2012. SwordFight: enabling a new class of phone-to-phone action games on commodity phones. In Proceedings of the 10th international conference on Mobile systems, applications, and services (MobiSys). ACM, 1--14. Google ScholarDigital Library
- Tong Zhu, Qiang Ma, Shanfeng Zhang, and Yunhao Liu. 2014. Context-free Attacks Using Keyboard Acoustic Emanations. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 453--464. Google ScholarDigital Library
Index Terms
- Your Table Can Be an Input Panel: Acoustic-based Device-Free Interaction Recognition
Recommendations
Mobile App Classification Method Using Machine Learning Based User Emotion Recognition
ICICM '18: Proceedings of the 8th International Conference on Information Communication and ManagementIn this paper, we propose a convolutional neural network based application method which shows superior performance in image classification. Recently, various requirements such as emotional UI, rather than the existing Touch UI method, have been ...
Learning contrastive feature distribution model for interaction recognition
We introduce an intra-inter-frame skeleton feature for interaction description.We learn CFDM for a discriminative representation of interactions.We capture a new database of interactions, CR-UESTC.We evaluate our proposed CFDM approach on CR-UESTC and ...
Scratch input: creating large, inexpensive, unpowered and mobile finger input surfaces
UIST '08: Proceedings of the 21st annual ACM symposium on User interface software and technologyWe present Scratch Input, an acoustic-based input technique that relies on the unique sound produced when a fingernail is dragged over the surface of a textured material, such as wood, fabric, or wall paint. We employ a simple sensor that can be easily ...
Comments