Yu Zhang
Chu Luo
Vassilis Kostakos
Skip Abstract Section
Abstract
We present FinDroidHR, a novel gesture input technique for off-the-shelf smartwatches. Our technique is designed to detect 10 hand gestures on the hand wearing a smartwatch. The technique is enabled by analysing features of the Photoplethysmography (PPG) signal that optical heart-rate sensors capture. In a study with 20 participants, we show that FinDroidHR achieves 90.55% accuracy and 90.73% recall. Our work is the first study to explore the feasibility of using optical sensors on the off-the-shelf wearable devices to recognise gestures. Without requiring bespoke hardware, FinDroidHR can be readily used on existing smartwatches.
- Gregory D Abowd. 2012. What next, ubicomp?: celebrating an intellectual disappearing act. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing. ACM, 31--40. Google Scholar
Digital Library
- Nasimuddin Ahmed, Rohan Banerjee, Avik Ghose, and Arijit Sinharay. 2015. Feasibility Analysis for Estimation of Blood Pressure and Heart Rate using a Smart Eye Wear. In Proc. WearSys 2015. ACM Press, 9--14. Google ScholarDigital Library
- Nazneen Akhter, Sumegh Tharewal, Vijay Kale, Ashish Bhalerao, and K.V. Kale. 2016. Heart-Based Biometrics and Possible Use of Heart Rate Variability in Biometric Recognition Systems. Advances in Intelligent Systems and Computing 395 (2016), 15--29.Google ScholarCross Ref
- Daniel Ashbrook, Kent Lyons, and Thad Starner. 2008. An Investigation into Round Touchscreen Wristwatch Interaction. In Proc. MobileHCI 2008. ACM Press, 311--314. Google ScholarDigital Library
- IEEE Standards Association. 2003. IEEE Standard for Transitions, Pulses, and Related Waveforms. In IEEE Std. 181-2003. IEEE Press.Google Scholar
- Hamed Azami, Karim Mohammadi, and Behzad Bozorgtabar. 2012. An Improved Signal Segmentation Using Moving Average and Savitzky-Golay Filter. Journal of Signal and Information Processing 3 (2012), 39--44.Google ScholarCross Ref
- Nick Barnes. 2005. Improved Signal To Noise Ratio And Computational Speed For Gradient-Based Detection Algorithms. In International Conference on Robotics and Automation 2005. IEEE Press, 4661--4667.Google Scholar
- Patrick Baudisch and Gerry Chu. 2009. Back-of-Device Interaction Allows Creating Very Small Touch Devices. In Proc. CHI 2009. ACM Press, 1923--1932. Google ScholarDigital Library
- Jorge Blasco, Thomas M. Chen, Juan Tapiador, and Pedro Peris-Lopez. 2016. A Survey of Wearable Biometric Recognition Systems. In Proc. CSUR 2016. ACM Press.Google ScholarDigital Library
- Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y. Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: Private and Subtle Interaction Using Fingertips. In Proc. UIST 2013. ACM Press, 255--260. Google ScholarDigital Library
- Jikai Chen, Yanhui Dou, Yang Li, and Jiang Li. 2016. Application of Shannon Wavelet Entropy and Shannon Wavelet Packet Entropy in Analysis of Power System Transient Signals. Entropy 2016 18 (2016), 14.Google Scholar
- Ke-Yu Chen, Kent Lyons, Sean White, and Shwetak Patel. 2013. uTrack: 3D Input Using Two Magnetic Sensors. In Proc. UIST 2013. ACM Press, 237--244. Google ScholarDigital Library
- Sangita Das, Saurabh Pal, and Madhuchhanda Mitra. 2016. Real Time Heart Rate Detection from PPG Signal in Noisy Environment. In Proc. ICICPI 2016. IEEE Press, 70--73.Google ScholarCross Ref
- Artem Dementyev and Joseph A. Paradiso. 2014. WristFlex: Low-Power Gesture Input with Wrist-Worn Pressure Sensors. In Proc. UIST 2014. ACM Press, 161--166. Google ScholarDigital Library
- Tizen Developers. 2017. Heart Rate Monitor LED Green Sensor. (2017). https://developer.tizen.org/development/guides/native-application/location-and-sensors/device-sensors#hrm_irGoogle Scholar
- Florian Grutzmacher, Johann-Peter Wolff, and Christian Haubelt. 2015. Exploiting Thread-Level Parallelism in Template-Based Gesture Recognition with Dynamic Time Warping. In Proc. WOAR 2015. ACM Press, 6. Google ScholarDigital Library
- Sidhant Gupta, Dan Morris, Shwetak N Patel, and Desney Tan. 2012. SoundWave: Using the Doppler Effect to Sense Gestures. In Proc. CHI 2012. ACM Press, 1911--1914. Google ScholarDigital Library
- Da-Yuan Huang, Liwei Chan, Shuo Yang, Fan Wang, Rong-Hao Liang, De-Nian Yang, Yi-Ping Hung, and Bing-Yu Chen. 2016. DigitSpace: Designing Thumb-to-Fingers Touch Interfaces for One-Handed and Eyes-Free Interactions. In Proc. CHI 2016. ACM Press, 1526--1537. Google ScholarDigital Library
- Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing gesture recognition to all devices. In Proc. NSDI 2014. ACM Press, 303--316. Google ScholarDigital Library
- Frederic Kerber, Markus Lochtefeld, Antonio Kruger, Jess Mcintosh, Charlie McNeill, and Mike Fraser. 2016. Understanding Same-Side Interactions with Wrist-Worn Devices. In Proc. NordiCHI 2016. ACM Press. Google ScholarDigital Library
- Jungsoo Kim, Jiasheng He, Kent Lyons, and Thad Starner. 2007. The Gesture Watch: A Wireless Contact-free Gesture based Wrist Interface. In Proc. ISWC 2007. IEEE Press, 1--8. Google ScholarDigital Library
- Min Sik Kim, Taekhyun Kim, Yong-June Shin, Simon S. Lam, and Edward J. Powers. 2008. A wavelet-based approach to detect shared congestion. In IEEE/ACM Trans. Netw 2008. IEEE Press, 763--776. Google ScholarDigital Library
- Vassilis Kostakos and Mirco Musolesi. 2017. Avoiding Pitfalls When Using Machine Learning in HCI Studies. INTERACTIONS (2017). Google ScholarDigital Library
- Sven Kratz and Michael Rohs. MobileHCI. Hoverflow: Exploring Around-Device Interaction with IR Distance Sensors. In Proc. MobileHCI 2009. ACM Press, 4. Google ScholarDigital Library
- Gierad Laput, Robert Xiao, Xiang Anthony' Chen, Scott E. Hudson, and Chris Harrison. 2014. Skin Buttons: Cheap, Small, Low-Power and Clickable Fixed-Icon Laser Projections. In Proc. UIST 2014. ACM Press, 389--394. Google ScholarDigital Library
- YANN LECUN, LEON BOTTOU, YOSHUA BENGIO, and PATRICK HAFFNER. 1998. Gradient-Based Learning Applied to Document Recognition. In Proc. IEEE 1998. IEEE Press, 2278--2324.Google Scholar
- Zhiyuan Lu, Xiang Chen, Qiang Li, Xu Zhang, and Ping Zhou. 2014. A Hand Gesture Recognition Framework and Wearable Gesture-Based Interaction Prototype for Mobile Devices. In Proc. HUMAN-MACHINE SYSTEMS 2014. IEEE Press, 293--299.Google ScholarCross Ref
- Yuka Maeda, Masaki Sekine, and Toshiyo Tamura. 2011. Relationship Between Measurement Site and Motion Artifacts in Wearable Reflected Photoplethysmography. Journal of Medical Systems 39, 5(2011), 969--976. Google ScholarDigital Library
- HENRY MARTIN, PAUL GROVES, and MARK NEWMAN. 2016. The Limits of In-Run Calibration of MEMS Inertial Sensors and Sensor Arrays. NAVIGATION: Journal of The Institute of Navigation 63 (2016).Google Scholar
- Jess Mcintosh, Asier Marzo, Mike Fraser, and Carol Phillips. 2017. EchoFlex: Hand Gesture Recognition using Ultrasound Imaging. In Proc. CHI 2017. ACM Press. Google ScholarDigital Library
- Kei Nakatsuma, Hiroyuki Shinoda, Yasutoshi Makino, Katsunari Sato, and Takashi Maeno. 2011. Touch Interface on Back of the Hand. In Proc. SIGGRAPH 2011. ACM Press, 1. Google ScholarDigital Library
- Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. In Proc. CHI 2016. ACM Press, 1515--1525. Google ScholarDigital Library
- Simon T. Perrault, Eric Lecolinet, James Eagan, and Yves Guiard. 2013. WatchIt: Simple Gestures and Eyes-free Interaction for Wristwatches and Bracelets. In Proc. CHI 2013. ACM Press, 1451--1460. Google ScholarDigital Library
- Raj Rakshit, V Ramu Reddy, and Parijat Deshpande. 2016. Emotion Detection and Recognition using HRV Features Derived from Photoplethysmogram Signals. In Proc. ERM4CT 2016. ACM Press. Google ScholarDigital Library
- Marko Robnik-Sikonja and Igor Kononenko. 2003. Theoretical and Empirical Analysis of ReliefF and RReliefF. In Machine Learning, 53, 2003. 23--69. Google ScholarDigital Library
- T. Scott Saponas, Chris Harrison, and Hrvoje Benko. 2011. PocketTouch: Through-Fabric Capacitive Touch Input. In Proc. UIST 2011. ACM Press, 303--308. Google ScholarDigital Library
- T. Scott Saponas, Desney S. Tan, Dan Morris, Jim Turner, and James A. Landay. 2010. Making Muscle-Computer Interfaces More Practical. In Proc. CHI 2010. ACM Press, 851--854. Google ScholarDigital Library
- Mads Soegaard and Rikke Friis Dam. 2013. The Encyclopedia of Human-Computer Interaction, 2nd Ed. (2nd ed.). The Interaction Design Foundation.Google Scholar
- Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. WiDraw: Enabling Hands-free Drawing in the Air on Commodity WiFi Devices. In Proc. Mobicom 2015. ACM Press, 77--89. Google ScholarDigital Library
- S.K. Deric Tang, Y.Y. Sebastian Goh, M.L. Dennis Wong, and Y.L. Eileen Lew. 2016. PPG signal reconstruction using a combination of discrete wavelet transform and empirical mode decomposition. In Proc. ICIAS 2016. IEEE Press.Google Scholar
- H. Emrah Tasli, Amogh Gudi, and Marten den Uyl. 2014. Integrating Remote PPG in Facial Expression Analysis Framework. In Proc. ICMI 2014. ACM Press, 74--75. Google ScholarDigital Library
- Ihor Vasyltsov and Seunghwan Lee. 2015. Entropy Extraction from Bio-Signals in Healthcare IoT. In Proc. IoTPTS 2015. ACM Press, 11--17. Google ScholarDigital Library
- Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jurgen Steimle. 2015. iSkin: Flexible, Stretchable and Visually Customizable On-Body Touch Sensors for Mobile Computing. In Proc. CHI 2015. ACM Press, 2991--3000. Google ScholarDigital Library
- Martin Weigel, Aditya Shekhar, Nittala1 Alex Olwal, and Jurgen Steimle. 2017. SkinMarks: Enabling Interactions on Body Landmarks Using Conformal Skin Electronics. In Proc. CHI 2017. ACM Press, 3095--3105. Google ScholarDigital Library
- Hongyi Wen, Julian Ramos Rojas, and Anind K. Dey. 2016. Serendipity: Finger Gesture Recognition using an Off-the-Shelf Smartwatch. In Proc. CHI 2016. ACM Press, 3847--3851. Google ScholarDigital Library
- Robert Xiao, Gierad Laput, and Chris Harrison. 2014. Expanding the Input Expressivity of Smartwatches with Mechanical Pan, Twist, Tilt and Click. In Proc. CHI 2014. ACM Press, 193--196. Google ScholarDigital Library
- Liu Xing and Qian Feng. 2016. Measuring and optimizing android smartwatch energy consumption: poster. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 421--423. Google ScholarDigital Library
- Chao Xu, Parth H. Pathak, and Prasant Mohapatra. 2014. Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition using Smartwatch. In Proc. HotMobile 2015. ACM Press. Google ScholarDigital Library
- Yalan Ye, Wenwen He, Yunfei Cheng, Wenxia Huang, and Zhilin Zhang. 2017. A Robust Random Forest-Based Approach for Heart Rate Monitoring Using Photoplethysmography Signal Contaminated by Intense Motion Artifacts. MDPI 385 (2017).Google Scholar
- Xiaobo Zhang and Xiangchu Feng. 2014. A New Gradient-based Nonlinear Diffusion Method Applied to Image Denoising. In Proc. PIC 2014. IEEE Press, 321--315.Google ScholarCross Ref
- Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016. SkinTrack: Using the Body as an Electrical Waveguide for Continuous Finger Tracking on the Skin. In Proc. CHI 2016. ACM Press, 1491--1503. Google ScholarDigital Library
Index Terms
- FinDroidHR: Smartwatch Gesture Input with Optical Heartrate Monitor
Recommendations
fSense: Unlocking the Dimension of Force for Gestural Interactions using Smartwatch PPG Sensor
AH2019: Proceedings of the 10th Augmented Human International Conference 2019While most existing gestural interfaces focus on the static posture or the dynamic action of the hand, few have investigated the feasibility of using the forces that are exerted while performing gestures. Using the photoplethysmogram (PPG) sensor of off-...
Hands-free gesture control with a capacitive textile neckband
ISWC '14: Proceedings of the 2014 ACM International Symposium on Wearable ComputersWe present a novel sensing modality for hands-free gesture controlled user interfaces, based on active capacitive sensing. Four capacitive electrodes are integrated into a textile neckband, allowing continuous unobtrusive head movement monitoring. We ...
Aerobic activity monitoring: towards a long-term approach
With recent progress in wearable sensing, it becomes reasonable for individuals to wear different sensors all day, and thus, global activity monitoring is establishing. The goals in global activity monitoring systems are among others to tell the type of ...
Comments