ABSTRACT
We present CapBand, a battery-free hand gesture recognition wearable in the form of a wristband. The key challenges in creating such a system are (1) to sense useful hand gestures at ultra-low power so that the device can be powered by the limited energy harvestable from the surrounding environment and (2) to make the system work reliably without requiring training every time a user puts on the wristband. We present successive capacitance sensing, an ultra-low power sensing technique, to capture small skin deformations due to muscle and tendon movements on the user's wrist, which corresponds to specific groups of wrist muscles representing the gestures being performed. We build a wrist muscles-to-gesture model, based on which we develop a hand gesture classification method using both motion and static features. To eliminate the need for per-usage training, we propose a kernel-based on-wrist localization technique to detect the CapBand's position on the user's wrist. We prototype CapBand with a custom-designed capacitance sensor array on two flexible circuits driven by a custom-built electronic board, a heterogeneous material-made, deformable silicone band, and a custom-built energy harvesting and management module. Evaluations on 20 subjects show 95.0% accuracy of gesture recognition when recognizing 15 different hand gestures and 95.3% accuracy of on-wrist localization.
- M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. Tensorflow: A system for large-scale machine learning. In OSDI, volume 16, pages 265--283, 2016. Google ScholarDigital Library
- B. Abbott and D. Wilkie. The relation between velocity of shortening and the tension-length curve of skeletal muscle. The Journal of Physiology, 120(1-2):214--223, 1953.Google ScholarCross Ref
- S. Agarwal, A. Mondal, G. Joshi, and G. Gupta. Gestglove: A wearable device with gesture based touchless interaction. In Proc. of the 8th Augmented Human International Conference, AH '17, pages 3:1--3:8, 2017. Google ScholarDigital Library
- C. Alippi and C. Galperti. An adaptive system for optimal solar energy harvesting in wireless sensor network nodes. IEEE Transactions on Circuits and Systems I: Regular Papers, 55(6):1742--1750, 2008.Google ScholarCross Ref
- ArcSoft. Arcsoft gesture recognition technology. https://goo.gl/bJmRea. {Online; accessed Apr 01, 2017}.Google Scholar
- K. Braam, T. C. Huang, C. H. Chen, E. Montgomery, S. Vo, and R. Beausoleil. Wristband vital: A wearable multi-sensor microsystem for real-time assistance via low-power bluetooth link. In 2015 IEEE 2nd World Forum on Internet of Things (WF-IoT), pages 87--91, Dec 2015. Google ScholarDigital Library
- Broadcom. Hsms-285c detector diode. https://goo.gl/iu3AZY. {Online; accessed Apr 05, 2018}.Google Scholar
- T. S. Buchanan, D. G. Lloyd, K. Manal, and T. F. Besier. Neuromusculoskeletal modeling: estimation of muscle forces and joint moments and movements from measurements of neural command. Journal of applied biomechanics, 20(4):367--395, 2004.Google Scholar
- T. S. Buchanan, M. J. Moniz, J. P. Dewald, and W. Z. Rymer. Estimation of muscle forces about the wrist joint during isometric tasks using an emg coefficient method. Journal of biomechanics, 26(4-5):547--560, 1993.Google ScholarCross Ref
- K.-Y. Chen, S. N. Patel, and S. Keller. Finexus: Tracking precise motions of multiple fingertips using magnetic sensing. In ACM CHI, pages 1504--1514, 2016. Google ScholarDigital Library
- C. Cornelius, R. Peterson, J. Skinner, R. Halter, and D. Kotz. A wearable system that knows who wears it. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services, MobiSys '14, pages 55--67, 2014. Google ScholarDigital Library
- A. Dementyev and J. A. Paradiso. Wristflex: Low-power gesture input with wrist-worn pressure sensors. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST '14, pages 161--166, 2014. Google ScholarDigital Library
- A. Dementyev and J. A. Paradiso. Wristflex: Low-power gesture input with wrist-worn pressure sensors. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST '14, pages 161--166, New York, NY, USA, 2014. ACM. Google ScholarDigital Library
- Y. Du, W. Jin, W. Wei, Y. Hu, and W. Geng. Surface emg-based inter-session gesture recognition enhanced by deep domain adaptation. Sensors, 17(3), 2017.Google Scholar
- EnOcean. Pushbutton transmitter module. https://goo.gl/oiHnVj. {Online; accessed Apr 05, 2018}.Google Scholar
- FCC. Fcc codes of regulation, part 15. https://goo.gl/8mcbdi. {Online; accessed Apr 05, 2018}.Google Scholar
- A. Ferrone, X. Jiang, L. Maiolo, A. Pecora, L. Colace, and C. Menon. A fabric-based wearable band for hand gesture recognition based on filament strain sensors: A preliminary investigation. In 2016 IEEE Healthcare Innovation Point-Of-Care Technologies Conference (HI-POCT), pages 113--116, Nov 2016.Google ScholarCross Ref
- A. Ferrone, F. Maita, L. Maiolo, M. Arquilla, A. Castiello, A. Pecora, X. Jiang, C. Menon, A. Ferrone, and L. Colace. Wearable band for hand gesture recognition based on strain sensors. In 2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob), pages 1319--1322, 2016.Google ScholarCross Ref
- A. E. Flatt. Grasp. Proceedings (Baylor University. Medical Center), 13(4):343, 2000.Google Scholar
- R. Fukui, M. Watanabe, T. Gyota, M. Shimosaka, and T. Sato. Hand shape classification with a wrist contour sensor: Development of a prototype device. In Proceedings of the 13th International Conference on Ubiquitous Computing, UbiComp 11, pages 311--314, 2011. Google ScholarDigital Library
- J. Gong, X.-D. Yang, and P. Irani. Wristwhirl: One-handed continuous smartwatch input using wrist gestures. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST '16, pages 861--872, New York, NY, USA, 2016. ACM. Google ScholarDigital Library
- T. Grosse-Puppendahl, C. Holz, G. Cohn, R. Wimmer, O. Bechtold, S. Hodges, M. S. Reynolds, and J. R. Smith. Finding common ground: A survey of capacitive sensing in human-computer interaction. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 3293--3315. ACM, 2017. Google ScholarDigital Library
- GSA. Lighting condition. https://goo.gl/2SfKq4. {Online; accessed Apr 05, 2018}.Google Scholar
- J. L. Hernandez-Rebollar, N. Kyriakopoulos, and R. W. Lindeman. The Accele-Glove: A whole-hand input device for virtual reality. In SIGGRAPH, page 259, 2002. Google ScholarDigital Library
- Y. Huang, W. Guo, J. Liu, J. He, H. Xia, X. Sheng, H. Wang, X. Feng, and P. B. Shull. Preliminary Testing of a Hand Gesture Recognition Wristband Based on EMG and Inertial Sensor Fusion, pages 359--367. Springer International Publishing, Cham, 2015. Google ScholarDigital Library
- T. Instruments. Bluetooth low energy cc2650 module. https://goo.gl/Ysc2dL. {Online; accessed Apr 01, 2017}.Google Scholar
- T. Instruments. Bq25570. https://goo.gl/XV8BzM. {Online; accessed Apr 05, 2018}.Google Scholar
- T. Instruments. Chip msp430g2553. https://goo.gl/szRjHL. {Online; accessed Apr 01, 2017}.Google Scholar
- T. Instruments. Multiplexer cd74hc4067. https://goo.gl/8KQzSm. {Online; accessed Apr 01, 2017}.Google Scholar
- S. Jiang, B. Lv, X. Sheng, C. Zhang, H. Wang, and P. B. Shull. Development of a real-time hand gesture recognition wristband based on semg and imu sensing. In 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 1256--1261, Dec 2016.Google ScholarCross Ref
- X. Jiang, L.-K. Merhi, Z. G. Xiao, and C. Menon. Exploration of force myography and surface electromyography in hand gesture classification. Medical Engineering & Physics, 41(Supplement C):63--73, 2017.Google Scholar
- P. G. Jung, G. Lim, S. Kim, and K. Kong. A wearable gesture recognition device for detecting muscular activities based on air-pressure sensors. IEEE Transactions on Industrial Informatics, 11(2):485--494, 2015.Google Scholar
- K. Kadirvel, Y. Ramadass, U. Lyles, J. Carpenter, V. Ivanov, V. McNeil, A. Chandrakasan, and B. Lum-Shue-Chan. A 330na energy-harvesting charger with battery management for solar and thermoelectric energy harvesting. In Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 2012 IEEE International, pages 106--108. IEEE, 2012.Google ScholarCross Ref
- N. Karlsson, B. Karlsson, and P. Wide. A glove equipped with finger flexion sensors as a command generator used in a fuzzy control system. In IMTC/98 Conference Proceedings. IEEE Instrumentation and Measurement Technology Conference. Where Instrumentation is Going (Cat. No.98CH36222), volume 1, pages 441--445, 1998.Google ScholarCross Ref
- J. M. Kauer, A. de Lange, H. H. Savelberg, and J. G. Kooloos. The wrist joint: Functional analysis and experimental approach. In Wrist Disorders, pages 3--12. Springer, 1992.Google ScholarCross Ref
- F. Kerber, M. Puhl, and A. Krüger. User-independent real-time hand gesture recognition based on surface electromyography. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, MobileHCI '17, pages 36:1--36:7, 2017. Google ScholarDigital Library
- D. Kim, O. Hilliges, S. Izadi, A. D. Butler, J. Chen, I. Oikonomidis, and P. Olivier. Digits: Freehand 3d interactions anywhere using a wrist-worn gloveless sensor. In Proc. of the 25th Annual ACM Symposium on User Interface Software and Technology, UIST '12, pages 167--176, 2012. Google ScholarDigital Library
- S. Kim, R. Vyas, J. Bito, K. Niotaki, A. Collado, A. Georgiadis, and M. M. Tentzeris. Ambient rf energy-harvesting technologies for self-sustainable standalone wireless sensor platforms. Proceedings of the IEEE, 102(11):1649--1666, 2014.Google ScholarCross Ref
- A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097--1105, 2012. Google ScholarDigital Library
- G. Laput, R. Xiao, and C. Harrison. Viband: High-fidelity bio-acoustic sensing using commodity smartwatch accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST '16, pages 321--333, 2016. Google ScholarDigital Library
- T. Li, X. Xiong, Y. Xie, G. Hito, X.-D. Yang, and X. Zhou. Reconstructing hand poses using visible light. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 1(3):71:1--71:20, 2017. Google ScholarDigital Library
- J.-W. Lin, C. Wang, Y. Y. Huang, K.-T. Chou, H.-Y. Chen, W.-L. Tseng, and M. Y. Chen. Backhand: Sensing hand gestures via back of the hand. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, UIST '15, pages 557--564, 2015. Google ScholarDigital Library
- J. McIntosh, A. Marzo, and M. Fraser. Sensir: Detecting hand gestures with a wearable bracelet using infrared transmission and reflection. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST '17, pages 593--597, 2017. Google ScholarDigital Library
- J. McIntosh, A. Marzo, M. Fraser, and C. Phillips. Echoflex: Hand gesture recognition using ultrasound imaging. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI '17, pages 1923--1934, 2017. Google ScholarDigital Library
- J. McIntosh, C. McNeill, M. Fraser, F. Kerber, M. Löchtefeld, and A. Krüger. Empress: Practical hand gesture classification with wrist-mounted emg and pressure sensing. In Proc. of the 2016 CHI Conf. on Human Factors in Computing Systems, CHI '16, pages 2332--2342, 2016. Google ScholarDigital Library
- Myo. Myo support. https://goo.gl/TgBtBP. {Online; accessed Apr 01, 2017}.Google Scholar
- H. W. Ng, X. Jiang, L.-K. Merhi, and C. Menon. Investigation of the Feasibility of Strain Gages as Pressure Sensors for Force Myography, pages 261--270. 2017.Google Scholar
- P. Nguyen, N. Bui, A. Nguyen, H. Truong, A. Suresh, M. Whitlock, D. Pham, T. Dinh, and T. Vu. Tyth-typing on your teeth: Tongue-teeth localization for human-computer interface. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services, pages 269--282. ACM, 2018. Google ScholarDigital Library
- P. Nguyen, U. Muncuk, A. Ashok, K. R. Chowdhury, M. Gruteser, and T. Vu. Battery-free identification token for touch sensing devices. In Proceedings of the 14th ACM Conference on Embedded Network Sensor Systems CD-ROM, pages 109--122. ACM, 2016. Google ScholarDigital Library
- P. Nintanavongsa, U. Muncuk, D. R. Lewis, and K. R. Chowdhury. Design optimization and implementation for rf energy harvesting circuits. IEEE Journal on emerging and selected topics in circuits and systems, 2(1):24--33, 2012.Google ScholarCross Ref
- M. Ogata and M. Imai. Skinwatch: Skin gesture interaction for smart watch. In Proceedings of the 6th Augmented Human International Conference, AH '15, pages 21--24, 2015. Google ScholarDigital Library
- U. Olgun, C.-C. Chen, and J. Volakis. Design of an efficient ambient wifi energy harvesting system. IET Microwaves, Antennas & Propagation, 6(11):1200--1206, 2012.Google Scholar
- S. Ortega-Avila, B. Rakova, S. Sadi, and P. Mistry. Non-invasive optical detection of hand gestures. In Proceedings of the 6th Augmented Human International Conference, AH '15, pages 179--180, 2015. Google ScholarDigital Library
- G. K. Ottman, H. F. Hofmann, A. C. Bhatt, and G. A. Lesieutre. Adaptive piezoelectric energy harvesting circuit for wireless remote power supply. IEEE Transactions on power electronics, 17(5):669--676, 2002.Google ScholarCross Ref
- Panasonic. Panasonic - bsg am-5610car solar cell. https://goo.gl/CtF6m6. {Online; accessed Apr 05, 2018}.Google Scholar
- A. Pantelopoulos and N. G. Bourbakis. A survey on wearable sensor-based systems for health monitoring and prognosis. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 40(1):1--12, 2010. Google ScholarDigital Library
- J. K. Perng, B. Fisher, S. Hollar, and K. S. J. Pister. Acceleration sensing glove (asg). In Digest of Papers. Third International Symposium on Wearable Computers, pages 178--180, 1999. Google ScholarDigital Library
- M. Piñuela, P. D. Mitcheson, and S. Lucyszyn. Ambient rf energy harvesting in urban and semi-urban environments. IEEE Transactions on Microwave Theory and Techniques, 61(7):2715--2726, 2013.Google ScholarCross Ref
- Q. Pu, S. Gupta, S. Gollakota, and S. Patel. Whole-home gesture recognition using wireless signals. In Proc. of the 19th International conference on Mobile computing & networking, pages 27--38. ACM, 2013. Google ScholarDigital Library
- Pyralux. Flex pcb material. https://goo.gl/1pWRbR. {Online; accessed Apr 01, 2017}.Google Scholar
- S. S. Rautaray and A. Agrawal. Vision based hand gesture recognition for human computer interaction: a survey. Artificial Intelligence Review, 43(1):1--54, Jan 2015. Google ScholarDigital Library
- J. Rekimoto. Gesturewrist and gesturepad: Unobtrusive wearable interaction devices. In Wearable Computers, 2001. Proceedings. Fifth International Symposium on, pages 21--27. IEEE, 2001. Google ScholarDigital Library
- Z. Ren, J. Yuan, and Z. Zhang. Robust hand gesture recognition based on finger-earth mover's distance with a commodity depth camera. In Proceedings of the 19th ACM international conference on Multimedia, pages 1093--1096. ACM, 2011. Google ScholarDigital Library
- SAMSUNG. Gesture in smart tv. https://goo.gl/36vFhx, 2013. {Online; accessed Nov 21, 2017}.Google Scholar
- T. S. Saponas, D. S. Tan, D. Morris, and R. Balakrishnan. Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08, pages 515--524, 2008. Google ScholarDigital Library
- T. S. Saponas, D. S. Tan, D. Morris, R. Balakrishnan, J. Turner, and J. A. Landay. Enabling always-available input with muscle-computer interfaces. In Proceedings of the 22Nd Annual ACM Symposium on User Interface Software and Technology, UIST '09, pages 167--176, 2009. Google ScholarDigital Library
- A. R. Sarkar, G. Sanyal, and S. Majumder. Hand gesture recognition systems: a survey. International Journal of Computer Applications, 71(15), 2013.Google Scholar
- Shopify. Lunar solar smart watch. https://goo.gl/nbzFDC. {Online; accessed Apr 05, 2018}.Google Scholar
- Smooth-On. Dragon skin 10 very fast. https://goo.gl/Uao6gP. {Online; accessed Apr 01, 2017}.Google Scholar
- Smooth-On. Dragon skin 30. https://goo.gl/uLMNhY. {Online; accessed Apr 01, 2017}.Google Scholar
- Smooth-On. Ecoflex 10. https://goo.gl/CjuJ8X. {Online; accessed Apr 01, 2017}.Google Scholar
- H. H. Szu. Non-convex optimization. In Proceedings of the SPIE, volume 698, pages 59--65, 1986.Google ScholarCross Ref
- J. Taleisnik, R. H. Gelberman, B. W. Miller, and R. M. Szabo. The extensor retinaculum of the wrist. The Journal of hand surgery, 9(4):495--501, 1984.Google Scholar
- V. Talla, B. Kellogg, S. Gollakota, and J. R. Smith. Battery-free cellphone. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 1(2):25, 2017. Google ScholarDigital Library
- V. Talla, B. Kellogg, B. Ransford, S. Naderiparizi, S. Gollakota, and J. R. Smith. Powering the next billion devices with wi-fi. In Proceedings of the 11th ACM Conference on Emerging Networking Experiments and Technologies, page 4. ACM, 2015. Google ScholarDigital Library
- Thalmic-Labs. Myo armband wearable guesture control. https://www.myo.com/, 2017. {Online; accessed Apr 04, 2017}.Google Scholar
- H. Truong, P. Nguyen, N. Bui, A. Nguyen, and T. Vu. Low-power capacitive sensing wristband for hand gesture recognition. In Proceedings of the 9th ACM Workshop on Wireless of the Students, by the Students, and for the Students, pages 21--21. ACM, 2017. Google ScholarDigital Library
- H. Truong, P. Nguyen, A. Nguyen, N. Bui, and T. Vu. Capacitive sensing 3d-printed wristband for enriched hand gesture recognition. In Proceedings of the 2017 Workshop on Wearable Systems and Applications, pages 11--15. ACM, 2017. Google ScholarDigital Library
- H. Truong, P. Nguyen, V. Nguyen, M. Ibrahim, R. Howard, M. Gruteser, and T. Vu. Through-body capacitive touch communication. In Proceedings of the 9th ACM Workshop on Wireless of the Students, by the Students, and for the Students, pages 7--9. ACM, 2017. Google ScholarDigital Library
- T. Vu et al. Distinguishing users with capacitive touch communication. In ACM MobiCom, pages 197--208, 2012. Google ScholarDigital Library
- R. J. Vyas, B. B. Cook, Y. Kawahara, and M. M. Tentzeris. E-wehp: A batteryless embedded sensor-platform wirelessly powered from ambient digital-tv signals. IEEE Transactions on microwave theory and techniques, 61(6):2491--2505, 2013.Google ScholarCross Ref
- B. Wan, R. Wu, K. Zhang, and L. Liu. A new subtle hand gestures recognition algorithm based on emg and fsr. In 2017 IEEE 21st International Conference on Computer Supported Cooperative Work in Design (CSCWD), pages 127--132, 2017.Google ScholarCross Ref
- C. Xu, P. H. Pathak, and P. Mohapatra. Finger-writing with smartwatch: A case for finger and hand gesture recognition using smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications, HotMobile '15, pages 9--14, 2015. Google ScholarDigital Library
- P. Xu. A real-time hand gesture recognition and human-computer interaction system. CoRR, abs/1704.07296, 2017.Google Scholar
- Y. Zhang and C. Harrison. Tomo: Wearable, Low-Cost Electrical Impedance Tomography for Hand Gesture Recognition. In ACM UIST, pages 167--173, 2015. Google ScholarDigital Library
- Y. Zhang, R. Xiao, and C. Harrison. Advancing Hand Gesture Recognition with High Resolution Electrical Impedance Tomography. In ACM UIST, pages 843--850, 2016. Google ScholarDigital Library
Index Terms
- CapBand: Battery-free Successive Capacitance Sensing Wristband for Hand Gesture Recognition
Recommendations
SmartLens: sensing eye activities using zero-power contact lens
MobiCom '22: Proceedings of the 28th Annual International Conference on Mobile Computing And NetworkingAs the most important organs of sense, human eyes perceive 80% information from our surroundings. Eyeball movement is closely related to our brain health condition. Eyeball movement and eye blink are also widely used as an efficient human-computer ...
Pull Outperforms Push as Vibrotactile Wristband Feedback for Mid-Air Gesture Guidance
MuC '23: Proceedings of Mensch und Computer 2023The use of mid-air gestures to control interactive systems is becoming increasingly important, particularly in mixed reality scenarios. However, these gestures are not always intuitive and can be challenging to learn as they lack visual guidance. ...
Sensing Hand Interactions with Everyday Objects by Profiling Wrist Topography
TEI '22: Proceedings of the Sixteenth International Conference on Tangible, Embedded, and Embodied InteractionWe demonstrate rich inferences about unaugmented everyday objects and hand object interactions by measuring minute skin surface deformations at the wrist using a sensing technique based on capacitance. The wristband prototype infers muscle and tendon ...
Comments