Bing Zhai
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Traditionally, sleep monitoring has been performed in hospital or clinic environments, requiring complex and expensive equipment set-up and expert scoring. Wearable devices increasingly provide a viable alternative for sleep monitoring and are able to collect movement and heart rate (HR) data. In this work, we present a set of algorithms for sleep-wake and sleep-stage classification based upon actigraphy and cardiac sensing amongst 1,743 participants. We devise movement and cardiac features that could be extracted from research-grade wearable sensors and derive models and evaluate their performance in the largest open-access dataset for human sleep science. Our results demonstrated that neural network models outperform traditional machine learning methods and heuristic models for both sleep-wake and sleep-stage classification. Convolutional neural networks (CNNs) and long-short term memory (LSTM) networks were the best performers for sleep-wake and sleep-stage classification, respectively. Using SHAP (SHapley Additive exPlanation) with Random Forest we identified that frequency features from cardiac sensors are critical to sleep-stage classification. Finally, we introduced an ensemble-based approach to sleep-stage classification, which outperformed all other baselines, achieving an accuracy of 78.2% and F1 score of 69.8% on the classification task for three sleep stages. Together, this work represents the first systematic multimodal evaluation of sleep-wake and sleep-stage classification in a large, diverse population. Alongside the presentation of an accurate sleep-stage classification approach, the results highlight multimodal wearable sensing approaches as scalable methods for accurate sleep-classification, providing guidance on optimal algorithm deployment for automated sleep assessment. The code used in this study can be found online at: https://github.com/bzhai/multimodal_sleep_stage_benchmark.git
Supplemental Material
Available for Download
zip
zhai.zip (5 MB)
Supplemental movie, appendix, image and software files for, Making Sense of Sleep: Multimodal Sleep Stage Classification in a Large, Diverse Population Using Movement and Cardiac Sensing
- Saeed Abdullah, Mark Matthews, Elizabeth L. Murnane, Geri Gay, and Tanzeem Choudhury. 2014. Towards Circadian Computing: "Early to Bed and Early to Rise" Makes Some of Us Unhealthy and Sleep Deprived. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14). Association for Computing Machinery, New York, NY, USA, 673--684. https://doi.org/10.1145/2632048.2632100Google Scholar
Digital Library
- M. Aktaruzzaman, M. Migliorini, M. Tenhunen, S. L. Himanen, A. M. Bianchi, and R. Sassi. 2015. The addition of entropy-based regularity parameters improves sleep stage classification based on heart rate variability. Medical and Biological Engineering and Computing 53, 5 (5 2015), 415--425. https://doi.org/10.1007/s11517-015-1249-zGoogle Scholar
- Emina Alickovic and Abdulhamit Subasi. 2018. Ensemble SVM Method for Automatic Sleep Stage Classification. IEEE Transactions on Instrumentation and Measurement 67, 6 (6 2018), 1258--1265. https://doi.org/10.1109/TIM.2018.2799059Google ScholarCross Ref
- Bruce M Altevogt, Harvey R Colten, et al. 2006. Sleep disorders and sleep deprivation: an unmet public health problem. National Academies Press, Washington (DC). https://doi.org/10.17226/11617Google Scholar
- Salikh Bagaveyev and Diane J Cook. 2014. Designing and Evaluating Active Learning Methods for Activity Recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '14 Adjunct). Association for Computing Machinery, New York, NY, USA, 469--478. https://doi.org/10.1145/2638728.2641674Google ScholarDigital Library
- Argelinda Baroni, Jean Marie Bruzzese, Christina A. Di Bartolo, and Jess P. Shatkin. 2016. Fitbit Flex: an unreliable device for longitudinal sleep measures in a non-clinical population., 853--854 pages. https://doi.org/10.1007/s11325-015-1271-2Google Scholar
- Richard B. Berry, Rita Brooks, Charlene E. Gamaldo, Susan M. Harding, Robin M. Lloyd, Carole L. Marcus, and Bradley V. Vaughn. 2016. American Academy of Sleep Medicine. The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology, and Technical Specifications, Version 2.2. American Academy of Sleep 28, 3 (2016), 391--397. www.aasmnet.org.Google Scholar
- Diane E. Bild, David A Bluemke, Gregory L Burke, Robert Detrano, Ana V Diez Roux, Aaron R Folsom, Philip Greenland, David R. Jacobs, Richard Kronmal, Kiang Liu, Jennifer Clark Nelson, Daniel O'Leary, Mohammed F Saad, Steven Shea, Moyses Szklo, and Russell P Tracy. 2002. Multi-Ethnic Study of Atherosclerosis: Objectives and design. American Journal of Epidemiology 156, 9 (11 2002), 871--881. https://doi.org/10.1093/aje/kwf113Google ScholarCross Ref
- M H Bonnet and D L Arand. 1997. Heart rate variability: sleep stage, time of night, and arousal influences. Electroencephalography and Clinical Neurophysiology 102, 5 (5 1997), 390--396. https://doi.org/10.1016/S0921-884X(96)96070-1Google Scholar
- Philippe Boudreau, Wei-Hsien Yeh, Guy A. Dumont, and Diane B. Boivin. 2013. Circadian Variation of Heart Rate Variability Across Sleep Stages. Sleep 36, 12 (12 2013), 1919--1928. https://doi.org/10.5665/sleep.3230Google Scholar
- Liqiong Chang, Jiaqi Lu, Ju Wang, Xiaojiang Chen, Dingyi Fang, Zhanyong Tang, Petteri Nurmi, and Zheng Wang. 2018. SleepGuard: Capturing Rich Sleep Information Using Smartwatch Sensing Data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1--34. https://doi.org/10.1145/3264908Google ScholarDigital Library
- R J Cole, D F Kripke, W Gruen, D J Mullaney, and J C Gillin. 1992. Automatic sleep/wake identification from wrist activity. Sleep 15, 5 (10 1992), 461--9. http://www.ncbi.nlm.nih.gov/pubmed/1455130Google Scholar
- Nediyana Daskalova, Bongshin Lee, Jeff Huang, Chester Ni, and Jessica Lundin. 2018. Investigating the Effectiveness of Cohort-Based Sleep Recommendations. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (9 2018), 1--19. https://doi.org/10.1145/3264911Google ScholarDigital Library
- Dennis A. Dean, Ary L. Goldberger, Remo Mueller, Matthew Kim, Michael Rueschman, Daniel Mobley, Satya S. Sahoo, Catherine P. Jayapandian, Licong Cui, Michael G. Morrical, Susan Surovec, Guo-Qiang Zhang, and Susan Redline. 2016. Scaling Up Scientific Discovery in Sleep Medicine: The National Sleep Research Resource. Sleep 39, 5 (5 2016), 1151--1164. https://doi.org/10.5665/sleep.5774Google Scholar
- Sigrid Elsenbruch, Michael J. Harnish, and William C. Orr. 1999. Heart Rate Variability During Waking and Sleep in Healthy Males and Females. Sleep 22, 8 (12 1999), 1067--1071. https://doi.org/10.1093/sleep/22.8.1067Google Scholar
- Nina E Fultz, Giorgio Bonmassar, Kawin Setsompop, Robert A Stickgold, Bruce R Rosen, Jonathan R Polimeni, and Laura D Lewis. 2019. Coupled electrophysiological, hemodynamic, and cerebrospinal fluid oscillations in human sleep. Science (New York, N.Y.) 366, 6465 (11 2019), 628--631. https://doi.org/10.1126/science.aax5440Google Scholar
- Jennifer Girschik, Lin Fritschi, Jane Heyworth, and Flavie Waters. 2012. Validation of self-reported sleep against actigraphy. Journal of epidemiology 22, 5 (2012), 462--8. https://doi.org/10.2188/jea.je20120012Google ScholarCross Ref
- Yu Guan and Thomas Plötz. 2017. Ensembles of Deep LSTM Learners for Activity Recognition using Wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2 (6 2017), 1--28. https://doi.org/10.1145/3090076Google ScholarDigital Library
- Haodong Guo, Ling Chen, Liangying Peng, and Gencai Chen. 2016. Wearable Sensor Based Multimodal Human Activity Recognition Exploiting the Diversity of Classifier Ensemble. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). Association for Computing Machinery, New York, NY, USA, 1112--1123. https://doi.org/10.1145/2971648.2971708Google ScholarDigital Library
- Vincent T van Hees, Severine Sabia, Samuel E Jones, Andrew R Wood, Kirstie N Anderson, Mika Kivimaki, Timothy M Frayling, Allan I Pack, Maja Bucan, Diego R Mazzotti, Phil R Gehrman, Archana Singh-Manoux, and Michael N Weedon. 2018. Estimating sleep parameters using an accelerometer without sleep diary. Scientific Reports 8, 1 (2018), 12975. https://doi.org/10.1101/257972Google ScholarCross Ref
- J. Allan Hobson, Robert W. McCarley, and Peter W. Wyzinski. 1975. Sleep Cycle Oscillation: Reciprocal Discharge by Two Brainstem Neuronal Groups. Science 189, 4196 (1975), 55--58. http://www.jstor.org/stable/1740806Google Scholar
- M Hornyak, M Cejnar, M Elam, M Matousek, and B G Wallin. 1991. Sympathetic muscle nerve activity during sleep in man. Brain 114 (Pt 3, 3 (6 1991), 1281--95. https://doi.org/10.1093/brain/1143.1281Google Scholar
- Luca Imeri and Mark R Opp. 2009. How (and why) the immune system makes us sleep., 199--210 pages. https://doi.org/10.1038/nrn2576Google Scholar
- D. A. Kirby and R. L. Verrier. 1989. Differential effects of sleep stage on coronary hemodynamic function. American Journal of Physiology-Heart and Circulatory Physiology 256, 5 (5 1989), H1378-H1383. https://doi.org/10.1152/ajpheart.1989.256.5.H1378Google ScholarCross Ref
- B. Koley and D. Dey. 2012. An ensemble system for automatic sleep stage classification using single channel EEG signal. Computers in Biology and Medicine 42, 12 (12 2012), 1186--1195. https://doi.org/10.1016/j.compbiomed.2012.09.012Google Scholar
- Daniel F. Kripke, Elizabeth K. Hahn, Alexandra P. Grizas, Kep H. Wadiak, Richard T. Loving, J. Steven Poceta, Farhad F. Shadan, John W. Cronin, and Lawrence E. Kline. 2010. Wrist actigraphic scoring for sleep laboratory patients: Algorithm development. Journal of Sleep Research 19, 4 (12 2010), 612--619. https://doi.org/10.1111/j.1365-2869.2010.00835.xGoogle ScholarCross Ref
- Jung-Min Lee, Wonwoo Byun, Alyssa Keill, Danae Dinkel, and Yaewon Seo. 2018. Comparison of Wearable Trackers' Ability to Estimate Sleep. International journal of environmental research and public health 15, 6 (2018), 1265. https://doi.org/10.3390/ijerph15061265Google Scholar
- Scott M. Lundberg, Gabriel G. Erion, and Su-In Lee. 2018. Consistent Individualized Feature Attribution for Tree Ensembles. ArXiv abs/1802.03888 (2018), 1--9.Google Scholar
- Marek Malik. 1996. Heart Rate Variability. Annals of Noninvasive Electrocardiology 1, 2 (4 1996), 151--181. https://doi.org/10.1111/j.1542-474X.1996.tb00275.xGoogle ScholarCross Ref
- Farid Melgani and Lorenzo Bruzzone. 2004. Classification of hyperspectral remote sensing images with support vector machines. IEEE Transactions on Geoscience and Remote Sensing 42, 8 (8 2004), 1778--1790. https://doi.org/10.1109/TGRS.2004.831865Google ScholarCross Ref
- Luca Menghini, Evelyn Gianfranchi, Nicola Cellini, Elisabetta Patron, Mariaelena Tagliabue, and Michela Sarlo. 2019. Stressing the accuracy: Wrist-worn wearable sensor validation over different conditions. Psychophysiology 56, 11 (2019), e13441. https://doi.org/10.1111/psyp.13441 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/psyp.13441Google ScholarCross Ref
- Joao Palotti, Raghvendra Mall, Michael Aupetit, Michael Rueschman, Meghna Singh, Aarti Sathyanarayana, Shahrad Taheri, and Luis Fernandez-Luque. 2019. Benchmark on a large cohort for sleep-wake classification with machine learning techniques. npj Digital Medicine 2, 1 (12 2019), 50. https://doi.org/10.1038/s41746-019-0126-9Google Scholar
- Sanjay R. Patel, Jia Weng, Michael Rueschman, Katherine A. Dudley, Jose S. Loredo, Yasmin Mossavar-Rahmani, Maricelle Ramirez, Alberto R. Ramos, Kathryn Reid, Ashley N. Seiger, Daniela Sotres-Alvarez, Phyllis C. Zee, and Rui Wang. 2015. Reproducibility of a Standardized Actigraphy Scoring Algorithm for Sleep in a US Hispanic/Latino Population. Sleep 38, 9 (9 2015), 1497--1503. https://doi.org/10.5665/sleep.4998Google Scholar
- Ignacio Perez-Pozuelo, Bing Zhai, Joao Palotti, Raghvendra Mall, Michaël Aupetit, Juan M Garcia-Gomez, Shahrad Taheri, Yu Guan, and Luis Fernandez-Luque. 2020. The future of sleep health: a data-driven revolution in sleep science and medicine. NPJ digital medicine 3, 1 (2020), 1--15.Google Scholar
- Huy Phan, Fernando Andreotti, Navin Cooray, Oliver Y. Chen, and Maarten De Vos. 2019. Joint Classification and Prediction CNN Framework for Automatic Sleep Stage Classification. IEEE Transactions on Biomedical Engineering 66, 5 (5 2019), 1285--1296. https://doi.org/10.1109/TBME.2018.2872652Google ScholarCross Ref
- Athi Ponnusamy, Jefferson L B Marques, and Markus Reuber. 2012. Comparison of heart rate variability parameters during complex partial seizures and psychogenic nonepileptic seizures. Epilepsia 53, 8 (8 2012), 1314--21. https://doi.org/10.1111/j.1528-1167.2012.03518.xGoogle Scholar
- Mustafa Radha, Pedro Fonseca, Arnaud Moreau, Marco Ross, Andreas Cerny, Peter Anderer, Xi Long, and Ronald M. Aarts. 2019. Sleep stage classification from heart-rate variability using long short-term memory neural networks. Scientific Reports 9, 1 (12 2019), 1--11. https://doi.org/10.1038/s41598-019-49703-yGoogle Scholar
- Valentin Radu, Catherine Tong, Sourav Bhattacharya, Nicholas D Lane, Cecilia Mascolo, Mahesh K. Marina, and Fahim Kawsar. 2018. Multimodal Deep Learning for Activity and Context Recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1--27. https://doi.org/10.1145/3161174Google ScholarDigital Library
- Avi Sadeh, P J Hauri, Daniel F Kripke, and P Lavie. 1995. The role of actigraphy in the evaluation of sleep disorders. Sleep 18, 4 (5 1995), 288--302. https://doi.org/10.1093/sleep/18.4.288Google Scholar
- Avi Sadeh, Katherine M Sharkey, and Mary A Carskadon. 1994. Activity-Based Sleep---Wake Identification: An Empirical Test of Methodological Issues. Sleep 17, 3 (1994), 201--207. https://doi.org/10.1093/sleep/173.201Google Scholar
- Edward Sazonov, Nadezhda Sazonova, Stephanie Schuckers, Michael Neuman, and CHIME Study Group. 2004. Activity-based sleep-wake identification in infants. Physiological measurement 25, 5 (10 2004), 1291--304. http://www.ncbi.nlm.nih.gov/pubmed/15535193Google Scholar
- Jonathan R L Schwartz and Thomas Roth. 2008. Neurophysiology of sleep and wakefulness: basic science and clinical implications. Current neuropharmacology 6, 4 (12 2008), 367--78. https://doi.org/10.2174/157015908787386050Google Scholar
- Eti Ben Simon, Aubrey Rossi, Allison G Harvey, and Matthew P Walker. 2020. Overanxious and underslept. Nature Human Behaviour 4, 1 (2020), 100--110.Google ScholarCross Ref
- Urminder Singh, Sucheta Chauhan, A Krishnamachari, and Lovekesh Vig. 2015. Ensemble of deep long short term memory networks for labelling origin of replication sequences. In 2015 IEEE International Conference on Data Science and Advanced Analytics (DSAA). IEEE, Paris, France, 1--7. https://doi.org/10.1109/DSAA.2015.7344871Google ScholarCross Ref
- Frederick Snyder, J. Allan Hobson, Donald F. Morrison, and Frederick Goldfrank. 1964. Changes in respiration, heart rate, and systolic blood pressure in human sleep. Journal of Applied Physiology 19, 3 (5 1964), 417--422. https://doi.org/10.1152/jappl.1964.193.417Google ScholarCross Ref
- Virend K. Somers, Mark E. Dyken, Allyn L. Mark, and Francois M. Abboud. 1993. Sympathetic-Nerve Activity during Sleep in Normal Subjects. New England Journal of Medicine 328, 5 (2 1993), 303--307. https://doi.org/10.1056/NEJM199302043280502Google ScholarCross Ref
- Jens B. Stephansen, Alexander N. Olesen, Mads Olsen, Aditya Ambati, Eileen B. Leary, Hyatt E. Moore, Oscar Carrillo, Ling Lin, Fang Han, Han Yan, Yun L. Sun, Yves Dauvilliers, Sabine Scholz, Lucie Barateau, Birgit Hogl, Ambra Stefani, Seung Chul Hong, Tae Won Kim, Fabio Pizza, Giuseppe Plazzi, Stefano Vandi, Elena Antelmi, Dimitri Perrin, Samuel T. Kuna, Paula K. Schweitzer, Clete Kushida, Paul E. Peppard, Helge B. D. Sorensen, Poul Jennum, and Emmanuel Mignot. 2018. Neural network analysis of sleep stages enables efficient diagnosis of narcolepsy. Nature Communications 9, 1 (12 2018), 5229. https://doi.org/10.1038/s41467-018-07229-3Google Scholar
- Xiao Sun, Li Qiu, Yibo Wu, Yeming Tang, and Guohong Cao. 2017. SleepMonitor: Monitoring Respiratory Rate and Body Position During Sleep Using Smartwatch. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (9 2017), 1--22. https://doi.org/10.1145/3130969Google ScholarDigital Library
- Hirofumi Tanaka, Kevin D. Monahan, and Douglas R. Seals. 2001. Age-predicted maximal heart rate revisited. Journal of the American College of Cardiology 37, 1 (2001), 153--156. https://doi.org/10.1016/S0735-1097(00)01054-8Google ScholarCross Ref
- Elizabeth A. Thomson, Kayla Nuss, Ashley Comstock, Steven Reinwald, Sophie Blake, Richard E. Pimentel, Brian L. Tracy, and Kaigang Li. 2019. Heart rate measures from the Apple Watch, Fitbit Charge HR 2, and electrocardiogram across different exercise intensities. Journal of Sports Sciences 37, 12 (6 2019), 1411--1419. https://doi.org/10.1080/02640414.2018.1560644Google ScholarCross Ref
- Joëlle Tilmanne, Jérôme Urbain, Mayuresh V Kothare, Alain Vande Wouwer, and Sanjeev V Kothare. 2009. Algorithms for sleep-wake identification using actigraphy: A comparative study and new results. Journal of Sleep Research 18, 1 (3 2009), 85--98. https://doi.org/10.1111/j.1365-2869.2008.00706.xGoogle ScholarCross Ref
- Eleonora Tobaldini, Lino Nobili, Silvia Strada, Karina R. Casali, Alberto Braghiroli, and Nicola Montano. 2013. Heart rate variability in normal and pathological sleep. Frontiers in Physiology 4 (10 2013), 1--11. https://doi.org/10.3389/fphys.2013.00294Google Scholar
- Emilio Vanoli, Philip B. Adamson, Ba-Lin, Gian D. Pinna, Ralph Lazzara, and William C. Orr. 1995. Heart Rate Variability During Specific Sleep Stages. Circulation 91, 7 (4 1995), 1918--1922. https://doi.org/10.1161/01.CIR.91.7.1918Google Scholar
- A. Varri, Bob Kemp, Thomas Penzel, and A. Schlogl. 2001. Standards for biomedical signal databases. IEEE Engineering in Medicine and Biology Magazine 20, 3 (2001), 33--37. https://doi.org/10.1109/51.932722Google ScholarCross Ref
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in neural information processing systems. Curran Associates, Long Beach, CA, United States, 5998--6008. http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdfGoogle ScholarDigital Library
- Emi Yuda, Yutaka Yoshida, Ryujiro Sasanabe, Haruhito Tanaka, Toshiaki Shiomi, Junichiro Hayano, Emi Yuda, Yutaka Yoshida, Ryujiro Sasanabe, Haruhito Tanaka, Toshiaki Shiomi, and Junichiro Hayano. 2017. Sleep Stage Classification by a Combination of Actigraphic and Heart Rate Signals. Journal of Low Power Electronics and Applications 7, 4 (11 2017), 28. https://doi.org/10.3390/jlpea7040028Google ScholarCross Ref
- Guo-Qiang Zhang, Licong Cui, Remo Mueller, Shiqiang Tao, Matthew Kim, Michael Rueschman, Sara Mariani, Daniel Mobley, and Susan Redline. 2018. The National Sleep Research Resource: towards a sleep data commons. Journal of the American Medical Informatics Association 25, 10 (10 2018), 1351--1358. https://doi.org/10.1093/jamia/ocy064Google ScholarCross Ref
Index Terms
- Making Sense of Sleep: Multimodal Sleep Stage Classification in a Large, Diverse Population Using Movement and Cardiac Sensing
Recommendations
Ubi-SleepNet: Advanced Multimodal Fusion Techniques for Three-stage Sleep Classification Using Ubiquitous Sensing
Sleep is a fundamental physiological process that is essential for sustaining a healthy body and mind. The gold standard for clinical sleep monitoring is polysomnography(PSG), based on which sleep can be categorized into five stages, including wake/...
Making Sense of Sleep Sensors: How Sleep Sensing Technologies Support and Undermine Sleep Health
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing SystemsSleep is an important aspect of our health, but it is difficult for people to track manually because it is an unconscious activity. The ability to sense sleep has aimed to lower the barriers of tracking sleep. Although sleep sensors are widely available,...
Performance comparison between wrist and chest actigraphy in combination with heart rate variability for sleep classification
The concurrent usage of actigraphy and heart rate variability (HRV) for sleep efficiency quantification is still matter of investigation. This study compared chest (CACT) and wrist (WACT) actigraphy (actigraphs positioned on chest and wrist, ...
Comments