Abstract
Recently, deep learning (DL) methods have been introduced very successfully into human activity recognition (HAR) scenarios in ubiquitous and wearable computing. Especially the prospect of overcoming the need for manual feature design combined with superior classification capabilities render deep neural networks very attractive for real-life HAR applications. Even though DL-based approaches now outperform the state-of-the-art in a number of recognition tasks, still substantial challenges remain. Most prominently, issues with real-life datasets, typically including imbalanced datasets and problematic data quality, still limit the effectiveness of activity recognition using wearables. In this paper we tackle such challenges through Ensembles of deep Long Short Term Memory (LSTM) networks. LSTM networks currently represent the state-of-the-art with superior classification performance on relevant HAR benchmark datasets. We have developed modified training procedures for LSTM networks and combine sets of diverse LSTM learners into classifier collectives. We demonstrate that Ensembles of deep LSTM learners outperform individual LSTM networks and thus push the state-of-the-art in human activity recognition using wearables. Through an extensive experimental evaluation on three standard benchmarks (Opportunity, PAMAP2, Skoda) we demonstrate the excellent recognition capabilities of our approach and its potential for real-life applications of human activity recognition.
- Gregory D Abowd. 2016. Beyond Weiser - From Ubiquitous to Collective Computing. IEEE Computer 49, 1 (2016), 17--23. Google ScholarDigital Library
- Mohammad A Alsheikh, Ahmed Selim, Dusit Niyato, Linda Doyle, Shaowei Lin, and Hwee-Pink Tan. 2016. Deep Activity Recognition Models with Triaxial Accelerometers. In Proc. AAAI Workshop on Artificial Intelligence Applied to Assistive Technologies and Smart Environments.Google Scholar
- Akin Avci, Stephan Bosch, Mihai Marin-Perianu, Raluca Marin-Perianu, and Paul Havinga. 2010. Activity Recognition Using Inertial Sensing for Healthcare, Wellbeing and Sports Applications: A Survey. In Proc. Int. Conf. Architecture of Compu. Systems (ARCS).Google Scholar
- Marc Bachlin, Meir Plotnik, Daniel Roggen, Inbal Maidan, Jeffrey M Hausdorff, Nir Giladi, and Gerhard Troster. 2010. Wearable assistant for Parkinson's disease patients with the freezing of gait symptom. IEEE Transactions on Information Technology in Biomedicine 14, 2 (2010), 436--446. Google ScholarDigital Library
- Sourav Bhattacharya and Nicholas D Lane. 2016. Sparsification and Separation of Deep Learning Layers for Constrained Resource Inference on Wearables. In Proc. Int. Conf. Embedded Networked Sensor Systems (SenSys). Google ScholarDigital Library
- Léon Bottou. 2010. Large-Scale Machine Learning with Stochastic Gradient Descent. In Prco. Int. Conf. Computational Statistics (COMPSTAT).Google Scholar
- Leo Breiman. 1996. Bagging predictors. Machine Learning 24 (1996), 123--140. Google ScholarDigital Library
- Leo Breiman. 2001. Random Forests. Mach. Learn. (2001), 5--32. Google ScholarDigital Library
- Peter Bühlmann. 2012. Bagging, Boosting and Ensemble Methods. Springer Berlin Heidelberg, 985--1022.Google Scholar
- Andreas Bulling, Ulf Blanke, and Bernt Schiele. 2014. A tutorial on human activity recognition using body-worn inertial sensors. Comput. Surveys 46, 3 (Jan. 2014), 1--33. Google ScholarDigital Library
- Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José Del R Millán, and Daniel Roggen. 2013. The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters 34, 15 (Nov. 2013), 2033--2042. Google ScholarDigital Library
- Linmi Chen, Jesse Hoey, Chris D Nugent, Diane J Cook, and Zhiwen Yu. 2012. Sensor-Based Activity Recognition. IEEE Trans. on Systems, Man, and Cybernetics - Part C: Applications and Reviews 42, 6 (2012), 790--808. Google ScholarDigital Library
- Krzysztof Dembczynski, Arkadiusz Jachnik, Wojciech Kotlowski, Willem Waegeman, and Eyke Hüllermeier. 2013. Optimizing the F-Measure in Multi-Label Classification: Plug-in Rule Approach versus Structured Loss Minimization. Li Deng and John C Platt. 2014. Ensemble deep learning for speech recognition.. In Proc. INTERSPEECH.Google Scholar
- Sander Dieleman and others. 2015. Lasagne: First release. (Aug. 2015).Google Scholar
- Nils Y. Hammerla, James Fisher, Peter Andras, Lynn Rochester, Richard Walker, and Thomas Plötz. 2015. PD Disease State Assessment in Naturalistic Environments using Deep Learning. In Proc. Int. Conf. Assoc. Advancement of Art. Intelligence (AAAI). Google ScholarDigital Library
- Nils Y. Hammerla, Shane Halloran, and Thomas Plötz. 2016. Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables.Google Scholar
- Nils Y. Hammerla, Reuben Kirkham, Peter Andras, and Thomas Plötz. 2013. On Preserving Statistical Characteristics of Accelerometry Data using their Empirical Cumulative Distribution.Google Scholar
- Nils Y. Hammerla and Thomas Plötz. 2015. Let's (not) Stick Together: Pairwise Similarity Biases Cross-Validation in Activity Recognition.Google Scholar
- Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009. The Elements of Statistical Learning. Springer.Google Scholar
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural computation 9, 8 (Nov. 1997), 1735--1780. Google ScholarDigital Library
- Jesse Hoey, Thomas Plötz, Dan Jackson, Andrew Monk, Cuong Pham, and Patrick Olivier. 2011. Rapid specification and automated generation of prompting systems to assist people with dementia. Pervasive and Mobile Computing (PMC) 7, 3 (2011), 299--318. Google ScholarDigital Library
- Martin Jansche. 2005. Maximum Expected F-measure Training of Logistic Regression Models. In Proc. Int. Conf. on Human Language Technology and Empirical Methods in Natural Language Processing. Google ScholarDigital Library
- Aftab Khan, Sebastian Mellor, Eugen Berlin, Robin Thompson, Roisin McNaney, Patrick Olivier, and Thomas Plötz. 2015. Beyond Activity Recognition: Skill Assessment from Accelerometer Data.Google Scholar
- David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proc. ACM Symp. User Interface and Software Technology (UIST). Google ScholarDigital Library
- Hyun-Chul Kim, Shaoning Pang, Hong-Mo Je, Daijin Kim, and Sung Yang Bang. 2003. Constructing support vector machine ensemble. Pattern Recognition 36, 12 (Dec. 2003), 2757--2767.Google ScholarCross Ref
- Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proc. Int. Conf. Learning Representation (IGLR).Google Scholar
- Josef Kittler, Mohamad Hatef, Robert P W Duin, and Jiri Matas. 1998. On Combining Classifiers. IEEE Trans. Pattern Analysis and Machine Intelligence (TPAMI) 20, 3 (1998), 226--239. Google ScholarDigital Library
- Matthias Kranz, Andreas Moeller, Nils Y. Hammerla, Stefan Diewald, Luis Roalter, Thomas Plötz, and Patrick Olivier. 2012. The mobile fitness coach: Towards individualized skill assessment using personalized mobile devices. Pervasive and Mobile Computing (PMC) (2012). Google ScholarDigital Library
- Ludmila Kuncheva. 2004. Combining pattern classifiers: methods and algorithms. John Wiley 8 Sons. Google ScholarDigital Library
- Cassim Ladha, Nils Y. Hammerla, Patrick Olivier, and Thomas Plötz. 2013. ClimbAX: Skill Assessment for Climbing Enthusiasts.Google Scholar
- Nicholas D Lane, Petko Georgiev, and Lorena Qendro. 2015. DeepEar: robust smartphone audio sensing in unconstrained acoustic environments using deep learning.Google Scholar
- Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (May 2015), 436--444.Google ScholarCross Ref
- James L McClelland and David E Rumelhart. 1986. Parallel distributed processing. MIT Press. Google ScholarDigital Library
- Francisco Javier Ordóñez Morales and Daniel Roggen. 2016. Deep convolutional feature transfer across mobile activity recognition domains, sensor modalities and locations.Google Scholar
- Francisco Ordóñez and Daniel Roggen. 2016. Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors 16, 1 (Jan. 2016), 115.Google ScholarCross Ref
- Thomas Plötz, Nils Y. Hammerla, and Patrick Olivier. 2011. Feature Learning for Activity Recognition in Ubiquitous Computing.Google Scholar
- Thomas Plötz, Nils Y. Hammerla, Agata Rozga, Andrea Reavis, Nathan Call, and Gregory D Abowd. 2012. Automatic Assessment of Problem Behavior in Individuals with Developmental Disabilities.Google Scholar
- Thomas Plötz, Paula Moynihan, Cuong Pham, and Patrick Olivier. 2010. Activity Recognition and Healthier Food Preparation. Activity Recognition in Pervasive Intelligent Environments (2010).Google Scholar
- Zhiquan Qi, Bo Wang, Yingjie Tian, and Peng Zhang. 2016. When Ensemble Learning Meets Deep Learning: a New Deep Support Vector Machine for Classification. Knowledge-Based Systems 107 (Sept. 2016), 54--60. Google ScholarDigital Library
- Nastaran M Rad, Andrea Bizzego, Seyed M Kia, Giuseppe Jurman, Paola Venuti, and Cesare Furlanello. 2015. Convolutional Neural Network for Stereotypical Motor Movement Detection in Autism. In arXiv.1511.01865. Attila Reiss and Didier Stricker. 2012. Introducing a New Benchmarked Dataset for Activity Monitoring.Google Scholar
- Charissa A Ronaoo and Sung-Bae Cho. 2015. Evaluation of deep convolutional neural network architectures for human activity recognition with smartphone sensors. In Proc. KIISE Korea Computer Congress.Google Scholar
- Robert E Schapire. 1990. The strength of weak learnability. Machine Learning (1990), 197--227. Google ScholarDigital Library
- Albrecht Schmidt, Michael Beigl, and Hans W Gellersen. 1999. There is more to Context than Location. Computer 8 Graphics 23, 6 (1999), 893--901.Google Scholar
- Urminder Singh, Sucheta Chauhan, A. Krishnamachari, and Lovekesh Vig. 2015. Ensemble of Deep Long Short Term Memory Networks for Labelling Origin of Replication Sequences. In Proc. Int. Conf. Data Science and Advanced Analytics (DSAA).Google ScholarCross Ref
- Yilin Song, Jonathan Viventi, and Yao Wang. 2016. Diversity encouraged learning of unsupervised LSTM ensemble for neural activity video prediction. arXiv preprint 1611.04899v (cs.CV) (2016).Google Scholar
- J. Michael Steele. 2004. The Gauchy-Schwarz Master Class: An Introduction to the Art of Mathematical Inequalities. Cambridge University Press, New York, NY, USA. Google ScholarDigital Library
- Thomas Stiefmeier, Daniel Roggen, Gerhard Tröster, Georg Ogris, and Paul Lukowicz. 2008. Wearable activity tracking in car manufacturing. IEEE Pervasive Computing 7, 2 (March 2008), 42--50. Google ScholarDigital Library
- Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints abs/1605.02688 (May 2016). http://arxiv.org/abs/1605.02688Google Scholar
- Giorgio Valentini and Thomas G. Dietterich. 2004. Bias-Variance Analysis of Support Vector Machines for the Development of SVM-Based Ensemble Methods. Journal of Machine Learning Research (JMLR) 5 (2004), 725--775. Google ScholarDigital Library
- Mark Weiser. 1991. The Computer for the 21st Century. Scientific American (1991).Google Scholar
- Jianbo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiaoli Li 0001, and Shonali Krishnaswamy. 2015. Deep Convolutional Neural Networks on Multichannel Time Series for Human Activity Recognition.Google Scholar
- Ming Zeng, Le T Nguyen, Bo Yu, Ole J Mengshoel, and Joy Zhang. 2014. Convolutional Neural Networks for Human Activity Recognition using Mobile Sensors. Proc. Int. Conf. Mobile Commuting, Applications and Services (MobiCASE) (2014).Google ScholarCross Ref
- Licheng Zhang, Xihong Wu, and Dingsheng Luo. 2015. Human activity recognition with HMM-DNN model.. In Proc. Int. Conf. Cognitive Informatics 8 Cognitive Computing (ICCI). 192--197.Google ScholarCross Ref
- Zhi-Hua Zhou, Jianxin Wu, and Wei Tang. 2002. Ensembling neural networks: Many could be better than all. Artificial Intelligence 137, 1 (2002), 239--263. Google ScholarDigital Library
Index Terms
- Ensembles of Deep LSTM Learners for Activity Recognition using Wearables
Recommendations
Human Activity Recognition Using Wearable Sensors by Deep Convolutional Neural Networks
MM '15: Proceedings of the 23rd ACM international conference on MultimediaHuman physical activity recognition based on wearable sensors has applications relevant to our daily life such as healthcare. How to achieve high recognition accuracy with low computational cost is an important issue in the ubiquitous computing. Rather ...
Deep Temporal Conv-LSTM for Activity Recognition
AbstractHuman activity recognition has gained interest from the research community due to the advancements in sensor technology and the improved machine learning algorithm. Wearable sensors have become more ubiquitous, and most of the wearable sensor data ...
Batch-based activity recognition from egocentric photo-streams revisited
Wearable cameras can gather large amounts of image data that provide rich visual information about the daily activities of the wearer. Motivated by the large number of health applications that could be enabled by the automatic recognition of daily ...
Comments