Elsevier

Neurocomputing

Volume 426, 22 February 2021, Pages 26-34
Neurocomputing

Cross-subject transfer learning in human activity recognition systems using generative adversarial networks

https://doi.org/10.1016/j.neucom.2020.10.056Get rights and content

Abstract

Application of intelligent systems especially in smart homes and health-related topics has been drawing more attention in the last decades. Training Human Activity Recognition (HAR) models - as a major module- requires a fair amount of labeled data. Despite training with large datasets, most of the existing models will face a dramatic performance drop when they are tested against unseen data from new users. Moreover, recording enough data for each new user is non-viable due to the limitations and challenges of working with human users. Transfer learning techniques aim to transfer the knowledge which has been learned from the source domain (subject) to the target domain in order to decrease the models’ performance loss in the target domain. This paper presents a novel method of adversarial knowledge transfer named SA-GAN stands for Subject Adaptor GAN, which utilizes the Generative Adversarial Network framework to perform cross-subject transfer learning in the domain of wearable sensor-based Human Activity Recognition. SA-GAN outperformed other state-of-the-art methods in more than 66% of experiments and showed the second-best performance in the remaining 25% of experiments. In some cases, it reached up to 90% of the accuracy which can be obtained by supervised training over the same domain data.

Introduction

Remarkable enrichment of sensor technology and consequently smart environments alongside with huge progress in machine learning techniques have pervasively brought intelligent solutions into every aspect of human life. Recognition of what the human subject is doing, widely considered to be one of the most important tasks of an intelligent system known as an active field of research named Human Activity Recognition (HAR). The Activity Recognition problem could include various sub-categories of tasks such as sensing, detection, and classification. In this research, the HAR approaches are mainly considered to be addressing the classification task.

Previous studies on HAR can be generally categorized based on sensor modalities and data utilized for recognition of activity details, including vision and sensors based approaches. Vision-based sensors are exploited to capture images, videos or surveillance camera features to recognize activity [1]. Despite the successful performance of vision-based solutions, non-visual sensors are still required to address their existing limitations such as laborious processing and privacy problems. Non-visual sensors can be installed on the human’s body (wearable sensors) or in the environment (ambient sensors). Utilizing a network of heterogeneous sensors has become widespread interest as well.

Let us consider collecting data from sensors measuring k attributes, while the human user is performing activities. Given:W={w0,w1,,wm-1},Si={S(i,0),,S(i,k-1)},A={a0,,an-1}while W is a set of m time windows such that each wi contains a set of time series values Si from each of the k measured attributes, and A is a set of activity labels, the task of Human Activity Recognition can be defined as finding a mapping function f:SiA such that f(Si) is as similar as possible to the actual activity performed during wi [2].

Diverse supervised and semi-supervised machine learning models have been proposed for activity recognition. These models deliver promising accuracy conditioning on training with enough labeled data. However, the pitfall is that their performance will dramatically fall against data, from new unseen distributions. The difference may root in feature space or label space distribution. Therefore, recognizing the activities of a new user remains challenging for the model which was trained by samples of other users’ behavior. Nevertheless, collecting and labeling sufficient training data is not feasible for every new user since it requires a relatively long time observation of human subjects’ behavior which is time-consuming and sometimes impractical [3], [4].

Transfer Learning techniques aim to prevent the aforementioned performance leak by adapting obtained knowledge from the source domain (training users) to the target domain (new users). These approaches should address the primary questions of What, How, and From where to transfer. Thought, it should be noted that when the prior experiences (from other sources of knowledge) are not relevant enough to the new domain, a brutal-force transfer may degrade the learning performance, resulting in the negative knowledge transfer phenomena [5]. Transfer learning is researched under a variety of different names such as life-long learning, knowledge transfer, learning to learn, inductive transfer, context-sensitive learning, and meta-learning in the field of machine learning [6].

The remainder of this paper is organized as follows in five sections. Section 2 examines the previous research works have been devoted to the study of HAR and Transfer Learning. Section 3 describes SA-GAN and its related training details. Evaluation, experimental results and their analysis are presented in Section 4. Finally, Section 5 summarizes the results of this work, draws conclusions, and highlights issues for future researches.

Section snippets

Human activity recognition

Traditional machine learning approaches including K-Nearest Neighbor (KNN), Hidden Markov Model (HMM), Support Vector Machine (SVM), Random Forest (RF), and Naïve Bayes (NB) have shown satisfactory results on recognizing human activities [7], [8]. A major criticism of these models is that they mainly rely on handcrafted feature extraction or heuristic information. Besides the demand for the domain specialist, extracted features are not abstracted enough. Therefore, models are not suitable for

Proposed Model: SA-GAN

Following our semi-supervised knowledge transfer setting, we have labeled data of the source domain (Xs,Ys)~Ps and unlabeled data of target domain Xt~Pt. In our Cross-Subject Transfer Learning problem the difference between the domains roots in the distribution of feature space and the conditional distribution of label space. It can be interpreted as the scenario when there is a model trained with limited samples from distribution Ps, and it is required to test the model against samples drawn

Experiments

Our experiments are broken down into two groups on the basis of their objectives. The first set of analysis was carried out in order to justify the necessity of applying knowledge transfer technique by investigating the performance drop in case of domain shifts. Another group of experiments was conducted with the aim of measuring the improvement achieved by SA-GAN model. Fig. 4, depicts a simple overview of the required steps to have the prediction of target domain Yt̂, using SA-GAN.

Conclusions and Future Work

One of the most important limitations of HAR models lies in lacking a sufficient amount of labeled data. Furthermore, the discovered patterns through available labeled data might not be well generalizable to the samples from unseen subjects. However, data acquisition and labeling are not feasible for newcomers due to limitations of interaction with human users. This paper has highlighted an innovative cutting-edge solution for cross-subject knowledge transfer in the domain of Human Activity

CRediT authorship contribution statement

Elnaz Soleimani: Writing - original draft, Investigation. Ehsan Nazerfard: Supervision, Writing - review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ehsan Nazerfard received the BS and MS degrees in Computer Engineering and Artificial Intelligence from Amirkabir University of Technology and Sharif University of Technology, Tehran, Iran, in 2003 and 2006, respectively. He also received the PhD degree in Computer Science from Washington State University, WA USA, in 2014. He is currently an Assistant Professor in the department of Computer Engineering at Amirkabir University of Technology. His research interests include Machine Learning, Deep

References (45)

  • A. Mannini et al.

    Machine learning methods for classifying human physical activity from on-body accelerometers

    Sensors

    (2010)
  • L. Bedogni et al.

    By train or by car? detecting the user’s motion type through smartphone sensors data

  • Q. Yang, Activity recognition: linking low-level sensors to high-level intelligence, in: Twenty-First International...
  • P. Vepakomma et al.

    A-wristocracy, Deep learning on wrist-worn sensing for recognition of user complex activities

  • W.-Y. Cheng et al.

    Human activity recognition from sensor-based large-scale continuous monitoring of parkinson’s disease patients

  • Y. Chen, Y. Xue, A deep learning approach to human activity recognition based on single accelerometer, in: 2015 IEEE...
  • J. Yang, M.N. Nguyen, P.P. San, X.L. Li, S. Krishnaswamy, Deep convolutional neural networks on multichannel time...
  • S. Ha, J.-M. Yun, S. Choi, Multi-modal convolutional neural networks for activity recognition, in: 2015 IEEE...
  • W. Jiang et al.

    Human activity recognition using wearable sensors by deep convolutional neural networks, in

  • J. Wang, Y. Chen, S. Hao, X. Peng, L. Hu, Deep learning for sensor-based activity recognition: A survey, Pattern...
  • Y. Li et al.

    Unsupervised feature learning for human activity recognition using smartphone sensors

  • Y. Guan, T. Plötz, Ensembles of deep lstm learners for activity recognition using wearables, Proceedings of the ACM on...
  • Cited by (0)

    Ehsan Nazerfard received the BS and MS degrees in Computer Engineering and Artificial Intelligence from Amirkabir University of Technology and Sharif University of Technology, Tehran, Iran, in 2003 and 2006, respectively. He also received the PhD degree in Computer Science from Washington State University, WA USA, in 2014. He is currently an Assistant Professor in the department of Computer Engineering at Amirkabir University of Technology. His research interests include Machine Learning, Deep Learning and Internet of Things.

    Elnaz Soleimani received the BS and MS degrees in Computer Engineering and Artificial Intelligence from Amirkabir University of Technology, Tehran, Iran, in 2016 and 2018, respectively. She also received master degree in AI from University of Paris Est Creteil, Paris, France, in 2019. Her main research interests include Machine Learning, Transfer Learning and Computer Vision.

    View full text