Elsevier

Pattern Recognition Letters

Volume 49, 1 November 2014, Pages 231-237
Pattern Recognition Letters

Compressive classification of human motion using pyroelectric infrared sensors

https://doi.org/10.1016/j.patrec.2014.07.018Get rights and content

Highlights

  • Designed a mesh-based feature extraction method for the infrared radiation domain.

  • Developed a motion classification system by using a hardware architecture.

  • Validated a compressive classification scheme for human motion with experiments.

Abstract

Vision-based approaches have been widely applied in motion classification. However, their applicability is often limited by much higher data-loads and computational costs, particularly in the case of constrained recourses. In this paper, a compressive sensing based approach is investigated for motion classification by using pyroelectric infrared (PIR) sensors. We represent a human motion as a spatio-temporal energy sequence (STES) and extract it from an infrared radiation domain. To generate this sequence, a mask is used to divide the object space into small meshes, from which the human-motion induced variances in the infrared radiation will be used to construct a feature. Because of the sparsity of STES, a hardware prototype that is composed of a PIR sensor array and a visibility mask is designed for measuring STES compressively, and a nearest neighbor classifier is then used for classification in the compressive measurement domain. To evaluate the proposed approach, we recorded 360 compressive STESs of ten aerobic exercises performed by six persons. Encouraging experimental results validate the feasibility and efficacy of our approach.

Introduction

Human motion classification is an important research topic, and it can be found in many applications, such as surveillance-based security, human–computer interfaces, automatic diagnostic of orthopedic patients and analysis of an athlete’s performance. A traditional vision-based motion recognition paradigm has been widely used, and it can provide intuitive and rich information. This paradigm exploits techniques from the field of computer vision [6], [18], [17], such as motion segmentation, feature extraction, and motion tracking to analyze visual observations for pattern recognition. Although the vision-based motion recognition paradigm has been well studied, the methods developed still have some limitations: they require large amounts of memory for storage of video sequences, and they rely on sophisticated computer vision algorithms that are highly sensitive to the light illumination and background changes. Moreover, they are inflexible to use in terms of sensing efficiency, as they are inevitably associated with much higher data-loads and computational costs, especially in the case of constrained resources such as wireless sensor networks.

To provide a supplement and alternative to traditional vision-based approaches, we propose a compressive sensing based approach for motion classification using infrared motion sensors. Human motion classification involves the essential step of feature extraction, whose efficiency highly influences the computational complexity of the whole classification system. Many approaches have been proposed for feature extraction [23], [16], [1]. One approach based on the concept of a mesh [19] has been reported as a simple and effective means for motion classification using video sequences [21], [15]. In this approach, the mesh features are first extracted from a binarized image through subtracting the background, and then, a feature vector sequence is constructed according to the mesh feature vectors from the time sequential images. This approach retains a high computational complexity in constructing the mesh and calculating the statistical mesh feature in the pixel domain. However, in contrast to the previous work in Yamato et al. [21], in this paper, we consider only the mesh feature in the infrared radiation domain, and we propose a hardware implementation for feature extraction without the need of heavy computations. In our method, a mesh-based feature vector sequence in the infrared radiation domain, namely a spatio-temporal energy sequence (STES), is exploited to represent the human motion. Here, without needing the complicated computations of mesh feature extraction in Yamato et al., Polana and Nelson [21], [15], STES can be directly measured by using the compressive sensing system that is composed of a pyroelectric infrared (PIR) sensor array associated with the appropriate visibility modulation.

PIR sensors are particular suitable for human motion detection, because they are sensitive to the human motion induced variances in the infrared radiation, but robust to variations of illumination conditions [11]. In addition, infrared motion sensing paradigm offers a promising alternative to optical sensing paradigm because of its potential to provide low cost and nonintrusive sensory approaches without interference in comparison to other motion sensing paradigms such as wearable motion sensors [22].

Previous studies [8], [12], [13] have shown that the PIR sensors with the visibility modulations are capable of identification or classification of human body motions with high sensing efficiency. The key to these studies is the concept of visibility modulation with geometric reference structures, which can facilitate efficient multidimensional imaging and the analysis of radiation sources under the reference structure tomography (RST) paradigm [2]. In comparison to vision-based systems, the RST enhanced motion recognition with PIR sensors has notable advantages in its sensing efficiency. The reference structure that uses a Fresnel lens array with visibility modulation masks segments the radiation source space into small meshes, which are also called sampling cells. The more meshes that are segmented by reference structures, the higher the achieved spatial resolution. In recent studies, in Luo et al. [13], the resultant mesh with 17 sampling cells is specifically suited for the classification of human body motions (fall, stand, sit). Nevertheless, PIR sensor-based classification of the fine-grained motion of the limb parts (upper limbs, a leg) is still a challenging problem when a mesh with such a low spatial resolution is used.

In contrast to the aforementioned studies, the advantage of our approach is that it accounts for the mesh feature in the infrared radiation domain and performs feature extraction using hardware without the need of additional complex computations. In this paper, the mesh feature is constructed by using a reference structure, specifically a Fresnel lens array with visibility modulation masks, in such a way that the classification results can be refined at a fine-grained level. To further improve the classification performance, our method is enhanced with a compressive sensing mechanism [4], which exploits the jointly sparse structure of the mesh feature vector sequence. Using a PIR sensor array and a visibility modulation mask, a hardware architecture is especially presented for measuring the feature vector sequence compressively, and a nearest neighbor classifier is then used for motion classification in this low-dimensional space. Our experimental results demonstrate the technical feasibility of our approach in performing motion classification in the compressive measurement domain.

The remainder of this paper is organized as follows: Section 2 introduces the concept of the mesh feature vector sequence and analyzes the structural properties of the STES. A compressive sensing mechanism that uses PIR sensors with a visibility modulation mask is presented here to measure compressive the STES. In Section 3, a nearest neighbor classifier is subsequently used to efficiently infer motions from the compressive STES. Section 4 describes the implementation of the hardware architecture for compressive classification. The experimental studies are given to validate the proposed approach in Section 5.

Section snippets

Feature extraction

A classic experiment by Johansson [10] demonstrated that humans can perceive gait patterns from point light sources that are placed at a few limb joints with no additional information. Point trajectories of moving objects are apparently distinctive enough to infer different motions of that object. The trajectories can be observed from the spatio-temporal distribution of the human motion induced variances in the infrared radiation. According to this observation, the STES that includes the

Compressive classification

Signal classification with compressive measurements [4] has been reported as a computationally efficient paradigm. With the compressive measurements of the STES, the issue of motion recognition can be solved through measuring motion similarities between the test sequence and the template sequence in the compressive measurement domain.

Let the compressive measurement VRM×T be a test sequence associated with a query motion. Suppose that a training set is selected from the compressive measurements

Hardware implementation

This section will describe the hardware architecture of a compressive motion classification setting. For demonstration of the proposed compressive motion classification method, the prototype is developed as shown in Fig. 3. The size of our prototype setting is 14cm×4cm×21cm. As shown in Fig. 3, 9 PIR sensors are deployed in a 3×3 square array. The PIR sensor and Fresnel lens used are D205B and 8005, respectively. A mask that is composed of 9 sub-masks is mounted in front of the PIR sensor array

Experimental results

In this section, we will validate our compressive classification method in the context of the classification of aerobic exercises.

In our experimental study, all aerobic exercises are performed in a 200cm×200cm×50cm cube. Our prototype system is displaced in front of the human body, and the distance between the PIR sensor array and the human body is 150 cm, as shown in Fig. 5.

In this experiment, we create an infrared motion dataset for 10 aerobic exercises. Example images of all motions are shown

Conclusions

In this paper, we present a compressive sensing based approach and its hardware implementation for human motion classification. In doing this the human motion feature is captured by a PIR sensor array with visibility modulation in the infrared radiation domain. The spatio-temporal energy sequence (STES) is proposed to characterize the human motion features. With the jointly sparse structure of this feature, the compressive sensing based paradigm is developed for human-motion classification. In

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant Nos. 61375080 and 61074167). The authors would like to thank all staff of Information Processing & Human-Robot Systems Lab in Sun Yat-sen University for their precious help, as well as the anonymous reviewers for their constructive comments and suggestions.

References (25)

  • G. Hyseni et al.

    Infrared thermal detectors parameters: semiconductor bolometers versus pyroelectrics

    WSEAS Trans. Circuits Syst.

    (2010)
  • G. Johansson

    Visual perception of biological motion and a model for its analysis

    Percept. Psychophys.

    (1973)
  • Cited by (30)

    • Pyroelectric infrared device with overlap dual capacitor structure sensor

      2018, Sensors and Actuators, A: Physical
      Citation Excerpt :

      Pyroelectric infrared device was widely concerned by the researchers due to its applications in body detection [1], gas detection [2] even the thermal imaging [3].

    • A literature review on driving factors and contextual events influencing occupants' behaviours in buildings

      2017, Building and Environment
      Citation Excerpt :

      The first point to deal with concerns technologies for users' presence recording. One of the most used are passive infrared sensors (PIR): they are sensitive to human motion, low-cost and nonintrusive [28,29]. However, such technology cannot detect immobile occupants (e.g. people working at desk in offices) and it is not able to decode the number of people.

    • Understanding occupancy pattern and improving building energy efficiency through Wi-Fi based indoor positioning

      2017, Building and Environment
      Citation Excerpt :

      There are a variety of occupancy detection technologies currently available. The deployment of passive infrared (PIR) sensors for occupancy detection is one of the earliest though its effectiveness is often restricted by the fact that these motion sensors cannot normally detect immobile occupants' presence [13]. However, research has been carried out by others to overcome this limitation through introducing and optimizing a time delay to PIR sensors data [14,15] or through adopting probabilistic models [16,17] to achieve a greater accuracy of presence detection.

    • Intelligent evacuation guidance systems for improving fire safety of Italian-style historical theatres without altering their architectural characteristics

      2016, Journal of Cultural Heritage
      Citation Excerpt :

      PIR sensors for detecting if anyone is moving in a space/room [48]; PIR array sensors for detecting motion speed and direction along the path; applied on the ceiling or on the walls [75,76]; RFID sensors (e.g. gates) for determining individuals crossing particular passages, such as door, and so determining flows of pedestrians during the time and number of people along the path stretch between two consecutive passages [39,51,77]; active and passive RFID would implies the distance between the crossing pedestrians and the detectors;

    • Human motion classification using 2D stick-model matching regression coefficients

      2016, Applied Mathematics and Computation
      Citation Excerpt :

      Clearly, human motion classification has become an active field of study. Recent human motion classification works [18,20–22,54] applied the classification method to the available or captured motion data, but the authors did not classify estimated matching motions. Our goal is to address the gap in previous works on estimation models for classification analysis.

    View all citing articles on Scopus

    This paper has been recommended for acceptance by Jie Zou.

    View full text