Continuous wavelet filtering on webcam photoplethysmographic signals to remotely assess the instantaneous heart rate
Introduction
Recognizing an emotion by its physiological signature is a field of research that presents a particular interest on the last 10 years. Understanding emotions can be useful, particularly in virtual therapies, where emotions are feedbacks that regulate the virtual environment level and intensity. Physiological parameters like heart rate (HR) and heart rate variability (HRV) are reliable inputs to emotion recognition [1], [2], [3]. However, contact sensors can be limited in some scopes of application where a specialist must install and monitor them. When dealing with serious games, contact sensors can disturb the interaction and may be intrusive to the privacy.
Non-contact measurements of physiological parameters can be achieved using thermal infrared imaging, a technology employed by Pavlidis et al. to collect physiological data on human faces like heart and respiratory rates, perspiration, supraorbital and periorbital blood flow [4], [5], [6], [7]. Similarly, Doppler radars are non-contact sensors that were used to detect heartbeats [8], [9] and respiration signals [10]. More recently, digital cameras and webcams were employed on the face to detect blood volume pulse (BVP) [11], [12], [13], [14], [15] and compute HR and breathing rate (BR). The principle, based on photoplethysmography (PPG) consists in observing light variations on the skin to recover the cardiovascular pulse wave. This optical technique is mainly implemented in contact pulse oximeters where infrared wavelengths are employed to detect the pulse wave. Considered in this case as noise, ambient light is now an illumination source used for PPG exploitation via high sensitivity cameras and webcams. The main drawback of this technique is that PPG signals are susceptible to motion-induced artifacts, particularly when dealing with webcams and ambient light. Independent component analysis, a blind source separation method, has been proposed by Poh et al. [13] to remove noise artifacts from face imaging PPG signal. Standards of measurement recommend the use of ECG sensors to measure HRV [16]. However, it has been shown that pulse rate variability derived from PPG signals can be a good surrogate of HRV at rest [17]. Sun et al. [14], [15] have compared performances between a low-cost webcam and a high-sensitivity camera to assess HR and pulse rate variability. They conclude that the functional characteristics of a 30 fps webcam are comparable to those of a 200 fps camera when interpolating signals to improve the time domain resolution [15].
The HRV is a parameter used in affective computing and psychophysiology to give an index of the autonomic nervous system (ANS) activity in order to detect workload changes in real time [19]. Its spectral analysis can provide the sympathovagal balance, a ratio that reflects reciprocal changes of sympathetic and vagal outflows [18]. The HRV tends to be rhythmic and ordered in positive emotional states and follows the respiration by a phenomenon called respiratory sinus arrhythmia (RSA). In contrast, the HRV tends to be chaotic and disordered in states of anger, anxiety or sadness. These rhythmic variations provide a state known as cardiac coherence [20], [21]. The use of non-contact means to detect physiological signals is particularly advantageous in affective computing, where the objective is to induce emotions like stress or fear for example. In these psychophysiological experiments, contact sensors may generate a bias by interfering with the user, resulting practically by an erroneous emotion classification [4]. Currently, published methods effectively recover HR, BR and HRV spectral components over a given time period. However, few attempts [15] have been made to measure the instantaneous HR (iHR) with a webcam, especially when considering head motion artifacts.
The immediate objective of this study was to provide a motion-tolerant method that reliably recovers the instantaneous pulse and breathing rates using a low-cost webcam. The presented methodology was developed to overcome signal variations generated by natural head movements. Firstly, we describe the approach, where robust image and signal processing are introduced to gather exclusively skin pixels that contain PPG information. A wavelet filtering algorithm was elaborated to recover both the instantaneous HR and BR. Secondly, we validate the accuracy of the proposed approach using approved contact probes. Webcam PPG signals were remotely recorded from 12 healthy volunteers during a set of two experiments, specifically at rest and during motion. Remote measurements of the instantaneous heart and breathing rates were respectively compared to those acquired from BVP and chest belt sensors.
Section snippets
Experimental procedure
Two experiments were conducted indoors to evaluate the iHR assessment method on 12 healthy volunteers (Table 1) of both gender and various ages. The skin type reported in Table 1 corresponds to a visual estimation of the participant skin color using the Fitzpatrick chromatic scale [22], defined between I for white skins and V for black skins. For practical purposes, categories I and II were regrouped in one set. All participants gave their informed consent before the beginning of a session.
Results
Image and signal processing detailed in the previous section were employed to compute the iHR series and respiration traces of all subjects. These two physiological parameters were simultaneously recorded by contact sensors. A typical example from subject 3 (male, age = 22 years and skin type = I–II) is presented in Fig. 6 where respective signals are directly comparable. Pearson's correlation coefficients along with a statistical analysis and Bland–Altman plots were used to quantify the level of
Discussion
The HRV is a physiological measurement used in several domains, especially in affective computing and personal health care. The present study demonstrates that the instantaneous HR can be assessed robustly using a low-cost HD webcam on human faces, even in presence of motion. Statistical and beat to beat analysis reveals that webcam-derived HR and BR are in close agreement with reference sensors. The respiration signal is recovered using the method presented in Section 2.3.3.2. The breathing
Conclusion
This study presents image and signal processing techniques to remotely assess the instantaneous HR. Using a skin detection filter, the proposed method selects only skin pixels that contain PPG information. The u* component of the CIE L*u*v* color space is used to increase robustness on motion and light variations. A typical example is presented in Fig. 9. A wavelet-based filtering operation is then applied to detrend and denoise raw signals. It has been shown that trends in PPG signals must be
References (33)
- et al.
Interacting with human physiology
Computer Vision and Image Understanding
(2007) - et al.
Heart rate measurement based on a time-lapse image
Medical Engineering and Physics
(2007) - et al.
Real-time detection of workload changes using heart rate variability
Biomedical Signal Processing and Control
(2012) - et al.
Heart rate variability. Applications in psychiatry
L’Encéphale
(2009) The CIE system of colorimetry
Digital video and HDTV: algorithms and interfaces
(2003)The absorption spectra of hemoglobin and its derivatives in visible and near infrared regions
Journal of Biological Chemistry
(1943)- et al.
Adaptive skin color modeling using the skin locus for selecting training pixels
Pattern Recognition
(2003) - et al.
Emotion recognition using physiological signals
Lecture Notes in Computer Science
(2006) - et al.
Emotion recognition through physiological signals for human–machine communication
Cutting Edge Robotics
(2010) - et al.
Physiological signals based human emotion recognition: a review