2.1.

Photoplethysmography Imaging

The PPGI technique is an advancement of the classical and well-established photoplethysmography (PPG).^3–5 Nowadays, PPG is widely used in many medical disciplines and is appreciated by physicians because of its simple design and relatively low cost per examination. PPG systems with optical sensor technology in the near-infrared ($\sim 900\mathrm{nm}$) light range can noninvasively detect changes in blood volume (related to arterial and venous hemodynamics) in the dermal blood vessel network by registering the optical attenuation $A$ of transilluminated tissue, according to the Beer-Lambert law.⁶

In this active optical sensing system, one or more selective light sources and one or more light detectors are integrated into the optical sensor unit, which can work in reflection or transmission mode.

Vital signs that can be observed include HR and RR (and their variability), which may provide information on the autonomic nervous system, perfusion phenomena with frequencies below respiration, venous reflux, venous refilling rates, and other diagnostically relevant physiological indexes or signals.^7–10 Reflection mode sensor technology (rPPG) can be applied to practically all body areas, thus overcoming the limitation of transmission (tPPG) sensors, which can only be applied to thinner body parts, such as the fingertip or ear lobe.

In rPPG, distances between light source and light detector are in the order of a few millimeters [Fig. 1(b)], and typical penetration depths of light into the tissue are around a few millimeters, depending on the wavelength and sensor geometry.⁷

Fig. 1

History of reflective photoplethysmography, from (a) Hertzman’s first optoelectronic photoplethysmograph³ to (c) modern remote photoplethysmography imaging (PPGI).¹⁶

Using two or more wavelengths at the same time also allows one to measure arterial oxygen saturation in skin, for example, using the pulse oximetry method, see, e.g., Ref. 11.

In PPGI, the classical reflective-mode, skin-contact PPG sensor is replaced by a highly sensitive camera detector and an external illumination unit consisting of multiple light-emitting diodes.^12–16 In particular, to detect the weak light modulation caused by blood hemodynamics, our setup¹⁷ uses the Pike210B camera (Allied Vision Technologies Ltd., Hod-Hasharon, Israel) chosen due to its high dynamic range of 96 dB and its high readout speed of up to 31 full frames per second. The imaging sensor is a Kodak KAI-2093 type 1 progressive scan CCD (diag. 15.3 mm) with a pixel resolution of $1928\times 1084$ and a spectral range of 320 to 950 nm with a quantum efficiency of $>40\%$ at 480 nm.

The setup shown in Fig. 2 allows one to measure arbitrary parts of the skin surface and to simultaneously assess dermal perfusion in different skin regions. The PPGI system can assess local changes in perfusion patterns and create a functional mapping of the perfusion status. After recording a video sequence of the skin surface, the operator can select different regions of interest (ROIs; virtual sensors) for which the backscattered light intensity is calculated. The resulting signals contain the same information as the classical PPG signals. Of considerable importance is the possibility to obtain measurements even in wounds, transplanted skin, or on a neonate’s sensitive immature skin.

Fig. 2

(a) Early PPGI setup realized at RWTH Aachen University in 2000.¹² (b) Screen shot of a typical perfusion signal.

The PPGI camera is connected to a high-performance PC via FireWire (IEEE1394b). The PPGI software was developed to control the camera settings, capture the image sequence, and perform postprocessing and analysis. Currently, the PPGI software is capable of acquiring the video in online mode and saving it for offline processing. In online mode, the live video can be displayed dynamically. The maximal sampling rate amounts to 31 fps at full resolution, but it can be increased by reducing sensitivity, acquiring only a subarray, or by binning. The file size of a typical PPGI sequence with spatial resolution of $640\times 480\mathrm{pixels}$, 15 Hz sampling frequency, and 1-min duration is $\sim 500\mathrm{MB}$. At full frame and maximal readout speed, 1-min recordings will produce $>6\mathrm{GB}$ of data. Hence, to prevent extensive storage consumption, the recording parameters should be limited according to the analyzing requirements: the necessary image detail and sampling rate. Inside the PPGI frames, different sizes and positions of ROIs can be selected so that virtual PPG sensors of variable measuring window can be freely selected at different places. Here, typical sizes of these ROIs range from 10 to 50 pixels edge length. By calculating the average pixel values of the ROIs, the reflected optical signal (the PPG signal), which corresponds to blood volume changes, can be monitored. This signal describes the skin hemodynamics inside the ROI in the time domain.

For offline processing, first, the video is calibrated. Calibration includes equally intervalled resampling and motion compensation. Motion artifacts may occur due to movements of the patient/infant, or the patient’s body part in relation to the camera. Calculating differences relevant to subsequent images provides a first indication of the occurrence of movement. At points in time with motion artifacts, the calculated differences increase considerably, especially in regions with high contrast. Tracking of easily detectable edges in the images allows conclusions to be drawn about, e.g., an infant’s activity or, when monitoring the rise and fall of the chest contour, the registration of breathing activity.

For evaluation of perfusion information from PPGI sequences, motion artifacts are generally disturbing. On one hand, a clear spatial assignment of detected signals to certain body regions is no longer possible. On the other hand, the detected signal itself is corrupted by coupled frequency components of the movement artifact. Depending on the movements, the detected signal may be totally disrupted. To avoid this, depending on the type of expected movement (e.g., slight, strong, translational, or rotatory), the computational time available, and required accuracy of the result, various compensatory techniques can be applied.¹⁵ In our research, we mainly use block-matching algorithms, hierarchical motion approximation with the modified Newton algorithm, or calculation of motion constraints. The temporal effort for these methods depends on the image resolution, the size and number of tracked ROIs as well as the required accuracy. On modern PCs, the movement compensation of a single ROI with our experimental setup can be computed in quasi-real-time. For IRTI, these motion detection and compensation methods are applied in an analogous way.

2.2.

Infrared Thermography Imaging

Similar to PPGI, IRTI is a contactless but passive imaging technique for spatial and temporal registration of an object’s surface temperature.

To understand the basic principle of IRTI, it is essential that any physical object with a temperature above the absolute zero point (0 K) emits radiation. Physically, the specific spectral radiance $M(\lambda ,T)$ of ideal black bodies is described by Planck’s law.¹⁸

In this equation, $T$ is the black body’s temperature, $h$ is the Planck constant, $c$ is the speed of light in a vacuum, $\lambda$ is the wavelength, and $k$ is the Boltzmann constant. For selected temperatures, the Planck spectra are shown in Fig. 3.

Fig. 3

Specific spectral radiance for black bodies at different temperatures. (a) Image shows distribution at 3000 to 6000 K. (b) Image for human body surface temperatures. For temperatures around 6000 K, the maximum intensity is radiated in the visible spectrum. For lower temperatures, this maximum decreases and shifts to the long-infrared region. The maximum spectral sensitivity of the infrared thermography camera (VarioCam hr head) lies $\sim 9.5\mu \text{m}$ in the far-infrared region¹⁹ and is therefore suitable for the study of human body temperature.

The maximal emitted intensity for a fixed-size wavelength interval is strongly dependent on the body’s temperature. In fact, according to Stefan-Boltzmann’s law, the complete emitted radiation power $P$ in the whole spectrum is a function proportional to the body’s temperature to the power of four.¹⁹

A shift of the maximum to longer wavelengths for lower temperatures can also be observed. This effect is described by Wien’s displacement law.⁴

For typical skin temperatures (ca. 35°C), this maximum shifts to $\sim 9.5\mu \text{m}$ lying in the infrared C-band (IR-C). To draw conclusions about an object’s temperature by measuring its passive radiant emittance, the sensing device should be most sensitive in this wavelength region.

To record thermal image sequences, the VarioCam hr head (InfraTec GmbH, Dresden, Germany) is used. It possesses a geometric resolution of $384\times 289\mathrm{pixels}$ and a sampling frequency up to 50 Hz.²⁰ It is sensitive in the long-wave infrared band (7 to 14 μm) and most sensitive at 9.5 μm. Its dynamic range of 72 dB allows measurement of temperatures from $-40$ to $+1200°\mathrm{C}$ with best temperature resolution at 30°C (better than 0.03 K). Recorded data may be transferred via FireWire to a PC. The file size of a typical thermal video sequence of 1-min duration at 50 Hz is $\sim 400\mathrm{MB}$.

The recorded signal intensity correlates with the incident radiation weighted with the sensor’s sensitivity characteristic. With respect to internal calibration of the camera with regard to damping of the object lens, sensor, and camera temperature, and duration of exposure, this intensity can be correlated to the black body temperature. The distance of the observed object relative to the camera does not affect the detected temperature because the radiation density of a fixed-sized object decreases with the square of the distance, but at the same time, the size of an object observable with a single sensor pixel increases with the square of the distance.

However, to obtain precise measurements an additional calibration is required because in this way only temperatures of ideal black bodies can be measured exactly. However, objects of interest generally act like gray bodies or selective radiators. Their specific radiation $M_{\mathrm{obj}}$ is less than the specific radiation of an equivalent black body $M_{\mathrm{BB}}$.

The emissivity coefficient $\epsilon$ is dependent on the wavelength of the object’s material and its thickness. In the wavelength interval of the sensor’s sensitivity range, $\epsilon$ can be defined unambiguously. Typical values of $\epsilon$ for different materials are provided in reference tables or by the camera manufacturer.^20,21 Table 1 presents emissivity constants for a selection of different materials valid in the IR-C band (8 to 14 μm).

Table 1

Emissivity coefficients ε in the infrared C band (8 to 14 μm).20,21

Although the aberration of skin from an ideal black body is relatively small (0.97 versus 1), its influence is not negligible for accurate measurement. According to Eq. (5), an aberration of 0.03 of the detected radiance intensity leads to a nontolerable error of 2 to 3 K for measured skin temperature. However, knowing the emissivity coefficient, the equivalent black body radiation and thus the temperature can be estimated.

An emissivity coefficient close to 1 also indicates almost black body behavior. This means that nearly all incident radiation is absorbed. Hence, the penetration depth of electromagnetic radiation into the subject is close to 0. In reverse, all measured radiation from a black body has its origin at its surface. For example, radiation from deeper skin layers is already absorbed by the overlying layers. As a consequence, the measured temperature corresponds to the upper surface skin temperature.

Besides calibration according to the type of material involved, the measuring section itself can influence the recorded temperatures and therefore needs additional calibration and compensation steps. Figure 4 shows the potential sources of errors depending on the measurement geometry.²²

Fig. 4

Composition of detected radiation in infrared thermography (IRT) monitoring (modified from Ref. 21). The amount of radiation entering the measuring section (1) toward the IRT camera (2) is a sum of the emitted radiation (3) of the measured object (4) according to its temperature, environmental radiation (5) reflected on the object’s surface (6) and background radiation (7) transmitted through the object (8). Therefore, the total detected radiation (9) depends on the absorption and emitting characteristics of the media (10) in the measuring section.

On one hand, total radiation coming from the measured surface is generally a sum of three separate components. The first is the emitted radiation of the body (which we are interested in), the second is the transmitted radiation from the background through the body, and the third component is the radiation reflected on the object’s surface from the environment. On the other hand, total radiation detected by the camera differs from the sum of the radiation coming from the measured surface depending on the media in the path of the rays. Here, part of the radiation might be absorbed, or even new components might be emitted toward the camera, according to the media’s material property, density, and temperature.

For medical IRT applications, the influence of transmitted radiation through tissue can be neglected because the human body is not transparent in the IR-C band (Table 1). Reflections at the surface are strongly suppressed because of the nearly ideal black body behavior of skin, absorbing incoming radiation; however, minor parts are reflected toward the camera. The percentage of this reflected radiation of the total radiation is a function of the environmental radiation intensity. For example, monitoring of neonates under an IR heating unit requires compensation of the reflected components. Disturbing media in the pathway are not totally transparent objects, like windows or even air moisture.²³ According to Eq. (1), their effect on measurement increases when increasing the measuring distance or thickness of the material.

As is usual, the environmental and geometric conditions of IRTI measurements are assumed to be constant, or at least changing only slowly (e.g., room temperature or monitoring of a calm patient). Therefore, even disturbing influences will not change during measurement. This allows one to either calculate the offset in the temperature measurement if all environmental factors are known or compensate for their influence by additional calibration measurement with known references (e.g., measurement on dummies with known temperature and emissivity).

3.1.

Results from PPGI

For the PPGI system, the extraction of vital parameters is comparable to the classical PPG. Due to the spatial resolution, multiple time series (instead of a single one) need to be analyzed. Time series can be obtained either from the mean time-varying gray values inside the ROIs or for every pixel (or pixel array) separately. Algorithms used for extraction of vital signs in classical PPG are applicable one-to-one on these time series. Therefore, vital parameters (like HR or HR variability) can also be recovered from the PPGI sequence. Depending on the observed perfusion frequency band, in addition to rather, other rhythmic phenomenon (like breathing pattern or vasomotor activity) can also be monitored.

Figure 5 shows a typical PPGI recording with the corresponding time series of three ROIs. The sequence was recorded on the left hand with a small wound on the middle finger [Fig. 5(a); red ROI). For each ROI, the time signal can be calculated by estimating the mean gray value inside the ROI for each frame successively. Hence, the absolute perfusion amplitudes may feature strong inter- and intrapersonal variability; the signals were normalized by dividing them through their dc components to improve comparability. In all three time series, a signal component that strongly correlated with the heartbeat is clearly visible at $\sim 74\mathrm{bpm}$ [Fig. 5(b)]. However, the perfusion patterns from healthy skin and from the wound on the middle finger show significant differences. When examining only the heartbeat, its amplitude is slightly increased inside the wound, whereas the slower rhythms at $\sim 0.1\mathrm{Hz}$ are markedly reduced. More detailed analysis of signal components can be obtained by applying band-pass filters to the signals. Depending on the selected cut-off frequencies, for example, either rhythms correlated to the respiration frequency band [Fig. 5(c)] or vasomotor rhythms [Fig. 5(d)] around 0.1 Hz can be extracted. Please note that minor inaccuracies in the first and last few seconds of the band-pass filtered sequence are due to filter characteristics. In this example, although the amplitude of the respiration correlated signal is much weaker than the heartbeat or 0.1-Hz rhythm, phase synchronous patterns can be monitored on any region on the skin. In contrast, $\sim 0.1\text{-}\mathrm{Hz}$ (vasomotor) rhythms differentiate between wounded and healthy skin. Inside the wound, the slow rhythms have almost vanished. In healthy skin, even in adjacent ROIs, local variations are clearly visible.

Fig. 5

Example of PPGI analysis. (a) PPGI sequence of subject’s left hand with a small wound on the middle finger. Inside the first frame, three regions of interest (ROIs) are indicated: one inside the wound (red) and two close to each other on the back of the hand (blue and green). (b) Corresponding time signals for each ROI. (c) Band-pass filtered signals with cutoff frequencies at 0.2 and 0.6 Hz correlated with respiratory activity. (d) Band-pass filtered signals with cutoff frequencies at 0.02 and 0.2 Hz representing slow signal rhythms. (e) Fast Fourier transformation of the red signal. Five peaks in the frequency domain are visible: $\sim 0.1\mathrm{Hz}$ rhythm (1), respiratory rhythm (2), heart rate (3), and harmonics of heart rate (4 and 5).

A similar behavior can be monitored with PPGI recordings on the forehead of a relaxed, seated subject. Here, spatial rhythm fluctuations in dermal perfusion are visible. This phenomenon is best observed in actual video sequences of the PPGI recordings as blood volume clouds, which move across the forehead in an organized but complicated pattern.

Figure 6 aims to illustrate this with screenshots of the recording; the actual video is available as an electronic supplement on the website of the Journal of Biomedical Optics.

Fig. 6

Phenomenon of distributed blood volume clouds observed with PPGI on the forehead of a seated subject in a relaxed steady state. The postprocessed and colored images show perfusion-coded images taken at 14, 16, 50, and 69 s of a PPGI video sequence (Video 1, MPEG, 2 MB) [URL: http://dx.doi.org/10.1117/1.JBO.19.1.016012.1].

An alternative approach to analyzing the frequency components of recorded time series (superposition of all rhythmic phenomena, of venous and arterial origin, in the dermal perfusion) is visualization of these phenomena by analyzing beat-to-beat distances of subsequent heartbeats (Fig. 7). It is assumed that the rhythmic changes of HR correlate with the respiratory sinus arrhythmia. For this kind of arrhythmia, the HR increases during inspiration and decreases during expiration. A combination of these video signal postprocessing strategies may provide new insight into perfusion dynamics.

Fig. 7

Beat-to-beat analysis of PPG signal. (a) Raw PPG signal with points of identical phase indicated (here: maxima) of single heart beats. (b) Beat-to-beat distance of the subsequent heart beats.

3.2.

Results from IRTI

Following approval from the Medical Ethics Committee of RWTH Aachen University Hospital (EK032/09 August 19, 2009), the development of temperature distribution in seven neonates inside an incubator was monitored for 10 min. The IRT setup [Fig. 8(a)] was calibrated taking into account environmental humidity, radiant heat sources of the incubator unit, and the optical parameters of the window inside the incubator. For the latter, the side door of the incubator was replaced by a thin polyethylene cling film, which seals the incubator’s microclimate from the environment but has several orders of magnitude higher IR transparency (0.92 to 0.94) than the usual acryl (Plexiglas) doors ($\sim 0.004$). A compensation routine was performed with a reference measurement, which was repeated every 2 min.

Fig. 8

(a) Setup for neonatal IRT monitoring, with (b) IRT image of neonate inside an incubator, and (c) temperature development for the selected ROI.

In Fig. 8, the middle and right images are a typical freeze-frame image of an IRT sequence and the temperature development for the selected ROI. Even inside the incubator, slight differences in skin temperature can be detected. For this image the extremities are ca. 0.5°C cooler than the head and $\sim 1°\mathrm{C}$ cooler than the abdomen due to good temperature insulation of the mattress and of the bent left arm.

It is interesting to take a closer look at the recordings of the infant’s nasal region. According to the breathing air flow, the surrounding skin is either cooled or heated by warm exhaled air or cooler inhaled air (Fig. 9). These temperature changes are clearly visible in the IRT recordings as long as the nasal region is in the field of view of the camera and the spatial and/or zooming ability of the camera offers sufficient image resolution of the small nasal region. Although temperature differences inside the incubator are relatively small compared to measurements on adults, with lower common environmental room temperatures of $\sim 20°\mathrm{C}$, fluctuations of 0.3°C are still visible. In the example presented in Fig. 9, the recorded RR was confirmed by additional neonatal bedside monitoring (IntelliVue MP30, Philips, Eindhoven, the Netherlands).

Fig. 9

Respiration rhythm of a neonate monitored with IRT, modified from Ref. 24.

In all recordings, the distribution of skin temperature of all not-covered skin regions could be monitored clearly. As long as the nasal regions of the neonates were visible in the IRTI image section and the neonates were calmly lying in the incubator (no turning of head or heavy struggling movements), the RRs could be calculated from analyzing the nasal region for all seven neonates at least in sections.