Introduction

Visually guided procedures provide instant feedback and meantime afford insights that would otherwise be difficult or even impossible to obtain.26 This is a primary reason for mankind’s continuous desire to extend vision beyond the boundaries of the human eye, which has resulted in many successful diagnostic imaging modalities like X-ray imaging, endoscopy, thermography, ultrasound scans and MRI that all found their way into the clinic.

While investigating the principal feasability of a camera for imaging blood oxygenation levels, we noticed that our multispectral images also contained information about subcutaneous vasculature, with improved contrast in the near infrared.24 Near infrared imaging of superficial blood vessels in itself is not new; soon after infrared photography2,9,25 early electronic cameras were used for experiments.13,15 Since then, numerous near infrared imaging methods have been developed using transillumination mode and/or reflection mode.5,6,10,14, 22 Our multispectral camera, however, provides the opportunity to obtain normal color images within the visible range (VIS) which are pixel-to-pixel matched with images obtained in the invisible near infrared range (NIR). By combining the image information content of both spectral bands with a processing algorithm, we developed a new technique that allows selective enhancement of superficial blood vessels with selectable pigment suppression within a normal color image.23

The underlying principle does not necessarily require stereoscopic image acquisition to derive increased vessel contrasts from the tissue. When, however, developing a new medical imaging technique, deriving image information from patient tissue is not the only issue. Especially for visually guided procedures it is also crucial to put effort in an ergonomic human interface and avoid conflicts between the proprioceptional and visual perceptions of the user. The added value of stereoscopic image recordings already was recognized and successfully used to document medical cases more than a century ago.4 We likewise reasoned that offering enhanced blood vessel contrast at the cost of depth perception would restrict the usefulness of our technique and therefore decided to realize a stereoscopic version of our vessel contrast enhancement device.

Visually guided procedures typically require a combination of well trained eyes and specific fine-motoric skills, since the coordination between eyes and hands is task dependent.16 For eye-hand coordination, brain processes for perception and action interact so closely that they cannot be separated and the influence of visual illusions (like human depth perception) to motoric tasks becomes stronger when input takes place via lower levels in the brain.3 The match between a human interface device and the visual and motoric brain processes strongly defines whether a technology can be applied intuitively or not. A new technology can be classified as intuitive if it speeds up the learning process for novices in a certain skill, without impairing the performance of persons already skilled in the existing art.20 Thus a useful increase in vascular contrast should neither imply a task-impairing decrease in depth perception (like in monoscopic techniques) nor distort spatial clues for human vision (like shadows) nor introduce false depth information (like projection parallax).

As a first reconnaissance of practical feasibility, we applied our new technique to several visual clinical procedures for which we expected that improved visualization of blood vessels would be of interest, being:

  • blood withdrawal;

  • vein inspection in dark skin;

  • detection of veins through iodide;

  • inspection of varicose veins and nevi pigmentosum.

Methods

Instrumental Setup

The instrumental hardware setup is schematically drawn in Fig. 1. Two synchronized identical single-chip CMOS-cameras (Vector Technologies Belgium, custom built) were equipped with apochromatic lenses and two identical custom-built dual-band LED-arrays (O2-View, the Netherlands). These LED-arrays were current controlled and each had two individually programmable channels for the emission of in total ±1.2 Cd visible white light with adjustable color temperature (consisting of 40 broadband white LEDs with a yellow accent and 20 broadband white LEDs with a blue accent) as well as one programmable channel for the emission of near infrared radiation (20 LEDs, 920 nm, max. 32 mW per LED). The light sources were constructed so that the geometrical beam profiles of VIS and NIR matched very closely (and thus also any resulting shadows and/or reflections). Left (L) and Right (R) image data was simultaneously acquired and streamed to a dual processor PC equipped with a stereoscopic monitor (Sharp LL-151–3D). It was also possible to connect a conventional cathode ray tube (CRT) monitor equipped with shutter glasses, for comparison of stereoscopic representation.

Figure 1.
figure 1

Experimental setup. Two CMOS-cameras, with apochromatic lenses and dual-band LED-arrays, simultaneously stream Left (L) and Right (R) image data to a dual processor PC. Both cameras captured color images within the visible range (VIS, 400–780 nm) and grey-scale images within the near infrared range (NIR, 910–920 nm) by sequentially switching between LED-array emission bands.

Full size image

A schematical representation of image aquisition and data structure is drawn in Fig. 2. The CMOS-camera detector chips were equipped with a Bayer filter mosaic to obtain an RGB color image within the VIS. The camera pixels, however, were also sensitive to near infrared radiation because all three filter channels of the Bayer mosaic (red, green and blue) were designed as highly transparant within the NIR. Thus four NIR-pixels were acquired for each VIS RGB color pixel group. By sequentially switching between emission bands of the LED-arrays, images within the Visual range (VIS, 400–780 nm) and Near Infrared range (NIR, 910–920 nm) were acquired in an alternating fashion. VIS and NIR image-size-settings could be varied independantly. For 8-bit encoding depth, at image-size-settings of 1280 × 1024 for VIS and 640 × 512 for NIR, 12 cycles/s were obtained (1 cycle = 1 VIS L&R-pair + 1 NIR L&R-pair). At the cost of decreasing image-size-settings downto 640 × 512 for VIS and 320 × 256 for NIR, camera speed could be increased up to 25 cycles/s. Encoding depths of 10-bit and 12-bit were also available, but only used for stills (due to the lower obtainable framerate).

Figure 2.
figure 2

Schematic diagram of image aquisition. The sequentially acquired alternating VIS and NIR raw image frames form 3D-matrices for the Left and Right channel. The NIR image size is smaller than the VIS image size to increase framerate while maintaining an overview of the imaged area. Enlarged details illustrate the NIR transparency of the Bayer pattern RGB-filters which are applied to obtain a VIS color image. The time domain axis is expressed in image cycles. Along this image cycle axis, the control signals for NIR and VIS LEDs (synchronized with respectively NIR and VIS camera exposures) are visualized.

Full size image

Data Aquisition

Multispectral stereoscopic movies were recorded in several typical clinical settings for which the technique was considered as possibly useful. LED currents and diaphragm settings were chosen so that for each movie saturated pixels were avoided. A preview mode allowed aiming, adjustment of converging angle α and focusing of the cameras. After software triggering the stereo-camera streamed a sequence of 8-bit digitally encoded image cycles to PC-memory (using auto-incremental numbering). All images were automatically saved on a fast SATA harddisk-array. Camera and light source settings were automatically stored in a text file and located in the same directory.

All patients and volunteers gave their informed consent for filming as well as for publishing the resulting image material. No diagnosis or therapy was based upon any of our results.

Data Processing

General Aspects

Processing was performed with custom developed software (programming language C++) using the stored text file with camera and light source settings as additional input values. Figure 3 schematically represents this process.

Figure 3.
figure 3

Schematic diagram of image processing. The images captured within the visible range (VIS) and the images captured within the near infrared range (NIR) are combined, which reveals blood vessel patterns below the skin. The left and middle column focus on edge-enhancement and suppression of superficial artifacts, the right column serves to fill-in the blood vessel lumen. For raw VIS & NIR images as well as processed results see figures 6, 7, 8 and 9.

Full size image

The processing method allowed discrimination between image information obtained from the tissue surface versus image information obtained from within the tissue. In order to achieve this, a distinction was made between shadows, reflections and absorption contrasts for both VIS and NIR.

Suppressing Shadows on the Surface

Since NIR and VIS beams were matched closely, shadows produced by irregular shapes at the tissue surface (e.g. skin structure, skin folds, nevi, hair, etc.) or by objects between the light sources and the tissue (fingers, needles, surgical tools, etc.) also matched well in both wavelength ranges. Our algorithm excluded such matching shadows from enhancement and left all useful aspects of shadows unaffected (e.g. depth clues).

Useful Effects of Shadows and Lighting Geometry

Due to the fact that the VIS and IR shadows matched very closely, the algorithm was able to selectively enhance contrast from below the surface while leaving shadows on the surface unaffected (thus not distorting these important depth clues).

Due to the shallow angle of the lightbeams, the skin texture was pronounced. Due to the lighting from two sides, objects that were brought towards the tissue surface (e.g. needles, scalpels, probes, etc.) when lighted from both sides, could produce two separate (not too heavy) shadows. These shadows met and typically formed a “V” pattern when an object touched the surface in the middle of the field of view, thus providing extra information for depth perception.

Enhancing Absorption Contrast of Blood Vessel Walls

Edge detection by a Prewitt image filter was performed on each VIS and NIR image. Pixel positions containing edge information above a certain adjustable threshold in both spectral regions were classified as surface artefacts and excluded from enhancement. The boundary regions of absorption contrasts caused by structures below the tissue surface, produced edges that were mainly present in the NIR image. Enhancement was selectively performed only for pixel positions where the NIR image contained more edge information than the corresponding VIS image. These “valid” pixels identified the vessel boundaries. The positions of these valid pixels were stored in a 1st NIR mask.

Discarding Reflections on the Surface

Shiny areas that produced reflections and/or saturated pixels also matched in both wavelength ranges. This characteristic allowed to calculate a 2nd NIR mask in which superficial reflections were also excluded from enhancement. Neighboring pixels of identified saturated pixels were excluded from enhancement. The radius of this exclusion region was programmable, but due to the favorable anti-blooming behavior of the CMOS-camera chips, suppressing direct neighbor pixels appeared sufficient.

Enhancing Absorption Contrast of Blood Vessel Lumen

The “content” of blood vessels was separately enhanced by raising pixel values of the normalized NIR image to the power of N (with N user adjustable between 0.5 and 2.5) while discriminating NIR pixels below a freely adjustable noise threshold and excluding information from identified shadows. Multiplication with the 2nd NIR mask then produced a final enhancement mask for subsequential backprojection into the VIS image by pixel-to-pixel multiplication.

Suppressing Contrasts Originating from Melanin Pigment

Superficial contrasts within the VIS, originating from melanin pigment concentrations, could either be filtered out or left unchanged. Figure 4 illustrates the normalized distribution of the intensities for the aquired separate spectral bands (R, G, B and NIR) in relation to the intensity (IVIS) of the composed visible RGB-image. Intensities were calculated using the Intel ippiRGBtoGray function.11 Four clouds of data points can be discerned, being IR/IVIS (red channel), IG/IVIS (green channel), IB/IVIS (blue channel) and INIR/IVIS (NIR channel). By calculating the ratio of (IR/IVIS )/(INIR/IVIS ) and comparing the result for each pixel with an adjustable threshold, it can be decided whether or not to apply backprojection to a pixel. This concerns the pixels located within the overlapping region of IR/IVIS and INIR/IVIS within Fig. 4. The resulting difference is clear when comparing between Fig. 9c and d with regard to the visualization of nevi pigmentosum and hair.

Figure 4.
figure 4

Relative intensity distribution of the different spectral bands. Dimensionless normalized distribution of the intensities for the 4 aquired red (I R), Green (I G), Blue (I B) and near infrared (I NIR) individual spectral bands expressed in ratio to the normalized intensity (I VIS) of the composed RGB-image. The four data clouds (R, G and B labeled by their natural colors and NIR labeled as pink) show a generally marked separation, but especially for a number of pixels within the red and infrared the data clouds partly overlap. This indicates regions where the VIS contrast potentially is superior to the NIR contrast. By comparing an adjustable threshold with the calculated ratio of (I R/I VIS)/(I NIR/I VIS) it can be decided which spectral band provides superior contrast for the pertaining pixel and thus whether or not it is used for enhanced backprojection.

Full size image

Data Presentation

The images could either be displayed on an auto-stereoscopic liquid crystal display (LCD) monitor or on a conventional CRT monitor equipped with shutter glasses.

The auto-stereoscopic LCD-monitor (Sharp LL-151–3D) was equipped with software controllable switching between stereoscopic (or 3D) and normal (monoscopic or 2D) mode. Monitor resolution was 1280 × 768 in monoscopic mode (XGA). In stereoscopic mode the available pixels were split-up in two separate images (L&R) by activation of a vertical LCD parallax barrier. The principle behind this technology is illustrated in Fig. 5.

Figure 5.
figure 5

Principle of applied autostereoscopic LCD-monitor (reprinted with permission from Sharp). In 2D mode, only one camera-channel is displayed (either from the L or R camera) and the parallax barrier is not actuated. Both eyes of an observer therefore receive the same image and a conventional flat image with full resolution is seen. In 3D mode, both camera channels are displayed (L&R) and the parallax barrier is actuated. The left eye and right eye of an observer now receive different images, and a stereoscopic (in-depth image) with halve resolution is seen.

Full size image

The CRT-monitor (iiyama vision master 21) was used at a resolution setting of 1280 × 1024 in combination with wireless shutter glasses (e-Dimensional) and thus provided stereoscopic information without sacrificing resolution.

The enhancement algorithm occurs on a pixel-to-pixel basis and does not affect resolution.

Our device offered several imaging modes for data presentation:

  • Monocopic raw VIS preview (normal full color vision)

  • Monoscopic raw NIR preview (greyscale)

  • Off-line stereoscopic looped VIS view with and without enhanced blood vessel back-projection (with freely adjustable enhancement settings).

  • Off-line stereoscopic looped raw NIR view or enhanced NIR view (with freely adjustable enhancement settings)

  • Stereoscopic stills in all modes

  • Monoscopic stills in all modes (freely switchable between Left and Right)

All modes offered the possibility for pause and scrolling forward or reverse frame-by-frame. By means of virtual slider controls and virtual pushbuttons the user interface allowed freely adjustable settings for shadow suppression, pigment suppression, noise threshold and vessel lumen fill-in contrast.

Results

Results for Blood Withdrawal

During routine blood withdrawal the inserted needle tip remained only slightly visible on the VIS image due to skin surface deformation (see Fig. 6a). On the raw NIR image, however, the inserted needle remained visible within the tissue for a few mm with some metallic reflection. Subcutaneous bleeding during needle removal could be detected in the raw NIR image while the needle tip was still in the tissue (see Fig. 6b). This was clearly highlighted by backprojection in the VIS image (see Fig. 6c). The processing settings used to obtain this backprojection mainly laid the accent upon enhancing the absorption contrast of blood vessel lumen, whereas edge enhancement was set to minimum. Reflections on the thumb nail did not lead to image distortion and the blood volume in the nail bed showed more contrast.

Figure 6.
figure 6

Routine blood withdrawal. Image pairs showing unprocessed images for VIS (a) and NIR (b) as well as the result after application of the new image processing method (c). Note the forked shadow (which is not effected by the enhancement algorithm), the clearly visualized subcutaneous bleeding and the improved visibility of the needle tip.

Full size image

Results for Dark Skin

Hardly any vascular contrast is present within the VIS image (see Fig. 7a). The NIR image, however, is not affected by skin pigmentation and reveals a subcutaneous vascular pattern (see Fig. 7b). The combined information results in an enhanced image (see Fig. 7c).

Figure 7.
figure 7

Influence of skin pigmentation. A dark skin color (a) has no significance for the applied NIR wavelength of 920 nm. Blood vessels provide good contrasts (b) and the resulting enhanced image (c) offers an improved visualization of the vasculature.

Full size image

Results for Vein Detection Through Iodide

Iodizing skin portions before surgery is a common clinical procedure which darkens the skin and lowers the visibility of blood vessels. Our method offers imaging right through iodide. To demonstrate this, we filmed a glass Petri dish, placed horizontally upon a volunteers arm while filling it up to a 3 mm thick layer of iodide solution. From the VIS image no vessels could be detected through the iodide solution (see Fig. 8b) and hardly any through the empty Petri dish (see Fig. 8a). The raw NIR image, however, clearly showed superficial blood vessels even through the Petri dish (see Fig. 8c).

Figure 8.
figure 8

Vein detection through iodide. Within the visible range, the superficial vasculature is only vaguely discernable (a). After filling the Petri-dish with a 3 mm thick layer of iodide solution, this fully blocks out the tissue view within the visual range (b), whereas a clear view of the vasculature remains possible at the applied NIR wavelength of 920 nm (fig 8c).

Full size image

Results for Varicose Vein and Nevi Pigmentosum Inspection

The device was used to film varicosis patients during dermatological outpatient clinic. The visualization of varicose veins could be drastically improved. Even in cases where the blood vessels were covered by subcutaneous fat, they could be detected quite well (see Fig. 9). By either de-activating or activating the first NIR mask in the processing algorithm (see Fig. 3) it was also possible to choose whether pigment contrasts in the visual range were left intact or filtered out. The difference is clear when comparing between Fig. 9c and d with regard to the visualization of nevi pigmentosum and hair. By clicking a software screen button, this suppression of pigment contrast could be freely switched on and off during viewing.

Figure 9.
figure 9

The VIS image does not contain much information about the underlying vascular pattern (a). The NIR image, however, clearly shows what’s hiding beneath the surface (b). Note that, when building the enhanced image, the nevus which is present in the VIS image can freely either be suppressed as a surface contrast (c) or be kept visible (d).

Full size image

Discussion

From the figures presented in this article, it is clear that all experiments resulted in improved visualization of superficial blood vessels. Based upon the absorption coefficients of Hb versus HbO2, arteries provide superior contrasts at 920 nm. The presented images, however, show enhanced venous contrasts, because the larger arteries are buried deeper underneath the skin. Stereoscopic movies provide a more lifely impression of the underlying, but presentation of stereoscopic movies is not possible in a printed journal. We therefore offer the possibility to download viewer software and movies via the internet http://www.erasmusmc.nl/ThoraxcenterBME/html/research/additional/bloodvesselcamera.htm

For both the autostereoscopic LCD and the CRT display plus shutter glasses, we found that the perception of depth was less for stills than for movies. This qualitative observation matches research findings on stereoscopic display techniques in X-ray technology8 and is also consistent with the differences between static and dynamic stereoacuity described by Mathias and Rudolf Sachsenweger.17 Stereoscopic movies also showed an increase in apparent image sharpness compared to stereoscopic stills, which may be explained by Shipley’s description of stereoscopic contour integration over blur.18 Compared to the autostereoscopic LCD-display, the increase in perceived sharpness was much stronger when using a CRT with shutter glasses. For monoscopic stills, however, the LCD-display (switched to monoscopic mode) was superior to the CRT. These observations are in agreement with the technology overview by Szold19 and can be explained by the sacrifice of resolution when using the LCD display in autostereoscopic mode, which price has not to be paid when using the CRT with shutter glasses.

The beneficial effect of separately adjustable enhancement paths for blood vessel contours and blood vessel lumen, which we experienced during the development of our processing algorithm, is consistent with the findings of Yin et al. that surface-features and edge processes make different contributions in determining an object’s unity and shape.27 The perceived usefulness of stereoscopic information, however, also supports the concept formulated by Tse that ‘‘mergeable’’ volumes, rather than relatable contours, are the critical elements in completion.21

With our technique we have aimed to avoid parallax errors and loss of vessel contrast by shadows which are inherent to other blood vessel contrast enhancement techniques projecting the vascular pattern via a projector onto the skin.28 We exclude parallax errors because the user truly looks beneath the tissue surface trough two camera’s. Absence of vessel contrast in shadows is avoided by lighting from two sides and by discrimination of superficial artifacts versus contrasts originating from below the surface.

This ability to selectively enhance contrasts from beneath the surface, while preserving the natural depth clues of shadows on the surface, supports both static and dynamic transmittance anchoring of the visual system which is crucial for depth perception.1 Preservation of natural shadows is furthermore important for the correct interpretation of depth clues by occlusion (e.g. from a hand or an instrument positioned between observer and tissue) thus precluding errors in depth perception.12

Since only carefully balanced white light is projected on the skin, there is no impairment of color perception. It is up to the user to freely switch between normal full color vision, color vision plus superimposed blood vessel backprojection and stereoscopic near infrared greyscale vision (with or without enhancement features).

The embodiment described in this article still has some drawbacks:

  • Due to the fact that the pertaining configuration aquires VIS and NIR images sequentially, motion artifacts can lead to a backprojection shift.

  • The deeper the vessels lie under the surface, the less we can visualize them. In the pertaining configuration, the depth range is limited to about 1 mm. Based upon the absorption coefficients of Hb and HbO2, arteries provide superior contrasts at 920 nm. The presented images, however, show enhanced venous contrasts, because the larger arteries are buried deeper underneath the skin.

The above mentioned drawbacks require further development. Fortunately, many technological improvents still can be added. Allthough the device worked quite well under normal ambient lighting conditions (100–200 Lux), the use of a switchable filter might further improve spectral separation. Furthermore, known techniques like application of crossed polarizers, in combination with more powerful light sources, can offer a considerably larger penetration depth.7

Conclusions

Compared to inspection with the naked eye under normal lighting conditions, the tested stereoscopic blood vessel contrast enhancer offered improved visualization in all investigated settings, providing the best stereoscopic image quality when using the CRT monitor with shutter glasses and the best monoscopic image quality when using the LCD set to monoscopic mode.

Our technique supports perception of depth, 3-dimensional motion and discrimination between tissue surface and underlying structures. It also has potential as an educational tool by offering the possibility to look and record trough the eyes of an experienced specialist. Further improvements on penetration depth, frame-rate and focus depth-of-field form targets for momentary ongoing further research.