Biologically inspired image enhancement based on Retinex

Abstract

This paper presents a biologically inspired adaptive image enhancement method, consisting of four stages: illumination estimation, reflection extraction, color restoration and postprocessing. The illumination of the input image is estimated using guided filter. We propose to utilize the smoothed Y channel in the YCbCr color space as the guidance image, since it can better capture the illuminance of the real scene. The reflection of the input image is extracted using the Retinex algorithm and refined through color restoration. In order to further improve the quality of the extracted reflection, we explore a learning strategy to select the optimal parameters of the nonlinear stretching by optimizing a novel image quality measurement, named as the Modified Contrast–Naturalness–Colorfulness (MCNC) function. Compared with the original CNC function, the proposed MCNC function employs a more effective objective criterion and can better agree with human visual perception. Both qualitative and quantitative experiments demonstrate that the proposed method is adaptive and robust to outdoor images and achieves favorable performance against state-of-the-art methods especially for images captured under extremely hazed or low-light conditions.

Introduction

In numerous applications of computer vision technology, such as visual tracking, anomaly detection and recognition, clear images are the critical prerequisite for good understanding of the real scenes. In practice, however, the quality of the images captured outdoors can be severely degraded due to various weather conditions, such as low illumination, fog and haze, which result in dimness or distortion. Therefore, enhancing and restoring degenerated images is particularly important.

Existing methods for image enhancement can be mainly classified into two categories [1]: (1) image restoration based on physical models, and (2) image enhancement based on image processing techniques.

For the first category, the optimal estimate of a haze-free image is obtained by establishing and inverting the process of image degradation. Various attempts have been explored, and tremendous progress has been made in haze removal for single image [2], [3], [4]. The dark channel prior (DCP) theory proposed by He et al. [4] directly estimates the thickness of haze using the statistics prior of haze-free outdoor images. Recent algorithms [5], [6], [7] improve the DCP method in some particular aspects. Though significant performance gain has been achieved by these methods, results restored from images captured under the overcast environment are still unsatisfactory. Take the DCP method for instance, its dehazing results are clear and natural for images with a certain brightness (Fig. 1(a) and (b)). However, it performs poorly when the lightness of the image is in the low level (Fig. 1(c) and (d)).

The second category of image enhancement techniques directly improves contrast and highlights details by global or local pixel-processing, regardless of the cause of image degradation. Some traditional methods, such as gamma correction, contrast stretching and histogram equalization (HE), are simple but easily fail to provide exact enhanced images and sometimes may even destroy the image contents [8]. In contrast, more advanced methods, like contrast-limited adaptive histogram equalization (CLAHE) [9], wavelet transformations [10] and homomorphic filtering [8], have shown strong robustness to images of various quality.

The Retinex theory is firstly introduced to image enhancement by Land et al. [11] based on the observation that sensations of color have a strong correlation with reflectance, even though the amount of visible light reaching the eye depends on the product of reflectance and illumination. Subsequently, a line of methods [12], [13], [14], [15], [16] has been proposed. Among them, the multi-scale Retinex with color restoration (MSRCR) method [14] proposes to estimate the illumination of the input image using three Gaussian surround filters with different scales and conduct enhancement by applying color restoration followed by linear stretching to the logarithm of reflectance. Though the MSRCR method has demonstrated a strong ability in providing dynamic range compression and preserving most of the details (see Fig. 2(a) and (b)), a large number of parameters are involved and set empirically, such as the scales and weights of Gaussian filters, and stretching factors, which limits the generalization ability and often results in pseudo halos and unnatural color (see Fig. 2(c) and (d)).

Recently, some improvements on MSRCR have been proposed. Rather than utilizing the Gaussian filters to perform the illumination estimation, the method in [17] employs a denoising technique called non-local means filter, assuming that the denoised image is equivalent to the illumination image. In [18], the illumination of the input image is estimated using a guided filter, which not only plays the smoothing role, but also transfers the structure of its guidance image to the filtering output. Jang et al. [19] propose the visual contrast measure (VCM) method to select the scales and weights of Gaussian filters for illumination estimation, while the parameters of MSRCR in [20] are optimized using the particle swarm optimization (PSO) method. Apart from the obtained reflectance, the mid-tone of the image is also considered in [21] by applying an inverse sigmoid function to the estimated illumination.

This paper focuses on improving the visual quality of outdoor images, especially for those captured under the overcast or low-light conditions. To this end, we propose a biologically inspired adaptive image enhancement method. The pipeline of the proposed method is shown in Fig. 3.

Firstly, we also exploit guided filter to estimate the illumination. However, different from [18] which directly uses the input image as the guidance image of the guided filter, we utilize the smoothed Y channel in the YCbCr color space, which can better reflect the luminance of the real scene. The smoothing operation of the Y channel is conducted by utilizing a weighted combination of three Gaussian filters with different scales, the weights of which depend on the local contrast [19] of the Y channel. According to the estimated illumination, the reflection of the input image is then predicted using the Retinex algorithm. Finally, the predicted reflection is further refined via color restoration followed by a novel automatic postprocessing method to obtain the final enhanced image. Specifically, in the postprocessing stage, we explore a learning strategy to select the optimal parameters of the nonlinear stretching by optimizing a novel image quality measurement, named as the Modified Contrast–Naturalness–Colorfulness (MCNC) function. Compared with the original CNC function [22], the proposed MCNC function employs a more effective objective criterion and thus better agrees with human visual perception. The optimization of parameters is conducted by using the QDPSO method [23], which has a stronger ability of global searching than the PSO method utilized in [20].

The contributions of this paper can be summarized as follows:

• We propose a novel design for the guided filter by utilizing the smoothed Y channel in the YCbCr color space as the guided image, which can reflect the luminance of the real scene and facilitates a better estimation of the illumination.

• We propose the MCNC function for the evaluation of image quality, which is more effective and better accord with the human visual perception than the CNC function considering both color and contrast of images.

• We explore a novel and effective postprocessing method, where the parameters of the stretching are adaptively determined by maximizing the value of MCNC function.

The remainder of this paper is organized as follows: the multi-scale Retinex with color restoration (MSRCR) method is explained in Section 2. In Section 3, a novel biologically inspired image enhancement method based on Retinex is proposed. Both quantitative and qualitative experimental results are reported in Section 4. Section 5 summarizes our work.