A locally adaptive L₁−L₂ norm for multi-frame super-resolution of images with mixed noise and outliers

Highlights

• A locally adaptive norm regularized method for super-resolution is proposed.

• The adaptive norm for the fidelity is chosen based on outliers’ detection.

• A weight to balance different norm constraints is estimated adaptively.

• The experiments show its superiority compared with other popular variational methods.

Abstract

In this paper, we present a locally adaptive regularized super-resolution model for images with mixed noise and outliers. The proposed method adaptively assigns the local norms in the data fidelity term of the regularized model. Specifically, it determines different norm values for different pixel locations, according to the impulse noise and motion outlier detection results. The $L_1$ norm is employed for pixels with impulse noise and motion outliers, and the $L_2$ norm is used for the other pixels. In order to balance the difference in the constraint strength between the $L_1$ norm and the $L_2$ norm, a strategy to adaptively estimate a weighted parameter is put forward. The experimental results confirm the superiority of the proposed method for different images with mixed noise and outliers.

Introduction

Due to the limitation of solid-state sensors such as CCD or CMOS, the digital images acquired may have a limited spatial resolution, thus reduce their practical value. In addition, digital images may be degraded by blurring or noise because the information each pixel records is easily polluted during the acquisition and transmission procedure. Super-resolution is a technique which takes resolution limitation and regular image degradation into consideration at the same time. Using the redundant information between multi-frame images with a relative sub-pixel motion, the super-resolution technique can construct a higher-resolution image or sequence, and it has been widely applied with medical images [1], remote sensing images [2], [3], [4], and video surveillance [5], [6].

The multi-frame super-resolution technique has been developed for almost three decades [7]. The frequency domain approaches were first addressed; however, these methods [7], [8], [9] are extremely sensitive to model errors, and have difficulty in including prior knowledge to solve the model. Therefore, methods in the spatial domain have become more popular in recent years [3], [10], [11], [12], [13]. These methods include iterative back projection (IBP) [10], projection onto convex sets (POCS) [11], and a group of Bayesian-based probability and statistical methods [3], [12], [13]. Among the methods, the Bayesian-based methods have the advantages of adding a prior and simultaneously handling the high-resolution (HR) image estimation. Furthermore, the model parameters such as the motion vector and blur kernel are estimated iteratively [12]. As a result, the Bayesian-based methods have become the most widely used. These methods in the spatial domain regard super-resolution as an ill-posed inverse problem. Supposing the degradation procedure during image acquisition involves warping, blurring, downsampling, and noise (Fig. 1), then the universal observation model is usually described as follows [14], [15]:

$${\mathit{y}}_k={\mathit{D}}_k{\mathit{B}}_k{\mathit{M}}_k\mathit{z}+{\mathit{n}}_k$$

where ${\mathit{y}}_k$ is the $k\mathrm{th}$ observed image of size $n_1\times n_2$, and $\mathit{z}$ is the desired HR image of size $N_1\times N_2$, which is determined by the downsampling factor. ${\mathit{D}}_k$, ${\mathit{B}}_k$, and ${\mathit{M}}_k$ are, respectively, the downsampling, blurring, and motion operators, and ${\mathit{n}}_k$ is the additive noise. If ${\mathit{D}}_k$ and ${\mathit{M}}_k$ are excluded, it is a model for image restoration dealing with problems with only noise and blurring.

The standard regularized minimizing function usually consists of a data fidelity term describing the model error, and a regularization to constrain the model to achieve a robust solution, which is often described as [12], [16], [17]

$$L(\mathit{z})=\sum _{k=1}^K\left|\right|{\mathit{y}}_k-{\mathit{D}}_k{\mathit{B}}_k{\mathit{M}}_k\mathit{z}|{|}_p^p+\lambda \Upsilon (\mathit{z})$$

In (2), K represents the number of low-resolution (LR) images. $\Upsilon (\mathit{z})$ is the regularization, which is generally chosen from Tikhonov [18], Huber–Markov random field (HMRF) [19], or total variation (TV) family [12], [15], [20], and λ is the regularized parameter which controls the tradeoff between the two terms. For the data fidelity term, the $L_2$ norm is widely used, but it only performs well for model errors satisfying a traditional Gaussian distribution [21]. It has been proved that the $L_1$ norm is more appropriate for impulse noise and motion outliers in image inverse problems. The reason for the robustness of $L_1$ when handling impulse errors is based on the fact that the $L_2$ model results in a pixel-wise mean, while the $L_1$ model results in a pixel-wise median of all the measurements after motion compensation [15], [16], [22].

In the image processing field, Gaussian-type noise is the most commonly assumed because the noise generated in image acquisition usually satisfies a Gaussian distribution. However, in many applications, practical systems suffer from impulse noise/outliers, which are typically caused by malfunctioning arrays in camera sensors, faulty memory locations in hardware, or transmission in a noisy channel [23], [24]. In the image super-resolution problem, motion outliers should be specially considered because some pixels in one frame may be unobservable in the other frames [25], [26], [27]. Considering the complicated degradation of images, as with mixed noise and motion outliers, both the $L_1$ and $L_2$ models have their own advantages and disadvantages. Therefore, some researchers have tried to combine the advantages of the $L_1$ and $L_2$ norms for the fidelity [28], [29], [30]. However, the combination of $L_1$ and $L_2$ may result in new challenges. The hybrid-norm problem is not a standard linear problem to solve, so an efficient optimization method should be considered. In addition, the reconstruction capability of the regularized model hinges on the selection of $\lambda$, and different norm constraints should correspond to different weights for the tradeoff between the fidelity and the regularization. The traditional methods usually tune the weight for the hybrid norm manually, which is time consuming [31]; thus, adaptive weight estimation is an issue worth considering.

In order to deal with super-resolution with mixed noise and/or motion outliers, we propose a variational model employing a regularized framework with a locally adaptive fidelity norm. Specifically, different norm values in the data fidelity for different pixel locations are determined according to the detection results for the impulse noise and motion outliers. The $L_1$ norm is employed for pixels with impulse noise and motion outliers, and the $L_2$ norm is used for the other pixels. To balance the difference in the constraint strength between the $L_1$ norm and the $L_2$ norm, a strategy to adaptively estimate the weighted parameter is put forward.

The remainder of this paper is organized as followed. We introduce the adaptive model construction and optimization in Section 2. The norm selection is described in Section 2.1, in which detection for impulse noise and motion outliers is presented separately. The solving strategy and the method of adaptively determining the weight for the norm-adaptive data fidelity term are given in Section 2.2. We present the experiments for super-resolution under mixed noise conditions, including real video sequence images with moving objects, in Section 3, along with a discussion of the comparative results and parameters. Section 4 is the conclusion.