NDSRGAN: A Novel Dense Generative Adversarial Network for Real Aerial Imagery Super-Resolution Reconstruction

Abstract

In recent years, more and more researchers have used deep learning methods for super-resolution reconstruction and have made good progress. However, most of the existing super-resolution reconstruction models generate low-resolution images for training by downsampling high-resolution images through bicubic interpolation, and the models trained from these data have poor reconstruction results on real-world low-resolution images. In the field of unmanned aerial vehicle (UAV) aerial photography, the use of existing super-resolution reconstruction models in reconstructing real-world low-resolution aerial images captured by UAVs is prone to producing some artifacts, texture detail distortion and other problems, due to compression and fusion processing of the aerial images, thereby resulting in serious loss of texture detail in the obtained low-resolution aerial images. To address this problem, this paper proposes a novel dense generative adversarial network for real aerial imagery super-resolution reconstruction (NDSRGAN), and we produce image datasets with paired high- and low-resolution real aerial remote sensing images. In the generative network, we use a multilevel dense network to connect the dense connections in a residual dense block. In the discriminative network, we use a matrix mean discriminator that can discriminate the generated images locally, no longer discriminating the whole input image using a single value but instead in chunks of regions. We also use $smoot{h}_{L_1}$ loss instead of the $L_1$ loss used in most existing super-resolution models, to accelerate the model convergence and reach the global optimum faster. Compared with traditional models, our model can better utilise the feature information in the original image and discriminate the image in patches. A series of experiments is conducted with real aerial imagery datasets, and the results show that our model achieves good performance on quantitative metrics and visual perception.

1. Introduction

Unmanned aerial vehicle (UAV) aerial remote sensing images can provide information about the Earth’s surface quickly, but medium- and low-resolution UAV aerial imagery has limitations regarding extraction of high-precision features, map updates and target detection. The development of high-resolution UAV aerial imagery makes the in-depth application of aerial imagery possible, thus providing favourable conditions for GIS data updates [1] and GIS applications [2]. It is also important for map updates [3], semantic segmentation [4] and target detection [5]. Computer vision technology has been widely applied in many fields such as multi-target recognition [6] and seismic performance evaluation [7].

In the field of aerial photography, obtaining high-resolution aerial images is difficult due to the influence of imaging technology and photographic equipment [8], which often requires UAVs to perform ultra-low-altitude aerial photography, a process that is both labour-intensive and costly. As a result, techniques to achieve super-resolution (SR) reconstruction of images from an algorithmic point of view have become an important research topic in various fields such as image processing and computer vision [9]. However, most of the existing super-resolution reconstruction models generate low-resolution images for training by downsampling high-resolution images through bicubic interpolation, and the models trained from these data have poor reconstruction results on real-world low-resolution images.

To solve this problem, we propose NDSRGAN. Unlike previous GAN-based image SR reconstruction models, this model has a dense connection in the residual dense block (DCRDB) in the generative network, which consists of multilevel dense connections (MDC) and residual connections [10]. Multiple DCRDBs are then connected by a multilevel dense network to form a generative network, which is optimised to enable the generative network to fully utilise the feature information of real aerial images. The discriminator is specially designed using a matrix mean discriminator (MMD) network, which discards the full connection of the last layer of the traditional discriminator and outputs a discriminator matrix instead, where each value of the matrix represents a piece of the receptive field in the image. In the objective function, we use $smoot{h}_{L_1}$ loss instead of the $L_1$ loss used in most image SR reconstruction models. This can better accelerate the convergence of the model to reach the global optimum [11].

Most existing image SR reconstruction models use LR images generated by bicubic interpolation using HR images, which are very different from the low-resolution images obtained by actual photography. To ensure that our model met the requirements of practical applications, we did not use LR images generated by bicubic interpolation paired with HR images for training. Instead, we created the dataset by using LR images and HR images of the real shots to perform training. In the reconstruction of the real aerial imagery dataset and benchmark dataset, our model achieved the best SR reconstruction results, compared with representative image SR reconstruction models.

Specifically, the main contributions of this work are as follows:

• We produce a new image dataset with paired high- and low-resolution real aerial remote sensing images, which are both obtained by actual photography.
• We propose a novel dense generative adversarial network for real aerial imagery super-resolution reconstruction (NDSRGAN). In the generative network, we use a multilevel dense network to connect the dense connections in a residual dense block. In the discriminative network, we use a matrix mean discriminator that can discriminate the generated images locally. We also use $smoot{h}_{L_1}$ loss to accelerate the model convergence and reach the global optimum faster.

The rest of the paper is organised as follows. Related studies on super-resolution are introduced in Section 2. In Section 3, we detail our NDSRGAN. The datasets and experimental results are given in Section 4, and we discuss our method on an open-source dataset in Section 5. Finally, we conclude our work and discuss future research directions in Section 6.

2. Related Work

The main SR methods are divided into three categories, which are discussed in the following subsections.

2.1. Interpolation-Based Methods

Interpolation-based methods calculate the value of a point according to a certain formula by using the values of several known points around the point and the relationship between the surrounding points and the location of this point. Interpolation-based methods are simple, efficient and easy to understand, but they are also prone to blurring and jaggedness when recovering images, and they cannot recover image details because no new information is generated during the interpolation process. Common interpolation-based methods include nearest neighbour [12], bilinear [13] and bicubic [14] methods.

2.2. Reconstruction-Based Methods

Reconstruction-based methods establish an observation model for the process of converting high-resolution (HR) images into low-resolution (LR) images, and then achieve SR reconstruction by solving the inverse problem of the observation model. Typical reconstruction-based methods are nonuniform interpolation [15], iterative inverse projection [16], maximum posterior probability [17] and convex set projection [18] methods.

2.3. Learning-Based Methods

A learning-based image SR algorithm learns the mapping relationship between LR and HR images by training the image dataset to predict the missing details of feature information in LR images to reconstruct high-quality images. Learning-based SR methods are the mainstream research direction at present, and they achieve the best results. Commonly used learning-based SR methods include the neighbour embedding method [19], sparse representation [20] and deep learning [21].

In recent years, with the rapid development of artificial intelligence technology [22], the use of deep learning methods to achieve SR reconstruction has become mainstream, and many deep-neural-network-based models have made good progress in the field of SR reconstruction. The methods using deep learning for single-image SR reconstruction [23] can be further divided into two categories: convolutional neural network (CNN)-based SR reconstruction and generative adversarial network (GAN)-based SR reconstruction.

2.3.1. CNN-Based SR Reconstruction Methods

A simple SRCNN was proposed by Dong et al. [24], which used a simple three-layer CNN [25] to achieve SR reconstruction and obtained better reconstruction results than traditional methods. However, due to the large size of the convolutional kernel and the shallow depth of the network, the reconstructed image was too smooth, and the reconstruction details were lost.

To overcome this shortcoming, Dong et al., proposed a fast SR reconstruction CNN (FSRCNN) [26], which uses smaller convolutional kernels than SRCNN to complete feature extraction and nonlinear mapping of images, and abandons the initial interpolation of low-resolution images for enlargement, instead directly inputting low-resolution images and finally performing deconvolution [27] to amplify the image. After this series of optimisations, FSRCNN obtained a higher evaluation score and better reconstruction effect than SRCNN. However, the reconstruction details still require improvement because the network structure is too simple.

Following the wide application of residual networks (ResNet) [28] in the field of target detection and target recognition, Lim et al. applied ResNet to the field of image SR reconstruction and proposed an enhanced SR reconstruction residual network (EDSR) [29]. In this network, the authors removed the batch normalisation (BN) layer from the ResNet [30] and increased the number of residual layers from 16 to 32. The BN layer consumes the same amount of memory as its preceding convolutional layers. Thus, after this operation step is removed, EDSR can stack more network layers, so that more features are extracted per layer with the same computational resources, resulting in a better performance.

These CNN-based SR network models have made good progress in continuous optimisation and improvement. However, a common loss function for CNN-based SR network models is the mean squared error (MSE), which causes the generated images to have a high signal-to-noise ratio but lack high-frequency detail and have an over-smoothed texture.

2.3.2. GAN-Based SR Reconstruction Methods

Compared with traditional CNNs, the biggest difference and success of GANs is the discriminator, which can be trained to discriminate the input generated images. In the field of aerial photography, the image complexity in aerial images is greater than that in natural images [31]. Conventional discriminators are prone to local discriminative errors in discriminating images. They cannot make correct discriminations based on the correlation of surrounding features, thus seriously affecting the quality of generated high-resolution images.

The main structure of a GAN consists of a generator and a discriminator. The generator generates fake images that deceive the discriminator, and the discriminator discriminates whether the input image is from a real image or a fake image generated by the generator, to achieve Nash equilibrium [32] by alternately training the generator and the discriminator, finally determining the network model parameters [33].

Ledig proposed a single-image SR reconstruction method using GANs (SRGAN) [34], which was the first application of GANs in image SR reconstruction, to solve the shortcomings of MSE in CNN-based SR reconstruction models [35]. SRGAN uses perceptual loss as an optimisation objective. Perceptual loss was proposed by Johnson et al., inspired by the studies in [36,37], and consists of adversarial loss [33] and content loss. Adversarial loss maps the image to a high-dimensional feature space and uses a discriminative network to discriminate the reconstructed image from the original image, and content loss is based on receptive similarity rather than pixel similarity [38].

An activated 19-layer VGG network [39] is used to obtain the feature maps of the generated image and the original high-resolution image, and then to improve the quality of the generated image by minimising the error between the generated image and the feature maps of the original high-resolution image. However, because the generative network of SRGAN partly uses ResNet, the generated image is usually not sufficiently natural and contains some noise.

To address the shortcomings of SRGAN, Wang et al., proposed an enhanced SRGAN (ESRGAN) [40]. This uses a multilevel network structure unit RRDB instead of ResNet in the generative network of the SRGAN and removes the BN layer. Inspired by a relativistic GAN, it allows the discriminator to predict the probability that the generated image is more false, relative to the high-resolution image, rather than just the probability that the generated image is false [41]. It also uses feature maps with stronger supervisory information before activation, to constrain the perceptual loss function. After a series of optimisations, the ESRGAN model achieved better visual quality in its reconstructed images but did not reconstruct more desirable results for the edge contours of some features. Therefore, Ma et al., proposed a gradient-guided SR reconstruction network model (SPSR) [42], which adds a gradient branch and gradient loss to the ESRGAN network and can recover the contour details of the original image to a great extent.

However, the overall visual effect of the image is still different from that of the original image. Although the above research on GAN SR models has achieved great improvements in the SR effect, the mainstream models usually use some more basic residual networks and L1 loss, resulting in the inability to fully learn the features of the real aerial images, while most models ignore the optimisation of the discriminator.

3. Method

This section focuses on the design of the generative network, discriminative network and loss function of NDSRGAN.

3.1. Generative Network

The generative network architecture is shown in Figure 1. Firstly, we use one convolutional layer to extract the original features in the LR image to obtain the original feature maps $I_f$. $I_f$ is further input into multiple DCRDBs connected by a dense network. Then, the input and output of the u-th DCRDB can be represented as follows:

$$Q_u=\{\begin{array}{ccc}\hfill I_f\hfill & ,\Vert & \hfill u=0\\ \hfill I_f+\alpha {\displaystyle \sum _{i=0}^{u-1}}{I}_i\hfill & ,\Vert & \hfill u\ge 1\end{array}$$

(1)

$$I_u=F_{DCRDB,u}\left(Q_u\right)$$

(2)

In Equation (1), $I_f$ represents the original feature maps of the LR image $I_{LR}$ after convolution and $\alpha$ represents the residual scaling factor [43]. In Equation (2), $Q_u$ represents the input of the u-th DCRDB, $I_u$ represents the output of the u-th DCRDB and $F_{DCRDB,u}(\cdot )$ represents the composite network formed by the u-th DCRDB through dense connections.

The specific network structure of the DCRDB is shown in Figure 2a. Each DCRDB consists of $\sigma$ dense blocks (DB) and a convolutional layer, connected using a dense network. These dense connections can ensure that the output feature maps of each dense block module can be reused in multiple levels, which can maximise the use of feature information on LR images. The v-th DCRDB internal calculation formula can be described as follows:

$$Q_{u,v}=\{\begin{array}{ccc}\hfill Q_u\hfill & ,\Vert & \hfill v=0\\ \hfill Q_u+\alpha {\displaystyle \sum _{i=0}^{v-1}}{I}_{u,i}\hfill & ,\Vert & \hfill v\ge 1\end{array}$$

(3)

$$I_{u,v}=F_{DB,v}\left(Q_{u,v}\right)$$

(4)

In Equation (4), $Q_{u,v}$ is the input of the v-th layer of the DB in the u-th DCRDB, $I_{u,v}$ is the output of the v-th layer of the DB in the u-th DCRDB and $F_{DB,v}$ represents the v-th layer of the DB in the u-th DCRDB. After this, we sum the outputs of the $\sigma$ DBs through the dense connection and input them into the convolutional layer, which is calculated as follows:

$$I_{DBConv}=F_{conv}\left(Q_u+\alpha {\displaystyle \sum }_{i=0}^{\sigma -1}{I}_{u,i}\right)$$

(5)

In Equation (5), $F_{conv}(\cdot )$ represents the last convolutional operation in the DCRDB and $I_{DBConv}$ represents the output of the convolutional layer, immediately after which we access a residual connection, calculated as follows:

$$I_u=Q_u+I_{DBConv}$$

(6)

In Equation (6), $Q_u$ represents the input after the u-th DCRDB and $I_u$ represents the output after the u-th DCRDB.

The DB structure is shown in Figure 2b. Each DB is composed of a series of CL modules and a convolutional layer, with dense connection. The CL module consists of a convolutional layer and a leaky ReLU (LReLU) layer [44,45]. The internal computational formula of the v-th DB in the u-th DCRDB can be described as

$$Q_{u,v,w}=\{\begin{array}{ccc}\hfill Q_{u,v}\hfill & ,\Vert & \hfill w=0\\ \hfill T_{concat}\left(Q_{u,v,},I_{u,v,0,},I_{u,v,1}\cdot \cdot \cdot \cdot \cdot \cdot I_{u,v,w-2},I_{u,v,w-1}\right)\hfill & ,\Vert & \hfill w\ge 1\end{array}$$

(7)

$$I_{u,v,w}=F_{CL,w}\left(Q_{u,v,w}\right)$$

(8)

In Equation (8), $Q_{u,v,w}$ is the input of the w-th CL module in the v-th dense block in the u-th DCRDB, $I_{u,v,w}$ is the output of the w-th CL module in the v-th dense block in the u-th DCRDB, $F_{CL,w}$ represents the w-th CL composite network in the v-th DB of the u-th CRDB and $T_{concat}(\cdot )$ represents the concatenate operation that sums the channels of the input feature maps. Suppose we use $\phi$ CL modules, then:

$$I_{CL}=F_{conv}\left(T_{concat}\left(Q_{u,v},I_{u,v,0,},I_{u,v,1}\cdot \cdot \cdot \cdot \cdot \cdot I_{u,v,\phi -2},I_{u,v,\phi -1}\right)\right)$$

(9)

In Equation (9), $I_{CL}$ is the output of $F_{conv}(\cdot )$ and $F_{conv}(\cdot )$ is the last layer of the convolution operation of the DB. Following this, we access a residual connection

$$I_{u,v}=Q_{u,v}+I_{CL}$$

(10)

In Equation (10), $Q_{u,v}$ is the input of the v-th DB in the u-th DCRDB and $I_{u,v}$ is the output of the v-th dense block in the u-th DCRDB.

After introducing the basic components of the DCRDB, we now introduce the generative network. Suppose we use $\epsilon$ DCRDBs. The output $I_{\epsilon -1}$ is connected with the output of each previous layer of DCRDBs and fed into the next convolutional layer:

$$I_y=F_{conv}\left(I_f+\alpha {\displaystyle \sum }_{i=0}^{\epsilon -1}{I}_i\right)$$

(11)

In Equation (11), $F_{conv}(\cdot )$ represents the final layer of the convolution operation of the generative network and $I_y$ represents the output of $F_{conv}(\cdot )$. Then, we access a residual connection

$$I_z=I_y+I_f$$

(12)

In Equation (12), $I_z$ represents the output of $I_y$ connected to the $I_f$ residuals. Finally, $I_z$ is fed to the upsampling network, set up as follows:

$$I_{SR}=F_{upnet}\left(I_z\right)$$

(13)

In Equation (13), $F_{upnet}(\cdot )$ represents our upsampling network, $I_z$ represents the input of the upsampling network and $I_{SR}$ represents the output of the upsampling network, which is also the input of the final discriminator. The upsampling network of ESRGAN [40] is exploited in our network. It includes two magnification steps. Each step uses a nearest neighbour algorithm to magnify the image and connect a convolutional layer and an LReLU [44,45] layer.

3.2. Discriminative Network

The role of the discriminator is to discriminate the feature map distribution difference between $I_{SR}$ and $I_{HR}$, and to discriminate whether $I_{SR}$ is real or fake. The output of the basic discriminator of a GAN is often a classification after a series of convolutions and a full concatenation score to indicate whether the overall category of the image is real or fake. The receptive field for classification is the whole image, which is why the network is insensitive to the local information of the image and cannot achieve a higher-fidelity image reconstruction in the case of rich feature details. Therefore, local feature extraction and discrimination of the image are needed to generate an SR reconstructed image that is closer to the real aerial image.

Inspired by [46], we used the idea of a matrix mean discriminator capable of local discrimination on the generated images, to construct our discriminative network. The network structure of the discriminative network is shown in Figure 1, set up as a fully convolutional network [47]. The first layer consists of a convolutional layer and an LReLU [44,45]. The second, third and fourth layers consist of a convolution layer, a BN layer and an LReLU [44,45]. The fifth layer consists of one convolutional layer. The size of the output matrix in each layer is shown in

Table 1. The size of the output matrix of each layer of the network, where $h$ represents the size of the input image, $k_1-k_5$ represent the sizes of each convolution kernel from the first layer to the fifth layer, $p_1-p_5$ represent the amounts of padding from the first layer to the fifth layer, $s_1-s_5$ represent the sizes of each convolutional step from the first layer to the fifth layer, $n_1-n_4$ represent the sizes of the output matrix from the first layer to the fourth layer and $n$ represents the size of the final output matrix from the discriminator.

Network Layers	Calculation Formula	Size of the Output Matrix
the first layer	$\left(h-k_1+2p_1\right)/s_1+1$	$n_1$
the second layer	$\left(n_1-k_2+2p_2\right)/s_2+1$	$n_2$
the third layer	$\left(n_2-k_3+2p_3\right)/s_3+1$	$n_3$
the fourth layer	$\left(n_3-k_4+2p_4\right)/s_4+1$	$n_4$
the fifth layer	$\left(n_4-k_5+2p_5\right)/s_5+1$	$n$

.

The discriminative network finally yields an $n\times n$ discriminative matrix. Each element in the matrix represents a receptive field in the original image and carries out a real or fake discrimination for each value in the $n\times n$ matrix to complete the local discrimination of the input image. The experimental results shown in Section 4 demonstrate that the image quality is significantly improved by our discriminator-optimised model.

3.3. Loss Function Design

Here, we introduce the design of the loss function of the generative network, which consists of three main parts: pixel loss, perceptual loss [38] and adversarial loss [33]. Pixel loss is calculated as follows:

$$L_{pixel}\left(I_{SR},I_{HR}\right)=smoot{h}_{L_1}\left(I_{SR}-I_{HR}\right)$$

(14)

In which

$$smoot{h}_{L_1\left(x\right)}=\{\begin{array}{ccc}\hfill 0.5x^2\hfill & ,\Vert & \hfill \left|x\right|<1\\ \hfill \left|x\right|-0.5\hfill & ,\Vert & \hfill \left|x\right|\ge 1\end{array}$$

(15)

Is a more powerful objective function. In fact, in the interval of $\left|x\right|\ge 1$, $L_1$ loss actually solves the problem of a too-large gradient and unstable training caused by a too-large difference between $I_{SR}$ and $I_{HR}$ at the early stage of training. In the $\left|x\right|<1$ interval, which is actually $L_2$ loss, the derivative of the $L_2$ loss is relatively small when the difference between $I_{SR}$ and $I_{HR}$ is small in the late training period, thereby making the loss convergence more stable and aiding convergence to the global optimum [11]. The perceptual loss is calculated as follows:

$$I_{Perceptual}\left(VGG\left(I_{SR}\right),VGG\left(I_{HR}\right)\right)= smoot{h}_{L_1}\left(VGG\left(I_{SR}\right)-VGG\left(I_{HR}\right)\right)$$

(16)

We use the $smoot{h}_{L_1}$ loss to design our perceptual loss. In Equation (16), $VGG(\cdot )$ represents a 19-layer VGG network [39], and we use this network before activation to extract the feature maps from the generated and real images. The goal of the perceptual loss function is to minimise the error between the feature maps. The addition of this loss function allows our model to generate images with a more realistic texture. We can now introduce the adversarial loss for the generator, which is calculated as follows:

$$L_{GA}=-E\left(log\left(1-D_{HS}\right)\right)-E\left(log\left(D_{SH}\right)\right)$$

(17)

In Equation (17), $D_{HS}=\sigma \left(D\left(I_{HR}\right)-E\left(D\left(I_{SR}\right)\right)\right)$ represents the difference between the matrix $D\left(I_{HR}\right)$ of the real image $I_{HR}$ output through the discriminator $D(\cdot )$ and the element mean of the output matrix $D\left(I_{SR}\right)$ of $I_{SR}$ after the discriminator $D(\cdot )$. Then, the resultant matrix is mapped to a probability matrix between 0 and 1 by the sigmoid function, and the $D_{HS}$ matrix has an optimisation target of 0 for each element. Each element represents an optimisation target of 0 for each receptive field. Here, $\sigma (\cdot )$ is the sigmoid function and $D_{SH}=\sigma \left(D\left(I_{SR}\right)-E\left(D\left(I_{HR}\right)\right)\right)$ represents the difference between the matrix $D\left(I_{SR}\right)$ of $I_{SR}$ output through the discriminator $D(\cdot )$ and the element mean of the output matrix $D\left(I_{HR}\right)$ of $I_{HR}$ after the discriminator $D(\cdot )$. Afterwards, the resultant matrix is mapped to a probability matrix between 0 and 1 by the sigmoid function, and the optimisation target of each element in the $D_{SH}$ matrix is 1. Each element also represents an optimisation target of 1 for each receptive field. The optimisation target of the receptive field is 1. The final objective function of the generator can be defined as

$$L_G=\beta L_{pixel}\left(I_{SR},I_{HR}\right)+I_{Perceptual}\left(VGG\left(I_{SR}\right),VGG\left(I_{HR}\right)\right)+\gamma L_{GA}$$

(18)

In Equation (18), $\beta$ and $\gamma$ represent the weight coefficients of the pixel loss and the generator adversarial loss.

Next, we introduce the discriminator loss function construction. The discriminator adversarial loss calculation formula can be expressed as

$$L_{DA}=-E\left(log\left(D_{HS}\right)\right)-E\left(log\left(1-D_{SH}\right)\right)$$

(19)

The design of the discriminator adversarial loss differs from that of the generator in which each element of the $D_{HS}=\sigma \left(D\left(I_{HR}\right)-E\left(D\left(I_{SR}\right)\right)\right)$ output matrix has an optimisation objective of 1, in that each element of the $D_{SH}=\sigma \left(D\left(I_{SR}\right)-E\left(D\left(I_{HR}\right)\right)\right)$ output matrix has an optimisation objective of 0. The similarity lies in the fact that each element of the matrix represents each receptive field. The loss function of the discriminator can be expressed as

$$L_D=\eta \ast L_{DA}$$

(20)

In Equation (20), $\eta$ represents the weight coefficient of $L_{DA}$. We optimise $L_G$ and $L_D$ by alternate training to update the parameters of our generative and discriminative networks and obtain our SR reconstruction model.

4. Experiments and Results

In this section, we introduce the production of the SR reconstruction dataset based on real HR and LR aerial imagery (RHLAI), the details of training parameters, the analysis of changes in the reconstructed image quality metrics during NDSRGAN training and the analysis of the reconstruction results of the RHLAI dataset.

4.1. RHLAI Dataset

Most SR reconstruction methods use LR images obtained by bicubic interpolation of HR images. A difference exists between the LR images obtained in this way and the actual LR images. As a result, the trained SR models often do not have the SR reconstruction capability of the real LR images. To solve this problem, in this study we discarded the bicubic interpolation method of generating LR images from HR images used in most previous studies to construct paired HR and LR datasets. Instead, we used real LR aerial imagery and the corresponding real HR aerial imagery to produce the datasets, to enable the model to learn more complex and more realistic mapping relationships instead of simply learning the inverse process of bicubic interpolation.

To ensure that the HR and LR images corresponded to each other, we took HR and LR images by aerial flight at 100 m and 400 m, respectively, in the same place. These images were taken on 9 January 2021, at 2:00 p.m., at Yichang City, Hubei Province, China. We obtained real HR aerial imagery with 0.05 m resolution and real LR aerial imagery with 0.2 m resolution.

We used the georeferencing tool in ArcGIS to align the HR images and LR images by manually setting the control points, in order to make the features in the aligned HR images and LR images correspond to each other as much as possible. We used OpenCV to crop the 0.05 m HR images into 256 × 256 images and the corresponding 0.2 m LR images into 64 × 64 images, resulting in a dataset of 9288 pairs of HR and LR images. We selected 8385 pairs of HR and LR images as the training dataset and 903 pairs of HR and LR images as the test dataset. We discuss the performed cross-validation on the training dataset in Section 4.3.

We named this dataset RHLAI. This dataset can avoid the problem of the model simply learning the inverse process of bicubic interpolation and can truly reflect the reconstruction ability of the SR model.

4.2. Training Details

We set the batch size to 16 for the input images. All the input data were randomly cropped and data-augmented. We randomly cropped the HR images to 128 × 128 and the LR images to 32 × 32. NDSRGAN uses 23 DCRDBs and 3 DBs in each DCRDB; each DB in turn uses 4 CR modules.

The residual scaling factor $\alpha$ was set to 0.2, $\beta$ was set to $e^{-2}$, $\gamma$ was set to $2.5e^{-3}$ and $\eta$ was set to 0.5. The learning rate at the beginning of training was set to $2e^{-4}$. The number of training iterations was set to $e^5$ and the learning rate was reduced by half at $2.5e^4$, $5e^4$ and $7.5e^4$ iterations.

For the optimiser settings, NDSRGAN used the Adam optimiser [48], with ${\beta }_1=0.9$ and ${\beta }_2=0.99$. The generative and discriminative networks were trained alternately during the training process until Nash equilibrium was achieved.

We developed our model code using the PyTorch framework and completed the model training on two GTX1080Ti GPUs with 12 GB of global memory.

4.3. Cross-Validation

In this experiment, we used three accuracy metrics to evaluate the model: peak signal-to-noise ratio (PSNR) [49], structural similarity (SSIM) [50] and learned perceptual image patch similarity (LPIPS) [51]. LPIPS is calculated according to the $L_2$ distance $d\left(x,x_0\right)$, which was defined in [51] as

$$d\left(x,x_0\right)=\sum _l\frac{1}{H_l{W}_l}\sum _{h,w} \Vert w_l\odot {({\widehat{y}}_{hw}^l-{\widehat{y}}_{0hw}^l)\Vert }_2^2$$

(21)

The widely used perceptual metrics (PSNR, SSIM) are very simple functions that cannot reflect human perception well. Therefore, in this experiment, we added the LPIPS metric to evaluate the model. LPIPS is a type of computational evaluation metric for image depth features, which is closer to human perception in visual similarity judgment and performs better. LPIPS is a metric calculated by mapping the evaluated image and the original image to a high-dimensional space using a deep learning model, and then calculating the distances between the feature images of the evaluated image and the original image. It can be used to evaluate the quality of the image. The smaller the distance between two feature images, the closer the evaluated image is to the original image. A lower LPIPS value represents better image quality [51].

We conducted nine-fold cross-validation experiments on the training dataset. A total of 8385 images were divided into nine sub-datasets. We extracted one sub-dataset each time as the validation dataset and input the remaining eight sub-datasets into the model for training.

Table 2. Mean values of PSNR, SSIM and LPIPS for cross-validation. The best performance in bold.

	$1$	$2$	$3$	$4$	$5$	$6$	$7$	$8$	$9$
PSNR	$19.282$	$19.251$	$19.304$	$\mathbf{19.513}$	$19.418$	$19.389$	$19.466$	$19.276$	$19.411$
SSIM	$0.4438$	$0.4537$	$0.4521$	$\mathbf{0.4675}$	$0.4520$	$0.4502$	$0.4533$	$0.4356$	$0.4523$
LPIPS	$0.3502$	$0.3436$	$0.3401$	$\mathbf{0.3306}$	$0.3391$	$0.3420$	$0.3361$	$0.3536$	$0.3383$

shows three metrics of the model for nine experiments. After observing that the fourth experiment had the highest performance, we chose the sub-datasets of the fourth experiment to obtain the final parameters of our model and selected them for conducting subsequent model comparison tests.

4.4. Image Quality Metrics Analysis during NDSRGAN Training

We saved the mean values of PSNR, SSIM and LPIPS for the validation dataset every 500 iterations while the model was trained and plotted them into three graphs for visual analysis.

Figure 3a shows the PSNR values during the training process. The horizontal coordinates are the number of iterations, and the vertical coordinates represent the PSNR values, from which we can observe that the oscillation amplitude of the PSNR values was very large at the beginning of the training period, and the difference between the highest and lowest peaks was more than 2. Our model gradually stabilised, and the amplitude stayed at around 0.2 when the number of iterations approached 100,000.

Figure 3b shows the curve of the SSIM value change during training, from which we can observe that the oscillation amplitude of the SSIM value was still very large at the beginning of training, and the difference between the highest peak and the lowest peak was more than 0.05. As the number of training iterations increased, the overall trend of the SSIM values gradually increased, the amplitude of oscillation gradually stabilised and the final amplitude remained around 0.01, which also indicates that our model gradually stabilises in the late training period.

Figure 3c shows the curve of the LPIPS value change during the training process. It shows that the quality of our model improved continuously as the number of training iterations increased, and the model stabilised and converged when the number of iterations approached 100,000.

4.5. Reconstruction Results Analysis of RHLAI Dataset

We conducted experiments using SRCNN [25], EDSR [29], SRGAN [34], ESRGAN [40], SPSR [42], Real-ESRGAN [52] and NDSRGAN on the RHLAI dataset, under the conditions where the training iterations, initial learning rates and training hardware environments were guaranteed to be the same.

Table 3. Mean values of PSNR, SSIM and LPIPS for different methods. The best performance in bold.

Metrics	Bicubic	CNN		GAN
		SRCNN	EDSR	SRGAN	ESRGAN	SPSR	Real-ESRGAN	Ours
PSNR	$\mathbf{20.763}$	$19.985$	$20.619$	$19.011$	$18.637$	$18.391$	$19.201$	$19.513$
SSIM	$0.5246$	$\mathbf{0.5308}$	$0.5307$	$0.4517$	$0.4320$	$0.4562$	$0.4876$	$0.4675$
LPIPS	$0.6802$	$0.6923$	$0.6777$	$0.3949$	$0.3791$	$0.3688$	$0.3952$	$\mathbf{0.3306}$

shows the PSNR, SSIM and LPIPS values of the reconstructed images on the test dataset. Higher PSNR and SSIM values and lower LPIPS values represent better image quality. Through experimental comparison, we found that the bicubic interpolation methods and the CNN-based SR reconstruction methods tended to obtain higher PSNR and SSIM values compared with the GAN-based methods, but a large gap still exists between the real reconstruction visual effect and the GAN-based methods’ reconstruction effect, and the image visual effect is blurred (see Figure 4). Therefore, we believe that the PSNR and SSIM metrics do not represent the real image reconstruction quality well. The authors who developed ESRGAN [40] and SPSR [42] presented the same insights in their papers. For this reason, we calculated a metric closer to human visual perception, to continue the accuracy evaluation, namely LPIPS [51].Our model achieved the highest accuracy score, significantly exceeding the LPIPS value of the second-ranked SPSR model. It can be seen from Figure 4 that this metric is consistent with human visual perception. The lower the LPIPS value othe reconstructed image, the higher the reconstruction quality.

We also performed a group of qualitative and quantitative analyses of the reconstructed images in terms of human visual perception. Figure 4 shows that the reconstructed images from our model have richer texture details than the reconstructed images from other models, and the reconstructed images are closer to the real HR images.

The figure also shows that the images reconstructed using bicubic interpolation and CNN-based methods are obviously blurred, and the images reconstructed by the GAN-based methods are clearer.

As can be seen in the first and second rows of Figure 4, our model is able to reconstruct a clearer car outline. In the third row of Figure 4, our model is able to restore more of the white texture of the roof and produce fewer artifacts and in the fourth row of Figure 4, our model is able to generate a clearer gutter that is closer to the real HR image. In the fifth row of Figure 4, our model generates more regular and more detailed parking lines.

These results indicate that our model is able to take full advantage of the features of the original image and that the feature detail contours in the generated image are closer to the real HR image, due to the matrix mean discriminator utilised by NDSRGAN. The performance of NDSRGAN will be further discussed in Appendix A.

5. Discussion

To further demonstrate the good performance of NDSRGAN, we conducted a series of experiments using SRCNN, EDSR, SRGAN, ESGAN, SPSR, Real-ESRGAN and NDSRGAN on the DIV2K dataset [53], under conditions where the training iterations, initial learning rates and training hardware environments were guaranteed to be the same.

Table 4. Mean values of PSNR, SSIM and LPIPS for different methods in Set5, Set14 and Urban100. The best performance in bold.

Dataset	Metrics	Bicubic	CNN		GAN
			SRCNN	EDSR	SRGAN	ESRGAN	SPSR	Real-ESRGAN	Ours
	PSNR	$28.420$	$30.254$	$\mathbf{32.237}$	$28.696$	$28.877$	$28.647$	$29.490$	$29.471$
Set5 [54]	SSIM	$0.8245$	$0.8721$	$\mathbf{0.9075}$	$0.8445$	$0.8459$	$0.8419$	$0.8680$	$0.8552$
	LPIPS	$0.3407$	$0.2182$	$0.1724$	$0.1057$	$0.0877$	$0.0902$	$0.0931$	$\mathbf{0.0820}$
	PSNR	$26.100$	$27.362$	$\mathbf{28.650}$	$25.921$	$25.764$	$25.722$	$26.476$	$26.135$
Set14 [55]	SSIM	$0.7850$	$0.8253$	$\mathbf{0.8528}$	$0.7669$	$0.7612$	$0.7591$	$0.7923$	$0.7688$
	LPIPS	$0.4393$	$0.3331$	$0.2814$	$0.1710$	$0.1613$	$0.1588$	$0.1661$	$\mathbf{0.1537}$
	PSNR	$23.145$	$24.239$	$\mathbf{26.255}$	$23.264$	$23.238$	$23.092$	$23.739$	$23.481$
Urban100 [56]	SSIM	$0.9011$	$0.9347$	$\mathbf{0.9614}$	$0.9100$	$0.9146$	$0.9132$	$0.9267$	$0.9220$
	LPIPS	$0.4726$	$0.3457$	$0.2226$	$0.1908$	$0.1781$	$0.1692$	$0.1729$	$\mathbf{0.1591}$

shows the PSNR, SSIM and LPIPS values of the reconstructed images in Set5 [54], Set14 [55] and Urban100 [56].

As can be seen in Table 4, the EDSR model obtained the highest PSNR and SSIM metrics in all three test sets. As described in Section 4.5, CNN-based models tended to obtain higher PSNR and SSIM values, but the image reconstruction visual results were not as good as those of the GAN-based models. The LPIPS metric is more consistent with human visual evaluation results, so the ranking of LPIPS values is more convincing regarding the superiority of the model. Our NDSRGAN model obtained the best LPIPS metrics in three test sets, further proving the superiority of our model’s reconstruction ability.

6. Conclusions

High-resolution aerial images are important in aerial photography applications, but they have a high acquisition cost and a long processing period. Moreover, the current models do not achieve a good reconstruction effect on real aerial image super-resolution reconstruction. Therefore, reconstructing high-quality high-resolution aerial images from real low-resolution aerial images is a great challenge.

To address the difficulty of reconstructing real aerial images by using existing super-resolution reconstruction models, we produced the RHLAI dataset with real aerial high- and low-resolution pairing for aerial image super-resolution reconstruction and proposed a new aerial image super-resolution reconstruction model called NDSRGAN.

We used multilevel dense connection to connect multiple dense connections in the residual dense block, so that the designed generative network could maximise the use of the features of real aerial images, and we used a matrix mean discriminator to optimise our discriminative network, which does not discriminate the whole image of the input but instead discriminates the input image locally in chunks.

The experimental results demonstrated that the reconstructed images obtained by image local discrimination were of higher quality and closer to the real high-resolution images. In our model, $smoot{h}_{L_1}$ loss was used instead of the $L_1$ loss used in the mainstream models (ESRGAN [40], SPSR [42]), and this optimisation accelerated the convergence of the model, making it reach the global optimum faster.

The previous super-resolution reconstruction models and NDSRGAN were trained on the real aerial image dataset we produced, and the experiments showed that our proposed super-resolution reconstruction model for aerial images could reconstruct the real aerial images with better results, and that the reconstructed images achieved better results than previous methods regarding both quantitative metrics and qualitative evaluation.

Although the proposed method could reconstruct real aerial images with better results, the quality of the reconstructed images was not good when the model reconstructed images taken by different sensors. The lack of reconstruction generalisation ability of the super-resolution model on images taken by different sensors remains a problem that needs to be studied and addressed in the future.