Next Article in Journal
Non-Linear Modeling of Detectability of Ship Wake Components in Dependency to Influencing Parameters Using Spaceborne X-Band SAR
Next Article in Special Issue
Multi-Block Mixed Sample Semi-Supervised Learning for SAR Target Recognition
Previous Article in Journal
Building Component Detection on Unstructured 3D Indoor Point Clouds Using RANSAC-Based Region Growing
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Novel LSTM Model with Interaction Dual Attention for Radar Echo Extrapolation

The Department of Computer Science, Harbin Institute of Technology, Shenzhen 518055, China
*
Author to whom correspondence should be addressed.
Remote Sens. 2021, 13(2), 164; https://doi.org/10.3390/rs13020164
Submission received: 28 November 2020 / Revised: 27 December 2020 / Accepted: 30 December 2020 / Published: 6 January 2021
(This article belongs to the Special Issue Deep Learning for Radar and Sonar Image Processing)

Abstract

:
The task of precipitation nowcasting is significant in the operational weather forecast. The radar echo map extrapolation plays a vital role in this task. Recently, deep learning techniques such as Convolutional Recurrent Neural Network (ConvRNN) models have been designed to solve the task. These models, albeit performing much better than conventional optical flow based approaches, suffer from a common problem of underestimating the high echo value parts. The drawback is fatal to precipitation nowcasting, as the parts often lead to heavy rains that may cause natural disasters. In this paper, we propose a novel interaction dual attention long short-term memory (IDA-LSTM) model to address the drawback. In the method, an interaction framework is developed for the ConvRNN unit to fully exploit the short-term context information by constructing a serial of coupled convolutions on the input and hidden states. Moreover, a dual attention mechanism on channels and positions is developed to recall the forgotten information in the long term. Comprehensive experiments have been conducted on CIKM AnalytiCup 2017 data sets, and the results show the effectiveness of the IDA-LSTM in addressing the underestimation drawback. The extrapolation performance of IDA-LSTM is superior to that of the state-of-the-art methods.

1. Introduction

Precipitation nowcasting refers to predicting the future rainfall intensity within a relatively short period (e.g., 0∼6 h) based on the observation of radar. The prediction is significant for alerting natural disasters caused by heavy rain and guiding the travel plan of people. The key part of the task is the radar echo map extrapolation, namely predicting the radar echo map sequences in the future based on the historical observations. Once the extrapolation is obtained, precipitation nowcasting can be easily obtained with many methods such as the Z-R relationship [1].
Existing radar echo map extrapolation methods can be mainly classified into two types: (1) optical flow-based models [2,3] and (2) deep learning-based algorithms [4,5,6,7,8]. The former calculates a motion field between the adjacent maps based on the assumption that the brightness of pixels is constant. Then, the extrapolation can be made by applying the motion field iteratively. However, the intensity of the echo map always keeps changing, for example, strengthening or weakening. Therefore, the constant assumption on brightness is unreasonable. Moreover, the generation of movement only uses a few recent radar images, which suggests the type of methods cannot utilize the valuable historical observations. The latter builds a mapping from previous radar observations to future echo maps by constructing a neural network, for example, the convolution neural network (CNN) [9,10], recurrent neural network (RNN) [11], and the Spatial Transformer Networks (STN) [12]. One of the representative approaches is the ConvLSTM. The method combines the Convolution (CNN) and Long Short-Term Memory (LSTM). Here, the LSTM captures the temporal dynamics of the hidden states into the temporal memory. CNN is responsible for extracting spatial information. The deep learning extrapolation approaches usually perform better than the optical flow methods [13], because they do not have the unreasonable constant assumption and they can effectively leverage the valuable historical observations.
However, there is a fatal problem for almost all deep learning-based methods, namely the high echo value part is often underestimated, as shown in Figure 1. We observe that the high radar echo region of the prediction has the trend of disappearing. This phenomenon is universal for other deep learning-based models, which will lead to a serious negative influence on the prevention of disasters caused by strong rainfall. It can be attributed to the following two primary reasons. First, in each step of operation, the various gates in Convolutional Recurrent Neural Network (ConvRNN) are generated by independent convolutions on input and hidden state and a sum fusion. The independent convolution has a limitation because the input and hidden state do not help each other to identify and preserve important information. Consequently, the ConvRNN may lose short-term dependency information in each step. Second, LSTM has a forgetting mechanism to control whether discarding information from previous temporal memory and this process is irreversible [14]. Therefore, in terms of the long-period, the representation of high echo value cannot be found once it is forgotten by temporal memory. That is, the long-term dependency is not nicely modeled.
Recently, some methods were proposed to address the issues. First, to preserve the short-term dependency, Wang. et al. proposed PredRNN [15] and PredRNN++ [16] by introducing a spatial memory into the original ConvLSTM. The spatial memory can preserve the spatial information from the bottom to the top layer. Also, Tran., et al. [17] showed that it can be applied in radar echo extrapolation and deliver better performance than ConvLSTM. However, the extra spatial memory does not help the input or the hidden state to select important features because they are convolved independently. Secondly, to preserve the long-term spatiotemporal correlation, Eidetic 3D LSTM(E3D-LSTM) [18] utilizes the self-attention [19,20] module. It can preserve more spatiotemporal representation because the information can be found from historical memories by the attention mechanism. Nevertheless, it merely uses the single spatial attention mechanism to recall the previous temporal memories. The channel correlations are not modeled [21].
To overcome the limitations of existing models, we propose a novel Interactional Dual Attention Long Short-term Memory (IDA-LSTM) by adding (1) an interaction scheme between the hidden state and the input to preserve more import representation, and (2) a dual attention module for both channel and temporal information to obtain better long-term spatiotemporal representation. For the interaction part, we develop a coupled convolution framework for the input and hidden state. In the framework, the input and hidden state are interacted and fused by a serial of coupled convolutions. As a result, a novel input and hidden state are formed, where the important information are selected with the coupled convolutions. For the dual attention part, we combine channel attention and spatial attention module to substitute for the update of the temporal memory at the forget gate. The single spatial attention module used in E3D-LSTM only considers how to selectively reorganize the feature at each position. However, it ignores the correlation between the different channels. Therefore, channel attention is introduced to further improve the representation in the long term. The experimental results show that the proposed IDA-LSTM is valid to improve the accuracy of precipitation nowcasting, especially in the high echo region. The contribution of our method can be summarized as follows:
  • We first develop the interaction scheme to enhance the short-term dependency modeling ability of ConvRNN approaches. The interaction scheme is a general framework, which can be applied in any ConvRNN model.
  • We introduce the dual attention mechanism to combine the long-term temporal and channel information for the temporal memory cell. The mechanism helps recall the long-term dependency and form better spatiotemporal representation.
  • By applying the interaction scheme and the dual attention mechanism, we propose our IDA-LSTM approach for radar echo map extrapolation. Comprehensive experiments have been conducted. The IDA-LSTM achieves state-of-the-art results, especially in the high radar echo region, on the CIKM AnalytiCup 2017 radar datasets. To reproduce the results, we release the source code at: https://github.com/luochuyao/IDA_LSTM.

2. Proposed Method

In this section, we present the proposed IDA-LSTM model. We first introduce the interaction framework which can be applied in any ConvRNN model. Then, we elaborate the dual attention mechanism and describe how to embed the dual attention mechanism and interaction into our model. Finally, we introduce the whole architecture of the proposed IDA-LSTM model.

2.1. Interaction Framework

The original ConvRNN models independently apply a convolution into the input and hidden state respectively. This process cannot effectively model the correlations between the input and hidden state. To address it we propose a novel interaction scheme by constructing a serial of coupled convolutions as shown in Figure 2. We assume that the original input and hidden state are x 0 and h 0 in ConvRNN models. The new input x 1 is obtained by the convolutions on x 0 and h 0 respectively and adding them together. Upon the summation result, a rectified linear threshold unit (ReLU) is appended. Similar to x 1 , the new hidden state h 1 is obtained according to x 1 and h 0 . The process can be repeated for i times to obtain the new hidden state h i and x i . The new hidden state h i and x i will be fed into the ConvRNN unit. Formally, the operation of h i and x i in ith iterator can be expressed as the following equation:
x i = r e l u ( x i 1 W x x + h i 1 W h x ) h i = r e l u ( x i W x h + h i 1 W h h )
Here, ‘∗’ denotes the 2D convolution. From this equation, we can see that each update of the input always uses the information of the hidden state, which means it merges the representation of the hidden state. Besides, the h i 1 W h x also can help the old input x i 1 to leverage significant representation by convolution. The same advantage also can be reflected in the hidden state. It aggregates the information from the input and can extract better representation under the guidance of the input. By repeating this process, the formed input and the hidden state can nicely exploit the context information before going into the ConvRNN unit.
To examine the performance of the interaction method, we depict Figure 3 which shows the eight one-hour prediction samples w . (with) and w / o . (without) interaction. We utilize the black box to mark the areas with high reflective to emphasize these parts. We can see that the model with the interaction mechanism can effectively improve the nowcasting, especially in the high rainfall regions. For the models without interaction, it even cannot generate the red parts. It implies the interaction scheme is more effective in predicting the high echo value parts. We will further validate its effectiveness in Section 3. We note that this scheme is a general framework which can be applied in any ConvRNN models.

2.2. Dual Attention Mechanism

To further model the long-range dependency, we develop a dual attention mechanism for both the spatial and channel module, shown as in Figure 4. Next, we detail the dual attention mechanism.
Spatial Attention Module Given the feature map f t R N × C × H × W and it can be regarded as the query Q s , where N, C, H and W denote the batch size, channel, height and width of the feature map respectively, we directly reshape it to Q s R N × ( H W ) × C . For a series of feature maps C t τ : t 1 R N × C × τ × H × W , it can be seen as the key K s and value V s . Here, the τ is length of the series. Similarly, it can be reshaped into K s R N × ( τ H W ) × C and V s R N × ( τ H W ) × C . Next, according to the Equation (2), we can obtain the output of spatial attention:
A s = A t t n s ( f t , C t τ : t 1 ) = n o r m ( C t 1 + s o f t m a x ( Q s · K s T ) · V s ) Q s = f t ; K s = V s = C t τ : t 1 ,
which is shown as the green part in Figure 4. Here, the s o f t m a x ( Q s · K s T ) R N × H W × τ H W denotes to apply a softmax layer to the matrix product operation of Q s and K s . It represents position similarity between the Q s and K s . The value implies the degree of correlation of given feature maps f t and the long-term series maps C t τ : t 1 . Then, it can be regarded as the weight to update spatial information by the calculation of matrix product with V s , which selectively integrates the location information from C t τ : t 1 into the final result A s in terms of the spatial similarity. Finally, we rebuild the shape of A s R N × C × H × W to let the f t and C t 1 as the output of the module.
Channel Attention Module aims to model the correlations between the channels. Different from the spatial attention module, the query f t R N × C × H × W are transformed and reshaped to the query Q c R N × C × ( H W ) . Also, the series of feature maps C t τ : t 1 R N × C × τ × H × W can be reshaped into key K c R N × τ C × ( H W ) and value V c R N × ( τ C ) × ( H W ) . The channel attention can be expressed as the following equation:
A c = A t t n c ( f t , C r τ : t 1 ) = n o r m ( C t 1 + s o f t m a x ( Q c · K c T ) · V c ) . Q c = f t ; K c = V c = C t τ : t 1 .
Here, the s o f t m a x ( Q c · K c T ) R N × C × ( τ C ) denotes the query f t ’s impact representation on the key C t τ : t 1 in terms of channels. Then, we perform matrix product between it and the value V c . In the same way, we reshape the result of Equation (3) into appropriate sizes shown as the blue part in Figure 4 and then sum it with the spatial attention result.
Sum Fusion finally integrates the output from the two attention modules. Figure 5 shows the structure of the part in detail. Specifically, it independently applies two convolution layers on A s and A c respectively. The first convolution layer involves a convolution with 3 kernel size, a layer normalization and the activate function of ReLU. The second utilizes a convolution operation where kernel size is 1. Then, the element-wise sum is performed on the two results. At last, a convolution layer is applied to generate the final result.
The combination of the results from both parts not only involves the local spatial features but also includes the channel relationship at corresponding position in the long period of τ . Therefore, the output of the dual attention module is more effective than each single module. In summary, the dual attention module can be shown as the following formula:
A d = A t t n d ( f t , C t τ : t 1 ) = S u m F u s i o n ( A t t n s ( f t , C r τ : t 1 ) , A t t n c ( f t , C r τ : t 1 ) ) .
Figure 6 shows the eight one-hour prediction examples with the position attention module, channel attention module and dual attention module, respectively. We can see the channel attention module and spatial attention model cannot accurately predict the high echo value regions. Both of them underestimate the parts. As for the dual attention module, its prediction is more accurate and the high echo value region parts are not underestimated, which implies that the combination of the two attention modules is very effective. We will further demonstrate its advantages in Section 3.

2.3. The IDA-LSTM Unit

In this subsection, we will introduce how to embed the dual attention mechanism into the LSTM unit to form our IDA-LSTM. Figure 7 illustrates the inside structure of our proposed IDA-LSTM unit. We can see that the input of IDA-LSTM block contains the current input X t , spatial memory M t l 1 , historical temporal memories C t τ : t 1 l and hidden state H t 1 l . The current input X t and hidden state H t 1 l are first transformed by the developed interaction block. The resulting new input and hidden state are then combined with C t τ : t 1 l and as the input of the developed dual attention mechanism to update the temporal memory C t l . By the dual attention on the multiple previous temporal memories C t τ : t 1 l , the C t l can recover the forgotten information. Finally, the temporal memory C t l is delivered to the next time. In addition to temporal memory, in our IDA-LSTM unit, we follow PredRNN and introduce the spatial memory M t l . Its update scheme is the same as that in the PredRNN. Formally, the calculation of the IDA-LSTM unit is expressed as:
i t = σ ( W x i X t + W h i H t 1 l + b i ) , g t = t a n h ( W x g X t + W h g H t 1 l + b g ) , f t = σ ( W x f X t + W h f H t 1 l + b f ) , i t = σ ( W x i X t + W m i M t l 1 + b i ) , g t = t a n h ( W x g X t + W m g M t l 1 + b g ) , f t = σ ( W x f X t + W m f M t l 1 + b f ) , C t l = i t g t + A t t n d ( f t , C t τ : t 1 ) , M t l = i t g t + f t M t l 1 , o t = σ ( W x o X t + W h o H t 1 l + W c o C t l + W m o M t l + b o ) , H t l = o t t a n h ( W 1 × 1 [ X t l , M t l ] ) ,
where ∘ is the matrix product and τ is the number of previous temporal memories. The * denotes the 2D convolution and the L a y e r N o r m is layer normalization [22] that is designed to stabilize the training process. A t t n d in Equation (5) denotes the dual attention module in Section 2.2. Here, the forget gate f t is the query, and the historical memories C t τ : t 1 denotes the key and value. The function of this mechanism is to control where and what information to emphasize on previous memories so as to produce good predictions. Obviously, the dual attention mechanism can nicely model the long-term dependency.

2.4. The IDA-LSTM Extrapolation Architecture

The architecture of our model is similar to the convolutional recurrent network such as PredRNN [15], PredRNN++ [16], and Eidetic 3D LSTM [18] model as shown in Figure 8. Our architecture is a four-layer network built upon the IDA-LSTM units. In the architecture, the temporal memory information is delivered along the horizontal direction (shown as black dot lines) and the spatial memory information is transmitted in an zigzag manner (shown as red dot lines). The prediction X ^ t is generated from the output of the top layer after going throw a convolution layer.

3. Experiment

3.1. Experimental Setup

In this part, we show the introduction of the experiment including the dataset, evaluation metrics, parameter and training setting.

3.1.1. Dataset

We utilize the Conference on Information and Knowledge Management (CIKM) AnalytiCup 2017 challenge dataset to evaluate our model. It contains 10,000 samples as the training set and 4000 samples as the testing set. The sample time of both is different, the data in the training set come from two consecutive years and the test-set is sampled within the next year. We randomly select 2000 samples from the training set as the validation set. Each sample includes 15 CAPPI radar images with an interval of 6 min and elevation-angles (0.5, 1.5, 2.5 and 3.5 km). In this paper, we select the 3.5 km level images to train and test our models. In each sample, the first five echo maps are treated as input and the last ten as the expected output. That is, we aim to predict the one-hour extrapolation based on the half-an-hour observations in the past. Each image covers 101 km × 101 km square with the size of 101 × 101 (pixel). Each pixel represents a resolution of 1 km × 1 km and the original range of it is [0, 255]. We give the Figure 9 to show the distribution of pixels of train-sets, validation-set and test-set. The distribution gap between the training-sets and test implies the challenge of this nowcasting task.

3.1.2. Evaluation Metrics

As a preprocessing step, we convert the pixel value of each pixel value as follows:
d B Z = p i x e l _ v a l u e × 95 / 255 10 .
As for evaluation, we convert the prediction echo map and the ground truth one by thresholding. If the value larger than the given threshold, the corresponding value is set to 1; otherwise it is set to 0. Then we calculate the number of positive predictions TP (prediction = 1, truth = 1), false-positive predictions FP (predictio = 1, truth = 0), true negative predictions TN (prediction = 0, turth = 0) and false-negative predictions FN (predition = 0, truth = 1). Specifically, we use three thresholds namely 5 dBZ, 20 dBZ and 40 dBZ. Finally, we compute the Heidke Skill Score (HSS) [23] and Critical Success Index (CSI) metrics to evaluation the results:
H S S = 2 ( T P × T N F N × F P ) ( T P + F N ) ( F N + T N ) + ( T P + F P ) ( F P + T N ) C S I = T P T P + F N + F P .
Moreover, we also apply MAE and SSIM to evaluate our model from a different view.

3.1.3. Parameter and Training Setting

The parameter configuration details of IDA-LSTM model are described in Table 1. Here, ‘In Kernel’, ‘In Stride’, and ‘In Pad’ denotes the kernel size, stride, and the padding in the input-to-state 3D convolution respectively. ‘State Ker.’ and ‘Spatial Ker.’ denotes the kernel size of the state-to-state and spatial memory 3D convolution respectively. Its stride and padding setting is the same as ‘In Stride’ and ‘In Pad’. For the interaction part as Figure 2 shown, each convolution block applies the convolution kernel with 32 filters, 5 × 5 kernel size, 1 stride and the same padding. Besides, each input with shape (101, 101, 1) is padded with zeros into the shape (128, 128, 1) and then is patched to the shape (32, 32, 16). Eventually, at any time step, the output with the shape (32, 32, 16) in the top layer transforms the shape to (128, 128, 1) as the final prediction.
Before training, all radar echo maps were normalized to [−1, 1] as the input. our model is optimized with an L1+L2 loss. In the training step, all models are trained by utilizing the ADAM optimizer [24] with the 0.001 learning rate. The batch size of each iteration process and the maximum number of iterations is set to 4 and 80,000 respectively. Besides, the early-stopping strategy was applied. All experiments are implemented in Pytorch and executed on NVIDIA TITAN GPU.

3.2. Experimental Results

Table 2 shows the results of all the methods. Here, ConvLSTM, ConvGRU, TrajGRU and PredRNN were previously applied and tested on the data set. From this table, DA-LSTM denotes our model with the dual interaction module but without the interaction part. We can see that the proposed model including DA-LSTM and IDA-LSTM achieves the best performance in terms of the HSS, CSI at all thresholds. Particularly, the evaluated metrics reach 0.2262 and 0.1287 when the threshold is at 40 dBZ, which is 21.74% and 24.47% higher than the second-rank algorithm (MIM model), respectively. It implies that the developed dual attention and interaction modules are helpful for the high rainfall region prediction, which is especially important to alert significant threats on human activity and economy. For the MIM model, it delivers the the second result at two relatively high thresholds (20 dBZ and 40 dBZ). This can be attributed to its memory in memory scheme, which can help to preserve the high echo value regions. Besides, the results of PredRNN, PredRNN++ and TrajGRU are superior to ConvLSTM and ConvGRU. E3DLSTM performs the worst among all the methods.
We draw Figure 10 to describe the HSS and CSI curves of all models at all nowcasting lead time stamps. We can see that our models always keep top positions at all thresholds and any time, which demonstrates the robust superiority of our approach. It is worth pointing out that the gaps between our IDA-LSTM model, and other models are most obvious at the 40 dBZ, which demonstrates our method significantly improves the high rainfall region prediction. Besides, the MIM model delivers the second best performance due to the design of memory in memory. Moreover, the result of PredRNN is always worse than the proposed models, particularly at the high threshold. It implies the effectiveness of the proposed interaction and dual attention mechanisms, because the two parts are the only differences between PredRNN and our IDA-LSTM. As for ConvGRU and ConvLSTM model, they deliver the worst performance.
To better compare and understand the results, we visualize some prediction examples produced by different methods in Figure 11. We can see that only PredRNN, DA-LSTM, and IDA-LSTM can preserve the high echo value regions (red parts) in the 10th prediction image. Among the three methods, IDA-LSTM is the best, followed by DA-LSTM and then PredRNN. The reason is that IDA-LSTM applies both the interaction and dual attention schemes, which better model the short-term and long-term dependency. DA-LSTM applies only the dual attention module. Hence, the performance degenerates a little bit. As for PredRNN, it does not have the interaction and dual attention modules, but only leverages the spatial memory cell. Thus, it can only preserve parts of the high echo value regions. Moreover, we can see from the ground-truth sequence that the high echo value regions increase and the intensity becomes higher as the lead time goes on. The proposed IDA-LSTM can nicely predict the trend. As for other deep learning models, they cannot predict the high echo value regions and the red parts gradually disappear as the lead time goes on. The fact further confirms our motivation of this paper.

3.3. Ablation Study

To further investigate the effectiveness of the two proposed mechanisms, we conduct ablation study in this subsection.
Context Interaction: To validate the effectiveness of the proposed interaction framework, we embed it into the ConvLSTM, PredRNN, PredRNN++ and DA-LSTM, respectively and test their performances. Table 3 shows the results of these models without the interaction and with different number of interactions. The methods with the prefix “i” denote the ones with interactions and the superscript number denotes interaction times. We can see that when equipped with interactions, the performances of these models all improve. The improvement is especially obvious at 40 dBZ threshold. The fact demonstrates the effectiveness of the proposed interaction framework.
Similarly, we also depict the HSS and CSI curves w.r.t. different nowcasting lead times at 40 dBZ threshold in Figure 12 and Figure 13, respectively. The methods with the prefix “i” denote the ones with the interactions. We observe that the performance with the interactions is better at different nowcasting lead times. The fact further validates the effectiveness of the developed interaction framework.
To visually compare the results with/without the interactions of the four methods ConvLSTM, PredRNN, PredRNN++, IDA-LSTM, we depict in Figure 14 four groups of prediction results. We can see that the models with the interactions predict the high echo value regions (red parts) better than their counterparts without the interactions. The results further demonstrate the superiority of the interaction framework.
Dual Attention Module: The purpose of the dual attention module is to exploit more spatiotemporal representation from huge history temporal memories and to preserve adequate information involving high echo value regions. To validate its effectiveness, we embed different types of attention schemes into the PredRNN model. The attention schemes include “without attention”, “with spatial attention”, “with channel attention” and “with the dual attention”. Table 4 shows the results of the four schemes. Here, CA-LSTM and SA-LSTM denote the PredRNN model with channel attention and spatial attention respectively. The DA-LSTM represents the PredRNN model with our dual attention. We observe from the table that the DA-LSTM delivers the best results, followed by CA-LSTM and SA-LSTM. The PredRNN method without any attention performs the worst. The observations validate the effectiveness of the attention schemes and the superiority of our dual attention mechanism.
Again, we show the HSS and CSI curves of the four methods against different nowcasting lead times in Figure 15. It can be seen that the DA-LSTM equipped with our dual attention mechanism consistently performs the best. Moreover, as the threshold increases from 5 dBZ to 40 dBZ, the improvement of our method becomes more and more obviously. The observation demonstrates that the developed dual attention mechanism is especially helpful for the prediction of high echo value parts.
Figure 16 shows the one-hour prediction echo maps of the four schemes on an example. We can see that the method with our dual attention mechanism can better preserve the high echo value parts (red parts) than the one with single attention or without attention. The result further validates the effectiveness of the developed dual attention mechanism.

4. Conclusions

In this paper, we propose a novel radar echo map extrapolation method, namely, IDA-LSTM. In the method, an interaction framework is developed for the ConvRNN unit to fully exploit the short-term context information, which can be applied in any ConvRNN based model. The ablation study shows that it can improve the prediction almost in all ConvRNN models after several interactions. Moreover, a dual attention mechanism is developed to combine the channel attention and spatial attention, which can recall forgotten information in ConvRNN to model long-term dependency. The experiment shows that embedding dual attention into the PredRNN achieve better performance. By combining the two improvements, we proposed IDA-LSTM which overcomes the underestimation issue of high echo value parts that existing deep learning extrapolation methods suffer from. By comparing other existing algorithms, the superiority and performance of it have been fully demonstrated. In the future, we will explore the prediction of radar echo maps at multiple heights to improve the nowcasting in some extreme weather events such as convection.

Author Contributions

Conceptualization, X.L. and C.L.; methodology, C.L.; software, C.L. and Y.W.; validation, C.L. and X.Z.; formal analysis, C.L.; investigation, Y.Y.; resources, Y.Y.; data curation, Y.W.; writing—original draft preparation, C.L.; writing—review and editing, Y.Y., X.L. and X.Z.; visualization, Y.W.; supervision, Y.Y., X.L.; project administration, X.L.; funding acquisition, Y.Y. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported in part by the Shenzhen Science and Technology Program under Grant JCYJ20180507183823045 and JCYJ20200109113014456.

Institutional Review Board Statement

The study did not involve humans or animals.

Informed Consent Statement

The study did not involve humans.

Data Availability Statement

The CIKM AnalytiCup 2017 Dataset can be download according to this address: https://tianchi.aliyun.com/dataset/dataDetail?dataId=1085.

Conflicts of Interest

The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

References

  1. Marshall, J.S.; Palmer, W.M.K. The distribution of raindrops with size. J. Meteorol. 1948, 5, 165–166. [Google Scholar] [CrossRef]
  2. Wong, W.; Yeung, L.H.; Wang, Y.; Chen, M. Towards the Blending of NWP with Nowcast—Operation Experience in B08FDP. In Proceedings of the WMO Symposium on Nowcasting, Hong Kong, China, 25–29 July 2009; Volume 30. [Google Scholar]
  3. Woo, W.C.; Wong, W.K. Operational application of optical flow techniques to radar-based rainfall nowcasting. Atmosphere 2017, 8, 48. [Google Scholar] [CrossRef] [Green Version]
  4. Shi, X.; Chen, Z.; Wang, H.; Yeung, D.Y.; Wong, W.K.; Woo, W.C. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 7–12 December 2015; pp. 802–810. [Google Scholar]
  5. Ballas, N.; Yao, L.; Pal, C.; Courville, A. Delving deeper into convolutional networks for learning video representations. In Proceedings of the International Conference on Learning Representations (ICLR), San Juan, PR, USA, 2–4 May 2016. [Google Scholar]
  6. Shi, X.; Gao, Z.; Lausen, L.; Wang, H.; Yeung, D.Y.; Wong, W.K.; Woo, W.C. Deep learning for precipitation nowcasting: A benchmark and a new model. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 5617–5627. [Google Scholar]
  7. Tran, Q.K.; Song, S.k. Computer vision in precipitation nowcasting: Applying image quality assessment metrics for training deep neural networks. Atmosphere 2019, 10, 244. [Google Scholar] [CrossRef] [Green Version]
  8. Wang, Y.; Zhang, J.; Zhu, H.; Long, M.; Wang, J.; Yu, P.S. Memory in memory: A predictive neural network for learning higher-order non-stationarity from spatiotemporal dynamics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 9154–9162. [Google Scholar]
  9. Ayzel, G.; Heistermann, M.; Sorokin, A.; Nikitin, O.; Lukyanova, O. All convolutional neural networks for radar-based precipitation nowcasting. Procedia Comput. Sci. 2019, 150, 186–192. [Google Scholar] [CrossRef]
  10. Shi, E.; Li, Q.; Gu, D.; Zhao, Z. A method of weather radar echo extrapolation based on convolutional neural networks. In Proceedings of the International Conference on Multimedia Modeling, Bangkok, Thailand, 5–7 February 2018; pp. 16–28. [Google Scholar]
  11. Sutskever, I.; Vinyals, O.; Le, Q.V. Sequence to sequence learning with neural networks. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; pp. 3104–3112. [Google Scholar]
  12. Jaderberg, M.; Simonyan, K.; Zisserman, A. Spatial transformer networks. In Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 7–12 December 2015; pp. 2017–2025. [Google Scholar]
  13. Marrocu, M.; Massidda, L. Performance Comparison between Deep Learning and Optical Flow-Based Techniques for Nowcast Precipitation from Radar Images. Forecasting 2020, 2, 194–210. [Google Scholar] [CrossRef]
  14. Jozefowicz, R.; Zaremba, W.; Sutskever, I. An empirical exploration of recurrent network architectures. In Proceedings of the International Conference on Machine Learning, Lille, France, 6–11 July 2015; pp. 2342–2350. [Google Scholar]
  15. Wang, Y.; Long, M.; Wang, J.; Gao, Z.; Philip, S.Y. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 879–888. [Google Scholar]
  16. Wang, Y.; Gao, Z.; Long, M.; Wang, J.; Philip, S.Y. PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning. In Proceedings of the International Conference on Machine Learning, Stockholm, Sweden, 10–15 July 2018; pp. 5123–5132. [Google Scholar]
  17. Tran, Q.K.; Song, S.K. Multi-channel weather radar echo extrapolation with convolutional recurrent neural networks. Remote Sens. 2019, 11, 2303. [Google Scholar] [CrossRef] [Green Version]
  18. Wang, Y.; Jiang, L.; Yang, M.H.; Li, J.; Long, M.; Li, F.F. Eidetic 3D LSTM: A Model for Video Prediction and Beyond. In Proceedings of the International Conference on Learning Representations, 900 Convention Center Blvd, New Orleans, LA, USA, 6–9 May 2019; pp. 1–14. [Google Scholar]
  19. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 5998–6008. [Google Scholar]
  20. Lin, Z.; Feng, M.; Santos, C.N.D.; Yu, M.; Xiang, B.; Zhou, B.; Bengio, Y. A structured self-attentive sentence embedding. arXiv 2017, arXiv:1703.03130. [Google Scholar]
  21. Fu, J.; Liu, J.; Tian, H.; Li, Y.; Bao, Y.; Fang, Z.; Lu, H. Dual attention network for scene segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 3146–3154. [Google Scholar]
  22. Ba, J.L.; Kiros, J.R.; Hinton, G.E. Layer Normalization. arXiv 2016, arXiv:1607.06450. [Google Scholar]
  23. Hogan, R.J.; Ferro, C.A.; Jolliffe, I.T.; Stephenson, D.B. Equitability revisited: Why the “equitable threat score” is not equitable. Weather Forecast. 2010, 25, 710–726. [Google Scholar] [CrossRef] [Green Version]
  24. Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv 2014, arXiv:1412.6980. [Google Scholar]
Figure 1. An instance of radar echo map prediction. The first line denotes the ground truth and the second line is the prediction by the ConvLSTM model. (Best view in color).
Figure 1. An instance of radar echo map prediction. The first line denotes the ground truth and the second line is the prediction by the ConvLSTM model. (Best view in color).
Remotesensing 13 00164 g001
Figure 2. The interaction block. It interacts with the representation between the hidden state h and the input x.
Figure 2. The interaction block. It interacts with the representation between the hidden state h and the input x.
Remotesensing 13 00164 g002
Figure 3. The first line is the eight ground truth examples and the rest are predictions after an hour without (second row) interaction and with (the last row). The color scheme is based on the color code in the right part (Best view in color).
Figure 3. The first line is the eight ground truth examples and the rest are predictions after an hour without (second row) interaction and with (the last row). The color scheme is based on the color code in the right part (Best view in color).
Remotesensing 13 00164 g003
Figure 4. The dual attention mechanism embeding in proposed model.
Figure 4. The dual attention mechanism embeding in proposed model.
Remotesensing 13 00164 g004
Figure 5. The structure of SumFusion modular.
Figure 5. The structure of SumFusion modular.
Remotesensing 13 00164 g005
Figure 6. The first line is the eight ground truth examples and the rest are predictions after an hour merely with spatial attention (second row), channel attention (third row), and dual attention (the last row). (Best view in color).
Figure 6. The first line is the eight ground truth examples and the rest are predictions after an hour merely with spatial attention (second row), channel attention (third row), and dual attention (the last row). (Best view in color).
Remotesensing 13 00164 g006
Figure 7. The inside structure of Interactional Dual Attention Long Short-term Memory (IDA-LSTM) unit.
Figure 7. The inside structure of Interactional Dual Attention Long Short-term Memory (IDA-LSTM) unit.
Remotesensing 13 00164 g007
Figure 8. The architecture of our model.
Figure 8. The architecture of our model.
Remotesensing 13 00164 g008
Figure 9. The histograms of non-zeros pixel values in the train-sets, validation-set and test-set respectively.
Figure 9. The histograms of non-zeros pixel values in the train-sets, validation-set and test-set respectively.
Remotesensing 13 00164 g009
Figure 10. The performance changes against different nowcasting lead time in terms of HSS and CSI scores. (Best view in color).
Figure 10. The performance changes against different nowcasting lead time in terms of HSS and CSI scores. (Best view in color).
Remotesensing 13 00164 g010aRemotesensing 13 00164 g010b
Figure 11. The prediction results of all methods on an example from the CIKM AnalytiCup 2017 competition. The first five images in the first row are the input, and the remainders denote the ground-truth output. Other rows are the prediction of various models.
Figure 11. The prediction results of all methods on an example from the CIKM AnalytiCup 2017 competition. The first five images in the first row are the input, and the remainders denote the ground-truth output. Other rows are the prediction of various models.
Remotesensing 13 00164 g011
Figure 12. The performance changes against different nowcast lead times in interaction ablation study in terms of HSS as the threshold is 40 dBZ. (Best view in color).
Figure 12. The performance changes against different nowcast lead times in interaction ablation study in terms of HSS as the threshold is 40 dBZ. (Best view in color).
Remotesensing 13 00164 g012
Figure 13. The performance changes against different nowcast lead times in interaction ablation study in terms of CSI as the threshold is 40 dBZ. (Best view in color).
Figure 13. The performance changes against different nowcast lead times in interaction ablation study in terms of CSI as the threshold is 40 dBZ. (Best view in color).
Remotesensing 13 00164 g013
Figure 14. The four group interaction ablation examples from the CIKM AnalytiCup 2017 competition. These groups are ConvLSTM, PredRNN, PredRNN++, and DAST-LSTM respectively from top to bottom. The first five images in the first row are the input, and the remainders denote the groundtruth output. Other rows are the prediction of various models. (Best view in color).
Figure 14. The four group interaction ablation examples from the CIKM AnalytiCup 2017 competition. These groups are ConvLSTM, PredRNN, PredRNN++, and DAST-LSTM respectively from top to bottom. The first five images in the first row are the input, and the remainders denote the groundtruth output. Other rows are the prediction of various models. (Best view in color).
Remotesensing 13 00164 g014aRemotesensing 13 00164 g014b
Figure 15. The performance changes against different nowcast lead times in attention mechanisms ablation study in terms of HSS and CSI scores. (Best view in color).
Figure 15. The performance changes against different nowcast lead times in attention mechanisms ablation study in terms of HSS and CSI scores. (Best view in color).
Remotesensing 13 00164 g015
Figure 16. The attention mechanism ablation example from the CIKM AnalytiCup 2017 competition. The first five images in the first row are the input, and the remainders denote the ground-truth output. Other rows are the prediction of various models. (Best view in color).
Figure 16. The attention mechanism ablation example from the CIKM AnalytiCup 2017 competition. The first five images in the first row are the input, and the remainders denote the ground-truth output. Other rows are the prediction of various models. (Best view in color).
Remotesensing 13 00164 g016
Table 1. The details of the IDA-LSTM model. The output of the Layer 5 will be transform the prediction with same shape of input.
Table 1. The details of the IDA-LSTM model. The output of the Layer 5 will be transform the prediction with same shape of input.
LayerIn KernelIn StridePadState Ker.Spatial Ker.Ch I/OIn ResOut ResType
Layer 15 × 51 × 12 × 25 × 55 × 516/3232 × 3232 × 32IDA-LSTM
Layer 25 × 51 × 12 × 25 × 55 × 532/3232 × 3232 × 32IDA-LSTM
Layer 35 × 51 × 12 × 25 × 55 × 532/3232 × 3232 × 32IDA-LSTM
Layer 45 × 51 × 12 × 25 × 55 × 532/3232 × 3232 × 32IDA-LSTM
Layer 51 × 11 × 10 × 0--32/1632 × 3232 × 32Conv2D
Table 2. Comparison results of ablation study on the CIKM AnalytiCup 2017 competition dataset in terms of Heidke Skill Score (HSS) and Critical Success Index (CSI). Bold denotes the best evaluate index among all models.
Table 2. Comparison results of ablation study on the CIKM AnalytiCup 2017 competition dataset in terms of Heidke Skill Score (HSS) and Critical Success Index (CSI). Bold denotes the best evaluate index among all models.
dBZ ThresholdHSS ↑CSI ↑MAE ↓SSIM ↑
52040avg52040avg
ConvLSTM [4]0.70350.48190.10810.43120.76560.40340.05780.408915.060.2229
ConvGRU [6]0.67760.47660.15100.43510.74730.39070.08230.406816.270.1370
TrajGRU [6]0.68280.48620.16210.44370.74990.40200.08880.413615.990.1519
PredRNN [15]0.70800.49110.15580.45160.76910.40480.08390.419814.540.3341
PredRNN++ [16]0.70750.49930.15740.45480.76700.41370.08620.422314.510.3357
E3D-LSTM [18]0.71110.48100.13610.44270.77200.40600.07340.417114.780.3089
MIM [8]0.70520.51660.18580.46920.76280.42790.10340.431314.690.2123
DA-LSTM0.71840.52510.21270.48540.77650.43760.12020.444814.100.3479
IDA-LSTM0.71790.52640.22620.49020.77520.43720.12870.447014.090.3506
Table 3. Comparison results with the different number of interactions on the CIKM AnalytiCup 2017 competition dataset in terms of HSS and CSI. Bold denotes the best evaluate index among all models.
Table 3. Comparison results with the different number of interactions on the CIKM AnalytiCup 2017 competition dataset in terms of HSS and CSI. Bold denotes the best evaluate index among all models.
dBZ ThresholdHSS ↑CSI ↑MAE ↓SSIM ↑
52040avg52040avg
ConvLSTM0.70350.48190.10810.43120.76560.40340.05780.408915.060.2229
IConvLSTM 1 0.71490.48890.12360.44240.77690.41190.06670.418414.620.3390
IConvLSTM 2 0.70550.50010.12150.44240.76680.41200.06520.414614.420.3365
IConvLSTM 3 0.70920.47400.12470.43600.77840.41180.06710.419115.110.3372
IConvLSTM 4 0.56450.40440.08300.35030.63050.33620.04530.337320.650.3111
IPredRNN0.70810.49110.15580.45160.76910.40480.08540.419814.540.3341
IPredRNN 1 0.71330.51080.20470.47620.76850.41880.11510.434114.030.3488
IPredRNN 2 0.70810.50390.15310.45500.77100.41540.08360.423314.400.3312
IPredRNN 3 0.70010.51790.19510.47100.77100.42890.10890.435914.520.3281
IPredRNN 4 0.71110.50190.21550.47620.77260.41010.12180.434814.200.3327
IPredRNN++0.70750.49930.15750.45480.76700.41370.08620.422314.510.3357
IPredRNN++ 1 0.71880.51000.20040.47640.77590.42510.11240.437814.130.3513
IPredRNN++ 2 0.71190.50370.20980.47510.77150.42040.11810.436714.330.3423
IPredRNN++ 3 0.70230.49950.16100.45430.76650.41100.08820.421914.590.3255
IPredRNN++ 4 0.71530.49680.21720.47640.77740.42390.12340.441614.620.3487
DA-LSTM0.71850.52510.21270.48540.77650.43760.12020.444814.100.3479
IDA-LSTM 1 0.70930.50650.16060.45880.76830.42180.08810.426114.380.3345
IDA-LSTM 2 0.71790.52640.22620.49020.77520.43720.12870.447014.090.3506
IDA-LSTM 3 0.71790.51560.18790.47380.77980.43420.10440.439514.180.3436
IDA-LSTM 4 0.70680.50850.19300.46940.76310.41250.10810.427914.210.3461
Table 4. Comparison results of attention mechanism ablation study on the CIKM AnalytiCup 2017 competition dataset in terms of HSS and CSI. Bold denotes the best evaluate index among all models.
Table 4. Comparison results of attention mechanism ablation study on the CIKM AnalytiCup 2017 competition dataset in terms of HSS and CSI. Bold denotes the best evaluate index among all models.
ModelHSS ↑CSI ↑MAE ↓SSIM ↑
52040avg52040avg
PredRNN0.70810.49110.15580.45160.76910.40480.08540.419814.540.3341
SA-LSTM0.70420.49820.14810.45020.76890.41430.08080.421314.680.3241
CA-LSTM0.71150.50660.15750.45850.77330.41720.08610.425514.230.3296
DA-LSTM0.71850.52510.21270.48540.77650.43760.12020.444814.100.3479
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Luo, C.; Li, X.; Wen, Y.; Ye, Y.; Zhang, X. A Novel LSTM Model with Interaction Dual Attention for Radar Echo Extrapolation. Remote Sens. 2021, 13, 164. https://doi.org/10.3390/rs13020164

AMA Style

Luo C, Li X, Wen Y, Ye Y, Zhang X. A Novel LSTM Model with Interaction Dual Attention for Radar Echo Extrapolation. Remote Sensing. 2021; 13(2):164. https://doi.org/10.3390/rs13020164

Chicago/Turabian Style

Luo, Chuyao, Xutao Li, Yongliang Wen, Yunming Ye, and Xiaofeng Zhang. 2021. "A Novel LSTM Model with Interaction Dual Attention for Radar Echo Extrapolation" Remote Sensing 13, no. 2: 164. https://doi.org/10.3390/rs13020164

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop