Traffic sign detection and recognition using fully convolutional network guided proposals

Abstract

Detecting and recognizing traffic signs is a hot topic in the field of computer vision with lots of applications, e.g., safe driving, path planning, robot navigation etc. We propose a novel framework with two deep learning components including fully convolutional network (FCN) guided traffic sign proposals and deep convolutional neural network (CNN) for object classification. Our core idea is to use CNN to classify traffic sign proposals to perform fast and accurate traffic sign detection and recognition. Due to the complexity of the traffic scene, we improve the state-of-the-art object proposal method, EdgeBox, by incorporating with a trained FCN. The FCN guided object proposals can produce more discriminative candidates, which help to make the whole detection system fast and accurate. In the experiments, we have evaluated the proposed method on publicly available traffic sign benchmark, Swedish Traffic Signs Dataset (STSD), and achieved the state-of-the-art results.

Introduction

Traffic scene analysis is a very important topic in computer vision and intelligent systems [1], [2], [3], [4], [5], [6]. Traffic signs are designed to inform drivers of the current condition and other important information of the road. They are rigid and simple shapes with eye-catching colors and the information they carry is easy to understand. However, accidents may still occur when drivers do not pay attention to a traffic sign in time. Hence, it is very important to design an automatic real-time driver assistance system to detect and recognize traffic signs.

Although the research community and industry in the field of computer vision have achieved significant progresses in traffic sign detection and recognition in the past decades, there are two main inevitable difficulties. One is poor image quality due to low resolution, motion blur and noise. The other is uncontrolled environmental factors including weather conditions, complex background, variable illumination, sign color fading, etc. Fig. 1 shows some difficult examples. These make the traffic sign detection and recognition task still an open problem.

In order to tackle these problems, we propose a novel and efficient method to detect and recognize traffic signs. Generally, traffic signs have a distinctive property that they always appear on the two sides of the road to be traveled. Taking advantage of this property, the proposed method extracts traffic sign proposals by the guidance of fully convolutional network, unlike traditional traffic sign detection and recognition methods, which usually start from color segmentation [7], [8], [9], [10], [11], shape detection [12], [13], [14], [15] or sliding-window scanning [16], [17] to find traffic sign regions. This can effectively reduce the search range of traffic signs, so as to reduce the number of proposals and improve the efficiency.

The pipeline of the proposed method for traffic sign detection is illustrated in Fig. 2, which is mainly composed of two parts. One is the proposal extraction stage guided by fully convolutional network [18] and EdgeBox [19]. The other is the traffic sign classification stage using convolutional neural network [20]. Given a scene image, coarse regions of traffic signs are generated by fully convolutional network firstly. Then, proposals of traffic signs are extracted by EdgeBox from the coarse sign regions. Finally, traffic signs are identified and false positives are eliminated by a trained convolutional neural network classifier and the optimal bounding boxes are retained by non-maximal suppression.

Different from the previous methods on both traffic sign detection and general object detection, a novel FCN guided object proposal method is proposed. In the case of traffic sign detection where pixel-level annotation is unavailable, we propose to train FCN using boundingbox-level annotation which is a typical weak supervision for the semantic segmentation method like FCN.

The proposed method is validated on a publicly available database: Swedish Traffic Signs Dataset (STSD) [21]. The experimental results demonstrate that the proposed method has a much higher detection rate than other methods while producing less false positives. In addition, the proposed method is not time consuming, which is potentially to be applied in real time applications by utilizing the computational power of GPU devices.

Object detection using proposal is popular in recent literatures, especially after R-CNN [22] series works have achieved the state-of-the-art object detection performance on PASCAL VOC [23] and ILSVRC detection task [24]. The proposed method follows R-CNN, but there are at least three major differences from R-CNN: (1) We extent the R-CNN in a new application, the challenging traffic sign detection problem. (2) We adopt a much faster object proposal method, EdgeBox, rather than the selective search method [25] which is more accurate but slower. (3) We proposed a novel FCN guided object proposal which is both fast and accurate.

In summary, the main contributions of this work can be concluded in three folds: first, this paper proposes a new framework for traffic sign detection and recognition based on object proposal, which works in a coarse-to-fine manner. Second, due to the guidance of heat maps of traffic signs that generated by fully convolutional networks, the searching area of signs is drastically narrowed. Third, an efficient multi-class CNN model is trained for sign recognition with the bootstrapping strategy. The proposed method achieves the state-of-the art performance on the traffic sign benchmark.

The remainder of this paper is organized as follows: Section 2 presents recent works related to the proposed method. Section 3 details the proposed method, including coarse sign region segmentation, proposal extraction and sign classification. In Section 4, the experimental results and discussions are presented. Finally, the conclusions and future works are given in Section 5.