Abstract
We propose an advanced Automatic number-plate recognition (ANPR) system, which not only recognizes the number and the issuing state, but also the type and location of the vehicle in the input image. The system is based on a combination of existing methods, modifications to neural network architectures and improvements in the training process. The proposed system uses machine-learning approach and consists of three main parts: segmentation of input image by Fully Convolutional Network for localization of license plate and determination of vehicle type; recognition of the characters of the localized plate by a Maxout CNN and LSTM; determination of the state that has issued the license plate by a CNN. The training of these neural network models is accomplished using a manually labeled custom dataset, which is expanded with data augmented techniques. The resulting system is capable of localizing and classifying multiple types of vehicles (including motorcycles and emergency vehicles) as well as their license plates. The achieved precision of the localization is 99.5%. The whole number recognition accuracy is 96.7% and character level recognition accuracy is 98.8%. The determination of issuing state is precise in 92.8% cases.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Patel, C., Shah, D., and Patel, A., Automatic number plate recognition system (ANPR): A survey, Int. J. Comput. Appl., 2013, vol. 69, no. 9, pp. 21–33.
Everingham, M., Eslami, S.A., Van Gool, L., Williams, C.K., Winn, J., and Zisserman, A., The Pascal visual object classes challenge: A retrospective, Int. J. Comput. Vision, 2015, vol. 111, no. 1, pp. 98–136.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., and Fei-Fei, Li, ImageNet Large Scale Visual Recognition Challenge, Int. J. Comput. Vision, 2015, vol. 115, no. 3, pp. 211–252.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L., Microsoft COCO: Common objects in context, European Conference on Computer Vision, 2014, pp. 740–755.
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y., Overfeat: Integrated recognition, localization and detection using convolutional networks, arXiv:1312.6229, 2013.
Simonyan, K. and Zisserman, A., Very deep convolutional networks for large-scale image recognition, arXiv:1409.1556, 2014.
He, K., Zhang, X., Ren, S., and Sun, J., Deep residual learning for image recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
Ren, S., He, K., Girshick, R., and Sun, J., Faster R-CNN: Towards real-time object detection with region proposal networks, Adv. Neural Inf. Process. Syst., 2015, pp. 91–99.
Redmon, J., and Farhadi, A., YOLO9000: Better, faster, stronger, arXiv:1612.08242, 2016.
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C., SSD: Single shot multibox detector, European Conference on Computer Vision, 2016, pp. 21–37.
Grangier, D., Bottou, L., and Collobert, R., Deep convolutional networks for scene parsing, ICML 2009 Deep Learning Workshop, 2009, vol. 3.
Farabet, C., Couprie, C., Najman, L., and LeCun, Y., Scene parsing with multiscale feature learning, purity trees, and optimal covers, arXiv:1202.2160, 2012.
Badrinarayanan, V., Handa, A., and Cipolla, R., Segnet: A deep convolutional encoder-decoder architecture for robust semantic pixel-wise labeling, arXiv:1505.07293, 2015.
Paszke, A., Chaurasia, A., Kim, S., and Culurciello, E., Enet: A deep neural network architecture for real-time semantic segmentation, arXiv:1606.02147, 2016.
Shelhamer, E., Long, J., and Darrel, T., Fully convolutional networks for semantic segmentation, IEEE Trans. Pattern Anal. Mach. Intell., 2017, vol. 39, no. 4, pp. 640–651.
Li, H. and Shen, C., Reading car license plates using deep convolutional neural networks and LSTMs, arXiv:1601.05610, 2016.
Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., Prenger, R., Satheesh, S., Sengupta, S., Coates, A., and Ng, A.Y., Deep speech: Scaling up end-to-end speech recognition, arXiv:1412.5567, 2014.
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y., Show, attend and tell: Neural image caption generation with visual attention, International Conference on Machine Learning, 2015, pp. 2048–2057.
He, P., Huang, W., Quao, Y., Chen, C.L., and Tang, X., Reading scene text in deep convolutional sequences, Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y., Maxout networks, arXiv:1302.4389, 2013.
Jaderberg, M., Vedaldi, A., and Zisserman, A., Deep features for text spotting, European Conference on Computer Vision, 2014, pp. 512–528.
Hochreiter, S. and Schmidhuber, J., Long short-term memory, Neural Comput., 1997, vol. 9, no. 8, pp. 1735–1780.
Dorbe, N., Kadikis, R., and Nesenbergs, K., Vehicle type and license plate localisation and segmentation using FCN and LSTM, New Challenges of Economic and Business Development—2017: Digital Economy, 2017, pp. 132–140.
Gers, F.A. and Schmidhuber, J., Recurrent nets that time and count, Neural Networks, 2000. IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on, 2000, vol. 3, pp. 189–194.
Graves, A. Fernández, S., Gomez, F., and Schmidhuber, J., Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks, Proceedings of the 23rd International Conference on Machine Learning, 2006, pp. 369–376.
Author information
Authors and Affiliations
Corresponding author
Additional information
The article was translated by the authors.
About this article
Cite this article
Dorbe, N., Jaundalders, A., Kadikis, R. et al. FCN and LSTM Based Computer Vision System for Recognition of Vehicle Type, License Plate Number, and Registration Country. Aut. Control Comp. Sci. 52, 146–154 (2018). https://doi.org/10.3103/S0146411618020104
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.3103/S0146411618020104