Abstract
Robust understanding of the lane position and type is essential for changing lanes in autonomous vehicles. However, accomplishing this task in real time with high level of precision is not trivial. In this paper, we propose a novel cascaded deep neural network (DNet-CNet) for real-time end-to-end lane detection (DNet) and classification (CNet). The proposed model can simultaneously predict the lanes position and types. DNet integrates the spatial features extracted from the encoder with those from the decoder to compensate for the lower dimensional encoded data and edge information. Furthermore, the output of DNet is fused with the input image for real time lightweight lane classification model (CNet). The combined features exploit the inherent colors and shape of lanes to improve classification accuracy. Experimental results on the benchmark TuSimple, Caltech-lanes and ELAS datasets show that, the model proposed achieves superior lane detection and classification accuracy in real-time as compared to Cascade-CNN.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alsarhan A, Kilani Y, Al-Dubai A et al (2020) Novel fuzzy and game theory based clustering and decision making for VANETs. IEEE Trans Veh Technol 69(2):1568–1581. https://doi.org/10.1109/TVT.2019.2956228
Aly M (2008) Real time detection of lane markers in urban streets. In: IEEE intelligent vehicles symposium (IV). IEEE, Eindhoven, Netherlands, pp 7–12. https://doi.org/10.1109/IVS.2008.4621152
Arora M, Mangipudi P, Dutta MK (2021) Deep learning neural networks for acrylamide identification in potato chips using transfer learning approach. J Ambient Intell Humaniz Comput 12(12):10,601–10,614. https://doi.org/10.1007/s12652-020-02867-2
Benligiray B, Topal C, Akinlar C (2012) Video-based lane detection using a fast vanishing point estimation method. In: IEEE international symposium on multimedia (ISM). IEEE, Irvine, CA, USA, pp 348–351. https://doi.org/10.1109/ISM.2012.70
Berriel RF, de Aguiar E, De Souza AF et al (2017) Ego-lane analysis system (ELAS): dataset and algorithms. Image Vis Comput 68:64–75. https://doi.org/10.1016/j.imavis.2017.07.005
Cela AF, Bergasa LM, Sanchez FL, et al (2013) Lanes detection based on unsupervised and adaptive classifier. In: International conference on computational intelligence, communication systems and networks (CICSyN). IEEE, Madrid, Spain, pp 228–233. https://doi.org/10.1109/CICSYN.2013.40
Chen LC, Zhu YK, Papandreou G, et al (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: European conference on computer vision (ECCV). Springer, Munich, Germany, pp 801–818. https://doi.org/10.1007/978-3-030-01234-2_49
Chira IM, Chibulcutean A, Danescu RG (2010) Real-time detection of road markings for driving assistance applications. In: International conference on computer engineering and systems (ICCES). IEEE, Cairo, Egypt, pp 158–163. https://doi.org/10.1109/ICCES.2010.5674844
Chougule S, Koznek N, Ismail A, et al (2018) Reliable multilane detection and classification by utilizing CNN as a regression network. In: European conference on computer vision (ECCV). Springer, Munich, Germany, pp 740–752. https://doi.org/10.1007/978-3-030-11021-5_46
Collado JM, Hilario C, De La Escalera A, et al (2006) Adaptative road lanes detection and classification. In: International conference on advanced concepts for intelligent vision systems (ACIVS). Springer, Antwerp, Belgium, pp 1151–1162. https://doi.org/10.1007/11864349_105
De Paula MB, Jung CR (2013) Real-time detection and classification of road lane markings. In: Conference on graphics, patterns and images. IEEE, Arequipa, Peru, pp 83–90. https://doi.org/10.1109/SIBGRAPI.2013.21
De Paula MB, Jung CR (2015) Automatic detection and classification of road lane markings using onboard vehicular cameras. IEEE Trans Intell Transp Syst 16(6):3160–3169. https://doi.org/10.1109/TITS.2015.2438714
Fan RC, Wang XR, Hou QB et al (2019) SpinNet: Spinning convolutional network for lane boundary detection. Computational Visual Media 5(4):417–428. https://doi.org/10.1007/s41095-019-0160-1
Fang HY, Zhu J, Fang Y (2021) ContinuityLearner: Geometric continuity feature learning for lane segmentation. Preprint at https://arxiv.org/abs/2108.03507
Ghani HA, Besar R, Sani ZM, et al (2021) Advances in lane marking detection algorithms for all-weather conditions. International Journal of Electrical and Computer Engineering 11(4):3365–3373. https://doi.org/10.11591/ijece.v11i4.pp3365-3373
Gopalan R, Hong T, Shneier M et al (2012) A learning approach towards detection and tracking of lane markings. IEEE Trans Intell Transp Syst 13(3):1088–1098. https://doi.org/10.1109/TITS.2012.2184756
Gupta A, Choudhary A (2018) A framework for camera-based real-time lane and road surface marking detection and recognition. IEEE Transactions on Intelligent Vehicles 3(4):476–485. https://doi.org/10.1109/TIV.2018.2873902
Hillel AB, Lerner R, Levi D et al (2014) Recent progress in road and lane detection: a survey. Mach Vis Appl 25(3):727–745. https://doi.org/10.1007/s00138-011-0404-2
Howard AG, Zhu ML, Chen B, et al (2017) MobileNets: Efficient convolutional neural networks for mobile vision applications. Preprint at https://arxiv.org/abs/1704.04861
Hssayeni MD, Saxena S, Ptucha R et al (2017) (2017) Distracted driver detection: Deep learning vs handcrafted features. Electronic Imaging 10:20–26. https://doi.org/10.2352/ISSN.2470-1173.2017.10.IMAWM-162
Huang T, Zhang Q, Tang XA et al (2022) A novel fault diagnosis method based on CNN and LSTM and its application in fault diagnosis for complex systems. Artif Intell Rev 55:1289–1315. https://doi.org/10.1007/s10462-021-09993-z
Irshad A, Khan AA, Yunus I, et al (2017) Real-time lane departure warning system on a lower resource platform. In: International conference on digital image computing: techniques and applications (DICTA). IEEE, Sydney, NSW, Australia, pp 1–8. https://doi.org/10.1109/DICTA.2017.8227481
Ismaïl EH, Salah M, Zakariae A et al (2020) A novel FPGA implementation of hough transform for straight lane detection. Eng Sci Technol 23(2):274–280. https://doi.org/10.1016/j.jestch.2019.05.008
Jégou S, Drozdzal M, Vazquez D, et al (2017) The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW). IEEE, Honolulu, HI, USA, pp 1175–1183. https://doi.org/10.1109/CVPRW.2017.156
Jeong P, Nedevschi S (2005) Efficient and robust classification method using combined feature vector for lane detection. IEEE Trans Circuits Syst Video Technol 15(4):528–537. https://doi.org/10.1109/TCSVT.2005.844453
Jung S, Choi S, Khan MA, et al (2020) Towards lightweight lane detection by optimizing spatial embedding. Preprint at https://arxiv.org/abs/2008.08311
Kim J, Kim J, Jang GJ et al (2017) Fast learning method for convolutional neural networks using extreme learning machine and its application to lane detection. Neural Netw 87:109–121. https://doi.org/10.1016/j.neunet.2016.12.002
Lee S, Kim J, Shin Yoon J, et al (2017) VPGNet: Vanishing point guided network for lane and road marking detection and recognition. In: IEEE international conference on computer vision (ICCV). IEEE, Venice, Italy, pp 1965–1973. https://doi.org/10.1109/ICCV.2017.215
Li J, Mei X, Prokhorov D et al (2017) Deep neural network for structural prediction and lane detection in traffic scene. IEEE Trans Neural Netw Learn Syst 28(3):690–703. https://doi.org/10.1109/TNNLS.2016.2522428
Li JY, Jiang FL, Yang J et al (2021) Lane-DeepLab: Lane semantic segmentation in automatic driving scenarios for high-definition maps. Neurocomputing 465:15–25. https://doi.org/10.1016/j.neucom.2021.08.105
Lin Q, Han YJ, Hahn H (2010) Real-time lane departure detection based on extended edge-linking algorithm. In: International conference on computer research and development (ICCRD). IEEE, Kuala Lumpur, Malaysia, pp 725–730. https://doi.org/10.1109/ICCRD.2010.166
Liu RJ, Yuan ZJ, Liu T, et al (2021a) End-to-end lane shape prediction with transformers. In: IEEE winter conference on applications of computer vision (WACV). IEEE, Waikoloa, HI, USA, pp 3694–3702. https://doi.org/10.1109/WACV48630.2021.00374
Liu WB, Yan F, Zhang JY et al (2021) A robust lane detection model using vertical spatial features and contextual driving information. Sensors 21(3):708. https://doi.org/10.3390/s21030708
Lopez A, Canero C, Serrat J, et al (2005) Detection of lane markings based on ridgeness and RANSAC. In: IEEE intelligent transportation systems conference (ITSC). IEEE, Vienna, Austria, pp 254–259. https://doi.org/10.1109/ITSC.2005.1520139
Narote SP, Bhujbal PN, Narote AS et al (2018) A review of recent advances in lane detection and departure warning system. Pattern Recognit 73:216–234. https://doi.org/10.1016/j.patcog.2017.08.014
Neven D, De Brabandere B, Georgoulis S, et al (2018) Towards end-to-end lane detection: an instance segmentation approach. In: IEEE intelligent vehicles symposium (IV). IEEE, Changshu, China, pp 286–291. https://doi.org/10.1109/IVS.2018.8500547
Pan XG, Shi JP, Luo P, et al (2018) Spatial as deep: spatial CNN for traffic scene understanding. In: AAAI conference on artificial intelligence (AAAI). AAAI Press, New Orleans, LA, USA, pp 7276–7283
Pizzati F, Allodi M, Barrera A, et al (2019) Lane detection and classification using cascaded CNNs. In: International conference on computer aided systems theory. Springer, Las Palmas de Gran Canaria, Spain, pp 95–103. https://doi.org/10.1007/978-3-030-45096-0_12
Qiao D, Wu XY, Wang TT (2020) A lane recognition based on Line-CNN network. In: Asia-Pacific conference on image processing, electronics and computers (IPEC). IEEE, Dalian, China, pp 96–100. https://doi.org/10.1109/IPEC49694.2020.9114966
Ren KY, Hou HC, Li SY et al (2021) LaneDraw: Cascaded lane and its bifurcation detection with nested fusion. Sci China Ser E-Technol Sci 64(6):1238–1249. https://doi.org/10.1007/s11431-020-1702-2
Romera E, Alvarez JM, Bergasa LM et al (2018) ERFNet: Efficient residual factorized ConvNet for real-time semantic segmentation. IEEE Trans Intell Transp Syst 1(19):263–272. https://doi.org/10.1109/TITS.2017.2750080
Schubert R, Schulze K, Wanielik G (2010) Situation assessment for automatic lane-change maneuvers. IEEE Trans Intell Transp Syst 11(3):607–616. https://doi.org/10.1109/TITS.2010.2049353
Sivaraman S, Trivedi MM (2013) Integrated lane and vehicle detection, localization, and tracking: A synergistic approach. IEEE Trans Intell Transp Syst 14(2):906–917. https://doi.org/10.1109/TITS.2013.2246835
Song WJ, Yang Y, Fu MY et al (2018) Lane detection and classification for forward collision warning system based on stereo vision. IEEE Sens J 18(12):5151–5163. https://doi.org/10.1109/JSEN.2018.2832291
Su JM, Chen C, Zhang K, et al (2021) Structure guided lane detection. Preprint at https://arxiv.org/abs/2105.05403
Suddamalla U, Kundu S, Farkade S, et al (2015) A novel algorithm of lane detection addressing varied scenarios of curved and dashed lanemarks. In: International conference on image processing theory, tools and applications (IPTA). IEEE, Orleans, France, pp 87–92. https://doi.org/10.1109/IPTA.2015.7367103
Sun TY, Tsai SJ, Chan V (2006) HSI color model based lane-marking detection. In: IEEE intelligent transportation systems conference (ITSC). IEEE, Toronto, ON, Canada, pp 1168–1172. https://doi.org/10.1109/ITSC.2006.1707380
Tabelini L, Berriel R, Paixao TM, et al (2020) PolyLaneNet: Lane estimation via deep polynomial regression. In: International conference on pattern recognition (ICPR). IEEE, Milan, Italy, pp 6150–6156. https://doi.org/10.1109/ICPR48806.2021.9412265
Tabelini L, Berriel R, Paixao TM, et al (2021) Keep your eyes on the lane: Real-time attention-guided lane detection. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Nashville, TN, USA, pp 294–302. https://doi.org/10.1109/CVPR46437.2021.00036
Tian J, Liu SW, Zhong XY et al (2021) LSD-based adaptive lane detection and tracking for ADAS in structured road environment. Soft Comput 25(7):5709–5722. https://doi.org/10.1007/s00500-020-05566-4
TuSimple (2017) TuSimple dataset. https://github.com/TuSimple/tusimple-benchmark/wiki/
Vajak D, Vranješ M, Grbić R, et al (2019) Recent advances in vision-based lane detection solutions for automotive applications. In: International symposium ELMAR. IEEE, Zadar, Croatia, pp 45–50. https://doi.org/10.1109/ELMAR.2019.8918679
Wang H, Deng PY, Sun SL, et al (2021) Network optimization based lane line classification study. Preprint at https://doi.org/10.20944/preprints202109.0387.v1
Xiao ZW, Xu X, Xing HL et al (2021) RTFN: A robust temporal feature network for time series classification. Inf Sci 571:65–86. https://doi.org/10.1016/j.ins.2021.04.053
Yang WJ, Cheng YT, Chung PC (2019) Improved lane detection with multilevel features in branch convolutional neural networks. IEEE Access 7:173,148–173,156. https://doi.org/10.1109/ACCESS.2019.2957053
Yaseen MU, Anjum A, Fortino G et al (2022) Cloud based scalable object recognition from video streams using orientation fusion and convolutional neural networks. Pattern Recognit 121:108–207. https://doi.org/10.1016/j.patcog.2021.108207
Yoo H, Yang U, Sohn K (2013) Gradient-enhancing conversion for illumination-robust lane detection. IEEE Trans Intell Transp Syst 14(3):1083–1094. https://doi.org/10.1109/TITS.2013.2252427
Yu CQ, Wang JB, Peng C, et al (2018) BiSeNet: Bilateral segmentation network for real-time semantic segmentation. In: European conference on computer vision (ECCV). Springer, Munich, Germany, pp 325–341. https://doi.org/10.1007/978-3-030-01261-8_20
Yu CQ, Gao CX, Wang JB et al (2021) BiSeNet V2: Bilateral network with guided aggregation for real-time semantic segmentation. Int J Comput Vis 129(11):3051–3068. https://doi.org/10.1007/s11263-021-01515-2
Zare MR, Alebiosu DO, Lee SL (2018) Comparison of handcrafted features and deep learning in classification of medical X-ray images. In: International conference on information retrieval and knowledge management: diving into data sciences (CAMP). IEEE, Kota Kinabalu, Sabah, Malaysia, pp 1–5. https://doi.org/10.1109/INFRKM.2018.8464688
Zhang L, Kong B, Wang C (2021) LLNet: A lightweight lane line detection network. In: International conference on image and graphics (ICIG). Springer, Haikou, China, pp 355–369. https://doi.org/10.1007/978-3-030-87355-4_30
Zhang YH, Wang J, Wang XN et al (2018) Road-segmentation-based curb detection method for self-driving via a 3D-LiDAR sensor. IEEE Trans Intell Transp Syst 19(12):3981–3991. https://doi.org/10.1109/TITS.2018.2789462
Zhang ZC, Ma X (2019) Lane recognition algorithm using the hough transform based on complicated conditions. Journal of Computer and Communications 7(11):65–75. https://doi.org/10.4236/jcc.2019.711005
Zou Q, Jiang HW, Dai QY et al (2020) Robust lane detection from continuous driving scenes using deep neural networks. IEEE Trans Veh Technol 69(1):41–54. https://doi.org/10.1109/TVT.2019.2949603
Acknowledgements
The authors wish to thank the editors and the anonymous reviewers for their insightful comments and suggestions. This work was supported by the National Natural Science Foundation of China (Grant No. 91320301), the Anhui Provincial Key Laboratory of Multimodal Cognitive Computation (Grant No. MMC202007), and the Natural Science Foundation of Education Bureau of Anhui Province (Grant No. KJ2020A0111). This work also was supported by the Nature Science Foundation of Anhui Province, China (Grant No. 2108085MF195), and the Talent Research Foundation of Hefei University (Grant No. 20RC16). Prof. Amir Hussain would like to acknowledge the support of the UK Engineering and Physical Sciences Research Council (EPSRC)(Grants No. EP/M026981/1, EP/T021063/1, EP/T024917/1).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Human and animal rights
This article does not contain any studies with human or animal subjects performed by the any of the authors.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhang, L., Jiang, F., Yang, J. et al. DNet-CNet: a novel cascaded deep network for real-time lane detection and classification. J Ambient Intell Human Comput 14, 10745–10760 (2023). https://doi.org/10.1007/s12652-022-04346-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-022-04346-2