Skip to main content
SpringerLink
Log in

End-to-end deep learning of lane detection and path prediction for real-time autonomous driving

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Inspired by the UNet architecture of semantic image segmentation, we propose a lightweight UNet using depthwise separable convolutions (DSUNet) for end-to-end learning of lane detection and path prediction (PP) in autonomous driving. We also design and integrate a PP algorithm with convolutional neural network (CNN) to form a simulation model (CNN-PP) that can be used to assess CNN’s performance qualitatively, quantitatively, and dynamically in a host agent car driving along with other agents all in a real-time autonomous manner. DSUNet is 5.12\(\times \) lighter in model size and 1.61\(\times \) faster in inference than UNet. DSUNet-PP outperforms UNet-PP in mean average errors of predicted curvature and lateral offset for path planning in dynamic simulation. DSUNet-PP outperforms a modified UNet in lateral error, which is tested in a real car on real road. These results show that DSUNet is efficient and effective for lane detection and path prediction in autonomous driving.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Huval, B., et al.: An empirical evaluation of deep learning on highway driving. arXiv:1504.01716 (2015)

  2. Bruls, T., Maddern, W., Morye, A.A., Newman, P.: Mark yourself: road marking segmentation via weakly-supervised annotations from multimodal data. In: IEEE International Conference on Robotics and Automation, pp. 1863–1870 (2018)

  3. Garnett, N., et al.: 3D-LaneNet: end-to-end 3D multiple lane detection. In: Proceedings of IEEE International Conference on Computer Vision, pp. 2921–2930 (2019)

  4. Kim, W., et al.: Vehicle path prediction using yaw acceleration for adaptive cruise control. IEEE Trans. Intell. Transp. Syst. 19, 3818–3829 (2018)

    Article  Google Scholar 

  5. Lin, C.F., Ulsoy, A.G., LeBlanc, D.J.: Vehicle dynamics and external disturbance estimation for vehicle path prediction. IEEE Trans. Control Syst. Technol. 8, 508–518 (2000)

    Article  Google Scholar 

  6. Cudrano, P., et al.: Advances in centerline estimation for autonomous lateral control. In: Proceedings of IEEE Intelligent Vehicles Symposium, pp. 1415–1422 (2020)

  7. Badue, C., et al.: Self-driving cars: a survey. Expert Syst. Appl. 165, 113816 (2021)

    Article  Google Scholar 

  8. Li, X., Li, J., Hu, X., Yang, J.: Line-CNN: end-to-end traffic line detection with line proposal unit. IEEE Trans. Intell. Transp. Syst. 21, 248–258 (2020)

    Article  Google Scholar 

  9. Zou, Q., et al.: Robust lane detection from continuous driving scenes using deep neural networks. IEEE Trans. Veh. Technol. 69, 41–54 (2020)

    Article  Google Scholar 

  10. Lu, P., et al.: SUPER: a novel lane detection system. IEEE Trans. Intell. Veh. 6, 583–593 (2021)

    Article  Google Scholar 

  11. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241 (2015)

  12. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006)

    Article  MATH  Google Scholar 

  13. Liu, L., et al.: A survey on U-shaped networks in medical image segmentations. Neurocomputing 409, 244–258 (2020)

    Article  Google Scholar 

  14. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

  15. Iandola, F.N., et al.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and \(<\)0.5MB model size. arXiv:1602.07360 (2016)

  16. Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)

  17. Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861 (2017)

  18. Zhang, X., Lin, M., Sun, J.: ShuffleNet: an extremely efficient convolutional neural network for mobile devices. arXiv:1707.01083 (2017)

  19. Neven, D., et al.: Towards end-to-end lane detection: an instance segmentation approach. In: Intelligent Vehicles Symposium Proceedings, pp. 286–291 (2018)

  20. Santos, A.G., et al.: Reducing SqueezeNet storage size with depthwise separable convolutions. In: International Joint Conference on Neural Networks, pp. 1–6 (2018)

  21. Hou, Y., Ma, Z., Liu, C., Loy, C.C.: Learning lightweight lane detection CNNs by self attention distillation. In: Proceedings of IEEE International Conference on Computer Vision, pp. 1013–1021 (2019)

  22. Hussain, R., Zeadally, S.: Autonomous cars: research results, issues, and future challenges. IEEE Commun. Surv. Tutor. 21, 1275–1313 (2019)

    Article  Google Scholar 

  23. Beheshti, N., Johnsson, L.: Squeeze U-Net: a memory and energy efficient image segmentation network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1495–1504 (2020)

  24. Gadosey, P.K., et al.: SD-Unet: stripping down U-net for segmentation of biomedical images on platforms with low computational budgets. Diagnostics 10, 110 (2020)

    Article  Google Scholar 

  25. Grigorescu, S., Trasnea, B., Cocias, T., Macesanu, G.: A survey of deep learning techniques for autonomous driving. J. Field Robot. 37, 362–386 (2020)

    Article  Google Scholar 

  26. Liu, R., Jiang, D., Zhang, L., Zhang, Z.: Deep depthwise separable convolutional network for change detection in optical aerial images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 13, 1109–1118 (2020)

    Article  Google Scholar 

  27. Lee, D.H., et al.: Deep learning and control algorithms of direct perception for autonomous driving. Appl. Intell. 51, 237–247 (2021)

    Article  Google Scholar 

  28. Tabelini, L., et al.: Keep your eyes on the lane: real-time attention-guided lane detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 294–302 (2021)

  29. Chen, C., Seff, A., Kornhauser, A., Xiao, J.: DeepDriving: learning affordance for direct perception in autonomous driving. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2722–2730 (2015)

  30. Wymann, B., et al.: TORCS: the open racing car simulator (2000)

  31. Wang, X., et al.: ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3462–3471 (2017)

  32. Santana, E., Hotz, G.: Learning a driving simulator. arXiv:1608.01230 (2016)

  33. Bojarski, M., et al.: End to end learning for self-driving cars. arXiv:1604.07316 (2016)

  34. Behrendt, K., Soussan, R.: Unsupervised labeled lane markers using maps. In: Proceedings of IEEE International Conference on Computer Vision Workshops, pp. 832–839 (2019)

  35. TuSimple Competitions for CVPR2017. https://github.com/TuSimple/tusimple-benchmark. Accessed 11 Aug 2021

  36. Welch, G., Bishop, G.: An Introduction to the Kalman Filter. Univ. North Carolina, Chapel Hill (2001)

    Google Scholar 

  37. Fitzpatrick, K.: Horizontal curve design: an exercise in comfort and appearance. Transp. Res. Rec. 1445, 47–53 (1994)

    Google Scholar 

  38. Lee, D.H., Liu, J.L.: End-to-end multi-task deep learning and model based control algorithm for autonomous driving. arXiv:2112.08967 (2021)

Download references

Acknowledgements

This work was supported by the Ministry of Science and Technology, Taiwan, under Grant MOST 109-2115-M-007-011-MY2.

Author information

Authors and Affiliations

  1. Department of Electrophysics, National Chiao Tung University, 1001 University Road, Hsinchu, 300, Taiwan, ROC

    Der-Hau Lee

  2. Institute of Computational and Modeling Science, National Tsing Hau University, 101 Section 2, Guangfu Road, Hsinchu, 300, Taiwan, ROC

    Jinn-Liang Liu

Authors
  1. Der-Hau Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jinn-Liang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinn-Liang Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, DH., Liu, JL. End-to-end deep learning of lane detection and path prediction for real-time autonomous driving. SIViP 17, 199–205 (2023). https://doi.org/10.1007/s11760-022-02222-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02222-2

Keywords

Search

Navigation