Skip to main content
SpringerLink
Log in

Evaluating Action Durations for Adaptive Traffic Signal Control Based On Deep Q-Learning

  • Published:
International Journal of Intelligent Transportation Systems Research Aims and scope Submit manuscript

Abstract

Adaptive traffic signal control is the control technique that adjusts the signal times according to traffic conditions and manages the traffic flow. Reinforcement learning is one of the best algorithms used for adaptive traffic signal controllers. Despite many successful studies about Reinforcement Learning based traffic control, there remains uncertainty about what the best actions to actualize adaptive traffic signal control. This paper seeks to understand the performance differences in different action durations for adaptive traffic management. Deep Q-Learning has been applied to a traffic environment for adaptive learning. This study evaluates five different action durations. Also, this study proposes a novel approach to the Deep Q-Learning based adaptive traffic control system for determine the best action. Our approach does not just aim to minimize delay time by waiting time during the red-light signal also aims to decrease delay time caused by vehicles slowing down when approaching the intersection and caused by the required time to accelerate after the green light signal. Thus the proposed strategy uses not just information of intersection also uses the data of adjacent intersection as an input. The performances of these methods are evaluated in real-time through the Simulation of Urban Mobility traffic simulator. The output of this paper indicate that the short action times increase the traffic control system performances despite more yellow signal duration. The results clearly shows that proposed method decreases the delay time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Ge, H., Song, Y., Wu, C., Ren, J., Tan, G.: Cooperative deep Q-learning with Q-value transfer for multi-intersection signal control. IEEE Access 7, 40797–40809 (2019)

    Article  Google Scholar 

  2. Spall, J.C., Chin, D.C.: Traffic-responsive signal timing for system-wide traffic control. Transportation Research Part C: Emerging Technologies 5(3–4), 153–163 (1997)

    Article  Google Scholar 

  3. McCrea, J., Moutari, S.: A hybrid macroscopic-based model for traffic flow in road networks. Eur. J. Oper. Res. 207(2), 676–684 (2010)

    Article  MathSciNet  Google Scholar 

  4. Ye, B.-L., Wu, W., Gao, H., Lu, Y., Cao, Q., Zhu, L.: Stochastic model predictive control for urban traffic networks. Appl. Sci. 7(6), 588 (2017)

    Article  Google Scholar 

  5. Bull, A., Thomson, I.: "Urban traffic congestion: its economic and social causes and consequences," Cepal Review (2002)

  6. Garcia-Nieto, J., Olivera, A.C., Alba, E.: Optimal cycle program of traffic lights with particle swarm optimization. IEEE Trans. Evol. Comput. 17(6), 823–839 (2013)

    Article  Google Scholar 

  7. Zhou, P., Fang, Z., Dong, H., Liu, J., & Pan, S.: "Data analysis with multi-objective optimization algorithm: A study in smart traffic signal system," in 2017 IEEE 15th International Conference on Software Engineering Research, Management and Applications (SERA),Int. J. Softw. Eng. its Appl. (SERA), pp. 307-310. (2017)

  8. HCM, "Highway capacity manual," in "Washington, DC," vol. 2. (2002)

  9. Araghi, S., Khosravi, A., Creighton, D.: A review on computational intelligence methods for controlling traffic signal timing. Expert Syst. Appl. 42(3), 1538–1550 (2015)

    Article  Google Scholar 

  10. Webster, F., "Traffic signal settings, road research technical paper no. 39," Road Research Laboratory, (1958)

  11. Wei, H., Zheng, G., Gayah, V., Li, Z.: "A survey on traffic signal control methods," arXiv preprint, 1904.08117 (2019)

  12. Hunt, P., Robertson, D., Bretherton, R., Royle, M. C.: "The SCOOT on-line traffic signal optimisation technique," Traffic Engineering & Control 23(4) (1982)

  13. Lowrie, P.: "SCATS–A Traffic Responsive Method of Controlling Urban Traffic. Roads and Traffic Authority, Sydney," New South Wales, Australia (1990)

  14. Roess, R.P., Prassas, E. S., McShane W. R.: Traffic Engineering. Pearson/Prentice Hall, (2004)

  15. Zhao, D., Dai, Y., Zhang, Z.: “Computational intelligence in urban traffic signal control: A survey,” IEEE Transactions on Systems, Man, and Cybernetics. Part C (Applications and Reviews) 42(4), 485–494 (2012)

    Google Scholar 

  16. Ali, M. E. M., Durdu, A., Çeltek, S. A., Gültekin, S. S.: "Fuzzy logic and webster's optimal cycle based decentralized coordinated adaptive traffic control method," (2020)

  17. Joo H., Ahmed, S. H., Lim, Y.: "Traffic signal control for smart cities using reinforcement learning," Comput. Commun. (2020)

  18. Wang, H., Sun, H., Li, C., Rahnamayan, S., Pan, J.-S.: Diversity enhanced particle swarm optimization with neighborhood search. Inf. Sci. 223, 119–135 (2013)

    Article  MathSciNet  Google Scholar 

  19. Dimitriou, L., Tsekeris, T., Stathopoulos, A.: Adaptive hybrid fuzzy rule-based system approach for modeling and predicting urban traffic flow. Transportation Research Part C: Emerging Technologies 16(5), 554–573 (2008)

    Article  Google Scholar 

  20. Jovanović, A., Nikolić, M., Teodorović, D.: Area-wide urban traffic control: A Bee Colony Optimization approach. Transportation Research Part C: Emerging Technologies 77, 329–350 (2017)

    Article  Google Scholar 

  21. De Oliveira, M.B. de Almeida Neto, A.: "Optimization of traffic lights timing based on Artificial Neural Networks," in 17th International IEEE Conference on Intelligent Transportation Systems (ITSC), IEEE, pp. 1921–1922 (2014)

  22. Celtek, S. A., Durdu, A., Alı, M. E. M.: "Real-time Traffic Signal Control with Swarm Optimization Methods," Measurement (2020)

  23. El-Tantawy, S., Abdulhai, B., Abdelgawad, H.: Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (MARLIN-ATSC): methodology and large-scale application on downtown Toronto. IEEE Trans. Intell. Transp. Syst. 14(3), 1140–1150 (2013)

    Article  Google Scholar 

  24. Abdoos, M.: "Fuzzy Graph and Collective Multi-Agent Reinforcement Learning for Traffic Signals Control," IEEE Intelligent Systems, (2020)

  25. Wiering, M.A.: "Multi-agent reinforcement learning for traffic light control," in Machine Learning: Proceedings of the Seventeenth International Conference (ICML'2000) pp. 1151–1158 (2000)

  26. Abdoos, M., Mozayani, N., Bazzan, A. L.: "Traffic light control in non-stationary environments based on multi agent Q-learning," in 2011 14th International IEEE conference on intelligent transportation systems (ITSC) IEEE, pp. 1580–1585 (2011)

  27. Chin, Y. K., Kow, W. Y., Khong, W. L., Tan, M. K., Teo, K. T. K.: "Q-learning traffic signal optimization within multiple intersections traffic network," in 2012 Sixth UKSim/AMSS European Symposium on Computer Modeling and Simulation IEEE, pp. 343–348 (2012)

  28. Genders, W., Razavi, S.: Asynchronous n-step q-learning adaptive traffic signal control. Journal of Intelligent Transportation Systems 23(4), 319–331 (2019)

    Article  Google Scholar 

  29. Liang, X., Du, X., Wang, G., Han, Z.: A deep reinforcement learning network for traffic light cycle control. IEEE Trans. Veh. Technol. 68(2), 1243–1253 (2019)

    Article  Google Scholar 

  30. Tan, T., Bao, F., Deng, Y., Jin, A., Dai, Q., Wang, J.: Cooperative deep reinforcement learning for large-scale traffic grid signal control. IEEE transactions on cybernetics 50(6), 2687–2700 (2019)

    Article  Google Scholar 

  31. Haydari, A., Yilmaz, Y.: "Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey (2020)

  32. Aslani, M., Seipel, S., Mesgari, M.S., Wiering, M.: Traffic signal optimization through discrete and continuous reinforcement learning with robustness analysis in downtown Tehran. Adv. Eng. Inform. 38, 639–655 (2018)

    Article  Google Scholar 

  33. Aslani, M., Mesgari, M.S., Wiering, M.: Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption events. Transportation Research Part C: Emerging Technologies 85, 732–752 (2017)

    Article  Google Scholar 

  34. Xu, L. H., Xia, X. H. Luo, Q.: "The study of reinforcement learning for traffic self-adaptive control under multiagent markov game environment," Mathematical Problems in Engineering, vol. 2013 (2013)

  35. Gokulan, B.P., Srinivasan, D.: Distributed geometric fuzzy multiagent urban traffic signal control. IEEE Trans. Intell. Transp. Syst. 11(3), 714–727 (2010)

    Article  Google Scholar 

  36. Chin, Y. K., Lee, L. K., Bolong, N., Yang, S. S., & Teo, K. T. K.: "Exploring Q-learning optimization in traffic signal timing plan management," in 2011 third international conference on computational intelligence, communication systems and networks, IEEE, pp. 269–274 (2011)

  37. Wei, H., Zheng, G., Yao, H., & Li, Z.: "Intellilight: A reinforcement learning approach for intelligent traffic light control," in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2496–2505 (2018)

  38. Mannion, P., Duggan, J., Howley, E.: "An experimental review of reinforcement learning algorithms for adaptive traffic signal control," in Autonomic road transport support systems: Springer, pp. 47–66 (2016)

  39. Brys, T., Pham, T.T., Taylor, M.E.: Distributed learning and multi-objectivity in traffic light control. Connect. Sci. 26(1), 65–83 (2014)

    Article  Google Scholar 

  40. Arel, I., Liu, C., Urbanik, T., Kohls, A.G.: Reinforcement learning-based multi-agent system for network traffic signal control. IET Intel. Transport Syst. 4(2), 128–135 (2010)

    Article  Google Scholar 

  41. Chu, T., Wang, J., Codecà, L., Li, Z.: Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Trans. Intell. Transp. Syst. 21(3), 1086–1095 (2019)

    Article  Google Scholar 

  42. Khamis, M. A., Gomaa, W., El-Mahdy, A., & Shoukry, A.: "Adaptive traffic control system based on Bayesian probability interpretation," IEEE, pp. 151–156. (2012)

  43. Choe, C. J., Baek, S., Woon, B., & Kong, S. H.: "Deep Q Learning with LSTM for Traffic Light Control," in 2018 24th Asia-Pacific Conference on Communications (APCC), IEEE, pp. 331–336 (2018)

  44. Sutton R. S., Barto, A. G.: Reinforcement learning: An introduction. MIT press, (2018)

  45. Jang, B., Kim, M., Harerimana, G., Kim, J.W.: Q-learning algorithms: A comprehensive classification and applications. IEEE Access 7, 133653–133667 (2019)

    Article  Google Scholar 

  46. Dion, F., Rakha, H., Kang, Y.-S.: Comparison of delay estimates at under-saturated and over-saturated pre-timed signalized intersections. Transportation Research Part B: Methodological 38(2), 99–122 (2004)

    Article  Google Scholar 

  47. Yau, K.-L.A., Qadir, J., Khoo, H.L., Ling, M.H., Komisarczuk, P.: A survey on reinforcement learning models and algorithms for traffic signal control. ACM Computing Surveys (CSUR) 50(3), 1–38 (2017)

    Article  Google Scholar 

  48. Krajzewicz, D., Hertkorn, G. Rössel, C. Wagner, P.: "SUMO (Simulation of Urban MObility)-an open-source traffic simulation," in Proceedings of the 4th middle East Symposium on Simulation and Modelling (MESM20002), pp. 183–187 (2002)

  49. Kotusevski, G., Hawick, K.: "A review of traffic simulation software," (2009)

  50. Vidali, A. Crociani, L., Vizzari, G., Bandini, S.: "A Deep Reinforcement Learning Approach to Adaptive Traffic Lights Management," in WOA, pp. 42–50. (2012)

  51. Rinne, R.: The Weibull distribution: a handbook. CRC press, (2008)

  52. Celtek, S. A., The dataset for ATSC-DQ, 15.04.2021, https://github.com/salperenceltek/Evaluating-Action-Durations-for-ATSC-DQ

Download references

Acknowledgements

Authors are thankful to RAC-LAB (www.rac-lab.com) for providing the computer for this study.

Author information

Authors and Affiliations

  1. Department of Energy Systems Engineering, KaramanogluMehmetbey University, Karaman, Turkey

    Seyit Alperen CELTEK

  2. Robotics Automation Control Laboratory (RAC-LAB), Konya Technical University, Konya, Turkey

    Akif DURDU

  3. Robotics Automation Control Laboratory (RAC-LAB), Selcuk University, Konya, Turkey

    Muzamil Eltejani Mohammed ALI

Authors
  1. Seyit Alperen CELTEK
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Akif DURDU
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Muzamil Eltejani Mohammed ALI
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seyit Alperen CELTEK.

Ethics declarations

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

CELTEK, S.A., DURDU, A. & ALI, M.E.M. Evaluating Action Durations for Adaptive Traffic Signal Control Based On Deep Q-Learning. Int. J. ITS Res. 19, 557–571 (2021). https://doi.org/10.1007/s13177-021-00262-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13177-021-00262-5

Keywords

Search

Navigation