Abstract
We propose an interpretable neural network architecture for multi-agent deep reinforcement learning to understand the rationale for learned cooperative behavior of the agents. Although the deep learning technology has contributed significantly to multi-agent systems to build coordination among agents, it is still unclear what information the agents depend on to behave cooperatively. Removing this ambiguity may further improve the efficiency and productivity of multi-agent systems. The main idea of our proposal is to adopt the transformer to deep Q-network for addressing the above-mentioned issue. By extracting multi-head attention weights from the transformer encoder, we propose a multi-agent transformer deep Q-network (MAT-DQN) and show that agents using attention mechanisms possess better coordination capability with other agents despite being trained individually for a cooperative patrolling task problem; thus, they can exhibit better performance results compared with the agents with vanilla DQN (which is a baseline method). Furthermore, we indicate that it is possible to visualize heatmaps of attentions, which indicate the influential input-information in agents’ decision-making process for their cooperative behaviors.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amarasinghe, A., Wijesuriya, V.B., Ganepola, D., Jayaratne, L.: A swarm of crop spraying drones solution for optimising safe pesticide usage in arable lands: poster abstract. In: Proceedings of the 17th Conference on Embedded Networked Sensor Systems, SenSys 2019, pp. 410–411. ACI, USA (2019). https://doi.org/10.1145/3356250.3361948
Bathaee, Y.: The artificial intelligence black box and the failure of intent and causation. Harvard J. Law Technol. 31, 889 (2018)
Chen, H., Liu, Y., Zhou, Z., Hu, D., Zhang, M.: GAMA: graph attention multi-agent reinforcement learning algorithm for cooperation. Appl. Intel. 50 (December 2020). https://doi.org/10.1007/s10489-020-01755-8
Diallo, E.A.O., Sugiyama, A., Sugawara, T.: Coordinated behavior of cooperative agents using deep reinforcement learning. Neurocomputing 396, 230–240 (2020). https://doi.org/10.1016/j.neucom.2018.08.094
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (2021). arXiv:2010.11929
Ibrahim, S.: A comprehensive review on intelligent surveillance systems. Commun. Sci. Technol. 1 (2016). https://doi.org/10.21924/cst.1.1.2016.7
Iqbal, S., Sha, F.: Actor-attention-critic for multi-agent reinforcement learning. In: Proceedings of the 36th International Conference on Machine Learning, 09–15 Jun 2019, vol. 97, pp. 2961–2970. PMLR (2019). http://proceedings.mlr.press/v97/iqbal19a.html
Jiang, J., Lu, Z.: Learning attentional communication for multi-agent cooperation. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS 2018, pp. 7265–7275. Curran Associates Inc., USA (2018)
Miyashita, Y., Sugawara, T.: Analysis of coordinated behavior structures with multi-agent deep reinforcement learning. Appl. Intell. 51(2), 1069–1085 (2021). https://doi.org/10.1007/s10489-020-01832-y
Partel, V., Charan Kakarla, S., Ampatzidis, Y.: Development and evaluation of a low-cost and smart technology for precision weed management utilizing artificial intelligence. Comput. Electron. Agric. 157, 339–350 (2019). https://doi.org/10.1016/j.compag.2018.12.048
Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming, 1st edn. Wiley, USA (1994)
Sreenu, G., Durai, M.A.: Intelligent video surveillance: a review through deep learning techniques for crowd analysis. J. Big Data 6, 48 (2019). https://doi.org/10.1186/s40537-019-0212-5
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30, pp. 5998–6008. Curran Associates, Inc. (2017)
Acknowledgements
This work was partly supported by JSPS KAKENHI Grant Numbers 17KT0044 and 20H04245.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Motokawa, Y., Sugawara, T. (2021). MAT-DQN: Toward Interpretable Multi-agent Deep Reinforcement Learning for Coordinated Activities. In: Farkaš, I., Masulli, P., Otte, S., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2021. ICANN 2021. Lecture Notes in Computer Science(), vol 12894. Springer, Cham. https://doi.org/10.1007/978-3-030-86380-7_45
Download citation
DOI: https://doi.org/10.1007/978-3-030-86380-7_45
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86379-1
Online ISBN: 978-3-030-86380-7
eBook Packages: Computer ScienceComputer Science (R0)