Abstract
In Multiagent Reinforcement Learning (MARL), a single scalar reinforcement signal is the sole reliable feedback that members of a team of learning agents can receive from the environment around them. Hence, the distribution of the environmental feedback signal among learning agents, also known as the “Multiagent Credit Assignment” (MCA), is among the most challenging problems in MARL.
In this paper, the authors propose an extended solution to the problem of MCA. In the proposed method, called “Trust-based Multiagent Credit Assignment” (TMCA), a trust and reputation based model is utilized to evaluate the trustworthiness of the learning agents. Unlike the existing methods, TMCA not only qualifies to benefit from the knowledge and expertise of the sole target agent (the agent for which the credit is being evaluated), but also from the knowledge and expertise of the whole as a team.
To evaluate this method, the effect of different task types (e.g. AND vs. OR) are studied. Our simulations show the superiority of the proposed method in comparison to the prior investigated methods even in noisy environments, despite a reduction (caused by the noise) in the performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Wooldridge, M.: An Introduction to MultiAgent Systems. Wiley, Hoboken (2009)
Ahmadabadi, M.N., Imanipour, A., Araabi, B.N., Asadpour, M., Siegwart, R.: Knowledge-based extraction of area of expertise for cooperation in learning. In: Proceedings of 2006 IEEE/RSJ, International Conference on Intelligent Robots and Systems, p. 915 (2006)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Watkins, C.J.C.H., Dayan, P.: Q-learning. Mach. Learn. 8(3–4), 279–292 (1992)
Harati, A., Ahmadabadi, M.N., Araabi, B.N.: Knowledge-based multi-agent credit assignment: a study on task type and critic information. IEEE Syst. J. 1(1), 55–67 (2007)
Rahaie, Z., Beigy, H.: Expertness framework in multi-agent systems and its application in credit assignment problem. Intell. Data Anal. 18(3), 511–528 (2014)
Patel, J.A.: Trust and reputation model for agent-based virtual organisations. Ph.D. thesis, Faculty of Engineering and Applied Science, School of Electronics and Computer Science, University of Southampton (2007)
Huynh, T.D.: Trust and reputation in open multi-agent systems. Ph.D thesis, Faculty of Engineering and Applied Science, School of Electronics and Computer Science, University of Southampton (2006)
Sabater, J., Sierra, C.: Regret: reputation in gregarious societies. In: Proceedings of the Fifth International Conference on Autonomous Agents. ACM (2001)
Zhong, Y., Gu, G., Zhang, R.: A new approach for structural credit assignment in distributed reinforcement learning systems. In: ICRA, pp. 1215–1220 (2003)
Holland, J.H.: Escaping brittleness: the possibility of general-purpose learning algorithms applied to rule-based systems. In: Michalski, R.S., Carbonell, J.G., Mitchells, T.M. (eds.) Machine Learning: An Artificial Intelligence Approach, vol. 2. Morgan Kaufmann, San Mateo (1986)
Kinny, D.N., Georgeff, M.P.: Commitment and effectiveness of situated agents. In: Proceedings of the 12th International Joint Conference on Artificial Intelligence, pp. 82–88 (1991)
Irwig, K., Wobcke, W.: Multi-agent reinforcement learning with vicarious rewards. Linkping Electron. Art. Comput. Inf. Sci. 4(34) (1999)
Raghavan, S., Raghavan, S.V.: Learning the point gathering task using shared value function in mobile robots. In: International Conference on Advanced Computer Control, pp. 9–13 (2009)
Rahaie Z., Beigy, H.: Toward a solution to multi-agent credit assignment problem. In: Proceedings of the International Conference on Soft Computing and Pattern Recognition (SOCPAR 2009), Malacca, Malaysia, pp. 563–568, December 2009
Abbasi, M.A., Abbasi, Z.: Reinforcement distribution in a team of cooperative Q-learning agents. In: The Ninth ACIS International Conference on Software Engineering, IEEE Transactions on Artificial Intelligence, Networking, and Parallel/Distributed Computing (2008)
Harati, A., Ahmadabadi, M.N.: Certainty and expertness based credit assignment for cooperative Q-learning agents with and type task. In: Proceedings of the 9th International Conference on Neural Information Processing, ICONIP 2002, vol. 1, pp. 306–310 (2002)
Zaliamin, R., Shiri, M.E.: Certainty and expertness based credit assignment for cooperative Q-learning agents with an and type task. Aust. J. Basic Appl. Sci. 2, 1047–1054 (2010)
Marsh, S.: Formalising trust as a computational concept. Ph.D. thesis, University of Stirling, UK, pp. 271–293, March 1994
Schillo, M., Rovatsos, M., Funk, P.: Using trust for detecting deceitful agents in artificial societies. Appl. Artif. Intell. J. 14, 825–848 (2000). Special Issue of the Deception, Fraud and Trust in Agent Societies
Harati, A., Ahmadabadi, M.N.: Experimental analysis of knowledge based multiagent credit assign. In: Rajapakse, J.C., Wang, L. (eds.) Neural Information Processing: Research and Development. Studies in Fuzziness and Soft Computing, vol. 152, pp. 437–459. Springer, Heidelberg (2004)
Zacharia, G., Moukas, A., Maes, P.: Collaborative reputation mechanisms for electronic marketplaces. Decis. Support Syst. 29, 371–388 (2000)
Fung, G.: A comprehensive overview of basic clustering algorithms (2001)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Nazari, S., Shiri, M.E. (2016). Trust-Based Multiagent Credit Assignment (TMCA). In: Rovatsos, M., Vouros, G., Julian, V. (eds) Multi-Agent Systems and Agreement Technologies. EUMAS AT 2015 2015. Lecture Notes in Computer Science(), vol 9571. Springer, Cham. https://doi.org/10.1007/978-3-319-33509-4_26
Download citation
DOI: https://doi.org/10.1007/978-3-319-33509-4_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-33508-7
Online ISBN: 978-3-319-33509-4
eBook Packages: Computer ScienceComputer Science (R0)