Abstract
This paper describes the application of a decentralised coordination algorithm, called Collaborative Reinforcement Learning (CRL), to two different distributed system problems. CRL enables the establishment of consensus between independent agents to support the optimisation of system-wide properties in distributed systems where there is no support for global state. Consensus between interacting agents on local environmental or system properties is established through localised advertisement of policy information by agents and the use of advertisements by agents to update their local, partial view of the system.
As CRL assumes homogeneity in advertisement evaluation by agents, advertisements that improve the system optimisation problem tend to be propagated quickly through the system, enabling the system to collectively adapt its behaviour to a changing environment. In this paper, we describe the application of CRL to two different distributed system problems, a routing protocol for ad-hoc networks called SAMPLE and a next generation urban traffic control system called UTC-CRL. We evaluate CRL experimentally in SAMPLE by comparing its system routing performance in the presence of changing environmental conditions, such as congestion and link unreliability, with existing ad-hoc routing protocols. Through SAMPLE’s ability to establish consensus between routing agents on stable routes, even in the presence of changing levels of congestion in a network, it demonstrates improved performance and self-management properties. In applying CRL to the UTC scenario, we hope to validate experimentally the appropriateness of CRL to another system optimisation problem.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Montresor, A., Meling, H., Babaoglu, O.: Towards self-organizing, self-repairing and resilient distributed systems. In: Schiper, A., Shvartsman, M.M.A.A., Weatherspoon, H., Zhao, B.Y. (eds.) Future Directions in Distributed Computing. LNCS, vol. 2584, pp. 119–123. Springer, Heidelberg (2003)
Visscher, P.: How self-organization evolves. Nature 421, 799–800 (2003)
Camazine, S., Deneubourg, J., Franks, N., Sneyd, J., Theraulaz, G., Bonabeau, E.: Self-Organization in Biological Systems. Princeton University Press, Princeton (2003)
Gelernter, D., Carriero, N.: Coordination languages and their significance. Commun. ACM 35(2), 97–107 (1992)
Goldin, D., Keil, K.: Toward domain-independent formalization of indirect interaction. In: 2nd Int’l workshop on Theory and Practice of Open Computational Systems, TAPOCS (2004)
Efstratiou, C., Friday, A., Davies, N., Cheverst, K.: Utilising the event calculus for policy driven adaptation in mobile systems. In: Proceedings of the 3rd International Workshop on Policies for Distributed Systems and Networks, June 13-24. IEEE Computer Society, Los Alamitos (2002)
Dorigo, M., Di Caro, G.: The ant colony optimization meta-heuristic. New Ideas in Optimization (1999)
Andrzejak, A., Graupner, S., Kotov, V., Trinks, H.: Adaptive control overlay for service management. In: Workshop on the Design of Self-Managing Systems. International Conference on Dependable Systems and Networks (2003)
De Wolf, T., Holvoet, T.: Towards autonomic computing: agent-based modelling, dynamical systems analysis, and decentralised control. In: Proceedings of IEEE International Conference on Industrial Informatics, pp. 470–479 (2003)
Boutilier, C., Das, R., Kephart, J., Tesauro, G., Walsh, W.: Cooperative negotiation in autonomic systems using incremental utility elicitation. Uncertainty in Artificial Intelligence (2003)
Khare, R., Taylor, R.N.: Extending the representational state transfer (rest) architectural style for decentralized systems. In: Proceedings of the International Conference on Software Engineering, ICSE (2004)
Curran, E., Dowling, J.: Sample: An on-demand probabilistic routing protocol for ad-hoc networks. Technical Report Department of Computer Science Trinity College Dublin (2004)
Jelasity, M., Montresor, A., Babaoglu, O.: A modular paradigm for building self-organizing peer-to-peer applications. In: Proceedings of ESOP 2003 International Workshop on Engineering Self-Organising Applications (2003)
Panagiotis, T., Demosthenis, T., Mackie-Mason, J.-K.: A market-based approach to optimal resource allocation in integrated-services connection-oriented networks. Operations Research 50(4) (July-August 2002)
Littman, M., Boyan, J.: A distributed reinforcement learning scheme for network routing. Technical Report CS-93-165 (1993)
Di Caro, G., Dorigo, M.: AntNet: Distributed Stigmergetic Control for Communications Networks. Journal of Artificial Intelligence Research 9, 317–365 (1998)
Sutton, R., Barto, A.: Reinforcement Learning. MIT Press, Cambridge (1998)
Bonabeau, E., Dorigo, M., Theraulaz, G.: Swarm Intelligence: from natural to artificial systems. Oxford University Press, New York (1999)
Crites, R., Barto, A.: Elevator group control using multiple reinforcement learning agents. Machine Learning 33(2-3), 235–262 (1998)
Kaelbling, L., Littman, M., Moore, A.: Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4, 237–285 (1996)
Perkins, C.: Ad Hoc on Demand Distance Vector (AODV) Routing. IETF Internet Draft (November 1997)
Johnson, D., Maltz, D., Broch, J.: DSR: The dynamic source routing protocol for multihop wireless ad hoc networks. In: Ad Hoc Networking, pp. 139–172. Addison-Wesley, Reading (2001)
NS-2 network simulator. Information Sciences Institute (2003)
Broch, J., Maltz, D., Johnson, D., Hu, J., Jetcheva, J.: A Performance Comparison of Multi-Hop Wireless Ad Hoc Network Routing Protocols. In: Mobile Computing and Networking 85–97 (1998)
Li, J., Blake, C., De Couto, D., Lee, H., Morris, R.: Capacity of ad hoc wireless networks. In: Proceedings of the 7th International Conference on Mobile Computing and Networking, pp. 61–69 (2001)
Klein, L.: Sensor Technologies and Data Requirements for ITS. Artech House (2001)
Hoar, R., Penner, J., Jacob, C.: Evolutionary Swarm Traffic: If Ant Roads had Traffic Lights. In: Proceedings of the IEEE Conference on Evolutionary Computation Honolulu Hawaii 1910–1916 (2002)
Abdulhai, B., Pringle, R., Karakoulas, G.: Reinforcement Learning for True Adaptive Traffic Signal Control. Transportation Engineering 129 (May 2003)
Findler, N.: Harmonization for Omnidirectional Progression in Urban Traffic Control. In: Computer-Aided Civil and Infrastructure Engineering, Honolulu Hawaii, vol. 14, pp. 369–377 (1999)
Pendrith, M.: Distributed Reinforcement Learning for a Traffic Engineering Application. In: Proceedings of the Fourth Internation Conference on Autonomous Agents, Barcelona, Spain (2000)
Dublin Transportation Office: DTO Strategy Update - Full Report - Platform For Change (2001), Available on: http://www.dto.ie/strategy.htm
Guestrin, C., Lagoudakis, M., Parr, R.: Coordinated reinforcement learning. In: Proceedings of The Nineteenth International Conference on Machine Learning, pp. 227–234 (2002)
Schneider, J., Wong, W., Moore, A., Riedmiller, M.: Distributed value functions. In: Proceedings of the Sixteenth International Conference on Machine Learning, pp. 371–378. Morgan Kaufmann Publishers, San Francisco (1999)
Stone, P.: TPOT-RL applied to network routing. In: Proceedings of the Seventeenth International Conference on Machine Learning (2000)
Mariano, C., Morales, E.: A new distributed reinforcement learning algorithm for multiple objective optimization problems. In: Monard, M.C., Sichman, J.S. (eds.) SBIA 2000 and IBERAMIA 2000. LNCS (LNAI), vol. 1952, p. 290. Springer, Heidelberg (2000)
Yagar, S., Dion, F.: Distributed Approach to Real-Time Control of Complex Signalized Networks. Transportation Research Record 1, 1–8 (1996)
Hunt, P., Robertson, R., Winton, R., Bretherton, R.: SCOOT- A Traffic Responsive Method of Coordinating Signals. Road Research Laboratory, TRRL Report 1014 (1981)
Sims, A.: The Sydney Coordinated Adaptive Traffic System. In: Proceedings of the ASCE Engineering Foundations Conference on Research Priorities in Computer Control of Urban Traffic Systems (1979)
Lo, H., Chow, A.: Control Strategies for Oversaturated Traffic. Transportation Engineering 130 (July 2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Dowling, J., Cunningham, R., Harrington, A., Curran, E., Cahill, V. (2005). Emergent Consensus in Decentralised Systems Using Collaborative Reinforcement Learning. In: Babaoglu, O., et al. Self-star Properties in Complex Information Systems. SELF-STAR 2004. Lecture Notes in Computer Science, vol 3460. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11428589_5
Download citation
DOI: https://doi.org/10.1007/11428589_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-26009-7
Online ISBN: 978-3-540-32013-5
eBook Packages: Computer ScienceComputer Science (R0)