Managing Large Computer Systems with Change Objects

Abstract

The management of large, complex computer systems is unimaginable without automation tools which help setting up and keeping the system in the desired state. We discuss the design and implementation of a distributed automated management system able to manage the complex state of large Linux installations. In the context of autonomic computing our approach belongs into the class of self-configuring systems. Popular self-configuration systems like CFEngine, Puppet, Chef, etc. focus on describing the configuration of each of the servers in a large computer system by a set of high-level policies called promises, manifests or recipes, and implement them in some mostly well-defined order. We go beyond the server focused approach of these tools and implement mechanisms for handling complex dependencies between managed components that cross server boundaries. The targeted system configuration and system state is described by a global, distributed graph of change objects (COBs) and their dependencies. Autonomous cobd agents running on each server are cooperating and driving the system towards the desired distributed state. The system can deal with reverting changes and correctly implementing a new generation of COBs where managed components are added or removed from the system, without needing to reinstall the servers. We apply the automated system to the configuration and management of a cluster of highly available file and storage servers delivering the Lustre parallel file system.