Fault-tolerant Distributed Computing (Lecture Notes in Computer Science)

Lecture Notes in Computer Science Edited by G. Goos and J. Hartmanis

448 B. Simons A. Spector (Eds.)

Fault-Tolerant Distributed Computing

Springer-Verlag New York Berlin Heidelberg London Paris Tokyo Hong Kong Barcelona

Editorial Board

D. Barstow W, Brauer R Brinch Hansen D. Gries D. Luckham C. Moler A. Pnueti G. Seegm~iller J. Stoer N. Wirth Editors

Barbara Simons IBM Almaden Research Center, Dept. K53/802 650 Harry Road, San .lose, CA 95120-6099, USA Alfred Spector Transarc Corporation, The Gulf Tower 70'7 Grant Street, Pittsburgh, PA 15219, USA

CR Subject Classification (198,7): D.4, C.2.4, E1.1,E2.0, C.3-4 ISBN 3-540-9?385-0 Springer-Verlag Berlin Heidelberg New York ISBN 0-38'7-9'7385-0 Springer-Verlag New York Berlin Heidelberg This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in other ways, and storage in data banks. Duplication of this publication or parts thereof is only permitted under the provisions of the Germ an Copyright Law of September 9, 1965, in its current version, and a copyright fee must always be paid. Violations fall under the prosecution act of the German Copyright Law, © Springer-Verlag Berlin Heidelberg 1990 Printed in Germany Printing and binding: Druckhaus Beltz, Hemsbach/Bergstr. 214513140-543210 - Printed on acid-free paper

Preface

The goal of the Asilom~r Workshop on Fault-Tolerant Distributed Computing was to facilitate interaction between theoreticians and practitioners. To achieve this goal, speaker:; were invited and topics chosen so that a broad overview of the field would be presented. Because the attendance at the Workshop was diverse, the presentations/papers were in many instances designed to appeal to a general audience. The presentations were also designed to span a body of research from the theoretical to the pragmatic. Since the material seemed ideal for a book, this book was planned together with the planning of the workshop. Held in the spring of 1986, the workshop brought together approximately 70 active researchers from academia and industry. Most of the chapters were written following the workshop and subsequently revised. Consequently, some of the results that are described were obtained after the workshop. However, six of the chapters (those by Bernstein, Cohn, Finkslstein, Liskov, Spector, and Wensley) were recorded, transcribed, and edited for inclusion. This transcription format makes for easy-toread, though somewhat chatty, articles. The chapters are arranged in the order of presentation at the workshop. Giwm the scope of this book, we feel that it should be of use to students, researchers, and developers. We hope that it will promote greater understanding within the world of fault-tolerant research and development.

Barbara Simons Alfred Spector

Workshop Program Committee: Chair: Barbara Simons, Haviu Cristian, Danny Dolev, Michael Fischer, Jim Gray, Leslie Lmnport, Nancy Lynch, Marshall Pease, Fred Schneider, Ray Strong.

We would like to thank IBM Almaden Research Center and the Office of Naval B.esearch for their support. We also apprec/ate the assistance of several volunteers who assisted at the workshop.

Contents

A Theoretician's View of Fault Tolerant Distributed Computing ................................................... 1

M. J. Fischer A Comparison of the Byzantine Agreement Problem and the Transaction Commit Problem .................................................................................................

10

J. Gray The State Machine Approach: A Tutorial .......................................................................................... 18

F. B. Schneider A Simple Model for Agreement in Distributed Systems ................................................................. 42

D. Dolev, ~ Strong Atomic Broadcast in a Real-Time Environment ............................................................................... 51

F. Cristian, D. Dolev, 1~ Strong, H. Aghili Randomized Agreement Protocols .......................................................................................................

72

M. Ben-Or An Overview of Clock Synchronization ...............................................................................................

84

B. Simons, J. L. Welch, 19. Lynch Implementation Issues in Clock Synchronization .............................................................................. 97

M. Beck, T. K. Srikanth, S. Toueg Systems Session I Argus .........................................................................................................................................................

108

B. Liskov TABS .........................................................................................................................................................

115

A. Z. Spector Communication Support for Reliable Distributed Computing ..................................................... 124

K. P. Birman, T. A. Joseph Algorithms and System Design in the Highly Available Systems Project .................................. 138

S. J. Finkelstein

VI

Easy Impossibility Proofs for Distributed Consensus Problems .................................................. 147

M. J. Fischer, N. A. Lynch, M. Merritt An Efficient, Fault-Tolerant Protocol for Replicated Data Management ................................. 171

D. Skeen, A. EI Abbadi, F. Cristian

Systems Session II Arpanet Routing ..................................................................................................................................... 192

S. Cohn On the Relationship Between the Atomic Commitment and Consensus Problems .......................................................................................................................201

V. Hadzilacos The August System ........................................................-~........................................................................ 209

Z Wensley The Sequoia System .:.............................................................................................................................217

P. Bemstein Fault Tolerance in Distributed UNIX ............................................................................................... 224 A. Borg, W. Blau, W. Oberle, W. Graetsch Faults and Their Manifestation ........................................................................................................... 244

D. P. Siewiorek The "Engineering" of Fault-Tolerant Distributed Computing Systems .......................................262

(). Babao~lu Bibliography for Fault.Tolerant Distributed Computing .....................................................:.........274

B. A. Coan

A Theoretician's View of Fault Tolerant Distributed Computing* M i c h a e l J. F i s c h e r Department of Computer Science Yale U n i v e r s i t y B o x 2158 Yale S t a t i o n N e w H a v e n , C T 06517

1

Introduction

Distributed computing systems have been objects of study ever since computers began to communicate with each other, and achieving reliability has always been a major problem of such systems. The theory of distributed computing is a relative newcomer, both to the fidd of distributed computing and to the general area of theoretical computer science. Thins paper is intended as a non-technical introduction to the r61e played by theory in the study of fault tolerant distributed computing systems. Rather than focus on particular accomplishments of theory, we will try to illustrate the theoretical approach and to point out its strengths and weaknesses as a paradigm for understanding distributed systems.

2

The Theoretical Paradigm

Theory could be called the science of asking (and answering) precise questions. A practitioner might be satisfied at finding a system that performs "well". A theoretician would want to know precisely how well. The two would also differ on what they considered acceptable evidence that a system performed as advertised. The practitioner is likdy to find the observed behavior of the system under practical operation most compelling. A theoretician would find such evidence unconvincing since not all of the relevant variables could be controlhd or understood, so the true causes of the observed behavior would remain in doubt. He would prefer instead a mathematical proof. *This work was supported in part by the National Science Foundation under grant DCR-8405478 and by the Office of Naval Research under Contract N00014-82-K-0154.

The power of theory is that it forces one to think clearly about the problem one is trying to solve. Until a question can be stated precisely, there is no theoretical question. The process of stating the question leads one to identify relevant variables, state explicitly any assumptions being made, and so forth. These very factors are often instrumental in leading one to a solution, and identifying them forces one to pay attention to them. A common criticism of theory is that it is too abstract. It deals with simplified ~ideal worlds' and ignores many complicating aspects of the ~real worldL Abstraction, however, is the real power of theory, for by ignoring the irrelevant details, one focuses attention on the relevant properties, enabling them to be understood to far greater depth than would otherwise be possible. Abstraction also produces generality, for results that depend on fewer assumptions are more widely applicable. Indeed, abstraction lies at the heart of scientific understanding, for to have an understanding of something is to have abstracted the essential features in a way that they can be applied to a variety of related situations. Of course, the insight gained from studying a particular abstraction is only as good as the abstraction itself. The results are only applicable in practice if the details ignored by the abstraction ready are irrelevant. Determining whether that is so is inherently a non-theoretical question which can only be answered like any other practical question--by experimentation and observation. Nevertheless, the power of the theoretical approach is that it confines the practical problem to that of validating the abstraction. Once that has been done, the theoretical results can be applied with confidence.

3

Constructing a Formal Theory

A theory of distributed computing involves three parts: a formal model, a formal problem statement, and a complexity measure for comparing one solution with another.

3.1

Formal Models

A formal model consists of a collection of set-theoretic objects that represent the various elements of the system being modeled. For example, if a distributed system consists of processes, communication links, algorithms, fault assumptions, message assumptions, and so forth, then the model will have corresponding formal dements. Just as a Tufing machine can be defined, abstractly, as a set of quintuples, so can a distributed system be defined as an appropriate tuple. The purpose of a formal model is to insure completeness of the specification. By saying that a formal model i8 a particular set-theoretic object, we have reduced the problem of specifying an abstract system to that of specifying a particular mathematical object, a problem for which good mathematical techniques have been developed

over the years. Understanding the implications of a particular formal model may not be easy, and that is the work of the theoretician, but at least there is little room for misunderstanding about what the formal model is.

3.2

Formal P r o b l e m S t a t e m e n t s

Along with a formal statement of the computational model, one needs a formal statement of the problem to be solved. Separating the problem from its solution is one of the real contributions of the theoretical approach, for it opens the door to alternative solutions. Often the problem to be solved is very basic and fundamental: choose a leader from among a collection of identical processes, reach an agreement s carry out an election, and so forth. In these cases, the problem itself is abstract, and a solution to such a problem can often be used as a building block in solving more complicated problems. But in many interesting cases, defining the problem precisely can be quite difficult, for it may not be at all clear exactly what one wants to achieve. Consider for example the problem of building a distributed name server to maintain a mapping between names and values (which might in practice be network addresses, user ids, mailboxes, and so forth). What properties should be required of the solution? One would like it to be reasonably efficient, highly available, reasonably robust against failures, remain reasonably current and consistent in the face of updates, and so forth, but to require perfection in any of these properties may make the problem impossible or prohibitively costly to solve. We have not one but many problems depending on the importance one attaches to the various properties, and a solution that is good in one respect might be very bad in other. The difficulty lies not in making a formal problem statement but in finding an abstract problem that reflects our intuition about the practical problem.

3.3

Complexity Measures

A complezity measure provides a means of measuring the goodness of the various possible solutions to a problem. Typical quantities to measure in a distributed system are time to solve the problem, total number of messages sent, numbers of faults tolerated, and so forth. As in sequential complexity theory, one can analyze either the worst case or the average case complexity. For the latter to be meaningful, one must know the underlying probability distributions on the choice of inputs, schedules, failure patterns, and so forth over which the "average" is being taken. In the absence of such information, one instead performs a worst case analysis, thus obtaining a pessimistic guarantee on the behavior of the system. Even when the distributions are known, the worst case behavior might be more important than the average case. For example, in a real-time system, response within a fixed amount of time might be required always, not just on the average. In carrying out a worst case analysis, one often pretends that choices are being

made by a malicious adversary who is trying to maximize the "cost" of the run. By definition, the worst-case complexity is the largest amount of the measured resource that the adversary can cause the system to use. However, this approach does not require that the choices actually be under the control of a malicious adversary, nor is it invalid in those situations where the choices are obviously made by non-malicious means such as by random coin flips. Rather~ a worst-case analysis is exactly what the name implies--it tells one the worst that can possibly happen. Whether or not that is likely to happen in practice is a separate question that may well require further assumptions to answer.

4

Theory of Distributed Computing

Distributed computing is the study of distributed systems and what they compute. We are thus led to the question, "What is a distributed system?" The obvious answer, that it is a collection of communicating, concurrent processes, is too broad and doesn't distinguish distributed'systems from parallel systems. Intuitively, a distributed system is a collection of geographically separated computers or node8 that communicate over a relatively low-speed network. However, from a theoretical point of view, this definition is unsatisfactory for a number of reasons: The physical geometry of the system is rarely relevant to the problems studied and is one of the first features to be discarded in building an abstract model. Detailed timing considerations are difficult to deal with in an abstract model and are often not relevant anyway when one is concerned only with what can or cannot be achieved. At a more abstract level, distributed systems and parallel computers seem to look quite similar~ and one can reasonably ask if there really is any qualitative difference between the two. Upon further examination, one sees that the characteristic features of distributed systems mentioned above do have a qualitative effect that we can identify--namely, they all lead to greater degrees of uncertainty. Geographical separation makes it more difficult to manage the individual nodes, leading to greater uncertainty about their status. Low-speed communication restricts the flow of information around ,the network, leading to greater undertalnty about the global state of the system. Typical communication channels are subject to various forms of unreliability, and one cannot realistically assume error-free communication. Thus, what distinguishes parallel computers from distributed systems is the degree of uncertainty to which they are subject and the extent to which one must take explicit account of such uncertainty. In studying a parallel computer, it is often reasonable to assume that the machine is working correctly, that one has full control over the code to be run on each of the processors, that interconnection topology is stable and known to all of the processors, and that communication is reliable and occurs within a predictable amount of time. In the study of distributed systems~ one often can not make such simplifying assumptions and still obtain realistic results. Individual nodes may be faulty, the network may lose or corrupt messages, one cannot always control

what programs other nodes run, nodes speeds can vary wildly from one another, and so forth. As a result of these differences, a major concern in the study of parallel computation is performance of algorithms, whereas a major concern in the study of distributed computing is dealing with uncertainty. This is not to say that there is a clear dichotomy between the two disciplines but rather to identify two ends of a spectrum. Thus, a p a r c e l system becomes more and more distributed as uncertainty factors become more and more significant in its operation. We now look in greater detail at some of the sources of uncertainty in distributed computing systems.

4.1

S y s t e m Configuration

In a distributed system, the eventual system configuration might not be known when the processes are designed, so the goal is to design processes that will work when embedded in a wide range of systems. Of course, some assumptions must be made about the rest of the system in order to say anything meaningful about system behavior. We look at some natural assumptions that have been considered. A process might have only partial information about the global structure of the system--how many processes there are altogether and how they are interconnected. For example, a process might know that the processes are connected together in a ring but not know the size of the ring, or it might know that the interconnection graph is connected but not know the topology, or it might know that the graph is fully connected but not know the identities of the processes attached to its ports. One can deal with such uncertainties either by finding algorithms to obtain the unknown information or by finding ways to accomplish a given task despite the uncertainty. A process might not know what behavior to expect from another process because it does not know the program being run by that process. In analyzing such a system, one thinks of the other process as a malicious adversary who does his best to disrupt the system. However, as in any worst-case analysis, this does not imply that the unknown process must possess some malevolent intelligence; only that in the absence of information to the contrary, one must assume the worst. This is the assumption that underlies much of the work in cryptographic protocols.

4.2

Faults

A fault! is an event that causes "incorrect" operation of a system component. Whether a particular behavior is considered a fault or simply normal behavior is rather arbitrary and depends on one's expectations. For example, failure to obtain exclusive access to a shared resource is expected if the resource is already in use, but it might be considered a fault if the resource is idle. As with uncertainty in system configuration, one must make some assumptions

about the kinds of faults that can occur in order to say anything meaningful about the system's overall behavior. These assumptions result in a fault model that describes the kinds of faults that are anticipated, and a system that operates in the presence of such faults is called fault tolerant. In the formal system, these are the only kinds of faults that can occur. In practice, other kinds of faults might occur~ but if they do, the formal results do not apply. Unlike the uncertainty due to system configuration discussed above, faults are often assumed to be benign, random events controlled by nature. Nevertheless, it is difficult to come up with plausible assumptions that govern such faults, and one generally treats faults as being controlled by an adversary (though perhaps a restricted one). Thus, both unpredictability of processes and failures of processes are treated the same by the theory, even though they arise from very different considerations. Many kinds of communication faults can be considered. Depending on the structure of the underlying communication system, messages may be delayed, lost, duplicated, reordered, or corrupted. For example, a simple point-to-point link may inject random noise into the data it carries, leading to lost or corrupted messages, but messages, if delivered correctly, are delivered only once and in the order sent. However, with more elaborate store-and-forward message systems, other behaviors are possible: messages may not necessarily arrive in the order sent, and multiple copies of the same message~ possibly traveling along different paths~ can be delivered to the receiver. Adding a link protocol to the communication system can radically alter its external behavior~ and the level at which one models a system depends on one's goals. For example, a link protocol implementing checksums and message retransmission can make the eventual delivery of corrupted messages highly unlikely at the expense of increased communication delay. Assuming reliable but slow message delivery might be reasonable when studying protocols built on top of such a link protocol but would not be reasonable when studying link protocols themselves.

4.3

Timing

Considerations

Time can be a major source of uncertainty in distributed systems. Unless processes operate off of the same clock, they will not proceed at exactly the same speed, and it is difficult to make reasonable assumptions about their relative speeds. For example, a process being run on a time-shared computer might be interrupted at an arbitrary point of its execution and suspended for a long and variable length of time. Quite separate from the question of whether all processes proceed at the same rate is whether or not clocks are available. Even if processes do not proceed in lockstep, the ability to read a common clock might be of considerable use in coordinating their activities. Lacking a common global clock, processes might have available local clocks that are accurate within known bounds. Nevertheless, keeping clocks synchronized is not easy in practice, and one often looks for algorithms that make no assumptions about time, thus leading one to consider the fully asynchronous model. It isn't so much that one really believes that every behavior is possible, but only that one

doesn't know where to draw the line between those that are possible and those that are not. The fully asynchronous model is the common denominator with respect to timing assumptions.

5

Sample R e s u l t s

This whole discussion has itself been pretty abstract. To give a flavor of the kinds of theoretical results that have been obtained, we look at the reliable broadcast problem, sometimes known as the Byzantine Generals problem.

5.1

The

Reliable

Broadcast

Problem

The model consists of a fixed number n of processes in a completely connected, reliable network. At most f of the processes are assumed to fail during an execution of the protocol, but which ones are faulty is not known (or even necessarily determined) in advance° The system may be synchronous or asynchronous. strained according to a particular fault model:

Faulty processes are con-

Failstop A faulty process ceases operation and other processes are notified of the fault. C r a s h A faulty process ceases operation but other processes are not notified. Omission A faulty process omits sending some of its messages but otherwise continues to operate correctly. B y z a n t i n e A faulty process may continue to operate but can send arbitrary messages. One process, ca~ed the sender, has an initial binary input value which it wants to broadcast to the other processes. The problem is to find a fault-tolerant protocol such that each reliable process decides on a value satisfying the following two conditions: Validity If the sender is nonfaulty, then each non-faulty process decides on the sender's value. A g r e e m e n t Whether or not the sender is faulty, all non-faulty processes decide on the same value. Here are just a few of the many results that have been obtained on this problem: 1. With Byzantine faults, there is a deterministic synchronous solution iff f < n/3 [PSL$O].

2. With crash faults, every synchronous deterministic solution requires at least ] + 1 rounds of communication [DS82]. 3. No asynchronous deterministic solution can tolerate even a single crash fault [FLP85]. 4. With Byzantine faults, there are both synchronous and asynchronous randomized solutions that use an expected constant number of rounds [FM88].

5.2

Insights Gained

What do such results tell us about practical distributed computing systems? First off, even simple-sounding problems may be much more difficult than they appear at first sight, and great caution is called for. The reliable broadcast problem-to agree on a single bit--is one of the simplest coordination problems imaginable, yet solutions only exist under certain conditions, and even when they do, they can be quite costly. Secondly, the conditions under which solutions are not possible set useful boundary conditions on the search for solutions. One does not need to look for 3-process solutions to tolerate a single Byzantine processor fault, for they do not exist. If someone purports to have such a solution, one knows that it is erroneous, even before the bug can be demonstrated. Third, such results help guide the refinement process in designing a system. Knowing that agreement problems are costly suggests that their need be avoided where possible. Perhaps weaker properties are enough. Finally, the theory itself might indicate a solution to the very problems it poses. One can notice that the time lower bound in result 2 above applies only to deterministic systems. Removing the determinicity restriction led to the much more efficient randomized solutions of result 4.

6

Conclusion

The theory of distributed systems so far lacks the cohesiveness that can only come with further development. In the future, we hope to see work on a greater variety of problems and on better models with the goal of eventually obtaining greater generality, sharper results~ and better insight.

Acknowledgements: We are grateful to Lenore Zuck and Barbara Simons for many helpful comments and suggestions.

References [DS82]

D. Dolev and H. R. Strong. Polynomial algorithms for multiple processor agremment. In Proe. l~th ACM Symposium on Theory o] Computing, pages 401-407, 1982.

[FLP85] M.J. Fischer, N. A. Lynch, and M. S. Paterson. Impossibility of distributed consensus with one faulty process. Journal o[ the A CM, 32(2):374-382, April 1985. [FM88] P. Feldman and S. Micali. Optimal algorithms for Byzantine agreement. In Proe. *Oth A CM Symposium on Theory of Computing, pages 148-161, 1988. [PSL80] M. Pease, R. Shostak, and L. Lamport. Reaching agreement in the presence of faults. Journal of the ACM, 27(2):228-234, 1980.

A Comparison of the Byzantine Agreement Problem and the Transaction Commit Problem

Jim Gray Tandem Computers, Cupertino, CA., USA

Abstract: Transaction Commit algorithms and Byzantine Agreement algorithms solve the

problem of multiple processes reaching agreement in the presence of process and mes~;age failures. This paper summarizes the computation and fault models of these two kinds of agreement and discusses the differences between them. In particular, it explains that Byzantine Agreement is rarely used in practice because it involves significantly more hardware and messages, yet does not give predictable behavior if there are more than a few

faults. 1. Introduction

The Workshop on Fault Tolerant Distributed Computing met at Asilomar, California on March 16-19, 1986. It brought together practitioners and theorists. The theorists seemed primarily concerned with variations of the Byzantine Generals problem. The practitioners seemed primarily concerned with techniques for building fault-tolerant and distributed systems. Prior to the conference, it was widely believed that the Transaction Commit Problem faced by distributed systems is a degenerate form of the Byzantine Generals Problem. One useful consequence of the conference was to show that these two problems have little in comInon.

2. The Transaction Commit Problem

The Transaction Commit Problem was lust solved by Niko Garzado noticed and solved it while working for IBM on a distributed system for the Italian Social Security Department in 1971. The problem was folklore for several years. Five years later, descriptions and solutions began to appear in the open literature [2], [5], [7]. Solutions to the Commit Problem make a collection of actions atomic -- either all happen or none happens. Atomicity is easy if all goes well, but the Commit Problem requires atomicity even if there are failures. Today, commit algorithms are a key element of most transaction processing systems. Maintenance of replicated objects (all copies must be the same) and maintenance of consistency within a system (a mail message must arrive at one node if it leaves a second one) require atomic agreement among computers.

11

To statethe Commit Problem more precisely,one needs a model of computation and a model of failures. Lampson and Sturgis formulated a simple and elegant model now widely cmbraced by practitioners[5], It is impossible to do the model justicehere; their paper is well worth reading. In outline: The Lampson-Sturgis computation model consistsof: • storagewhich may be read and written. • processeswhich execute programs composed of threekinds of actions: • change process state, • send or receive a message and, • read or write storage. Processes run at arbitrary speed, but eventually make progress. The Lampson-Sturgisfault model postulates that: • Storage writes may fail, or may corrupt another piece of storage. Such faults are rare, but when they happen, a subsequent read can detect the fault. • Storage may spontaneously decay, but such faults are rare. In particular, a pair of ,.torage units will not both decay within the repair time for one storage unit. When decayed storage is read, the reader can detect the corruption. These are the fault assumptions of duplexed discs. • Processes may lose state and be reset to a null state, but such faults are rare and detectable. • Ivlessages may be delayed, corrupted, or lost, but such faults are rare. Corrupted messages are detectably corrupted. Based on these computation and fault models, Lampson and Sturgis showed how to build single-fault tolerant stable storage which does not decay (duplexed discs), and stable processes which do not reset (process pairs). This single-fault tolerance is based on the assumptions of rare faults and eventual progress (Mean Time To Repair is orders of magnitude less than Mean Time To Failure).

The cost model implicit in the Lampson-Sturgis computation model is: • Computation cost is proportional to the number of storage accesses and messages. • Time cost, i.e. delay, is proportional to serialized the number of storage accesses plus serialized messages. Good algorithms have low cost and low delay in the average ease. In addition they tolerate arbitrarily many lost messages, process resets, and storage decays.

12 Given this computation and fault model the Commit Problem may be stated as: The Commit Problem: Given N stable processes each with the state diagram:

An algorithm solves the Commit Problem if it forces ALL processes to the COMMITTED state or ALL to the ABORTEDstate depending on the input to the algorithm. Q The Commit Problem is easily solved; a stable process just sends the decision to each process, and keeps resending it until the process changes state and acknowledges. Because messages may be lost and processes may run slowly, there is no limit on how long the algorithm may take. But, eventually all processes will agree and the algorithm will terminate. The expected cost is proportional to 2N messages plus N stable (i.e. single-fault tolerant) writes. Assuming constant service times, the expected delay is two message delays plus one stable write delay. There is a more general version of the Commit Problem called the Two-Phase Commit Problem. It allows an ACTIVEprocess to unilaterally abort its part of the transaction, by entering the ABORT state, and consequently aborting the whole transaction. To allow consensus among the processes, a PREPARED state is introduced. After entering the PREPARED state an active process abdicates its right to unilateral abort. By contrast, the One-Phase Commit Problem does not allow processes to unilaterally abort. Unilateral abort is very convenient; the CANCEL keys of most user interfaces are examples of unilateral abort. The Two Phase Commit Problem may be stated as: The Two-Phase Commit Problem: Given N stable processes each with the state diagram:

An algorithm solves the Two-Phase Commit Problem if it forces ALL the processes to the COMMITTED state or all to the ABORTEDstate depending on the input to the algorithm and whether any process has already spontaneously made the ACTIVEto ABORTEDtransition.~l

13 Algorithms for two-phase commit are fancier and costlier than the one-phase algorithms described first. Needless to say, three-phase algorithms have been invented and implemented, so called non-blocking commit protocols, but it has stopped there. All of the Commit Problems have the following properties: • All processes are assumed to correctly obey the state machine and commit/abort messages. • There may be arbitrarily many storage failures, process failures, and message failures. • F!ventually, all processes agree. 3. The Byzantine Generals Problem The Byzantine Generals Problem grew out of attempts to build a fault-tolerant computer to control an unstable aircraft. The goal was to fred a design_that could tolerate arbitrary faults in the computer. The designers assumed that they could verify the computer software, so the only remaining problem was faulty hardware. Faulty computer hardware can execute even CO~Tectprograms in crazy ways. So, the designers postulated that most computers functioned correctly, but some functioned in the most malicious way conceivable. Thus, the Byzantine Generals Problem was formulated [6], [4]. The computation and cost models of the Byzantine Generals problem are similar to the Lampson-Sturgis model for processes and messages (storage is not explicitly modeled). Processe~ communicate with one another via messages. The Byzantine fault model is quite different from the Lampson-Sturgis fault model. The Byzantine Generals Problem assumes that some processors are good and some arefaulty. The faulty ones can forge messages, delay messages sent via them, send conflicting or contradictory messages, and masquerade as others. If a message from a good process is lost or damaged, then the good process is treated as a bad one. The Lampson-Sturgis model assumes processes execute correctly or detectably fail, and that messages are delivered, detectably corrupted, or lost. Forgeries or undetected corruption are defined as "impossible" (i.e. very rare squared) by Lampson and Sturgis; actually, they just define such an ,event as a catastrophe. The Byzantine Generals Problem is intended for process control applications which must respond within a fixed time in order to fly the airplane or control the reactor. So, the problem statement insists that any solution must have a bounded execution time. This in turn implies a bounded number of messages and faults (fault rates are finite). These bounded-time, bounded-faults assumptions are the key distinction between the Commit and Byzantine problems.

14 The Byzantine Generals problem is defined as: The Byzantine Generals Problem Given N generals (processes) each with the state diagram:

An algorithm solves the Byzantine Generals Problem if it gets all the good generals to agree YES or ALL to agree NO within a bounded time.O The gist of the theory on solutions to the Byzantine Generals Problem is: • If at least 1/3 of the generals are bad, then the good generals cannot reliably agree [6]. • If fewer than 1/3 of the generals are bad, then there arc many algorithm~, • Solutions to the algorithm have polynomial cost (e.g. ~N 2 messages) and, assuming constant service time for a broadcast, have constant delay (typically ~N3). 4. Comparing the Problems The Commit and the Byzantine Generals problems are identical in the fault-free case -- this is hinted by the sin~larity in their state transition diagrams. In the fault-free case all the participants must agree. In the typical case (no faults) Commit algorithms send many fewer messages than the Byzantine Generals algorithms because they need not guard against ambiguity or forgery. Fundamental differences between the problems and their solutions emerge when there are faults.The basic differences arc: • Commit protocols tolerate finitely many faults. Byzantine protocols tolerate at most N/3 faults. • ALL processors agree in the conunit case. SOME processors agree in the Byzantine case. Commit algorithms are fail-fast. They give either a common answer or no answer. Byzantine algorithms give random answers without warning if the fault threshold is exceeded. • Commit agreement may require unbounded time. Byzantine agreement terminates in bounded time. • Commit algorithms require no extra processors and few extra messages. Byzantine algorithms require many messages and processors. •

15

The following examples show that neither of these problems is especially realistic Byzantine ATMS: Consider an Automated Teller Machine (ATM) doing a debit of a bank account. Three computers storing the account plus the ATM give the requisite four processors needed for Byzantine agreement. If the phone line to the ATM fails, then the three computers quickly agree to debit the account but the ATM refuses to dispense the money. This is Byzantine agreement.

Commit ATMS: If the same ATM is controlled by a single computer and they obey a commit protocol, then they will eventually agree (debit plus dispense or no debit and no dispense). But, the customer may have to wait at the ATM in PREPARED state for many days before a faulty phone line is luted and the money is dispensed. That is commitment[

Neither of these solutions is particularly nice. Most "rear' systems focus on single-fault tolerance because single faults are rare, double faults are very rare (rare squared). Commit algorithms are geared to this single-fault view. They give single-fault tolerance by duplexing fail-fast modules [3]. Byzantine algorithms require at least four-plexed modules to tolerate a single fault. This high processor cost and related high message cost is the probable reason no commercial system uses Byzantine algorithms. Look at the two pictures above and judge for yourself. The Lampson-Sturgis model distinguishes between message failure and process failure because long-haul messages typically are corrupted once an hour while processors typically fail once a month. This is an important practical distinction -- especially since Byzantine

16

messages outnumber processors by a polynomial multiplier. The Byzantine faultmodel typicallyequates message failureswith process failures. As the number of nodes grows, the number of message failures grows polynomial and produces a system much less reliablethan a singleprocessor. Paradoxically,both Byzantine and Commit systems have worse mean-time-to-failure (MTBF) than a single processor system, but for different reasons and in different ways Large Byzantine systems fail because of polynomial message failures [8], [i]. Commit systems fail because a single processor failuremay introduce a delay for all (unlessthree-phase algorithm¢ are used). Commit algorithms are fail-fastwhile Byzantine algori~rns give an answer in any case. Each non-faulty proccssor executing a Byzantine Generals algorithm gives an answer within a fixed time -- ~ N 3 message delays for a typicalalgorithm, ff there are few faults, thcn all the non-faultyprocessors will give the same answer. If at leastN/3 processors arc faulty or if at least N/3 mcssagcs are damaged, then two "correct" processors may give differentanswers. By contrast,commit algorithms cvcntuaUy get all the processes to give the same answer. This may take a very long time and many messages. N o one wants to wait forever for the right answer. Unfortunately, there is no solution to thisdilemma. If all processors must agree, and if there is no finitebound on the number of message or processor faults,then the processors must be prepared to wait an unbounded time. The salientproperties of the two problems are summarized in the chart below. It shows thatthere islittleovcrlap between the two problems. ",,,,,Degreeof Fault~g ~em~ T o l e r a n c c ' ~ ...... Limited Time & Errors

Unlimited Time &Errors

Some Agree

All Agree

Byzantine

Ideal

Inferior

Commit

Of course the ideal solution would combine the best of both: all agree within a fixed time limit. One can prove thatthere is no ideal solution [2]. Similarly an algorithm which has no time limit,and does not get universal agreement is uniformly inferiorthan Byzantine or Cornmitrncnt, and so is not interesting.

In summary, based on these comparisons between the two problems, practitioners embraced the Commit problem over the Byzantine Generals problem because it has an efficient solution to the single-fault case, gives correct answers in the multi-fault case, and has good no-fault performance.

17

5. Acknowledgments Phfll Garrett, Fritz Graf, Pat Helland, Pete Homan, Fritz Joem, and Barbara Simons made valuable comments on this paper. 6. References [1] Babaoglu, O., "on The Reliability of Consensus Based Fault Tolerant Distributed Computer Systems", ACM TOCS, V. 5.4, 1987. [2] Gray, J., "Notes on Database Operating Systems", Operating Systems, An Advanced Course, Lecture Notes in Computer Science, V. 60, Springer Verlag, 1978. [3] Gray, L, "Why Do Computers Stop, and What Can We Do About It?", Tandem TR: 85.7, Tandem Computers, 1985. [4] Lamport, L., Shostak, R., Pease, M., "The Byzantine Generals Problem", V. 4.3, 1982.

ACM TPLS,

[5] Lampson, B.W., Sturgis, H., "Atomic Transactions", Distributed Systems Architecture and Implementation; An Advanced Course, Lecture Notes in Computer Science, V. I05, Springer Verlag, 1981. [6] Pease, M., Shostak, R., Lamport, L., "Reaching Agreement in the Presence of Faults", ACM Journal, V. 27.2, 1980. [7] Rosenkrantz, D.J., R.D. Stearns, P.M. Lewis, "System Level Concurrency Control for Database Systems", ACM TODS, V. 3.2, 1977. [8] Tay, Y.C., "The Reliability of (k,n)- Resilient Distributed Systems", Proc. 4th Symposium in Distributed Software and Database Systems, IEEE, 1984.

The State Machine Approach: *

A Tutorial

Fred B. Schneider Department of Computer Science Cometl University Ithaca, New York 14853

Abstract. The state machine approach is a general method for achieving fault tolerance and implementing decentralized control in distributed systems. This paper reviews the approach and identifies abstractions needed for coordinating ensembles of state machines. Implementations of these abstractions for two different failure models Byzantine and fail-stolr--are discussed. The state machine approach is illustrated by programming several examples. Optimization and system reconfiguration techniques are explained.

1. Introduction The state machine approach is a general method for managing replication. It has broad applicability for implementing distributed and fault-tolerant systems. In fact, every protocol we know of that employs replication--be it for masking failures or simply to facilitate cooperation without centralized control---can be derived using the state machine approach. Although few of these protocols actually were obtained in this manner, viewing them in terms of state machines helps in understanding how and why they work. When the state machine approach is used for implementing fault tolerance, a computation is replicated on processors that are physically and electrically isolated. This permits the effects of failures to be masked by voting on the outputs produced by these independent replicas. Triple-modular redundancy (TMR) is a familiar example of this scheme. Although when the approach is used additional *This material is based on work supporledin part by the Office of Naval Research under contract N00014-86-K-0092, the National Science Foundation under Grant No. CCR-8701103, and Digital Equipment Corporation. Any opinions, findings, and conclusions or recommendationsexpressed in this publication are those of the author and do not reflect the views of these agencies.

19

coordination is necessary to distribute inputs and collect outputs from replicas, failures cannot increase task completion times. This makes the approach ideally suited for real-time systems, where deadlines must be met and timing is critical. Other approaches, such as failure detection and retry, are ill suited for real-time applic,ations because unpredictable delays can be observed in response to a failure. The state machine approach permits separation of fault-tolerance from other aspects of system design. The programmer is not forced to think in terms of a particular programming abstraction, such as transactions ~Liskov 85] [Spector 85], fault-tolerant actions [Schlichting & Schneider 83], reliable objects [Bimaan 85], replicated remote procedure calls [Cooper 84] or the multitude of other proposals that have appeared in the literature. Instead, a programming abstraction suited for the application at hand can be defined and used; the state machine approach is employed to realize a fault-tolerant implementation of that abstraction. This p~tper is a tutorial on the state machine approach. It describes the approach and its implementation for two representative environments. Small examples suffice to illustrate the points; however, the approach has been successfully applied to larger examples. Section 2 describes how a system can be viewed in t e l l s of a state machine, clients, and output devices. Measures of fault-tolerance are discussed in section 3. Achieving fault-tolerance is the subject of the following three sections: Section 4 discusses implementing fault-tolerant state machines; section 5 discusses tolerating faulty output devices; and section 6 discusses coping with faulty clients. An important class of optimizations---based on the use of tinae--is discussed in section 7. Optimizations possible by making assumptions about failures are discussed in section 8. Section 9 describes dynamic reconfiguration. Related work is discussed in section 10. 2. State Machines A state machine consists of state variables, which implement its state, and commands, which

transform its state. Each command is implemented by a deterministic program; execution of the command is atomic with respect to other commands and modifies the state variables and/or produces some output. A client of the state machine makes a request to specify execution of a command. The request names a state machine, names the command to be performed, and contains any information needed by the command. Output from request processing can be to an actuator (e.g. in a process-control system), to some other peripheral device (e.g. a disk or terminal), or to clients awaiting responses from prior requests. The name "state machine" is a poor one, since it is suggestive of a finite-state automata. Our state machines are more powerful than finite-state automata because they contain program variables and, therefore, need not be finite state. However, state machines intended for execution on real machines should be finite state because real computers have finite memories. Our state machines are also easier to specify than finite-state automata because any programming notation can be used. "State machine" is used in this paper for historical reasons--it is the term used in the literature. State machines can be described by explicitly listing state variables and commands. As an example, state machine memory of Figure 2.1 implements a mapping from locations to values. A read command permits a client to determine the value associated with a location, and a write command associates a new value with a location. Observe that there is little difference between our state machine description of merrugry and an abstract datatype or (software) module for such an object. This is deliberate---it makes it clear that state machines are a general programming construct. In fact, a state machine can be implemented in a variety of ways. It can be implemented as a collection of procedures that share data,

20

memory: s t a t e m a c h i n e var store : array [O..n] of word

read: command(loc : O..n) send store [loc ] to client end read; write: eommand(loc : O..n, value : word) store [loc] := value end write end memory

Figure 2.1. A memory

as in a module; it can be implemented as a process that awaits messages containing requests and performs the actions they specify; and, it can be implemented as a collection of interrupt handlers, in which case a request is made by causing an interrupt. (Disabling interrupts permits each command to be executed to completion before the next is started.) For example, the state machine of Figure 2.2 implements commands to ensure that at atl times at most one client has been granted access to some resource. 1 It would likely be implemented as a collection of interrupt handlers as part of the kernel of an operating system. Requests are processed by a state machine one at a time, in an order consistent with causality. Therefore, clients of a state machine can be programmed under the assumptions that O1:

requests issued by a single client to a given state machine sm are processed by sm in the order they were issued, and

O2:

if the fact that request r was made to a state machine sm by client c could have caused a request r' to be made by a client c" to sm, then sm processes r before r'.

In this paper, for expository simplicity, client requests are specified as tuples of the form

(state_machine.command, arguments) and the return of results is done using message passing. For example, a client could execute

(memory.write, 100, 16.2); (memory.read, 100); receive v from memory to set the vatue of location 100 to 16.2, request the value of location 100, and await that value, setting v to it upon receipt. 1We use xq~ to append y to the end of list x.

21

mutex: state machine var u s e r : client id init ~ ; waiting : list of client id init acquire: command if user=dp --> send OK to client; user := client user ~

--> waiting := waiting oclient

fi

end acquire release: command if waiting =d~ .-->user := D waiting ~

--->send OK to head(waiting); user := head(waiting); waiting := tail(waiting)

fi

end release end mutex

Figure2.2. Aresourceallocator

The defining characteristic of a state machine is not its syntax, but that it specifies a deterministic computation that reads a stream of requests and processes each, occasionally producing output: Semantic Characterization of State Machine. Outputs of a state machine are completely determined by the sequence of requests it processes, independent of time and any other activity in a system. Any program satisfying this definition will be considered a state machine for the purposes of this paper. For example, the following program solves a simple process-control problem in which an actuator is adjusted repeate~y based on the value of a sensor. Periodically, a client reads a sensor and communicates the value read to state machine pc: monitor : process do true --~ val := sensor; (pc.adjust, vat); delay D od end monitor State machine pc adjusts an actuator based on past adjustments saved in state variable q, the sensor reading, and a control function F.

22

pc: s t a t e m a c h i n e var q : real;

adjust: command(sensorval : real) q := F(q, sensorval); send q to actuator end adjust end pc Although it is tempting to structure pc as a single command that loops--reading from the sensor, evaluating F, and writing to actuator--if the value of the sensor is time-varying, then the result would not satisfy the semantic characterization given above and therefore would not be a state machine. This is because values sent to actuator (the output of the state machine) would not depend solely on the requests made to the state machine but would, in addition, depend on the execution speed of the loop. In the structure used above, this probIem has been avoided by moving the loop into monitor. Having to structure a system in terms of state machines and clients does not constitute a restriction. Anything that can be structured in terms of procedures and procedure calls can also be structured using state machines and clients--a state machine implements the procedure, and requests implement the procedure calls. In fact, state machines permit more flexibility in system structure than is usually available with procedure calls. With state machines, a client making a request is not delayed until that request is processed, and the output of a request can be sent someplace other than to the client making the request. We have not yet encountered an application that could not be programmed cleanly in terms of state machines and clients. 3. F a u l t - T o l e r a n c e A component is faulty once its behavior is no longer consistent with its specification. In this paper, we consider two representative classes of faulty behavior from a spectrum of possible ones: Byzantine Failures. The component can exhibit arbitrary and malicious behavior, perhaps involv-

ing collusion with other faulty components [Lamport et a182]. Fail-stop Failures. In response to a failure, the component changes to a state that permits other components to detect that a failure has occurred and then stops [Schneider 84]. Byzantine failures can be the most disruptive, and there is anecdotal evidence that such failures do occur in practice. Allowing Byzantine failures is the weakest possible assumption that could be made about the effects of a failure. Since a design based on assumptions about the behavior of faulty components runs the risk of failing if these assumptions are not satisfied, it is prudent that life-critical systems tolerate Byzantine failures. However, for most applications, it suffices to assume fail-stop failures. A system consisting of a set of distinct components is f fault-tolerant if it satisfies its specification provided that no more than f of those components become faulty during some interval of interest. 2 Fault-tolerance traditionally has been specified in terms of MTBF (mean-time-between-failures), probability of failure over a given interval, and other statistical measures [Siewiorek & Swarz 82]. While it is clear that such characterizations are important to the users of a system, there are advantages in ZAnf fault-tolerant systemmightcontinueto operate correctlyif more thanf failures occur, but correct operationcannot be guaranteed.

23

describing fault tolerance of a system in terms of the maximum number of component failures that can be tolerated over some interval of interest. Asserting that a system is f fault-tolerant makes explicit the assumptions required for correct operation; MTBF and other statistical measures do not. Moreover, f fault-tolerance is unrelated to the reliability of the components that make up the system and therefore is a measure of the fault tolerance supported by the system architecture, in contrast to fault tolerance achieved simply by using reliable components. Of course, MTBF and other statistical reliability measures of an f fault-tolerant system will depend on the reliability of the components used in constructing that system---in particular, the probability that there will be f o r more failures during the operating interval of interest. Thus, f should be chosen based on statistical measures of component reliability. O n c e f has been chosen, it is possible to derive MTBF and other statistical measures of reliability by computing the probabilities of various configurations of 0 throughf failures and their consequences [Babaoglu 86].

4. Fault-tolerant State Machines A n f fault-tolerant state machine can be implemented by replicating it and running a copy on each of the processors in a distributed system. Provided each copy being run by a non-faulty processor starts in the same initial state and executes the same requests in the same order, then each wiU do the same thing and produce the same output. If we assume that each failure can affect at most one processor, hence one state machine copy, then by combining the output of the state machine copies in this ensemble, the output for a t fault-tolerant state machine can be obtained. When processors can experience Byzantine failures, an ensemble implementing a f fault-tolerant state machine must have at least 2f +1 copies, and the output of the ensemble is the output produced by the majority of the state machine copies. This is because with 2f +1 copies, the majority of the outputs remain correct even after as many a s f failures. If processors experience only fail-stop failures, then an ensemble containing f + 1 copies suffices, and the output of the ensemble can be the output produced by any of its members. This is because onty correct outputs are produced by fail-stop processors, and after ffailures one non-faulty copy will remain among the f + 1 copies. Our scheme for implementing a n f fault-tolerant state machine is based on fault-tolerant implementations of two abstractions. Agreement. Every non-faulty copy of the state machine receives every request. Order. Requests are processed in the same order by every non-faulty copy of the state machine. However, knowledge of command semantics sometimes permit weaker (i.e., cheaper to implement) abstractions to be used. For example, when fall-stop processors are used, a request whose processing does not modify state variables need only be sent to a single non-faulty state machine copy, thus permitting reIaxation of Agreement. It also possible to exploit request semantics to relax Order. Two requests r and r" commute in a state machine if the sequence of outputs that would result from processing r followed by r ' is the same as would result from processing r' followed by r. Not surprisingly, the schemes outlined above for combining outputs of the members of an ensemble work even when two requests that commute axe processed in different orders by different state machine copies in an ensemble, thus permining relaxation of Order. An example of a state machine where Order is not necessary appears in Figure 4.1. State machine tally determines the first from among a set of alternatives to receive at least MAJ votes and sends this choice to SYSTEM. If clients cannot vote more than once and the number of clients Cno satisfies 2MAJ > Cno, then every request commutes with every other. Thus, implementing Order would be

24

tally: state_machine var votes : array[candidate] of integer init 0 cast vote: command(choice : candidate) votes [choice] := votes [choice ] + 1; if votes [choice ] >_MAJ --4 send choice to SYSTEM; halt

0 votes [choice] <MAJ --> skip

fi end cast vote end tally

Figure 4.1. Election

unnecessary--different copies of the state machine will produce the same outputs even if they process requests in different orders. On the other hand, if clients can vote more than once or 2MAJ u i d ( r ' ) , because r ' was relayed by s m i. The solution to this problem is for sm,~.w to consider requests received directly from c stable only after no relayed requests from

$9

c can arrive. Thus, the stability test must be changed: Stability Test During Restart. A request r received directly from a client c by a restarting state machine copy smngw is stable only after the last request from c relayed by another processor has been received by smnew. An obvious way to implement this is for a message to be sent to s m , ~ when no further requests from c will be relayed. 10. Related W o r k The state machine approach was first described in [Lamport 78a] for environments in which failures could not occur. It was generalized to handle fail-stop failures in [Schneider 82], a class of failures between fail-stop and Byzantine failures in [Lamport 78b], and full Byzantine failures in [Lamport 84]. The vm-ious abstractions proposed for these models are unified in [Schneider 85]. A critique of the approach for use in database systems appears in [Garcia-Molina et al 84]. Experiments evaluating the performance of various of the stability tests in a network of SUN Workstations are reported in [Pittelli & Garcia-Molina 87]. The state machine approach has been used in the design of significant fault-tolerant process control applications [Wensley et al 78]. It has also been used to implement distributed synchronization-including read/write locks and distributed semaphores [Schneider 80], input/output guards for CSP and conditional Ada SELEC~ statements [Schneider 82J--and, more recently, in the design of a fail-stop processor approximations in terms of processors that can exhibit arbitrary behavior in response to a failure [Schlichting & Schneider 83] [Schneider 84]. The state machine approach is rediscovered with depressing frequency, though rarely in its full generality. For example, the (late) Auragen 4000 series system described in [Borg et al 83] and the Publishing crash recovery mechanism [Powell & Presotto 83], both use variations of the approach. A stable storage implementation described in [Bernstein 85] exploits properties of a synchronous broadcast network to avoid explicit protocols for Agreement and Order and employs Transmitting a Default Vote (as described in section 7). The notion of A common storage, suggested in [Cristian et al 85], is a state machine implementation of memory that uses the Real-time Clock Stability Test. The method of implementing highly available distributed services in [Liskov & Ladin 86] uses the state machine approach, with clever optimizations of the stability test and Agreement abstraction that are possible due to the semantics of the application and the use of fail-stop processors. The ISIS project [Birman & Joseph 87] has recently been investigating fast protocols to support fault-tolerant process groups--in the terminology of this paper, state machines in a system of fail-stop processors. Their ABCAST protocol is a packaging of our Agreement and Order abstractions based on the Logical Clock Stability Test Tolerating Fail-stop Failures; CBCAST allows more flexibility in message ordering and permits designers to specify when requests commute. Another project at Comell, the Realtime-Reliabitity testbed, is investigating semantics-dependent optimizations to state machines. The goal of that project is to systematically develop efficient, faulttolerant, process control software for a hard real-time environment. Starting with a system structured as state machines and clients, various optimizations are performed to combine state machines, thereby obtaining an fast, yet provably fault-tolerant distributed program.

40

Acknowledgments Discussions with O. Babaoglu, K. Birman, and L. Lamport over the past 5 years have helped me to formulate these ideas. Helpful comments on a draft of this paper were provided by J. Aizikowitz, O. Babaoglu, A. Bernstein, K. Birman, D. Giles, and B. Simons.

References [Babaoglu 86] Babaoglu, O. On the reliability of consensus-based fault-tolerant distributed systems. ACM TOCS 5, 4 (Nov. 1987), 394-416. [Bernstein 85] Bemstein, A.J. A loosely coupled system for reliably storing data. IEEE Trans. on Software Engineering SE-11, 5 (May 1985), 446-454. [Birman 85] Birman, K.P. Replication and fault tolerance in the ISIS system. Proc. Tenth ACM Symposium on Operating Systems Principles. (Oreas Island, Washington, Dec. 1985), ACM, 79-86. [Birman & Joseph 87] Birman, K.P. and T. Joseph. Reliable communication in the presence of failures. ACM TOCS 5, 1 (Feb. 1987), 47-76. [Borg et al 83] Borg, A., J. Banmbach, and S. Glazer. A message system supporting fault tolerance. Proc. of Ninth ACM Symposium on Operating Systems Principles, (Bretton Woods, New Hampshire, October 1983), ACM, 90-99. [Cooper 84] Cooper, E.C. Replicated procedure call. Proc. of the Third ACM Symposium on Principles of Distributed Computing, (Vancouver, Canada, August 1984), ACM, 220-232. [Cristian et al 85] Cristian, F., H. Aghili, H.R. Strong, and D. Dolev. Atomic Broadcast: From simple message diffusion to Byzantine agreement. Proc. Fifteenth International Conference on Fault-tolerant Computing, (Ann Arbor, Mich., June 1985), IEEE Computer Society. [Dijkstra 74] Dijkstra, E.W. Self Stabilization in Spite of Distributed Control. CACM 17, 11 (Nov. 1974), 643-644. [Fischer et al 85] Fischer, M., N. Lynch, and M. Paterson. Impossibility of distributed consensus with one faulty process. JACM 32, 2 (April 1985), 374-382. [Garcia-Molina et al 84] Garcia-Molina, H., F. Pittelli, and S. Davidson. Application of Byzantine agreement in database systems. TR 316, Department of Computer Science, Princeton University, June 1984. [Gray 78] Gray, J. Notes on Data Base Operating Systems. Operating Systems: An Advanced Course, Lecture Notes in Computer Science, Vol. 60, Spilnger-Verlag, New York, 1978, 393-481. [Hammer and Shipman 80] Hammer, M. and D. Shipman. Reliability mechanisms for SDD-I: A system for distributed databases. ACM TODS 5, 4 (December 1980), 431-466. [Lamport 78a] Lamport, L. Time, clocks and the ordering of events in a distributed system. CACM 21, 7 (July 1978), 558565, [Lamport 78b] Lamport, L. The implementation of reliable distributed multiprocess systems. Computer Networks 2 (1978), 95-114. [Lamport 84] Lamport, L. Using time instead of timeout for fault-tolerance in distributed systems. ACM TOPLAS 6, 2 (April 1984), 254-280. [Lampert et al 82] Lamport, L., R. Shostak, and M. Pease. The Byzantine generals problem. ACM TOPLAS 4, 3 (July 1982), 382-401. [Liskov 85] Liskov, B. The Argus language and system. Distributed Systems----~ethods and Tools for Specification, Lecture Notes in Computer Science, Vol. 190, Springer-Verlag, New York, N.Y, 1985, 343.430. [Liskov & Ladin 86] Liskov, B. and R. Ladin. Highly-available distributed services and fault-tolerant distributed garbage collection. Proc. of the Fifth ACM Symposium on Principles of Distributed Computing, (Calgry, Alberta, Canada, August 1986), ACM, 29-39. [Pittelli & Garcia-Molina 87] Pittelli, F.M. and H. Garcia-Molina. Efficient scheduling in a TMR database system. Proc. Seventeenth International Symposium on Fault-tolerant Computing, (Pittsburgh, Pa, July 1987), IEEE. [Powell & Presotto 83] Powell, M. and D. Presotto. PUBLISHING: A reliable broadcast communication mechanism. Proc. of Ninth ACM Symposium on Operating Systems Principles, (Bretton Woods, New Hampshire, October 1983), ACM, 100-109.

41

[Schlichting & Schneider 83] Schlichting, R.D. and F.B. Schneider. Fail-Stop processors: An approach to designing faulttolerant computing systems. ACM TOCS 1, 3 (August 1983), 222-238. [Schneider 80] Schneider, F.B. Ensuring Consistency on a Distributed Database System by Use of Distributed Semaphores. Proc. International Symposium on Distributed Data Bases (Paris, France, March 1980), INRIA, 183-189. [Schneider 82t Schneider, F.B. Synchronization in distributed programs. ACM TOPLAS 4, 2 (April 1982), 179-195. [Schneider 84] Schneider, F.B. Byzantine generals in action: Implementing fail-stop processors. ACM TOCS 2, 2 (May 1984), 145-154. [Schneider 85]

Schneider, F.B. Paradigms for distributed programs. Distributed Systems---Methods and Tools for Specification, Lecture Notes in Computer Science, Vol. 190, Springer-Verlag, New York, N.Y. 1985, 343-430.

[Schneider 86] Schneider, F.B. A paradigm for reliable clock synchronization. Proc. Advanced Seminar on Real-Time Local Area Networks (Bandol, France, April 1986), INRIA, 85-I04. [Schneider et al 84] Schneider, F.B., D. Gries, and R.D. Schlichting. Fault-Tolerant Broadcasts. Science of Computer Programming 4 (1984), 1-15. [Siewiorek & Swarz 82] Siewiorek, D.P. and R.S. Swarz. The Theory and Practice of Reliable System Design. Digital Press, Bedford, Mass, 1982. [Skeen 82] Skeen, D. Crash Recovery in a Distributed Database System. Ph.D. Thesis, University of California at Berkeley, May 1982. [Spector 85] Spector, A.Z. Distributed transactions for reliable systems. Proc. Tenth ACM Symposium on Operating Systems Principles, (Orcas Island, Washington, Dec. 1985), ACM, 127-146. [Slrong & Dolev 83] Strong, H.R. and D. Dolev. Byzantine agreement. Intellectual Leverage for the Information Society, Digest of Papers, (Compcon 83, IEEE Computer Society, March 1983), 1EEE Computer Society. 77-82. [Wenstey et a178] Wensley, J., et al. SIFT: Design and Analysis of a Fault-Tolerant Computer for Aircraft Control. Proc. IEEE 66, 10 (Oct. 1978), 1240-1255.

A Simple Model for Agreement in Distributed Systems Danny Dolev Raymond Strong IBM Almaden Research Center 650 Harry Rd San Jose, CA 95120

1

Introduction

The goal of our research into fault tolerant algorithms has been to discover and prove the best results possible in any environment consisting of processing elements that communicate by means of messages via some communication medium. Thus, rather than settle for the existence of a practical algorithm that accomplishes some task, we explore the whole range of practical and impractical algorithms that could exist, attempting to establish trade-offs and lower bounds that explain why some problems are inherently more difficult to solve than others. Rather than settle for with algorithms that tolerate a single fault of a type that is deemed likely to occur~ the ultimate aim of our explorations is to provide the best possible multiple fault tolerant algorithms and to provide families of solutions to various problems, so that the best solution for each context is available. Thus we are not satisfied with tolerating a single or a simple fault until we understand how to tolerate more complex varieties, how to determine the most cost effective solution, understand the possible trade-offs that are involved. A particular application has driven much of the research reported here. Many algorithms for maintaining the consistency of distributed data are designed to tolerate the failure of a participating processing element (processor) by blocking, i.e. by holding up the completion of ongoing operations until the failed component is repaired. Such a strategy can obviously interfere with the timely completion of a distributed task. Instead, we study strategies that allow a duster of processors, communicating by messages, to behave like a single highly available processor so that data consistency preserving algorithms are prevented from blocking in the presence of faults. We are not directly concerned with improved Mgorithms for distributed data management. The algorithms studied here should not be compared directly with algorithms for preserving the atomicity of distributed transactions, such as two phase commit algorithms. Instead we wish to make two phase commit and other algorithms

43 more robust by providing systems of components that guarantee the assumptions under which they perform optimally. While studying solutions to this problem, we have discovered that techniques for fault tolerance in real time process control can be quite relevant, especially since we are interested in algorithms that are correct even in the worst case of coincidental multiple failure. Thus we have focussed attention on a paradigm that was introduced as a real time process control problem called the Byzantine Generals Problem ([LSP]). In this paradigm, a source participant (or general) must communicate some message to the other participants. These participants must either all receive and act identically on the same message or all take some default action. What makes the problem interesting, and often impossible, is that the coordination implicit in the problem must be carried out in the presence of faulty components, including possibly the source. The word "Byzantine" was applied to this paradigm by Leslie Lamport to indicate that the behavior of faulty components could be arbitrary and worst case. We treat the Byzantine Generals Problem as a problem of reaching agreement among processors in a distributed system. This kind of agreement problem is similar to but simpler than the problem of atomic commit of a distributed database transaction. The distributed commit problem requires that either all sites managing parts of a distributed database commit the changes prescribed by a transaction or that no site commits the changes. To insure progress, there must also be fault free conditions under which the changes are committed. Moreover, there must be some way for faulty sites to recover and commit changes committed by others and undo changes not committed by others. Within the context of the reasonable assumption that eventually any faulty component crashes and is repaired, a solution to the distributed commit problem must include commit and recovery algorithms that tolerate faults and allow progress in the absence of faults. Atomicity must be insured among correctly behaving components and among faulty components when they have been repaired. But faults may postpone or "block" progress indefinitely. Unlike the distributed commit problem, the agreement or atomic broadcast problem requires atomicity only among the components that have not failed and does not refer explicitly to recovery. However, for the agreement problem faults are not allowed to block progress; the correct components must agree on some action and take that action, independent of the faulty components. Now suppose that we obtained a solution to the agreement problem that can be used by a cluster of processors so that they appear to be a single highly available processor in solutions to the distributed commit problem. Using it we can make any distributed commit solution more reliable by making it less vulnerable to blocking failures of components ([Sch]). Moreover, many algorithms for distributed commit are not designed to tolerate worst case types of faulty behavior. For example, standard two phase commit algorithms tolerate what are called omission faults, but not more Byzantine two faced behavior ([MSF]). Thus, if we can provide ways to prevent or mask such behavior, we have again made distributed commit solutions potentially more reliable.

44 One problem with some agreement algorithms is that the increased reliability may be more expensive. But these algorithms provide an the alternative which is available for those times when it is needed. Another problem with some agreement algorithms is that they are efficient and practical in small clusters but not necessarily feasible in large networks. We elaim~ however, that rather then compare distributed commit and agreement algorithms directly, one should compare the reliability and performance of distributed commit over large networks~ enhanced by agreement over local dusters, with that of distributed commit alone. This approach may still have problems if the duster is to provide the services of a highly available node in some distributed commit protocol. These problems include the detection of cluster failure (no agreement algorithm can tolerate a partition of correctly functioning processors) and recovery after cluster failure ([$85]).

2

The Simple Model

In what follows we shall discuss only a simple version of the problem~ so that we can focus on the inherent difficulties and costs of its solution. It is beyond the scope of this paper to cover the more realistic models for agreement or the history of actual implementations of solutions. Instead, we survey theoretical results obtained in what we call the simple model. Our characterization of the simple model is taken from [$85]. The general context is a network of processors communicating by means of messages over links~ Variants of the simple model may be found in [PSL], [DS], [DFFLS], [FLM], [TPS], [C], and many others. Papers discussing more realistic models include [CASD], [DHSS], and [$86]. In [AGKS] and [GS] the reader can find descriptions of a recent implementation based on some of these theoretical algorithms. In the simple model we make the following assumptions: • Complete connectivity: the network of processors is fully connected. • Perfect communication: messages are never altered or lost by communication links. • Perfect synchronization: one absolutely reliable clock synchronizes all processors. • Isolation: only one protocol is executed at a time (no other concurrent processes can interfere). • Perfect information: the identities of the participants and the time and place of origin of the process under study are common knowledge in advance, as is the exact time required for transmission and processing of each message. (The latter time is called the message delay).

The assumptions of perfect synchronization and perfect information make it possible for algorithms in the simple model to be organized into rounds of message

45 exchange. Each round consists of the sending of messages followed by the receipt and processing of messages and the preparation of messages to be sent at the next round. In this model time is usually measured in rounds. The relevant performance characteristics for an algorithm in this model include time, number of messages required, number of bits per message, and number of faults of various types tolerated. A more detailed examination of performance might also include the size and complexity of a program implementing the algorithm, the amount of information that must be stored by each processor (either temporarily or permanently), and the amount of work performed by each processor in processing its messages.

3

Fault m o d e l s

The introduction of a hierarchy of fault models of increasingly complex and rare type has provided explicit understanding of trade-offs available to the system designer that replace more traditional rules of thumb. We need not describe the goal of fault tolerance as the masking of a single failure, except for bizarre unexpected behavior. Instead we can classify component failures in terms of the behavior experienced and thus in terms of the measures needed to mask their occurrences, whether they be single or coincidental. We now have the tools to express the number and types of failures that can be tolerated. Many distributed database management and other distributed algorithms were originally designed to tolerate failstop failures. In particular, a component is considered "fail-stop" if upon failing that component is guaranteed not only to halt immediately before violating any of its input-output specifications, but also to notify all components directly communicating with it of its failure ([Sch]). We, however, classify the fault rather than the component. We assume that each component has a complete input-output specification that describes its correct behavior. To be faithful to the origin of the term, we say that a fault that causes a component to stop functioning after notifying all its correspondents that it is about to stop is a ]ailstop fault. If the component simply ceases functioning without notification, we call the fault a crash fault. There has been some confusion about these terms, to which we unfortunately have contributed, so we hope that this more precise terminology will be adopted. Many of the a~gorithms that were implicitly designed to tolerate failstop or crash faults can easily be converted to tolerate a much broader class of fault. The term omission was introduced by [MSF] to classify the set of faults in which some action is omitted. That is, if the specification for a component requires a particular action and the component fails to perform this action but otherwise continues to function correctly, then the fault is classified as an omission fault. Note that the fault is classified according to the manner in which the component fails to meet its inputoutput specification. Thus if component A is supposed to send messages X and Z after receipt of message Y, and component A fails to send message X after receiving message Y but does send message Z, then component A is said to have suffered an

46

omission (or output omission) fault. If component B falls to receive message X and behaves as if message X had not arrived, then component B is said to have suffered an input omission fault. Input omission faults are more powerful and harder to tolerate than output omission faults. The class of crash faults is a subset of the class of (output) omission faults, which in turn is a subset of the class of timing faults. Suppose that if component A receives message X, it is supposed to send message Y immediately and to send message Z between 5 and 15 milliseconds thereafter. If, instead, component A receives message X and sends both messages Y and Z immediately, it is said to have suffered an early timing fault. Alternately, if A sends message Z 20 milliseconds after message Y, it is said to have suffered a late timing fault. Input timing faults are defined in analogy with input omission faults. A more general class of faults is those that can be ascribed to the arbitrary behavior of a subcomponent clock. If a component that contains a clock as a subcomponent behaves correctly except for an arbitrary malfunction of its clock, the component is said to have suffered a clock fault. The most general fault class encompasses all possible faults, including arbitrary, even malicious~ behavior. This class is called Byzantine Faults. There is a special subclass of Byzantine faults that has a great deal of practical significance. This is the class of faults that allows for "almost" arbitrary behavior; some authentication or error detection protocol is specifically excluded from corruption. When an algorithm is designed to tolerate any fault that does not corrupt an authentication protocol, the algorithm is referred to as an authenticated algorithm and the class of faults tolerated Byzantine with authentication. To emphasize that worst case faults could corrupt an authentication protocols we sometimes refer to such faults as unauthenticated Byzantine faults.

4

Problems

The specific problems discussed below are all variants on the notion of agreement and require some kind of consistency among the outputs of correct processors. When there is one input (to one processor called the source or originator), we refer to the consistency problem as an agreement problem. When each participant has an input that is to affect all outputs, we refer to the consistency problem as a consensus problem. Each of the variants discussed below has two correctness criteria: (1) output consistency, which may be equality or approximate equality in the presence of uniformly correct behavior on the part of all participants; and (2) progress, which prevents trivial solution by forcing the output to be a function of the input. A third criterion that sometimes allows finer distinctions is (3) termination, which may be specified as synchronous or asynchronous, depending on whether or not all correct processors are to produce their outputs at the same round. The Byzantine Generals ~ problem is a case of simple synchronous agreement. There is one input. All correct processors must (1) agree on identical outputs. And,

47 if the source functions correctly~ then all correct processors must (2) produce the input as output ([LSP]). Implicit in the statement of the problem is the third requirement that all correct processors must (3) produce their outputs at the same round ([DRS]). In the corresponding consensus problem, each processor has its own input. The output consistency criterion is the same as for the Byzantine Generals ~ problem. The usual criterion for progress states that if all inputs to processors are identical, then the input value must be produced as output. Asynchronous agreement and consensus have also been studied. When considering these problems studied in a less synchronous model, we replace the terms synchronous and asynchronous by simultaneous and eventual, respectively. The weak Byzantine Generals problem has the same consistency criterion, but it requires progress only if there are no faults ([L83]). It easily generalizes to four flavors corresponding to the pairs and <simultaneous~ eventual>.

Crusader Agreement represents a weakening of the consistency requirement so that correct processors need produce the same outputs only if the source is correct ([D]). It is not easy to generalize Crusader to a consensus problem, though possible (cf. [ST]). Alternatively~ a problem that is more easily defined in the consensus context is that of approxin~te consensus (often called approximate agreement) [DLPSW, MS]. Here the consistency requirement is that the range of output values be smaller than the range of inputs unless all inputs are identical, in which case the outputs must be equal to the inputs. Corresponding to progress is a requirement that outputs come from the range of the inputs. Many variations on these criteria are possible. Finally, the firing squad problem emphasizes the termination criterion, requiring simultaneous or synchronous termination. One simple formulation of the problem is a variant of weak simultaneous consensus on a binary value. This variant requires that if any input is 1 and if all processors are correct, then all processors output the value 1. Other versions of the problem are beyond the scope of our simple model. Each of the problems described above can be further specified by enumerating the numbers and types of faults that are to be tolerated. For example, one can consider Byzantine agreement, authenticated Byzantine consensus, omission crusa~ler agreement, etc.

5

Results

Results in the simple model are summarized in [F], [SD], and [FLM]. In particular [FLM] contains a unifying proof technique for lower bounds on the number and connectivity of processors required to tolerate unauthenticated Byzantine faults. As a special case, they reprove the well known result that neither Byzantine agreement nor consensus can be achieved if at least one-third of the processors are faulty ([PSL]). This result does not hold in the Authenticated model. Thus, if faults are constrained not to corrupt an authentication protocol (or some error detection scheme), then the

48 one third limitation does not exist. Lower bounds on the number of rounds required for Byzantine agreement (and consensus) can be found in [DRS]. In fact, these lower bounds hold for crash faults as well as for Byzantine faults. Denote by n the number of processors and by F the upper bound on the number of faults to be tolerated. Any algorithm that guarantees simultaneous agreement in the presence of up to F < n - 1 crash faults has scenarios that use at least F + 1 rounds of message exchange; any algorithm that guarantees eventual agreement in the presence of up to F < n - 1 crash faults, has, for each 0 _ f < F, scenarios in which there are at most ,f faults and ] + 2 rounds ofmessage exchange. Agreement protocols tolerant of different types of faults are described in other chapters of this book. Here we focus on a simple protocol that might be used in the application emphasized in our introduction. Assume that we have a small duster of n completely connected processors. For authentications we presume some simple error detection scheme such as combining a processor id with a checksum on the message. Suppose that the clocks of these processors are synchronized. An agreement protocol in which each processor signs and sends each new message received to all its neighbors can be designed to tolerate any number o f omission failures up to a partition of the network. The number of messages required per input can be ( n - 1) 2, and the time required can be n - 1 message delays. A proportionately shorter time is required to tolerate fewer than n - 2 faults. Provided neither authentication nor clock synchronization protocols are corrupted, Byzantine faults can be tolerated at a cost of only twice as many messages and with no additional message delay required. So, at least for this synchronous case, the cost of tolerating any imaginable fault is only double the cost of tolerating only omission failures.

References [AGKS]

H. Aghili, A. Griefer~ R. Kistler, R. Strong, "Highly available communication," Proceedings of the IEEE International Conference on Communication, 1439-1443, Toronto, 1986.

[c]

B. Coan, "A communication-efficient canonical form for fault-tolerant distributed protocols," Proc. 5th ACM Symp. on the Principles of Distributed Computing, Calgary, Canada, August, 1986.

[CASD]

F. Cristian, H. Aghili, R. Strong, and D. Dolev, "Atomic broadcast: from simple message diffusion to Byzantine agreement," proceedings, the 15th Int. Conf. on Fault Tolerant Computing, June 1985.

[D]

D. Dolev, "The Byzantine generals strike again," Journal of Algorithms 3:1, 1982.

49 [DFFLS 1 D. Dolev, M. Fischer, R. Fowler, N. Lynch, and R. Strong, "An efficient algorithm for Byzantine agreement without authentication," Information and Control 52:3,257-274, March 1982. [DHSS]

Do Dolev, J. Halpern, B. Simons, and R. Strong, "Fault Tolerant Clock Synchronization," Proc. 3rd ACM Symp. on the Principles of Distributed Computing, Vancouver, 1984.

[DLPSW] D. Dolev, N. Lynch, S. Pinter, E. Stark, and W. Weihl, "Reaching approximate agreement in the presence of faults," Journal of the ACM 33, 499-516, 1986. [DRS]

D. Dolev, R. Reischuk, and R. Strong, "Early stopping in Byzantine agreement," IBM Research Report RJ3915, June, 1983.

[DS]

D. Dolev and R. Strong, "Authenticated algorithms for Byzantine agreement," SIAM Journal of Computing 12:4, 656-666, 1983.

IF]

M. Fischer, "The consensus problemln unreliable distributed systems," Proc. of the International Conference on Foundations of Computing Theory, Sweden, 1983, see also Yale University Report YALEU/DCS/RR-273, June, 1983.

[FLM]

M. Fischer, N. Lynch, and M. Merritt, "Easy impossibility proofs for distributed consensus problems~" Distributed Computing 1, 26-39, 1986.

[Gs]

A. Griefer and R. Strong, "DCF: Distributed communication with fault tolerance," Proc. 7th ACM Symp. on the Principles of Distributed Computing, Vancouver, 1988.

[LSP]

L. Lamport, R. Shostak, and M. Pease, "The Byzantine generals problem," ACM TOPLAS 4:3, 382-401, July, 1982.

[L83]

L. Lamport, "The weak Byzantine generals problem," JACM 30, 668-676, 1983.

[MS]

S. Mahaney and F. Schneider, "Inexact agreement: accuracy, precision, and graceful degradation," Proc. 4th ACM Syrup. on the Principles of Distributed Computing, Minaki, 1985.

[MSF]

C. Mohan, R. Strong, S. Finkelstein, "Method for distributed commit and recovery using Byzantine agreement within clusters of processors," Proc. 2nd ACM Syrup. on the Principles of Distributed Computing, Montreal, 1983.

[PSL]

M. Pease, R. Shostak, and L. Lamport, "Reaching agreement in the presence of faults," JACM 27:2, 228-234, 1980.

[Sch]

F. Schneider, "Byzantine generals in action: implementing fail-stop processors," ACM TOCS 2:2, 146-154, May, 1984.

50

[ST]

T. Srikanth and S. Toueg, "Simulating authenticated broadcasts to derive simple fault-tolerant algorithms," Distributed Computing, 2:2, 80-94, August 1987.

[SD]

R. Strong and D. Dolev, "Byzantine agreement," Digest of Papers from Spring COMPCON, IEEE Computer Society Press, 1983, see also IBM Research Report RJ3714, December, 1982.

[s85]

R. Strong, "Problems in fault tolerant distributed systems," Digest of Papers from Spring COMPCON, IEEE Computer Society Press, 1985, see also IBM Research Report RJ4220, December, 1984.

[s86]

R. Strong, "Problems in maintaining agreement," Proc. 5th IEEE Symp. on Reliability in Distributed Software and Database Systems, 20-27, Los Angeles, January, 1986.

[TPS]

S. Toueg, K. Perry, and T. Srikanth, "Fast distributed agreement," SIAM 3. Comp. 16, 445-457, 1987.

ATOMIC BROADCAST IN A REAL-TIME ENVIRONMENT Flaviu Cristian

Danny Dolev Ray Strong IBM Research Almaden Research Center 650 Harry Road San Jose, CA 95120

Houtan Aghili*

Abstract

This paper presents a model for zeal-time distributed systems that is intermediate in complexity between the simple, perfectly synchronous model in which there are rounds of communication exchange among processors in a completely connected network and an asynchronous model in which there is no reasonable upper bound on the time required for transmission and processing of messages. In this model algorithms are described for atomic broadcast that can be used to update synchronous replicated storage, a distributed storage that displays the same contents at every correct processor as of any clock time. The algorithms are all based on a simple communication paradigm and differ only in the additional checking required to tolerate different classes of failures.

1

Introduction

The fundamental characteristic of a real-time distributed operating system is a known upper bound on the time required to transmit a message from one processor to another and to process the message at the receiver~ processing being assumed to include the preparation of any responsive messages. This bound in turn provides a bound on the time required to propagate information throughout the system, provided there is a known b o u n d on the n u m b e r of processors in the network and provided the network remains sufficiently connected. In such an environment, small numbers of failures can be tolerated by a distributed system that manages to provide logically synchronous replicated storage. The use of synchronous replicated storage considerably simplifies the programming of distributed processes, since a programmer is not confronted with inconsistencies *H. Aghili is now with the IBM T. J. Watson Resenrc~ Center, Hawthorne, New York.

52 among local knowledge states that can result from random communication delays or faulty processors and links. Moreover it allows a programmer to assume a shared clock in addition to the assumed shared memory. It is easy to adapt known concurrent programming paradigms for shared storage environments to distributed environments that provide the abstraction of a synchronous replicated storage. Several examples of such adaptations are given in ILl. The objective of this paper is to discuss fault tolerant protocols for updating synchronous replicated storage in an arbitrary point-to-point network and to contrast these protocols and their performance characteristics with protocols that might be obtained by a straightforward lifting from a simple model based on rounds of communication in a completely connected network [DS]. The real-time model and protocols presented here are based on work that appeared in [CASD]. In our model, processor clocks are synchronized to within some given precision. To implement the state machine approach, global system state information is replicated in all processors. Updates to this global system state may originate at any processor in the network. These updates are disseminated by means of an atomic broadcast protocol, so that all correct processors have identical views of the global state at identical local clock times. An a t o m i c b r o a d c a s t protocol is a communication protocol that possesses a fixed termination time A and satisfies: atomicityevery update whose broadcast is initiated by some processor at time T on its clock is either delivered at each correct processor at local clock time T + A or is never delivered at any correct processor, o r d e r - - all updates delivered at correct processors are delivered in the same order at each correct processor, and t e r m i n a t i o n - - every update whose broadcast is initiated by a correct processor at time T on its clock is delivered at all correct processors.

The Highly Available System project at the IBM Almaden Research Center, designed an atomic broadcast protocol to update replicated system directories and reach agreement on the failure and recovery of system components (v. [Or], [GS]). Much previous work on atomic broadcast protocols has been performed within the Byzantine Generals framework [LSP] (v. [F],[SD] for surveys of this work). This framework includes guaranteed communication in a completely connected network of perfectly synchronized processors that communicate in synchronous rounds of information exchange. This framework is called the s i m p l e model. In the simple model, processors send messages at the beginning of each round, and every processor has time to receive and process all messages sent to it during the round before the end of the round. Note that a round must have a time duration as great as the worst case delay in transmission and processing from one end of the network to the other. In the real-time model networks have arbitrary topology and are subject to link as well as processor failures. Moreover, processors may respond to messages immediately

53 rather than wait for the beginning of the next round. The real-time model has only approximately synchronized docks, but retains the upper bound on message transmission and processing time. Atomic broadcast protocols in the real-time model are not limited by any structure of rounds, S° they are generally faster and more efficient than protocols based on rounds. Indeed, a straightforward translation of a round based protocol (e.g. [DS]) into our model would require that routing be used to achieve full ~logical" connectivity among processors and that each round include not only the worst case time for sending a message between any two correct processors, but also an extra delay corresponding to the worst case duration between the end of a round on one processor clock and the end of the same round on another processor clock. Such protocols clearly send more messages and take longer than necessary.

2

Failure C l a s s i f i c a t i o n

The real time model is composed of components called processors and links. A set of input events and a set of output actions are associated with each component. Included in the output actions of processors are the set of possible message transmissions. Likewise, included in the set of input events are the set of possible message receipts. (For purposes of this failure classification, messages are not decomposed into constituent bits.) In addition to message receipt, the passage of a specific time duration also constitutes an input event for a processor. Input and output events for links are analogous. Each component is assumed to have an input-output specification describing its correct response (output) in relation to a history of previous inputs and outputs. For exampte~ a link connecting processor s to processor r is specified to deliver a message sent by s to r at some tin/e between the time s sent the message and a fixed number of time units later. Any output of a correct component depends only on its history of previous inputs and outputs and is consistent with its specification. A component specification prescribes both the output that should occur in response to a sequence of input events and the real-time interval within which this output should occur. A component failure occurs when a component does not behave in the manner specified. An omission failure occurs when, in response to an input event sequence, a component never gives the specified output. A timing failure occurs when, in response to a trigger event sequence, a component either omits the specified output or gives it too early or too late. A Byzantine failure [LSP] occurs when a component does not behave in the manner specified: either no output occurs, or the output is outside the real-time interval specified, or some output different from the one specified occurs. An important subclass of Byzantine failures is that for which any resulting corruption of output messages is detectable by message authentication protocols. These failures are called authentication-detectable Byzantine failures. Error detecting codes [PW] and public-key cryptosystems based on digital signatures [RSA] are two examples of well-known authentication techniques.

54

A processor crash, a link breakdown, a processor that occasionally does not forward a message that it should, and a link that occasionally loses messages are examples of omission failures. An excessive message transmission or processing delay due to a processor or network overload is an example of a late timing failure. Another example of a late timing failure is the delivery of messages out of order by a link specified to deliver them in first-in first-out order. When some coordinated action is taken too soon by a processor (perhaps because of a faulty internal timer), an early timing failure occurs. A message alteration by a processor or a link (because of a random fault) is an example of a Byzantine falhre that is neither an omission nor a timing failure. If the authentication protocol employed enables the receiver of the message to detect the alteration, then the failure is an authentication-detectable Byzantine failure. Crash failures are a proper subclass of omission failures (a crash failure occurs when a component systematically omits all outputs from some time on). Omission failures are a proper subclass of timing failures. Timing failures are a proper subclass of authentication-detectable Byzantine failures. Finally, authentication-detectable Byzantine failures are a proper subclass of the class of all possible failures, the Byzantine failures. The nested nature of the failure classes defined above makes it easy to compare "the power" of fault-tolerant protocols: a protocol A that solves some problem is "less fault-tolerant" than a protocol B which solves the same problem if A tolerates only a subclass of the failures that B tolerates. Observe that a failure cannot be classified without reference to a component specification. Moreover, the type of failure depends on the decomposition into components: if one component is made up of others, then a failure of one type in one of its constituent components can lead to a failure of another type in the containing component. For example, a clock on which the "time" never changes is an example of a crash failure. If that clock is part of a processor that is specified to associate different timestamps with different replicated synchronous storage updates, then the processor may be classed as experiencing a Byzantine failure. In our decomposition of a distributed system into processors and finks, neither type of component is considered part of the other. Also, when considering output behavior, we do not decompose messages, so a message is either correct or incorrect, as a whole. With' these conventions we can classify failures unambiguously. We are not concerned with directly tolerating or handling the failures experienced by such sub-components as clocks. We discuss fault tolerance in terms of the survival and correct functioning of processors that meet their specifications in an environmerit in which some other processors and some finks may not meet theirs (usually because they contain faulty sub-components). Thus when we speak of tolerating omission failures, we mean tolerating omission failures on the part of other processors or links, not tolerating omission failures on the part of sub-components like timers or clocks that might cause much worse behavior on the part of their containing processors.

55

3

The

Model

The real-time model consists of a connected network G of arbitrary topology, with n processors and m point-to-point links. Processors that share a link are called neighbors. Each processor p possesses a clock Cr that reads Cp(t) at real time t. We witl use upper case letters for clock times and lower case letters for real times. The model is characterized by the following assumptions. 1. All processor names in G are distinct and there is a total order on processor names. 2. The docks of correct processors are monotone increasing functions of real time and the resolution of processor docks is fine enough, so that separate clock readings yield different values (this will ensure that no correct processor issues the same timestamp twice). 3. The clocks of correct processors are approximately synchronized within a known, constant, maximum deviation e. That is, for any correct processors p and q, and for any real time t, I G ( t ) - o,(e)l < (Clock synchronization protocols tolerant of omission, late timing, and authentication detectable Byzantine failures are presented in [CAS,DHSS]. For a survey see [Sc], and "An Overview of Clock Synchronization" in this book.) 4. For the message types used in our protocols, transmission and processing delays (as measured on any correct processor's dock) are bounded by a constant 6. This assumption can be stated formally as follows. Let p and q be two correct neighbors linked by a correct link and let r be any correct processor. If p sends a message m to q at real time u, and q receives and processes m at real time v, then 0 < - c,(•) < 6. (We assume that processing time is negligible so 6 covers the interval of time from the transmission of a message to the time of subsequent transmission of any' message resulting from processing the message at a receiving neighbor.) 5. The underlying operating system provides a "schedule(A,T,p)" command that allows a task A to be scheduled for execution at time T with input parameters p. An invocation of "schedule(A,T, p)" at a local time U > T has no effect, and multiple invocations of "schedule(A,T,p)" have the same effect as a single invocation.

56

4

Protocols

We consider three properly nested failure classes: (1) omission failures, (2) timing failures, and (3) authentication-detectable Byzantine failures. For each one of these classes, we present an atomic broadcast protocol that tolerates up to 7r processor and up to A link failures in that class, provided these failures do not disconnect G. The termination time of each protocol is computed as a function of the failure class tolerated, of the rr and )t parameters, of the known constants ~i and e, and of the largest diameter d of a surviving communication network G - F, for all possible subnetworks F containing up to ~r processors and ,~ links. All protocols are based on a common communication technique called information diffusion: (1) when a correct processor learns a piece of information of the appropriate type, it propagates the information to its neighbors by sending messages to them, and (2) if a correct neighbor does not already know that piece of information, it in turn propagates the information to its neighbors by sending them messages. This ensures that, in the absence of network partitions, information diffuses throughout the network to all correct processors. A possible optimization is to eliminate messages to neighbors that are already known to possess the information.

5

First P r o t o c o l : T o l e r a n c e o f O m i s s i o n Failures

Each message diffused by our first protocol carries its initiation time (or timestamp) T, the name of the source processor s, and a replicated storage update ~r. Each atomic broadcast is uniquely identified by its timestamp T and its initiator's name s (by assumptions 1,2, and 3). As diffused messages are received by (the atomic broadcast layer) of a processor, these are stored in a local history log H until delivery (to the local synchronous storage layer). The order property required of atomic broadcasts is achieved by letting each processor deliver the updates it receives in the order of their timestamp, by ordering the delivery of updates with identical timestamps in increasing order of their initiatot's name, and by ensuring that no correct processor begins the delivery of updates with timestamp T before it is certain that it has received all updates timestamped T that it may ever have to deliver (to satisfy the atomicity requirement). For omission failures, the local time by which a processor is certain it has received copies of each message'timestamped T that could have been received by some correct processor is T + ~r~ + d~ + e. We call this time the delive~ deadline for updates with timestamp T. The intuition behind this deadline is as follows. The term ~r~ is the worst case delay between the initiation of a broadcast (7,8, ~r) and the moment a first correct processor r learns of that broadcast. It corresponds to the case in which the broadcast source s is a faulty processor, between s and r there is a path of r faulty processors which all forward just one message (T, s, a) on one outgoing link, and each of these messages experiences a maximum delay of 8 clock time units. The term d~ is the time sufficient for r to diffuse information about the broadcast (7', 8, a) to any correct

57 processor p in the surviving network. The last term ensures that any update accepted for delivery by a correct processor q whose clock is in advance of the sender's clock is also accepted by a correct processor p whose clock is behind the sender's clock. We assume all processors know the protocol termination time A -- ~r5 + d5 + e. To keep the number of messages needed for diffusing an update finite, each processor p that receives a message (T, 8, a) relay8 the message (to all its neighbors except the one that sent the message) only if it receives (T, s, a) ]or the first time. Ifp inserts all received messages in its local history H (and never removes them), p can easily test whether a newly arrived message m was or was not seen before by evaluating the test m E H. We call this test the "deja vu" acceptance test for the message m. The main drawback of the "deja vu" solution described above is that it causes local histories to grow arbitrarily large. To keep the size of H bounded~ a history garbage collection rule is needed. A possible solution is to remove from H a message (T, 8, ~r) as soon as the deadline T ÷ A for delivering a passes on the local clock. However, a simple-minded application of the above garbage-collection rule would not be sufficient for ensuring that locM histories remain bounded, since it is possible that copies of a message (T, s, a) continue to be received by a correct processor p after the delivery deadline T + A has passed on p's clock. Such duplicates would then pass the "deja vu" acceptance test and would be inserted again in t'he history of p. Since such "residual" duplicates will never be delivered (see assumption 5), they will cause p's history to grow without bound. To prevent such residual messages from accumulating in local histories, we introduce a "late message" acceptance test. This test discards a message (T, s~ a) if it arrives at a local time U past the delivery deadline T ÷ A, i.e. if U > T + A. The "deja vu" and "late message" acceptance tests ensure together that updates require only a finite number of messages and that local histories stay bounded (provided~ of course, processors broadcast only a bounded number of updates per time unit). A detailed description of our first atomic broadcast protocol is given in Figures 1, 2, and 3. Each processor has three concurrent tasks: a Start broadcast task (Figure 1) that initiates an atomic broadcast, a Relay task (Figure 2) that forwards at0rrfic broadcast messages to neighbors, and an End task (Figure 3) that delivers broadcast updates (to be performed on the synchronous replicated storage). In what follows we refer to line j of figure i as (i.j). A user of the atomic broadcast layer triggers the broadcast of an update a of some type Text by sending it to the local Start task. The command "take" is used by this task to take a as input (1.4). The broadcast of a is identified by the local time T at which a is received (1.4) and the identity of the sending processor, obtained by invoking the function "myid" (1.5). This function returns different processor identifiers when invoked on distinct processors. The broadcast of a is initiated by invoking the "send-all ~ command, which sends messages on all outgoing links (1.5). We do not assume that this command is atomic with respect to failures: a processor failure can prevent messages from being sent on some links. The fact that the broadcast of a has been initiated is then recorded in a local history variable H shared by all broadcast layer tasks:

58

1 task Start; 2 const Delta = (~r + d)8 + e; 3 vat T: Time; a: Text; s: Processor-Name; 4 cycle take(a); T ~- clock; 5 send-all(T, myid, a); 6 H ,--- H @ (T, myid, a); 7 schedule(End,T + A, T); 8 endcycle ; Figure 1: Start Task of the first protocol

1 task Relay; 2 const A = (lr + d)6 + e; 3 vat U, T: Time; a: Text; s: Processor-Name; 4 cycle receive((T, s, a), I); U ~-- clock; . 5 [g > T + A: "late message" iterate]; 6 IT e dom(tt)grs e dom(II(W)): "deja vu" iterate]; 7 send-all-but(l, (T, s, a)); 8 H ~ H @ (T, s, a); 9 echedule(End,T + A, T); 10 endcycle ; Figure 2: Relay Task of the first protocol v a t / / : Time ~ ( Processor-Name ~ Text ). We assume t h a t / / is initialized to the empty function at processor start. The variable//keeps track of ongoing broadcasts by associating with instants T in Time a function H(T) (of type Processor-Name --~ Text). The domain of H(T) consists of names of processors that have initiated atomic broadcasts at local time T. For each such processor p, H(T)(p) is the update broadcast by p at T. We define the following two operators on histories. The update "@ " of a history H by a message (T, s, a) yields a (longer) history, denoted H $ (T, s, a), that contains all the facts in H, plus the fact that s has broadcast a at local time T. The deletion '*V' of some instant T from a history H yields a (shorter) history, denoted H \ T, which does not contain T in its domain, i.e. everything about the broadcasts that were initiated at time T is deleted. Once the history H is updated (1.6), the End task is scheduled to start at local clock time T + A to deliver the update a (1.7). The Relay task uses the command "receive" to receive messages formatted as (T, s, a) from neighbors (2.4). In describing this task, we use double quotes to delimit comments and the syntactic construct ~[B: iterate]" to mean "if Boolean condition B is true, then terminate the current iteration and begin the next iteration". After a message is received (2.4), the parameter t contains the identity of the link over which

59

1 2 3 4

task End(T:Time); vat p: Processor-Name; val: Processor-Name ~ Text; vale--- H(T); while dom(val) ~ {}

5

dop

6 7 8 9

deliver(val(p)); val ~ val \ p; od;

H*-H\T; Figure 3: End Task

the message arrived. If the message is a duplicate of a message that was already received (2.6) or delivered (2.5) then it is discarded. A message is accepted if it passes the acceptance tests of the Relay task. If (T, s,~) is accepted (i.e. passes the "late message '~ (2.5) and "deja vu" (2.6) tests), theh it is relayed on all outgoing links except l using the command "send-all-but" (2.7), it is inserted in the history variable (2.8)~ and the End task is scheduled to start at local time T + A to deliver t h e received update (2.9). The End task (Figure 3) starts at clock time T + A to deliver updates timestamped T in increasing order of their sender's identifier ((3.5)-(3.8)) and to delete from the local history H everything about broadcasts initiated at time T (3.9). A proof of correctness for the protocol may be found in [CASD].

6

S e c o n d Protocol: Tolerance of T i m i n g Failures

The first protocol is not tolerant of timing failures because there is a fixed clock time interval, independent of the number of faulty processors, during which a message is unconditionally accepted by a correct processor. This creates a real time "window" during which a message might be "too late" for some (early) correct processors and "in time" for other (late) correct processors. To achieve atomicity in the presence of timing failures, we must ensure that if a first correct processor p accepts a message m, then all correct neighbors to which p relays m also accept m. A neighbor q does not know whether the message source p is correct or not. However, if p is correct, q must accept m if the information stored in it tells q that the clock time delay between p and q is at least - e (as when f s clock is very close to being e time units behind q's and the message propagation delay between p and q is very close to being 0) or at most ~ + e (as when p~s clock is very close to being e time units in advance of q~s and the message from p to q takes ~ time units). To be able to evaluate the time a message spends between two neighbors, we store in the message the number h of hops traversed by it. This leads to the following timeliness acceptance test: a correct processor q accepts a message timestamped T with hop count h if it receives it at a

60 1 task Start; const A = r:(8 + e) + d~ + e; vat T: Time; a: Text; s: Processor-Name; cycle take(a); T ~ clock; send-all(T, myid, 1, a); //*--- H ® (7", myid, o'); ~chedule(End,T + A, T); endeycle ; Figure 4: Start Task of the second protocol local time U such that: T-

he < U < T + h(b +e).

Since by hypothesis there can be at most a path of Ir faulty processors from a (faulty) sender s to a first correct processor p, and the message accepted must pass the above test at p, it follows that a message can spend at most lr(b+e) clock time units in the network before being accepted by a first correct processor. From that moment, it needs at most d~ clock time units to reach all other correct processors. Given the e uncertainty on clock synchrony the termination time of the second protocol is therefore: A = 7r(~ + e) + d~ + e. The Start task of the second protocol (Figure 4) is identical to that of the first except for the addition of a hop count (initially set to 1 (4.5)) to all messages. In addition to the tests used for providing tolerance of omission failures (5.7, 5.8), the Relay task of the second protocol (Figure 5) also contains the timeliness tests discussed above (5.5, 5.6). The hop count h carried by messages is incremented (5.9) every time a message is relayed. The End task of the second protocol is identical to that of the first protocol. A proof of correctness for the protocol together with an indication of why it will not tolerate Byzantine faults may be found in [CASD].

7

Tolerance of A u t h e n t i c a t i o n D e t e c t a b l e Byzantine Failures

A "Byzantine" processor can confuse a network of correct processors by forwarding appropriately altered messages on behalf of correct processors at appropriately chosen moments. One way of preventing this phenomenon is to authenticate the messages exchanged by processors during a broadcast [DS], [LSP], so that messages corrupted by "Byzantine" processors can be recognized and discarded by correct processors. In this way~ we are able to handle authentication-detectable Byzantine failures in a

61 1 task Relay; const A = ~r(8 + e) + d6 + e; 2 vat U, T: Time; a: Text; s: Processor-Name; h: Integer; 3 cycle receive((T, s, h, a), l); U ~ clock; 4 [U < t - he: "too early" iterate]; 5 [U > T + h(6 + e): "too late" iterate]; 6 7 [U > T + A: "late message" iterate]; IT E dom(H)&s E dom(H(T)): "deja vu" iterate]; 8 9 send-all-but(l, (T, s, h + 1, a)); H ~- g @ (T, s, a); 10 11 schedule(End,T + A, T); 12 endcyele ; Figure 5: Relay Task of the second protocol manner similar to the way we handle timing failures. Ignoring (for simplicity) the increase in message processing time due to authentication, we set the termination time of the third protocol to be the same as the termination time of the second protocol: A = ~r(6 + e) + d6 + e. The detailed implementation of our third protocol is given in Figures 6-12. We assume that each processor p possesses a signature function ~r, which, for any string of characters z, generates a string of characters y = @p(z) (called the signature ofp on z). Every processor q knows the names of all other processors in the communication network, and for each p E G, q has access to an authentication predicate O(z,p, y) which yields true if and only if y = ~p(z). We assume that if processor q receives a message (z,p, y) from processor p, and O(z,p, y) is true, then p actually sent that message to q. (If the authentication predicate fails to detect message forgery, then our last protocol can no longer guarantee atomicity in the presence of Byzantine failures.) The proper selection of the @r(m) and O functions for a given environment depends on the likely cause of message corruption. If the source of message corruption is unintentional (e.g., transmission errors due to random noise on a link or hardware malfunction), then simple signature and authentication functions like the error detecting/correcting codes studied in [PW] are appropriate. If the source of message corruption is intentional, e.g., an act of sabotage, then more elaborate authentication schemes like those discussed in [RSA] should be used. In any case there is always a small but non-zero probability that a corrupted message will be accepted as authentic. In practice this probability can be reduced to acceptably small levels by choosing signature and authentication functions appropriate for the (adverse) environment in which a system is intended to operate. We implement message authentication by using three procedures "sign", "cosign", and "authenticate", and a new signed message data type "Smsg" (Figure 6). These are all described in a Pascal-like language supporting recursive type declaration and the exception mechanism of [C].

62

1 type Smsg = record case tag: (first,relayed) o] first: (timestamp: Time; update: Text); relayed: (incoming: Smsg); procid: Processor-Name; signature: string; end; Figure 6: The Signed-Message data type 1 procedure sign(in T: Time; a: Text; out z: Smsg); 2 begin z.iag ~ ~first'; z.timestamp ~ T; 3 z.update ~-- a; z.procid :~-- myid; 4 z.signature ~- ~,,yid(z.tag, T, ~r); 5 end; Figure 7: The sign procedure A signed message (of type Smsg) that has been signed by k processors Pl,--.,P~ has the structure (relayed,

... (relayed, (first, T, a,p,, sl),p~, s2), ... Pk, sk)

where T and a are the timestamp and update inserted by the message source pl, and si, i E 1, ..., k, are the signatures appended by processors pl that have accepted the message. The sign procedure (Figure 7) is invoked by the originator of a broadcast (T, s, tr) to produce a message z containing the originator's signature. The co-sign procedure (Figure 8) is invoked by a processor r which forwards an incoming message z already signed by other processors; it yields a new message y with r's signature appended to the list of signatures on z. The authenticate procedure (Figure 9) verifies the authenticity of an incoming message. If no alteration of the original message content is detectable, it returns the timestamp and original update of the message as well as the sequence S of processor names that have signed the message. The identity of the initiator is the first element of the sequence, denoted first(S), and the

1 procedure co-sign(in z:Smsg; out y: Smsg); 2 begin y.tag ~ ~relayed~; y.incomin9 ~- z; 3 y.proeid :*-- myid; y.signature :~--- ffm~ia(y.tag, 4 end; Figure 8: The co-sign procedure

Z);

63

1 procedure authenticate(in z:Smsg; out T:Time; a:Text, S:Sequence-of-Processor-Name) [forged]; begin 2 [z.tag ='first'Sz -~®((z.tag, z.timestamp, z.val), z.procid,z.signature): signal forged]; [z.tag ='relayed'& -~O((z.tag, z.incoming), z.procid, z.signature): signal forged]; 4 if z.tag ='first' then T ~- z.timesiamp; a ~- z.updale; S ~-- 5 6 else authenticate(z.incoming, T, a, S) [forged: signal forged] fi; S ~ append(S, z.procid); 7 end; 8 Figure 9: The authenticate procedure

1 task Start; 2 eonst A = ~r(~ + e) + d~ + e; 3 vat T: Time; a: Text; z: Smsg; 4 cycle take(o); T ~- clock; sign(T, a, z); 5 send-all(z); 6 H ~ H @ (T, myid, a); 7 schedule (End,T + A, T); 8 endcycle ; 9 Figure 10: Start Task of the third protocol number ,of hops (i.e., the number of intermediate links) traversed by the message is the length of the sequence, denoted ISI. If the message is determined to be corrupted, the "forged" exception is signalled. Except for the change concerning the authentication of messages, the structure of the Start task of the third protocol (Figure 10) is the same as that of the second protocol. In order to handle the case in which a faulty processor broadcasts several updates with the same timestamp, the type of the history variable H is changed to var H: Time ---* (Processor-Name ---, (Text U @)), where the symbol 0 denotes a "null" update (0 ~Text). Specifically, if a processor receives several distinct updates with the same broadcast identifier, it associates the null update with that broadcast. Thus, a null update in the history is an indication of a faulty sender. The Relay task of the third protocol (Figure 11) works as follows. Upon receipt of a message (11.4), the message is checked for authenticity (11.5) and if corrupted,

64

1 task Relay; const A = 1r(~ + e) + d8 + e.; 2 3 vat T, U: Time; a : Text; s: Processor-Name; z: Smsg; S: Sequence-of-Processor-Name; cycle receive(z, l); U *---clock; 4 5 authenticate(z, T, a, S) [forged: iterate]; 6 [duplicates(S): "duplicate signatures" iterate]; 7 [U < T -[S[e: "too early" iterate]; 8 [U _> T q-IS[(~ q-e): "too late" iterate]; 9 [U > T + A: "late message" iterate]; 10 s ~ first(S); 11 if T e dom(lt)&s e dom(II(T) then 12 [a = H(T)(s): "deja vu" iterate]; 13 [H(T)(s) = 0: "faulty sender"iterate]; 14 H(T)(s) ~ 8; 15 else 16 H ~ H @ (T, s, o'); 17 schedule (End,T + A, T); 18

19 20 21

cosign(z, y); send-all-but(l, y); endcycle ; Figure 11: Relay Task of the third protocol

65 1 task End(T:Time); vat p: Processor-Name; val: Processor-Name --~ Text; val ~- H(T); while dom(vaI) # 0 dop , -

min(dom(aO);

if val(p) ¢

then deliverO,

val ~-- val \ p;

od; TI ,-- t t \ T ;

Figure 12: End Task of the third protocol the message is discarded. Then, the sequence of signatures of the processors that have accepted the message is examined to ensure that there are no duplicates; if there are any duplicate signatures, the message is discarded (11.6). Since processor signatures are authenticated, the number of signatures ISI on a message can be trusted and can be used as a hop count in determining the timeliness of the message (11.7, 11.8). No confusions such as those illustrated in the previous counter example can occur unless the authentication scheme is compromised. If the incoming message is authentic, has no duplicate signatures, and is timely, then the history variable H is examined to determine whether the message is the first of a new broadcast (11.11). If this is the case, the history variable H is updated with the information that the sender s = first(S) has sent update a at time T (11.16), the End task is scheduled to start processing and possibly delivering the received update at (local clock) time T + A (11.17), and the received message is cosigned and forwarded (11.19, 11.20). If the received update a has already been recorded i n / / (because it was received via an alternate path), it is discarded (11.12). If a is a second update for a broadcast identified (T, s), then the sender must be faulty. This fact is recorded by setting H(T)(s) to the null update (11.14). The message is then cosigned and forwarded so that other correct processors also learn the sender's failure (11.19, 11.20). Finally, if a is associated with a broadcast identifier to which H has already associated the null update (i.e., it is already known that the originator of the broadcast (t, s) is faulty), then the received update is simply discarded (11.13). The End task (Figure 12) delivers at local time T + A all updates broadcast correctly at time T. If exactly one update has been accepted for a broadcast initiated at clock time T, then that update is delivered (12.6), otherwise no update is delivered. In either ease, the updates associated with broadcasts (T, s), for all processors s, are deleted from H (12.9) to ensure H stays bounded.

66

8

Performance: Messages and Termination Time

In the absence of failures, the initiator s of an atomic broadcast sends d, messages to its neighbors, where d, denotes the degree of s (i.e. the number of its adjacent links). Each processor q ¢ s that receives a message from a processor p sends dq - 1 messages to all its neighbors (except p). Since the sum of all node degrees of a network is twice the number of network links, it follows that each atomic broadcast costs 2m - (n - 1) messages: one message for each link of a spanning tree in G and two messages in each direction for all the other links in G. For example, an atomic broadcast among 8 processors arranged in a 3-dimensional cube requires 17 messages in the absence of failures. The message cost of a diffusion based atomic broadcast protocol compares f~vorably to that of a round-based Byzantine agreement protocol designed for a fully connected network. To achieve the requirement that in each round each processor communicates with each other processor, if the underlying physical network is not fully connected, a message routing service can be used to implement the "logically" fully connected network required by the round structure. The "logical" messages sent by processors are then implemented as sequences of (one-hop) messages sent among neighbors, using some message forwarding technique. The full connectivity required by a round structure has its cost: some of the messages sent in each round will be redundant. Indeed, if a ~'logical'~ message has to be sent from a processor s to a non-neighbor processor r, and p is the neighbor of s on the path to r selected by the message routing algorithm used, then the message s sends to p to be forwarded to q is redundant with the message that s sends to p for direct consumption. For the example of 8 processors arranged in a 3-dimensional cube, a round of logical messages sent by one processor to the 7 others costs 12 (one-hop) messages. Thus, for Ir > 1, a round based agreement protocol tolerant of timing or authentication-detectable Byzantine failures sends in the absence of failures at least 12 + 7.12 = 216 messages, compared to the 17 messages needed by a diffusion based protocol for any 7r > 1. The termination time for an atomic broadcast depends on the network topology and on the class of failures to be tolerated. In the absence of information about the network topology except that the number of processors is bounded above by n, n - 1 can be taken to be an upper bound on ~"+ d. Clock synchronization algorithms which can provide an e close to d8 are investigated in [CAS,DHSS]. For simplicity, we assume here an e of (~r + d)6. Thus, for omission failures, the termination time of an atomic bro~lcast is linear in n : A = 2(7r + d)6 is bounded above by 2(n - 1)6. For timing and Byzantine failures, the termination time is proportional to the product of the number of processors and the number of processor failures to be tolerated: A = (Tr + 2)(~r + d)8 is bounded above by (~r + 2)(n - 1)6. As a numerical example, consider the case of 8 processors arranged in some arbitrary way to form a network. Assume that the link delay bound ~ is 0.01 seconds and that we want to tolerate up to two processor failures. The termination time for omission failures is 0.14 seconds, and for timing (or authentication-detectable Byzantine) failures is 0.28 seconds. If more information about network topology is available, then a better expression can be

67 computed for the network diffusion time db. Note that the expression ~r+d corresponds to a worst case path consisting of ~r hops between faulty processors followed by d hops along a shortest path in the surviving network of correct processors and links. For example, if the eight processors above are arranged in a 3-dimensional cube and we need tolerate no link failures, the approximate termination times for omission and timing (or authentication-detectable Byzantine) failures are cut to 0.10 and 0.20 seconds respectively. This is because lr + d is bounded above by 5: if the two faulty processors are adjacent, then the diameter o f the surviving network is at most 3; if they are not adjacent, the diameter can be 4, but 2 faulty processors cannot be encountered on a path before a correct processor is encountered. We now show that our protocols dominate round based protocols in speed. A straightforward translation into our system model of the algorithms designed for the simpler round-based model (fully connected network, exactly synchronized clocks) would require that each round include not only the worst case time for sending a message from any correct processor to any other, but also an extra delay corresponding to the worst case duration between the end of a round on one processor clock and the end of the same round on another processor clock. Thus, the length of round is at least d/~ + e (in our example, d5 + (~r + d)5) clock time units. To tolerate 7r failures, a round based protocol needs at least ~r+ 1 rounds, that is at least (~"+ 1)(~r + 2d)~ time units. This time is always equal or greater than the termination time (~r+2)(lr+d)$ of a diffusion based protocol (with equality for a fully connected surviving network with d = 1). For example, to atomically broadcast in a 3-dimensional cube with $ = 0.01 seconds despite up to two timing or authentication-detectable Byzantine fa~ures, a round based protocol needs 0.3 seconds, compared to the 0.2 seconds sufficient for a diffusion based protocol.

9

Conclusion

This paper presented an investigation of the atomic broadcast problem in a real-time model, proposed a classification of the failures observable in distributed systems, described three protocols for atomic broadcast in systems with bounded transmission delays and no partition failures, and has discussed and contrasted their performance with that of round based protocols designed for a simpler model. Atomic broadcast simplifies the design of distributed fault-tolerant programs by enabling correct processes to access global state information in synchronous replicated storage. The beauty of this notion is that it reduces distributed programming to "shared storage" programming without having a single point of system failure. In the Highly Available Systems prototype, we use synchronous replicated storage to store crucial system configuration information that must remain available despite arbitrary numbers of processor crashes. Our practical programming experience with synchronous replicated storage indicates that it leads to very simple and elegant decentra~zed program structures. Because of its nested structure, the proposed failure classification provides a good

68 basis for comparing the "power" of various published fault-tolerant distributed protocols. One of the most frustrating aspects to the reader of papers that present fault-tolerant distributed protocols is that the class of failures tolerated is rarely precisely specified. This makes the comparison of different protocols for solving a given problem difficult. We feel that the adoption of the failure classification proposed in this paper as a standard would simplify the task of comparing the power of various fault-tolerant distributed algorithms. The three protocols presented above share the same specification have the same diffusion-based structure. They differ in the classes of failures tolerated, ranging from omission failures, to authentication-detectable Byzantine failures. Clearly, the complexity increases as more failures are tolerated, but the complexity of the final protocol that handles authentication-detectable Byzantine failures is not orders of magnitude greater than that of the initial protocol. A variant of this protocol (which uses error correcting codes to authenticate messages) has been implemented and runs on a prototype system designed by the Highly Available Systems project at the IBM Almaden Research Center [GS]. The experience accumulated during the implementation and test of this prototype demonstrates-that the failures most likely to be observed in distributed systems based on general purpose operating systems such as VM or Unix are performance (or late timing) failures caused by random variations in system load. Our protocols are based on a relatively realistic communication model (arbitrary network topology, approximately synchronized clocks, unreliable communication links). Abandoning the simple rounds of communication model has led to better performance. Further improvements in performance may be obtained using a probabilistic clock synchronization approach like that of [Cri]. At the time when our .protocols were invented (1983), we were unaware of other protocols for atomic broadcast designed for system models more realistic than those assumed in the Byzantine agreement literature [F], [LSP], [SD]. Since then, several other protocols for atomic broadcast in system models similar to ours have been proposed (e.g. [BJ], [BSD], [Ca], [CM], [D], [PG].) All protocols proposed so far can be divided into two classes: diffusion based protocols providing bounded termination times even in the presence of concurrent failures, and acknowledgement-based protocols that do not provide bounded termination times if failures occur during a broadcast. Examples of protocols in the first class (other than those given in this paper) are [BSD] and [PG]. Examples of acknowledgement-based protocols are [BJ], [Ca], [CM], and [D]. Although the acknowledgement-based protocols can tolerate the late timing failures that can cause a logical network partitioning for diffusion protocols, they provide the additional tolerance at the cost of sacrificing bounded termination time. We have investigated methods for detecting and reconciling inconsistencies caused by partitions in systems using diffusion based atomic broadcast (e.g. [SSCA]), but such "optimistic" approaches cannot be used in applications in which there are no natural compensation actions for the actions taken by some processors while their state was inconsistent with the state of other processors. The existence of these two classes of protocols pose a serious dilemma to distributed system designers:

69 either avoid network partitioning by using massive network redundancy and reai-time operating systems to guarantee bounded reaction time to events in the presence of failures, or accept partitioning as an unavoidable evil (for example because the operating systems are not hard real-time) and abandon the requirement that a system should provide bounded reaction times to events when failures occur.

10

Acknowledgements

We would like to thank Shel Finkelstein, Joe Halpern, Nick Littlestone, Fred Schneider, Mario Schkolnik, Dale Skeen, Barbara Simons, and Irv Traiger for a number of useful comments and criticisms.

References [BSD]

O. Babaoglu, P. Stephenson, R. Drurnond: "Reliable Broadcasts and Communication Models: Tradeoffs and Lower Bounds" Distributed Computing, No. 2, 1988, pp. 177-189.

[BJ]

K. Birman, T. Joseph: "Reliable Communication in the Presence of Failures", ACM Transactions on Computer Systems, Vol 5, No. 1, February 1987, pp. 47-76. 1984.

[ca]

R. Cart, "The Tandem Global Update Protocol", Tandem Systems Review, June 1985, pp. 74-85.

It]

F. Cristian, "Correct and Robust Programs," IEEE Transaction8 on Software Engineering, vol. SE-10, no. 2, pp. 163-174, 1984.

[CAS]

F. Cristian, H. Aghili, and R. Strong, "Clock Synchronization in the Presence of Omission and Performance Faults, and Processor Joins," 16th Int. Conf. on Fault-Tolerant Computing, Vienna, Austria, 1986.

[CASD]

F. Cristian, H. Aghili, R. Strong, and D. Dolev, "Atomic Broadcast: from simple message diffusion to Byzantine agreement," IBM Research Report RJ5244, July 30, 1986.

[Cr]

F. Cristian, "Issues in the Design of Highly Available Computing Services," Invited paper, Annual Symposium of the Canadian In]ormation Processing Society, Edmonton, Alberta, 1987, pp. 9-16 (also IBM Research Report RJ5856, July 1987).

[Cri]

F. Cristian, "Probabilistic Clock Synchronization"~ IBM Research Report R36432, September, 1988 (also in Proc. 8th Int. Conf. on Distributed Computing, June 1989).

70

[CM]

J.M. Chang, and N.F. Maxemchuk, "Reliable Broadcast Protocols," A CM Transactions on Computer Systems, vol. 2~ no. 3~ pp. 251-273~ 1984.

[D]

"The Delta-4: Overall System Specification", D. Powell, editor, January 1989.

[DS]

D. Dolev~and R. Strong, "Authenticated Algorithms for Byzantine Agreement," SIAM Journal of Computing, vol. 12, no. 4, pp. 656-666~ 1983.

[DHSS]

D. Dolev, J. Halpern, B. Simons, and R. Strong, "Dynamic Fault-Tolerant Clock Synchronization," IBM Research Report R~6722, March 3, 1989. See also "Dynamic Fault-Tolerant Clock Synchronizations" Proceedings of the 3rd Annual A CM Symposium on Principles of Distributed Computing, 1984.

IF]

M. Fischer, "The Consensus Problem in Unreliable Distributed Systems," Proceedings of the International Conference on Foundations of Computing Theory, Sweden, 1983.

[GS]

A. Griefer, and H. R. Strong, DCF: Distributed Communication with Fault-tolerance, Proceedings of the 7th Annual A CM Symposium on Principles of Distributed Computing, 1988.

[r,]

L. Lamport~ "Using Time instead of Time-outs in Fault-Tolerant Systems," A CM Transactions on Programming Languages and Systems, vol. 6, no. 2, pp. 256-280~ 1984.

[Lsp]

L. Lamport, R. Shostak, and M. Pease, "The Byzantine Generals Problem," A CM Transactions on Programming Languages and Systems, vol. 4, no. 3~ pp. 382-401, July 1982.

[PG]

F. Pittelli, H. Garcia-Molina, "Recovery in a Triple Modular Redundant Database System'~ Technical Report CS-076-87, Princeton University, January, 1987.

[PW]

W. Peterson, and E. Weldon, "Error Correction Codes," (2nd Edition), MIT Press, Massachusetts, 1972.

[RSA]

R. Rivest~ A. Shamir, and L. Adelman, "A Method for Obtaining Digital Signatures and Public-Key Cryptosystems," CACM, Vol 21., no. 2, pp. 120-126, 1978.

IS]

F. Schneider: "Abstractions for Fault Tolerance in Distributed Systems," Invited paper, Proceedings IFIP Congress '86 September 1986.

[sc]

F. Schneider: "Understanding Protocols for Byzantine Clock Synchronization', Technical report 87-859, Cornell University, August 1987.

[SD]

R. Strong, and D. Dolev, "Byzantine Agreement," Proceedings of COMPCON, Spring 1983.

7]

[SSCA]

R. Strong, D. Skeen, F. Cristian, H. Aghili, "Handshake Protocols" 7th Int. Con]. on Distributed Computing, September~ 1987, pp. 521-528.

Randomized Agreement Protocols Michael Ben-Or Institute of Mathematics and Computer Science The Hebrew University, Jerusalem/Israel

Introduction Reaching agreement in the presence of faults is one of the most important problems in fault-tolerant distributed computation, and it is also a beautiful example of the power of randomized algorithms. This problem, first introduced by Pease, Shostak and Lamport [PSL80] as the "Byzantine Agreement Problem", considers a situation in which each process /~, i = 1 , . . . , n , holds an initial value Mi, and they have to agree on a common value M, such that (1) Consistency: All correct processes agree on the same value, and (2) Meaningfulness: If all correct processes hold the same initial value M~ = M, then the correct processes will agree on M. Moreover, these properties should hold even if some processes are maliciously faulty, and skillfully coordinate their action as to foil agreement among the correct processes. The problem of Byzantine Agreement has been studied extensively in the literature. In particular, Pease, Shostuk and Lamport [PSL80] proved that a solution exists if and only if the number of faulty processes, ], satisfies ] < n/3. Fischer and Lynch [LF82] (see also [DS82]) have shown that any deterministic solution tolerating ] faults may need ] ~ 1 communication rounds to reach agreement. Another important result of Fischer, Lynch and Paterson [FLP82] shows that the Byzantine Agreement problem has no deterministic solution in the asynchronous case. This impossibility result holds even if only a single process may fail by stopping. The negative results described above collapse when we consider randomized protocols. Allowing each process to sometimes use a coin flip to determine its next step, we can solve the Byzantine Agreement problem for asynchronous systems. Moreover~ we can sometimes reach agreement within a constant expected number of rounds. This proves that randomized protocols are strictly more powerful than deterministic protocols in the area of fault-tolerant distributed computation, and in some cases, can solve problems which have no deterministic solutions. Another important feature of these randomized protocols is that they are much simpler to describe and prove correct than most deterministic Byzantine Agreement protocols. In this paper, we describe the main ideas behind these randomized agreement protocols. This is not a survey of the many research papers on this topic, as we combine old and recent results to simplify the protocols presented here. In particular, Cryptography is not used throughout this paper. In an effort to highlight the main ideas and to avoid technical difficulties, we have adopted here weaker versions of the

73 best known results. We direct the reader to the forthcoming survey [CD], and to the original papers for details of the best solutions known.

The M o d e l of C o m p u t a t i o n In a deterministic system, we can model the worst case behavior by selecting in advance f processes to be faulty. This does not model the worst case behavior of a non-deterministic system. One cannot rule out the possibility that a process may become faulty following some particular sequence of coin flips but remain correct otherwise. The correct worst case assumption here is to assume that at any time during the computation, an adversary can select further processes to become faulty provided that the total number of faulty processes does not exceed f. We allow our adversary to select faulty processes and coordinate their action using complete information about the system. The adversary cannot predict, of course, a future coin ftip of some correct process. However once this coin flip is performed, the adversary can base its action on the outcome. This strong notion of adversary models the fact that any change in the global state of the system may result in a different faulty behavior. In the next section, we present a randomized asynchronous Byzantine Agreement protocol that will reach agreement with probability 1 against any such adaptive adversary. Assuming that the adversary can adaptively select the faulty processes, but requiring that its decision be based only on information held locally by the current faulty processes, we obtain the "Weak Adversary" model. Here the adversary cannot access the internal state or listen to the messages passed between correct processes. This model is a reasonable restriction of the general adversary if we assume that our processes reside in separate processors and, therefore, a local action of some correct process cannot affect the behavior of some remote faulty process. We shall present here fast agreement protocols for this weak adversary model tolerating f = O(n) faults. The reader should note that care must be taken when applying the protocols in the weak adversary m o d e l Consider a situation where each process is a compound process that resides on several processors, where different processes may share common processors. A failure of one processor may cause one process to fail while the other remains correct, tolerating this fault. In ~his case, the local action of a correct process can clearly affect the faulty one, and solutions in the weak adversary model may not be applied. In the next section, we describe a very simple randomized agreement protocol in the strong adversary model.

The Simple Two Threshold Scheme Let PI~ .... , P. be an asynchronous system of n processes where each process P~ can directly exchange messages with every other P/. We assume that messages sent by a correct process Pi will eventually arrive at their destination but may experience

74

arbitrary finite delay (controlled by our adversary). To simplify the presentation of our protocol we assume that the maximal number of faulty processes f is less than n/t6. A somewhat more complicated version tolerating up to f < n/3 faults can be found in [Br84]. Let M(i) be the input value of process P~. Our protocol will consist of rounds of Polling, where each P/ polls the other processes on their value of the message. After each such poll _P/will update its value M(i) to be one of the initial messages or else the default value 0. Since we may have several polling rounds, we assume that each player adds the round number r to all its messages. Initially r = 0. Polling: For each process Pi:

S t e p 1: Set r : = r + l ; Broadcast (M(i), r) (to all other processes); Collect incoming messages from this step until n - f messages arrive (including own value). Let N(i) be the maximal number of times a value appears among incoming messages, and let

Temp(i) =

Most common value if N(i) >_n/2 0 (Default value) otherwise

S t e p 2: Broadcast "End Poll r" message to all processes and walt for n - f such messages to arrive. (This delay step is needed as we explain below). Both Temp(i) and N(i) represent P~'s view of the poll. Since each player misses at most / messages from correct processes and receives at most f message from bad processes we have Fact (I): For any two (correct) processes/~ and Pi I#(i) - #(i)I
,~/2 + 2f then for any other correct Pj we have Temp(i) = Temp(j). Our first goal will be to speed up the agreement if the system is trying to reach an agreement on a message M that has been broadcasted by a correct process. In such a case, all the correct processes start the polling with the same message M(i) = M. Since n > 16f each correct Pi wilt have N(i) ~_ n - 2f > n/2 -{- 4f, and set Temp(i) = M. We can therefore use R u l e (A): (High threshold) If N(i) > n/2 + 4 f then set M(i) = Temp(i) and decide on M(i). i one

IPi decides on M bat will eontlnae to participate in the next round by broadcasting its value more time.

75 While this rule guarantees that if all correct processes start with the same value they all decide in one round, this simple rule forces us to set a second rule to handle the case when this does not hold. Note that if some correct Pi decides on M by rule (A), then any other Pj must have N(j) > n/2 + 2f, and also Temp(j) = M. Setting R u l e (B): (Low threshold) If N(i) > n/2 ÷ 2f then set M(i) = Temp(i) and continue to the next polling round. ensures that if some correct Pi decides on M at some round, all other correct Pi will set M(j) = M at this same polling round. Therefore, coming into the next round all correct processes will start with the same value M and by rule (A) decide also on this value within the next round. Rules (A) and (B) make sure that if some process decides, then all other processes will decide on the same value by the end of the next round. These rules do not cover the case of N(i) < n/2 + 2f. Knowing that no deterministic rule can help, we make our first use of randomization. Definition: Let 0 < p < 1/2 and let Coin be a distributed protocol that terminates within a constant number of communication rounds, in which each correct process P; selects a bit bi. We say that Coin is an f-resilient Global Coin Protocol with probability p, if for any adversary and any b E {0, 1), with probability at least p, all the correct processes Pi select bl = b. Intuitively, a "global coin" may be viewed as a random and unpredictable bit, on which the faulty processes, or in our model - the adversary, do not have complete control. We continue the description of our agreement protocol assuming the existence of such a global coin protocol Coin, by adding the following step to the Polling protocol. S t e p 3: Run the protocol Coin, and let bl be the bit selected by Pi. We can now complete the description of our randomized agreement protocol by adding

Rule (C): If N(i) O. The protocol described above guarantees that all correct processes will reach agreement with probability 1 (against any adaptive adversary!). Furthermore, the ezpected number of rounds to reach agreement is O(1/p).

Proo]: By the remarks above it is clear that if some correct process decides on some value, then all other processes will decide on the same value by the next round. In a similar manner, if for some round r, at least n - 2 f correct processes begin the round with the same value, then any correct process will tally this value at least n - 4f > n/2 + 4f, and therefore by rule (A), decide on this value during this round. In particular if all correct processes begin with a common value, all processes will decide (within one round) on that value. It remains to show that the probability that the adversary can prevent agreement for infinitely many rounds is 0. Let Pi be the first process to finish its r-th iteration of Polling. At this time, Pi has accumulated n - f notices from other processes that they have finished their step 1 of the poll and have gone on to step 2. Of these, at least n - 2 f are correct. Therefore no correct process will begin the coin flipping protocol before n - 2 f correct processes have finished their poll. (This is the reason for the delay in step 2. The adversary can delay the messages to at most f correct processes, so that their poll will be determined after the execution of Coin, and their poll may therefore depend on Coin's outcome.) First assume that among these n - 2 f correct processes some P is now forced to set its value to M by rule (B). By fact (II), all other processes Pj have Temp(j) = M. By our assumption on Coin, with probability at least p, all the correct processes P~ will have bi = 1. In this case according to all our rules, including rule (C), all the processes begin the next round with the same value M, and will therefore decide on M by round r + 1. Otherwise, before any Correct process starts the protocol Coin, we know that at least n - 2 f correct processes will set their value by rule (C). By our assumption on Coin, with probability at least p the coin will be a unanimous 0. By rule (C), on next round we have at least n - 2 f correct processes that will broadcast the value 0, and will therefore decide on the value 0 by round r + 1. We have seen that for any adversary schedule, on each round r with probability at least p all correct processes will reach agreement by round r + 1. Therefore the probability of not reaching agreement after r + 1 rounds is tess than (1 - p)" ~ 0 as r --* oc, and the expected number of rounds to reach agreement is O(1/p). QED.

A Simple Global Coin P r o t o c o l We now complete our simple two threshold scheme by presenting a very simple global coin protocol that is resilient against any adaptive adversary. Assuming that each process can flip a local unbiased coin, our goal is to generate a global coin from the local coins of the processes, while minimizing the influence of the faulty processes, or the adversary, on the outcome. A simple way to achieve this goal is to let each

77 process generate and broadcast a random bit, and take the majority of these values as a "global coin". Protocol

Simple-Coin

Each process Pi:

S t e p 1: Generate an unbiased bit rl and broadcast ri to all other processes. S t e p 2: Wait for n - f values of ri's. Set b i = majority of the rj's. Output hi. We have L e m m a I:

(a) Let f

< n/16. Simple-Coin is an f-resilieni global coin protocol with probability p > 1/2",

(b) lf f = O(v~) then Simple-Coin is an f-resilient protocol with probability po > O,

where Po is a constant not depending on n. Proof: (a) Let b E {0, 1}. If all the correct processes flip the value r~ = b, then on step 2 they will all compute the same majority b; = b. Since this event has probability at least p = 1/2", we are done. To prove (b) we note the expected deviation from n/2, among n independent coin flips, is O(v/-ff). Therefore if f = c v ~ , and b E {0, 1}, there is some constant probability P0 = p(c), that among n independent coin flips the value b will appear more than n/2 + 2 f times. If this happens, then on step 2 all the correct processes will set b l = b. Q E D . Combining Theorem 1 and Lemma 1 we have T h e o r e m 2: Let f < n/16.

There is a completely asynchronous [-resilient randomized Byzantine agreement protocol that guarantees agreement with probability 1 against any adaptive adversary that acts with complete in]ormation. Furthermore, i] f = 0(~/-~) then the ezpected number of rounds to reach agreement is constant.

This randomized solution is by no means practical. Since p in our proof can be as small as 2-", the expected number of rounds to reach agreement may be exponential! Nevertheless, this does show that randomized algorithms give us the power to solve problems that have no deterministic solution, and raises the possibility of better randomized solutions. Our protocol behaves much better when the number of faulty processes f is bounded by O(v/-n-). In fact, for synchronous systems, our protocol will still have

78 the same worst case constant expected time, while any deterministic protocol may require up to f ÷ 1 rounds to reach agreement. The Simple-Coin protocol described above is resilient to an adversary that selects and coordinates the action of the faulty processes based on complete information about the system including the internal states of all the processes. Much simpler and more efficient "global coin" protocols exist if we severely restrict the adversary's information, thereby restricting the type of faults we allow. We can, for example, hand out to all the processes a common long list of random bits as part of their protocol, and use the.r-th bit as the r-th global coin flip. This would provide a reasonable solution if we can assume that the faults in the system do not depend in any way on the values on this list. An even simpler solution is to use a common random number generator as our ~'global coin". Under this assumption we have an unbiased global coin and therefore, a constant expected time agreement protocol "tolerating" f = O(n) faulty processes. It is clear that our protocol is no longer valid if we allow the faulty processes to act based on future "coin flips". It is therefore, hard to justify such simple minded Coin protocols because we must make the unreasonable assumption that the faulty processes, though not acting according to the protocol~ do not make use of some information they have. In the next section we show, following Rabin IRa83], how to distribute pre-dealt random coins among all the n processes, in such a way that any set of at most f < n/5 faulty processes will be not have any information about the r-th coin before it is time to reveal the coin.

Secret Sharing Protocol Let s be a secret. We want to share the secret 8 among n processes so that (1) Any set of at most f processes will have no information about s, and (2) The secret can be reconstructed from all the pieces even if some of the pieces have been changed by faulty processes. To this end, let p > n be a prime number and let Zp = { 0 , . . . , p - 1} be the field of p elements with addition and multiplication modulo p. Let g E Zp be a primitive element in Zp so that ak = gk-1 h = 1 , . . . , f l - 1, are all distinct. To share a secret s E Zp, the dealer selects a random f-degree polynomial S(z), whose constant term S(0) = s. The dealer then hands process Pk the value sk = S(ak), [Sh79,BGWS8]. It is easy to see that any set of f players has no information about the secret because the values they hold~ together with any value of the secret, define a unique f-degree polynomial interpolating through these f ÷ 1 points. Moreover, by our special choi~:e of the evaluation points ak, the possible sequences ( s l , . . . , s,) are just a generalized BCH error correcting code that can correct up to (n - f - 1)/2 wrong s~ (see [PW72]). Thus, for f < n/3 we can correct f errors and for f < n/5 we can correct up to 2 f errors. To reveal the secret, all the processes broadcast their values.

?9

Each correct process receives at most f wrong values and can, therefore, correct the errors using one of the well-known error correction algorithms [PW72, page 286] and compute the correct interpolating polynomial to recover the secret s. From this point on we shall restrict our attention to the weak adversary model. As discussed above, instead of handing the random bits directly to all the processes, we can prepare a tong list of random bits and share their value using the robust secret-sharing scheme described above. By our assumptions, the weak adversary has no inforraation about the value of the next bit on the list before some correct process begins the secret recovery procedure and broadcasts its value. Combining this with Theorem 1 we have T h e o r e m 3: (Weak adversary, Pre-dealt coins) In a properly initialized system, under the weak adversary model, there is a constant ezpeeted number of rounds Byzantine Agreement protocol tolerating f = O(n) ]aults. Theorem 3 does not address the problem of generating unbiased secret coins by the system itsdf. We must therefore rely on a trusted "dealer" to prepare in advance enough secret coins for the lifetime of the system. In the next section we describe how the system of processors can prepare the needed unbiased random coins without any outside help.

Generating Secret Coins To generate an unbiased coin we let each process/~ select a secret random number sl, 0 < sl < p, and share this secret sl among all the other processes using the secret sharing procedure described above. Having done this, we can at a later step of the protocol reveal all the secrets and take the parity of the sum (modulo p) of all the secrets as our common "global coin". If a~ the secrets are properly shared, the sum will be random if at least one correct process selected its number randomly, and therefore our global coin will be unbiased. Before attempting to use these secrets we must verify that the secret shares sent by a possibly faulty process D are shares of a real secret and not some n random numbers. We want to do so without revealing any information about the secret itself. This is easily done using the following Zero Knowledge proof technique, first introduced by [GMR]. Let us assume that the system can reach Byzantine Agreement, and let S(z) be the f-degree polynomial used by process D to share the secret s. The dealer of the secret, D, generates n additional random f-degree polynomials R 1 , . . . , Rn, and sends to each P~ its share of the secret sh = S ( a k ) along with the values of ri,k = R j ( a k ) , j = 1 , . . . , n , of all the polynomials at the point ak. At this point, each player P~ randomly picks a random bit bk E {0, 1} and broa~lcasts its value to all the other processes. After reaching Byzantine Agreement on the value of all the broadcasted bits, bk, (missing values take the default value 0) the dealer of the secret, D, broadcasts the polynomials Fj(z), where Fj(z) = R~(z) if bi = 0, and Fj(z) = S(z) + Ri(z) if bi = 1, for j = 1 , . . . , n (Second Byzantine Agreement). At this point each process

80 Pk checks that the polynomials Fi(z) are of degree at most f and that at the point ak the shares it received satisfy the required equations, that is ri,k + bks~ = Fi(ak), for all j. If some Pk finds an error it broadcasts a complaint against the dealer D. (Third Byzantine Agreement). If f + 1 or more processes file a complaint, we decide that the dealer D is faulty and take the default value 0 as the value of its secret and a~ its shares. Claim: Let G be the set of correct processes that did not complain against PiSs secret. Let So(z) be the interpolation polynomial through the points in G of the original polynomial $(z). Then with probability exponentially close to 1, S.G(z) is a polynomial of degree (_ f .

Proof: (Sketch) Let R~(z) be the interpolation polynomials through the points in G of Ri(z ). Assume that the degree of SG(Z), deg S G > f. Then if deg R~ _( f and 5i = 1, then no polynomial of degree f will fit the points ri,k + 5isk for k E G. If on the other hand deg R~ > f and 6i = 0 then there is no f degree polynomial fitting all the point ri,k for k E G. Thus if 5~ was picked at random, with probability at least 1/2 some of the processes in G would file a complaint. Since at least n - f of the 5h were selected randomly by correct processes, the probability that deg S c ) f is exponentially small. Q E D . Note that if n _) 5 f + 1 then our secret sharing scheme can correct up to 2f errors. If a secret is accepted by the system, then at most f good processes may have values not on the polynomial $G(z). This, together with at most f additional wrong values coming from the faulty processes, gives altogether at most 2 f errors. Thus, in this case the secret is well defined (as the free coe~eient of S G) and there is a simple procedure to recover its value from the shares using the error correcting procedure. Our scheme to generate unbiased secret coins requires, in itself, three rounds of Byzantine Agreements. It is therefore hard to see how such a scheme can be of any use for the Byzantine Agreement itself. The scheme is helpful only because we can prepare many unbiased coins at a time. By distributing m secrets together we can prepare rn independent secret coins using the same three rounds of Byzantine Agreements. This leads to the following modification of Theorem 3: • Initialize the system with m = 100n z secret random coins. s Use these coins to run the fast agreement protocol of Theorem 3. • Whenever the number of random coins falls below m/2 generate in parallel ra additional secret coins. This will take only a constant expected number of communication rounds using some of the remaining m/2 random coins. It is easy to see that the probability of ever running out of coins is exponentially small (in m). If this ever happens, we can revert to slower Byzantine Agreement protocols [PSL80,Be83], generate again ra random coins, bringing the system back to its initialized state. Likewise, if there is no "trusted dealers, to initialize the system,

8] we can start up the system with a slow initialization stage, prepare m coins, and only then start the system. Summarizing this section we have T h e o r e m 4: (Weak Adversary) After an initialization stage by the system or by a trusted dealer, there is a constant ezpeetcd number of rounds Byzantine Agreement protocol tolerating f = O(n) ]aults.

The

Feldman-Micali

Protocol

The problem of initializing the system in a constant number of rounds without the aid of a "trusted dealer" was recently solved by Feldman and Micali [FM88]. In this section we briefly describe the main ideas underlying this beautiful protocol. The reader can find the full details and proof of this protocol in the original papers [FM88] and [F88]. Returning to the original Two Threshold scheme, we recall if process P sends a message to all other processes, then in just one more round we have (a) If P is correct, then all correct processes agree on that message, and (b) Even if P is faulty, if any correct process accepts the message, then all other processes receive the value of this same message, (but may or may not accept it). Running this constant round protocol, we say that process P has Cast the message to all other process. In a similar way, we can replace the broaxtcasts, and the needed Byzantine Agreements, in our secret sharifig protocol by the simple constant round Casting protocol. Doing this carefully we get a constant round Secret-Casting protocol, such that if P distributes a secret to all the players then (a) If P is correct then all the processes agree that a good secret has been shared, an{] (b) Even if P is faulty, if any good process accepts the secret as good, then all the good players know that there is some well defined secret, and furthermore, this un!que secret is recoverable from all the pieces. The crucial idea of the constant round initialization protocol is how to use the Casting and Secret-Casting protocols to generate a Common Global Coin protocol. To generate the coin the processes all participate in an election where each process is assigned a random number in the range [0, n~], and the player with the smallest number is selected. As the common global coin we can take the parity of this smallest number. To guarantee that thenumber assigned to each process P is indeed random, all the processes participate in the choice of this number. Each process selects a random number in the range [0, n~], and uses the Secret-Casting protocol to share its

82 pieces among ~he other processes. At this point our process P selects n - f secrets that it has accepted, and announces its selection using the Casting protocol. As P's number we take the sum of the secrets it has selected modulo n ~ + 1. This is done concurrently for each process P. A good player will accept P's selection only if it knows that all the secrets on the list are well defined. At this point all the shares of the secrets are opened, the secrets are recovered and our global coin is determined. Note that if P is correct, then the secrets it included in his list are known to all the players and they are all recoverable. Therefore P's number is well defined. If P is faulty, its number may not be well defined if it includes nonrecoverable secrets. A correct process will accept P~s selection only if all the secrets on the list are recoverable; therefore all the correct process that do accept P~s choice get the same value. Since each accepted list must contain n - f recoverable secrets, some of which were randomly selected by correct processes, the number assigned to P is indeed random. Since all the numbers assigned to all the processes are random, with probability at least 1 - f / n a correct process number wil~ be the minimum. In this case all the processes have the same minimal number and therefore the same common coin. In the unlucky case where a faulty process draws the minimal number, some of the processes may not receive this number and may come up with a different coin. Since this happens only with probability less than ]/n, combining this Global Coin Protocol with Theorem 1 we finally have T h e o r e m 5: (Weak Adversary) There is a constant ezpected number o] round, Byzantine Agreement protocol tolerating f = O(n) faults. The Feldman and Micali protocol is much more complicated then the simple protocol of, say, Theorem 3, and the constants involved are considerably bigger. It is therefore best to use this fast agreement protocol to preform the initialization step needed in Theorem 4, and after this first stage continue with the protocol of Theorem 4.

Final R e m a r k s We have presented here the main ideas underlying the best to date randomized agreement protocols for the strong and weak adversary models. In the weak adversary model~ these protocols give us the added benefit of being able to generate within the system a global unbiased coin. This by itself is an important and nontrivial task that may find other applications. In particular, just as the power of Byzantine Agreement allows the system to carry any deterministic computation in a consistent way, the ability to generate a global unbiased coin provides the system with the ability to carry any randomized computation. In the strong adversary model our solution is much less satisfactory. For constant expected time agreement our solution tolerates only f = O(v/'n-) faults, leaving the the question of fast agreement when ] = O(n) open to further research.

83

References [Be83]

M. Ben-Or, Another Advantage of Free Choice: Completely Asynchronous Agreement Protocols, Proc. 2nd Annum ACM Symposium on Principles of Distributed Computing, pp. 27-30, 1983.

[BGW88] M. Ben-Or, S. Goldwasser and A. Wigderson, Completeness Theorems for Non-Cryptographie Fault-Tolerant Computation, Proc. 20th Annual ACM Symposium on Theory of Computing, pp. 1-10, 1988. [Br84]

G. Bracha, An Asynchronous (n-1)~3-Resilient Consensus Protocol, Proc. 3rd Annum ACM Symposium on Principles of Distributed Computing, pp. 154-162, 1984.

[CD]

C. Chor and C. Dwork, Randomization in Byzantine Agreement, to appear.

[DS82]

D. Dolev and R. Strong, Polynomial Algorithms for Multiple Processor Agreement, Proc. 14th Annual ACM Symposium on Theory of Computing, pp. 401-407, 1982.

[F88]

P. Feldman, Optimal Algorithms for Byzantine Agreement, MIT Ph.D. Thessis, 1988.

[FM88]

P. Feldman and S. MicMi, Optimal Algorithms for Byzantine Agreement, Proc. 20th Annum ACM Symposium on Theory of Computing, pp. 148161, 1988.

[FLP83]

M. Fischer, N. Lynch and M. Paterson, Impossibility of Distributed Consensus with One Faulty Process, JACM 32, pp. 374-382, 1985.

[LF82]

M. Fischer and N. Lynch, A Lower Bound for the Time to Assure Interactive Consistency, Information Processing Letters 14, pp. 183-186, 1982.

[GMR85] S. Goldwasser, S. MicMi and C. Rackoff, The Knowledge Complexity of Interactive Proof Systems, Proc. 17th Annual ACM Symposium on Theory of Computing, pp. 291-304, 1985. [PSL80]

M. Pease, R. Shostak and L. Lamport, Reaching Agreement in the Presence of Faults, JACM 27, pp. 228-234, 1980.

[PW72]

W . W . Peterson and E. J. Weldon, Error Correcting Codes, Second Ed., MIT Press, 1972.

[R S3]

M. Rabin, Randomized Byzantine Generals, Proc. 24th Annual Symposium on Foundations of Computer Science, pp. 403-409, 1983.

[Sh79]

A. Shamir, How to Share a Secret, CACM 22, pp. 612-613, 1979.

An Overview of Clock Synchronization Barbara Simons, IBM Almaden Research Center Jennifer Lundelius Welch, GTE Laboratories Incorporated Nancy Lynch, MIT

1

Introduction

A distributed system consists of a set of processors that communicate by message transmission and that do not have access to a central clock. Nonetheless, it is frequently necessary for the processors to obtain some common notion of time, where ~time:~ can mean either an approximation to real time or simply an integer-valued counter. The technique that is used to coordinate the notion of time is known as

clock synchronization. Synchronized clocks are useful for many reasons. Often a distributed system is designed to realize some synchronized behavior, especially in real-time processing in factories, aircraft, space vehicles, and military applications. If clocks are synchronized, algorithms can proceed in "rounds :' and algorithms that are designed for a synchronous system can be employed. In database systems, version management and concurrency control depend on being able to assign timestamps and version numbers to files or other entities. Some algorithms that use timeouts, such as communication protocols, are very time-dependent. One strategy for keeping docks synchronized is to give each processor a receiver and to use time signals sent by satellite. There are obvious questions of reliability and cost with this scheme. An alternative approach is to use software and to design synchronization algorithms. This paper discusses the software approach to clock synchronization, using deterministic algorithms. The results surveyed in this paper are classified according to whether the distributed system being modeled is asynchronous or partially synchronous, reliable or unreliable. An asynchronous model is one in which relative processor speeds and message delivery times are unbounded. Partially synchronous can be interpreted in several ways - - processors may have real-time docks that are approximately the same or that move at about the same rate or that drift slightly. The message detivery time may always be within some bounds, or it may follow a probability distribution. A reliable system is one in which all components are assumed to operate correctly. In an unreliable system, communication faults such as sporadic message losses and link failures may occur, or processors may exhibit a range of faulty behavior.

85 This paper presents some of the theoretical results involving clock synchronization. A more thorough discussion of our basic assumptions and definitions, especially concerning faults, is contained in section 2. In section 3 we discuss the completely asynchronous, reliable model. Section 4 deals with asynchronous, unreliable models. In section 5, we discuss partially synchronous, reliable models. Section 6 is the longest and contains descriptions of several algorithms to synchronize clocks in some partially synchronous, unreliable models. In section 7 some problems closely related to the clock synchronization problem of the previous section are mentioned. We close with open problems in section 8.

2

Basic Assumptions

We assume that we are given a distributed system, called a networks of n processors (or nodes) connected by communication links. The processors do not have access to a source of random numbers, thus ruling out probabilistic algorithms. We allow the network to have up to f faults, where a fault can be either a faulty processor or a faulty link. We say that a system is reliable if f is always 0. Otherwise, the system is unreliable or faulty. Although there is some work on fault tolerance that distinguishes between node faults and link faults (e.g. see [DHSS]), for simplicity we shall assume that only node faults occur. If a link is faulty, we can arbitrarily choose one of the two nodes that are the endpoints of the faulty link and label that node as faulty. This is clearly a conservative assumption, since the node that is selected to be faulty might be the endpoint of many nonfaulty links~ all of which are now considered faulty. Having limited ourselves to node faults, there remains a variety of different models in which to work. The simplest of these models, called fail safe, is based on the assumption that the only type of failure is a processor crash. There is the further assumption that just before a processor crashes~ it informs the system that it is about to c r a s h . This is the only model in which the faulty processor is thoughtful enough to so inform the others. A more insidious form of failure is unannounced processor crashes, sometimes called a faiIstop fault. Next in the hierarchy of faults is the omission fault model. In this case a processor might simply omit sending or relaying a message. A processor that has crashed will of course omit sending all its messages.

Timing faults can be more complicated than omission faults, especially when dealing with the problem of clock synchronization. The class of timing faults is itself divided into the subcases of only late messages and of both early and late messages. For many systems the types of faults most frequently encountered are processor crashes (without necessarily notifying the other processors)~ omission faults, and late timing faults. Finally, a fault that does not fall into any of the above categories is called a

86

Byzantine .fault. (For a more thorough discussion of Byzantine faults, see the article by Dolev and Strong in this book). This includes faults that might appear to the outside observer to be malicious. For an example of such a fault that brought down the ARPANET for several hours, see the article by Cohn in this book.

3

Asynchronous Reliable M o d e l

We assume in this section that message delays are unbounded, and that neither processors nor the message delivery system is faulty. For this environment we examine the differences caused by whether relative processor speeds are lockstep or unbounded, i.e., whether processors are synchronous or asynchronous. Lamport ILl] presents a simple algorithm allowing asynchronous processors to maintain a discrete clock that remains consistent with communication. When processor i sends a message to processor j, i tags the message with the current time o n / ' s clock, say tl. Processor j receives the message at time tj. If tj < tl, processor j updates its clock to read time t~. Otherwise, processor j does nothing to its clock. Note that this algorithm depends heavily on the assumption that there are no faults in the system, since clearly a faulty processor could force correct processors to set their docks to arbitrary times. The Lamport algorithm can be used to assign timestamps for version management. It can also provide a total order on events in a distributed system, which is useful for solving many problems, such as mutual exclusion ILl]. The power of local processor clocks in an otherwise asynchronous system is further explored by Arjomandi, Fischer and Lynch [AFL]. They prove that there is an inherent difference in the time required to solve a simple problem, depending on whether or not processors are synchronous (i.e., whether or not processors have synchronized clocks). The problem is that of synchronizing output events in real time: there is a sequence of events, each of which must occur at each processor and each taking unit time, with the constraint that event i cannot occur at any processor until event i - 1 has occurred at all processors. With synchronous processors, the time for k events is k, and no communication is needed. With asynchronous processors, a tight bound on the time for k events is k times the diameter of the network. Note that since Lamport clocks can be used to make a completely asynchronous system appear to the processors to have synchronous processors, the problem presented in [AFL] is of necessity one of external synchronization.

4

Asynchronous Unreliable Models

Devising algorithms for a model in which faults may occur can be much more difficult than devising algorithms for the comparable reliable model. In fact, there might not even exist an algorithm for the unreliable version, as is the case for the agreement problem [FLP]. In particular, it is possible for all (correct) processors to reach agreement on some value in an asynchronous reliable model, but not in an asynchronous

87 unreliable one. By contrast, there exist methods [A] to convert algorithms designed for a synchronous reliable system into algorithms that are correct for an asynchronous reliable system. Welch [W] has shown that a system with asynchronous processors and asynchronous reliable communication can simulate a system with synchronous processors and asynchronous reliable communication, in the presence of various kinds of processor faults. The method used in the simulation is a variant of Lamport clocks - each message is tagged with the sender's time, and the recipient of a message delays processing the message until its local time is past the time tag on the message. One application of this simulation is that the result of Dolev, Dwork, and Stockmeyer [DDS], that the agreement problem is impossible in an unreliable model with synchronous processors and asynchronous communication, follows directly from the result of Fischer, Lynch, and Paterson [FLP], that agreement is impossible in an unreliable model in which both processors and communication are asynchronous. (Neiger and Toueg [NT] independently developed the same simulation, but they did not consider faults, and they studied different problems). A subtle point is determining exactly what is preserved by this transformation. (Cf. [NT] for the fault-free case). Since a partially synchronous system and an asynchronous system appear quite different when viewed externally, the behavior preserved by this simulation is that which is observed locally by the processors. Thus~ the transformation cannot be used in the asynchronous model to create simultaneous events at remote processors, even though this is easy to do in the model with synchronous processors and asynchronous communication. It is also possible to design Lamport-like clocks for an asynchronous system that tolerate some number, say f, of Byzantine faults. A common technique is to wait until hearing from ] ÷ 1 (or all but ]) of the processors that time i has passed, before setting one's clock to time i + 1. This type of clock imposes a round structure on an asynchronous computation, and is used in some probabilistic agreement algorithms. (See the article by Ben-Or in this book, and also [Be, St]). Dwork, Lynch and Stockmeyer [DLS] solve the agreement problem in unreliable models that lie somewhere between strictly asynchronous and synchronous. Their algorithms use interesting discrete docks reminiscent of Lamport clocks, but more complicated.

5

Partially Synchronous Reliable Models

Several researchers have considered a partially synchronous, reliable model in which processors have real-time clocks that run at the same rate as real time, but are arbitrarily offset from each other initially. In addition, there are known upper and lower bounds on message delays. The goal is to prove limits on how closely clocks can be synchronized (or, how close in time remote events can be synchronized). In a completely connected network of n processors, Lundelius and Lynch [LL] show that the (tight) lower bound is ~/(1 - l / n ) , where n is the difference between the bounds

88 on the message delay. This work was subsequently extended by ttalpern, Megiddo and Munshi [HMM] to arbitrary networks. A version of the Lamport clocks algorithm for real-time clocks has been analyzed ILl] in a different reliable, partially synchronous model, one in which clock drift rate and message uncertainty are bounded, to obtain upper bounds on the closeness of the clocks. Together with the results mentioned in the previous paragraph, we have upper and lower bounds on closeness imposed by uncertainty in system timing. Marzullo [M] also did some work in the same reliable, partially synchronous model as ILl]. The key idea is for each processor to maintain an upper bound on the error of its clock. This bound allows an interval to be constructed that includes the correct real time. Periodically each processor requests the time from each of its neighbors. As each response is received, the processor sets its new interval to be the intersection of its current one with the interval received in response, after adjusting for further error that could be introduced by message delays,

6

Partially Synchronous Unreliable Models

There has been much work done on the problem of devising fault-tolerant algorithms to synchronize real-time clocks that drift slightly in the presence of variable message delays [LM, M, WL, HSSD, MS, ST]. Although most of the algorithms are simple to state, the analyses tend to be very complicated, and comparisons between algorithms are difficult to make. The difficulty arises from the different assumptions, some of which are hidden in the models, and from differing notations. There has been some work by Schneider IS] attempting to unify all these algorithms into a common framework and common proof. Our goal in this section is simply to describe some of these algorithms and attempt some comparisons. First, though, we discuss the assumptions, notations and goals.

6.1

Assumptions

Recall that n is the total number of processors in the system, and .f is the maximum number of faulty processors to be tolerated. The required relationship between n and f in order for the clock synchronization problem to be solvable depends on the type of faults to be tolerated, the desired capabilities of the algorithm, and what cryptographic resources are available, as we now discuss. To overcome the problem in the case of Byzantine faults of deciding what message a processor actually sent to some other processor, algorithms may use authentication. The assumption customarily made for an authenticated algorithm is that there exists a secure encryption system such that if processor A tells processor B that processor C said X, then B can verify that X is precisely what C said. Dolev, Halpern arid Strong [DHS] show that without authentication, n must be greater than 3 f in order to synchronize clocks in the presence of Byzantine faults.

89 With authentication, any number of Byzantine faults can be tolerated. The paper [DHS] also shows that without authentication~ the connectivity of the network must be greater than 2 f in order to synchronize clocks in the presence of Byzantine faults. (See [FLM] for simpler proofs of the lower bounds in [DHS]). Even if authentication is used~ clearly each pair of processors must be connected by at least f ÷ 1 distinct paths (i.e., the network is ( f ÷ 1)-connected), since otherwise f faults could disconnect a portion of the network. Some of the algorithms in the literature assume that the network is totally connected~ i.e. every processor has a direct link to every other processor in the network. In such a model a processor can poll every other processor directly and does not have to rely on some processor's say-so as to what another processor said. The assumption of total connectivity often results in elegant algorithms, but it is~ unfortunately, an unrealistic assumption if the network is very large. Consequently~ there are other algorithms that assume only that the network has connectivity f ÷ 1 (and use authentication). One assumption that all of the algorithms make is that the processors' real-time (or hardware) clocks do not keep perfect time. We shall refer to the upper bound on the rate at which processor clocks ~drifC frbm real time as p. In particular, the assumption is usually made that there is a '~linear envelope" bounding the amount by which a correct processor's (hardware) clock can diverge from real time. In the formulations of this condition given below, C represents the hardware clock~ modeled as a function from real time to clock time; u, v and t are real times. The papers [HSSD, DHS, ST] use the following condition: (v -

+ p) < c ( v ) -


0. That is, if Ri(t) is the reading of the physical clock of processor i at time t, then for all t2 > tl, (1 + p)-l(t2 - t ~ ) - 1,

~UNIXis a trademark of AT&TBell Laboratories.

I01

and plotted network events against it. Our system was implemented on an Ethernet broadcast network, providing us with full interconnection between processors. We assumed the broadcast network to be reliable, and hence that every message is guaranteed to be received by all correct processors. This obviated the need for message relaying, and thus eliminated the possibility of faulty processors tampering with, and then forwarding, relayed messages. Furthermore, the Ethernet network driver marks each outgoing message with the unique internet address of the sending processor. Given our assumption that the network is reliable, the receiving processor can use these addresses to safely identify the sender of each message. In other words, the Ethernet broadcast hardware provides all the properties of authentication needed to run the authenticated version of our algorithm. 4.1. Our E x p e r i m e n t Figure 2 is a plot of the aggregate behavior of the processors' clocks, both synchronized and unsynchronized. After some preliminary experiments, we decided to assume a maximum message delivery time t4el=O.l seconds, and a maximum clock drift rate of p = 2 × 10 -4. We wanted to achieve a maximum difference between synchronized clocks D max =0.4 seconds. This choice of parameters led us to use a period between synchronizations P = 30 minutes, and a = 0.5 sec. W i t h a total number of synchronizing processors N = 10, we configured the protocol to withstand f = 3 processors with arbitrary faults. The graph represents the range and mean of the difference between the processors' local clocks and the VAX's "real time", C ( t ) - t , at the moment that each processor resynchronizes. Note that the horizontal axis of Figure 1 is the round number, as opposed to real time. The time between successive synchronization rounds in our experiment is measured in hours, while synchronization typically lasts for less than a second. If plotted on the same time scale, the times of events within a synchronization round would be indiscernible. From Figure 1, we can see that during the 10 hour experiment the slowest unsynchronized clocks drifted from real time at a rate close to 17 sec/day, so the rate of drift was bounded by p = 2 × 1 0 -4. During this experiment, unsynchronized clocks drifted increasingly apart from each other. The synchronized clocks that satisfied the assumed specifications on p and tdeL deviated by less than 0.4 sec. from each other, as desired. However, two processors experienced message delivery times as high as tdei = 0.5 sec. The maximum difference between all clocks in the experiment, including these faulty processorsj was less than 0.8 sec. Furthermore, all synchronized clocks were considerably closer to real time than the slowest unsynchronized clocks. As a reference point, the Berkeley Time Synchronization Protocol [Guse84], which synchronizes every 4 minutes, reportedly achieves a maximum difference between clocks of less than 0.04 sec. This shows that the performance of our experimental system,

102

C(t)-t in sec. I

IS

I0 0

l

I,ilI I

20

!

k

T

l I

"2

£I

"5 .t

4

"5

p,

o2-):'...

~

mox overagesynchronizedctock rain

mOX

T?overageunsynchronizedclock !

"~ rain

'ExperimentalParameter",

i

nf =tO " 3

~1 :0.1SeC X !,~ 4

1 P • 50mm a

On~x-O.4sec

• O, S . c

F i g u r e 2: D r i f t F r o m R e a l Time synchronized vs. unsynchronized clocks

running at a low user-level priority, is within an order of magnitude of a highly tuned production system. Our experiment demonstrated an important aspect of the robustness of our algorithm: It periodically recovered the unsynchronized clocks of some faulty processors, and gracefully reintegrated these clocks into the system of synchronized clocks. Specifically, "lazy" or temporarily faulty processors that occasionaly "missed" a resynchronization round (e.g., by losing or reacting slowly to a resynchronization message) were often reintegrated and correctly resynchronized at the next resynchronization. In fact, our algorithm ensures that all faulty processors are asked to start a new logical clock at the correct time (by the f + i correct processors) at each resynchronization round, independent of their past faulty behavior. In other words, at each resynchronization round, the correct processors give the opportunity to all the faulty processors to correct their clocks by resynchronizing on time. Thus, the clocks of processors that experience transient failures are never "lost" forever: instead, they usually correct themselves and fully recover at the next resynchronization round. For example, we observed a consistently slow clock that was forced to ""pull itself

103

up" and synchronize with the correct clocks at every resynchronization. The same type of periodic corrections would also happen to a consistently fast clock. Several previous synchronization algorithms exhibit the following undesirable behavior. Once a processor becomes unsynchronized because of a transient failure, it may forever refuse to accept synchronizing messages from correct processors: its own clock has already strayed "too far" from the correct clocks. With such algorithms, unsynchronized clocks have to be detected and then explicitlyreintegrated into the system.

5. Other Implementation Issues Our experience in building an experimental system was that the algorithm itselfwas straightforward to implement. The whole program consisted of under 250 lines of C code, and was written and debugged in under 2 weeks. While our experiments were not a complete implementation of a fault tolerant, synchronized time utility,our experience sheds some light on the task of building one. W e have considered, but not yet solved, the problems of overcoming high variance in the message delivery time (Section 5.2), and of changing c o m m o n algorithm parameters dynamically to achieve greater fault tolerance (Section 5.3).

5.1. Maintaining continuous clocks To simplify the presentation and analysis, we adopted the standard convention that a processor starts a new logical clock after each resynchronization [Halp84]. When a new clock is started, it is set to a value greater than that shown by the previous clock, thus ensuring that clocks are never set back. If clocks are set backwards, the intuitive rela. tionship between time and causality is lost. Since logical time is often used to order events in real systems (e.g., file system timestamps), monotonicity of clocks should be preserved. For some applications, this scheme of starting a new logical clock at every resynchronization creates some problems. For example, an application may repeatedly ask for the current time, and then use the returned clock values to accurately measure elapsed time. If tile system always returns the time according to the current (latest) logical clock, and several logical clocks are started during the execution of that application, the elapsed times measured will not be accurate: every time the system switches to a new logical clock, the logical time jumps forward. On the other hand, if we force the application to stick to the same logical clock across several resynchronizations, then this clock will deviate from the clocks of other processors beyond the allowed limit Dmax, preventing close synchronization of actions between different processors. A simple and elegant solution to the above problem is to maintain a single continuous logical clock at each processor, thus removing the ambiguity about which clocks to use. As Lamport and Melliar-Smith noted, [n [Lamp85], an algorithm for discontinuously resynchronizing clocks can be transformed into one where logical clocks are continuous. This can be achieved by" spreading out each

104

resynchronization adjustment over the next resynchronization period. We now show how to use our algorithm to implement a single continuous logical clock for each processor. Each processor i runs the algorithm described in Section 3 to start its logical clocks C~ (the initialization process starts CO). For all k >0, let t~ be the real time of the k th resynchronization of processor i, i.e., the time at which processor i starts the new clock C~. Let h~ be the forward adjustment that processor i makes to its logical clock at the h th resynchronization, namely ~ = Ci(t~) k k - C~-l(t~), for k - - l , and h0 = a. Let R~(t) be the value of the physical clock of processor i at time t. Using the forward adjustments A~, and the underlying physical clock Ri(t), we can easily implement a single continuous clock Ci for processor i as follows: c : t °) = o

el(t)

= Ci(tk) + x~(t)&~ + R~(t) - R~(t~) [

Ri(t)-R~(t')]

wherexk(t) = rain 1, P - a - D m a x

''

for tk 1, this construction would imply that Byzantine agreement is possible forf -- I in a subgraph of the triangle graph, contradicting Theorem 1. This is essentially the proof strategy for the general bound given in [PSL].

155

3.2. Connectivity Now we carry out the 2f+1 connectivity lower bound proof. Let c(G) = connectivity of G. We assume we can achieve Byzantine agreement in a graph G with c(G) < 2f, and derive a contradiction. For now, we consider the case f = l and the communication graph G of four nodes a, b, c and d, running devices A, B, C and D, as indicated below.

/\ B \/

A D

C The connectivity of G is two; the two nodes b and d disconnect G into two pieces, the nodes a and c. We consider the following system, with the eight-node graph S and devices and inputs as indicated.

/f\ /I

D ~ B

]\

A

A

ON

BraD

/1

oN / o C 0

The resulting behavior of the system is S. We consider three scenarios in S: S I, S 2 and S 3.

156

The first scenario, S1, is shown below.

S

/c\

A

D~B

\ A /]

B ~ D

"

/0\ B 0 \ y~ F

\,o \

C

,._

Oy

",,,0// This is also a scenario in a correct behavior E 1 of G. In El, nodes a, b and c are correct. Node d is faulty, exhibiting the same behavior to node a as one node running D in the covering graph, and the same behavior to b and c as the other node running D exhibits in the covering graph. Then nodes a, b and c must choose 0 in E l, and so must the nodes running A, B and C in S 1. Now consider the second scenario, S2.

S

E2 A

/1 A--

o,,

D ~ B

/

//1~ 1 X",. / / A~\,

/1/

/o/

B ~ D

",,,,0//

~

,,,,,,

,

N\ C 0

This scenario in S is also a scenario in a correct behavior E 2 of G in which nodes c, d and a are correct. This time, node b is faulty, exhibiting the same behavior to nodes c and d as one node running B in the covering, and the same behavior to node a as the other node running B. So nodes a, c and d must agree in E 2, and so do the corresponding nodes in S2. Since the node running C chooses 0 from the argument above, the nodes running D and A in S2 choose 0, too.

157

Finally, consider the last scenario S 3.

A /1N N

/ 1 A

",, 1 \ ",, \,, A ")

B

D

oN / o

B

rF

C

C 0

This scenario is again the same as a scenario in a behavior E 3 of G in which nodes a, b and c are correct, but have input 1. Node d is faulty, exhibiting the same behavior to node a that one node running D in the covering graph exhibits, and the same behavior to nodes b and c as the other D in the covering exhibits. Then nodes a, b and c choose 1 in E 3, and so must the nodes running A, B and C in S3, contradicting the argument above that the node running A chooses 0. The general case for arbitrary c(G) < 2 f i s an easy generalization of the case f o r f = 1. The same picutres are used. Just choose b and d to be sets consisting of at m o s t f nodes each, such that removing the nodes in b and d from G disconnects two nodes u and v of G. Let G' be the graph obtained by removing b and d from G, let the set a contains those nodes connected to u, and the set c contains the remaining nodes of G' (c contains at least one node, v). Construct S as before, by taking two copies of G and rearranging edges between the ' a ' sets and their neighbors. The nodes and edges in our figures are now a shorthand for the actual nodes and edges of G and S. This completes the proof of Theorem 1. The succeeding impossibility results for other consensus problems follow the same general form as the two arguments above. We assume a problem can be solved by specific devices in an inadequate graph, G, install the devices in a graph S that covers G, and provide appropriate inputs. Using the Locality and Fault axioms, we argue the existence of a sequence of correct behaviors o f G that have node and edge behaviors identical to some of those in the behavior of S. (This sequence was (E 1, E 2, E3)), in the arguments above.) By the agreement condition, correct nodes in each of the behaviors of G have to agree. Because each successive pair of system behaviors has a correct node behavior in common, all of the correct nodes in all the behaviors in the sequence have to agree. But by the validity condition, correct nodes in the first behavior in the sequence must choose different values than those in the last behavior, a contradiction.

158

As we indicated in the introduction, a less general version of Theorem 1 was previously known, and the structure of our proof is very similar to that of earlier proofs [PSL], [D]. Our proof differs in the construction of the system behaviors E 1, E 2 and E 3. Earlier results construct these behaviors inductively using less general models of distributed systems. The detailed assumptions of the models are necessary to carry out the tedious and involved constructions. Rather than construct the behaviors explicitly, we build them from pieces (node and edge behaviors) extracted from actual runs of the devices in a covering graph. The Locality and Fault axioms imply that scenarios in the covering graph are found in correct behaviors of the original inadequate graph. The model used to obtain these results is an extremely general one, but it does assume that systems behave deterministically. (For every set of inputs, a system has a single behavior). By considering a system and inputs as determining a set of behaviors, nondeterminism may be introduced in a straightforward manner. One changes the Locality axiom to express the following. If there exist behaviors of two systems in which the inedge borders of two isomorphic subsystems are identical, there exist such behaviors in which the behaviors of the subsystems are also identical. Using this axiom, the same proofs suffice to show that nondeterministic algorithms cannot guarantee Byzantine agreement.

4. Weak Agreement Now we give our impossibility results for the weak agreement problem. As in the Byzantine agreement case, nodes have Boolean inputs and must choose a Boolean output. The agreement condition is the same as for Byzantine agreement--all correct nodes must choose the same output. The validity condition, however, is weaker.

Agreement. Validity.

Every correct node chooses the same value.

If all nodes are correct and have the same input, that input must be the value chosen.

The weaker validity condition has an interesting impact on the agreement problem. If any correct node observes disagreement or faulty behavior, then all are free to choose a default value, so long as they still agree. Lamport notes that there are devices for reaching a form of approximate weak consensus which work when IGI < 3f. Running these for an infinite time produces exact consensus (at the limit) [L]. In such infinite behaviors any correct node observing disagreement or faulty behavior has plenty of time to notify the others before they choose a value. Thus, strengthening the choice condition by prohibiting such infinite solutions is necessary to obtain the lower bound. If the communication delays are not bounded away from zero, a similar type of infinite behavior is possible. In fact, if there is no lower bound on transmission delay, and if devices can control the delay and have synchronized clocks, then we can construct an algorithm for reaching

159

weak consensus. This algorithm requires at most two broadcasts per node, each having non-zero transmission delay, and works with any number of faults. Again, this is because any correct node which observes disagreement or faulty behavior has plenty of time to notify the others before they choose a value. 4 In more realistic models it is impossible to reach weak consensus in inadequate graphs. To show this, the minimal semantics introduced in the previous sections must be extended to exclude infinitary solutions. We do this as follows. Previously, behaviors of nodes and edges were elements of some arbitrary set. Henceforth, we consider them to be mappings t~om [0,oo) (our definition of time), to arbitrary state sets. Thus, if E u is a behavior of node u, then u is in state Eu(t) at time t. We add the following condition to the weak agreement problem.

Choice. A correct node must choose 0 or 1 after a finite amount of time. This means there is a function CHOOSE from behaviors of nodes running weak agreement devices to {0,1 }, with the following property: Every such behavior E has a finite prefix Et (E restricted to the interval [0,t]) such that all beha~,iors E' extending Et have CHOOSE(E) = CHOOSE(E'). This choice condition prohibits Lamport's infinite solution. To prohibit the second solution, we bound the rate at which information can traverse the network. To do so, we add the following stronger locality axiom to our model.

Bounded-Delay Locality Axiom. There exists a positive constant 8 such that the following is true. Let G and G ' be systems with behaviors E and E', respectively, and isomorphic subsystems U and U', (with vertex sets U and U'). ff the corresponding behaviors o f the inedge borders of U and U' in E and E ' are identical through time t, then scenarios E U and E U, are identical through time t+8.

Thus, news of events k edges away from some subgraph G' takes time at least k8 to arrive at G'. In a model with explicit messages, this axiom would hold if the transmission delay is at least 8; the edge behaviors in our model would correspond to state descriptions of the transmitting end of each communications link. T h e o r e m 2: Weak agreement is not possible in inadequate graphs for models satisfying the Bounded-Delay Locality axiom. Again, we first sketch the 3f+l node bound. In this case the previously published proof [L]

4Nodes start at time 0, and decide at time 1. They broadcast their value at time 0, specifying it to arrive at time

1/2. If a node first detects disagreement or failure (at time 1-0, it broadcasts a "failure detected, ehoosa default value" message, specifying it to arrive at time 1-t/2. The obvious decision is made by everyone at time 1.

160

was very difficult. As before, we restrict our attention to the case IGI = n = 3 , f = I. (l'he case for generalf follows immediately, just as above.) Assume there are weak agreement devices A, B and C, for the triangle graph G containing nodes a, b and c. Consider the two behaviors of G in which all nodes are correct, and all have input 0 or all have input 1. Let t' be an upper bound on the time it takes all nodes to choose 0 or 1 in both behaviors. Choose k > t'/8 to be a multiple of 3. The covering graph S consists of 4k nodes, arranged in a ring and assigned devices and inputs as follows:

C--B-A 1

1

A--B--C 0

0

A--C--B--A-C

1

1

1

1

1

1

0

0

B--C--A-B-C

0

0

0

0

.....

C--B--~- 1

.....

A--B-C

1

0

1

0

0

Consider the resulting behavior S, and each pair of successive two-node scenarios, such as the two below.

....

4a÷

CtA

q i\ol ~,,+./~, 1

As before, each scenario is identical to a scenario in a behavior in G of the appropriate two weak consensus devices. Since each pair of successive scenarios overlaps in one node behavior (here, that of the node running B), the agreement condition requires that all the nodes in both scenarios must choose the same value in G and in S. Thus, every node in S must choose the same value. Without loss of generality, assume they choose 1. Consider the k scenarios indicated below.

C--B--A ~~1 I 1

oi o Sli

S2

A--C--B--A ..... 1 1 1 1

o

iol • "

o¢olo

C--B--A 1 1 1~

o

C ol o I

J

....................................

Let E be the behavior of G in which a, b and c are correct and each has input 0, and denote the resulting behaviors of a, b and c by Ea, Eb and E c, respectively.

161

Lemma 3: The behavior in scenario S i of a node running device A (or B or C) is identical to E a (or E b or E c) through time i8. Proof: An easy induction using the Bounded-Delay Locality axiom, essentially arguing that no device in S i can hear from a device with input 1 until after time i5. By Lemma 3, the nodes running devices C and A in scenario Sk have behaviors identical to E c and E a through time k8 > t'. Since nodes c and a in G have chosen output 0 by this time, so have the corresponding nodes in Sk, a contradiction. The general case of IGI < 3fand the connectivity bound follow as for Byzantine agreement. There are strong similarities between this argument and a proof by Angluin, concerning leader elections in rings and arbitrarily long lines of processors [A]. Both results depend crucially on the existence of a lower bound on the rate of information flow. Under this assumption, devices in different communication networks can be shown to observe the same local behavior for some fixed time.

5. Byzantine Firing Squad The Byzantine f'wing squad problem addresses a form of synchronization in the presence of Byzantine failures. The problem is" given an input stimulus, "synchronize the response of entering a designated FIRE state. This problem was studied originally in [BL]. In [CDDS], a reduction of weak agreement to the Byzantine firing squad problem demonstrates that the latter is impossible to solve in inadequate graphs. We provide a direct proof that a simple variant of the original problem is impossible to solve in inadequate graphs. (In the original version, the stimulus can arrive at any time. We require that it arrive either at time 0, or not at all. Our validity condition is slightly different.) The proof is very similar to that for weak agreement. One or more devices may receive a stimulus at time 0. We model the stimulus as an input of 1, and the absence of the stimulus as an input of 0. Correct executions must satisfy the following conditions. Agreement. If a correct node enters the FIRE state at time t, every correct node enters the FIRE state at time t. Validity. If all nodes are correct and the stimulus occurs at any node, all nodes enter the FIRE state after some finite delay. If the stimulus does not occur and all nodes are correct, no node ever enters the FIRE state. As in the case of weak agreement, solutions to the Byzantine firing squad problem exist in models in which there is no minimum communication delay. Thus, the following result requires the Bounded-Delay Locality axiom, in addition to the Fault axiom.

162

Theorem 4: The Byzantinefiring squad problem cannot be solved in inadequate graphsfor modelssatisfying the Bounded-DelayLocalityaxiom. We sketch the 3f+1 node bound. As before, we examine the case IGI = n = 3,f = 1. Assume there are Byzantine firing squad devices A, B and C for the triangle graph G containing nodes a, b and c. Consider the two behaviors of G in which all nodes are correct, and all have input 0 or all have input 1. Let t be the time at which the correct devices enter the FIRE state in the case that the stimulus occurred (the input i case). Choose k >-t/~ to be a multiple of 3. (Recall that 5 is the minimum transmission delay defined in the Bounded-Delay l.x~ality axiom). The covering graph S consists of 4k nodes, arranged in a ring and assigned devices and inputs as follows:

C--B--A ( 1 1 1 A_B_C

0

0

.....

0

.....

A--C--B--A-C 1 1 1 1 1 B--C--A--B-C

0

0

0

C--B--A 1 1 1

0

0

.....

A--B--C

0

0

1

0

Similarly to the proof for weak agreement, the middle two devices receiving the stimulus enter the FIRE state at time t, as their behavior through time t is the same as that of the correct nodes in G which have received the stimulus and fire at time t. Because of the communication delay, there is not enough time for "news" from the distant nodes to reach these devices. By repeated use of the agreement property, all the devices in S must f'tre at time t. But through time t, the middle two devices not receiving the stimulus behave exactly as correct nodes in G which do not receive the stimulus (the input 0 case). Thus, they do not fn'e at time t, a contradiction.

6. Approximate Agreement Next, we turn to two versions of the approximate agreement problem [DLPSW,MS]. We call have real values as inputs and choose real numbers as a result. The goal is to have the results close to each other and to the inputs. To obtain the strongest possible impossibility result, we formulate very weak versions of the problems.

them simple approximateagreementand (~,5,y)-agreement. In these problems nodes

For the following two theorems we use only the Locality and Fault axioms. We do not need the Bounded-Delay Locality axiom used for the weak agreement and firing squad results.

6.1. SimpleApproximateAgreement First, examine a version of the simple approximate agreement problem [DLPSW]. Each correct node has a real value from the interval [0,1] as input, runs its device, and chooses a real value. Correct behaviors (those in which at least n - f nodes are correct) must satisfy the following conditions.

163

Agreement. The maximum difference between values chosen by correct nodes must be strictly smaller than the maximum difference between the inputs, or be equal to the latter difference if it is zero. Validity. Each correct node chooses a value within the range of the inputs of the nodes. Theorem 5: Simple approximate agreement is not possible in inadequate graphs. The proof is almost exactly that for Byzantine agreement. Here, we consider devices which take as inputs numbers from the interval [0,1], and choose a value from [0,1] to output. (Outputs are modeled by a function CHOOSE from behaviors of nodes running the devices to the interval [0,113 As before, assume simple approximate agreement can be reached in the triangle graph G. Consider the following three scenarios from the indicated behavior in the covering graph S.

A -/0

C 1

B

B ,':

iC

:

-Ai

11

Again, each scenario is also a scenario in a correct behavior of G. In the f'n'st scenario, the only value C can choose is 0. In the third, the only value A can choose is 1. This means the values chosen by A and C in the the second scenario are 0 and 1, so that the outputs are no closer than the inputs, violating the agreement condition. The general case of IGI < 3f and the connectivity bounds follow as for Byzantine agreement.

6.2. (e,~,7)-Agreement This version of approximate agreement is based on that in [MS]. Let e, ~ and "/be positive real numbers. The correct nodes receive real numbers as inputs, with rmin and rmax the smallest and largest such inputs, respectively. These inputs are all at most ~ apart (i.e. the interval of inputs [rmin, rmax] has length at most 5). They must choose a real number as output, such that correct behaviors (those in which at least n - f nodes are correct) satisfy the following conditions.

Agreement. The values chosen by correct nodes are all at most E apart. Validity. Each correct node chooses a value in the interval [rmin-7,rmax+'y].

164

Note that if e -> 8, (e,8,y)-agreement can be achieved trivially by choosing the input value as output. Theorem 6: If e < 8, (e,8,~l)-agreement is not possible in inadequate graphs.

Proof." Let e, 8 and y be positive real numbers with e < 8. We prove only the 3f+1 bound on the number of nodes. Assume that devices A, B and C exist which solve the (e,8,y)-approximate agreement problem in the complete graph G on three nodes for particular values of e, 8 and y, where e < 8. Choose k sufficiently large that 8 > 2y/(k-1) + e, and k+2 is divisible by three. The covering graph S contains k+2 nodes arranged in a ring, with devices and inputs assigned to create the following system.

fa_, node index

input

0

1 .

0

2.

k

k+l

Let Si, for 0 _< i < k, denote the two-node scenario in S containing the behaviors of nodes i and i+l. By the Fault Axiom, each scenario Si is a scenario of a correct behavior of G, in which the largest input value to a correct node is (i+1)8.

Lemma 7: For 0 < i < k, the value chosen by the device at node i+1 is at most 8 + y +ie. Proof: The proof is a simple induction. By validity applied to scenario S 0, the device at node 1 chooses at most 8 + y. Assume inductively that the device at node i chooses at most 8 + y + (i-1)e, for 0 < i < k+l. By agreement applied to scenario S i, the device at node i+1 chooses at most 8 + y + ie. In particular, Lemma 7 implies the device at node k chooses at most 8 + y + (k-1)E. But validity applied to scenario Sk implies the device at node k chooses at least k8 - y. So k8 - y < 8 + y + (k-1)e. This implies 8 < 27/(k-1) + e, a contradiction. The general case of IGI < 3f and the connectivity bounds follow as in previous proofs.

7. Clock Synchronization Each node has a hardware clock and maintains a logical clock. The hardware clocks are realvalued, invertible and increasing functions of time. Since different hardware clocks run at different rates, it may be necessary to synchronize the logical clocks more closely than the

165

hardware clocks. In addition, logical clocks should be reasonably close to real time--setting them to be constantly zero, for example, should be forbidden. Thus, we require the logical clocks to stay within some envelope of the hardware clocks. [See the paper by Lundelius, Lynch and Simons in this volume.] This problem was studied in [DHS] for the case of linear clock and envelope functions, where it was shown that it is impossible to synchronize to within a constant in inadequate graphs. Some more general synchronization issues were raised, such as that diverging linear clocks can be synchronized to within a constant if nodes can run their logical clocks as the logarithm of their hardware clocks. For a large class of clock and envelope functions (increasing and invertible clocks, non-decreasing envelopes), we can characterize the best synchronization possible in inadequate graphs. This synchronization requires no communication whatsoever. We model node i's hardware clock, D i, as an input to the device at node i that has value Di(t) at time t. The value of the hardware clock at time t is assumed to be part of the state of the node at time t, The time on node i's logical clock at real time t is given by a function of the entire state of node i. Thus, if E i is a behavior of node i (such that node i is in state Ei(t) at time t), then we express i's logical clock value at time t as Ci(Ei(t)). We assume that any aspect of the system which is dependent upon time (such as transmission delay, minimum step time, maximum rate of message transmission) is a function of the states of the hardware clocks. Having made this assumption, it is clear that speeding up or slowing down the hardware clocks uniformly in a behavior E cannot be observable to the nodes, so that the only impact on a E should be to change the (unobserved) real times at which events occur. To formalize this assumption, we need to talk about scaling clocks and behaviors. Let h be any invertible function of time. If E is a behavior (of a edge or node), then Eh, the behavior E scaled by h, is such that Eh(t)=E(h(t)), for all times t. Similarly, Dh is the hardware clock D scaled by h: Dh(O=D(h(t)). If E is a system behavior or scenario, Eh is the system behavior or scenario obtained by scaling every node and edge behavior in E by h. Similarly, if S is a system, then Sh is the system obtained by scaling every clock in S by h. Intuitively, a scaled clock or behavior is in the state at time t that the corresponding unsealed clock or behavior is in at time

h(t). Scaling Axiom

If E is the behavior of system S, then Eh is the behavior of system Sh.

If this axiom is significantly weakened, as by bounding the transmission delay, clock synchronization may be possible in inadequate graphs. In the following we use the Locality, Fault and Scaling axioms. We do not need the BoundedDelay Locality axiom used for the weak agreement and fitting squad results. The synchronization problem can be stated as follows. Let correct hardware clocks run either

166

at p(t) or q(t), where p and q are increasing, invertible functions, with p(t)

167

u(q(r)).

The covering graph S contains k+2 nodes arranged in a ring, with devices and clock inputs assigned to create the following system.

8__c)

A -- B

.k

k+l

node index

0

1

h a r d w a r e clock

g

gh

• gh~ gt~(k+~)

behavior

Eo

E1

.E k

-1

Ek+1

Let S be the behavior of this system. An initially troubling concern is that the hardware clocks in S are much slower in most of the devices in S than they would be in a correct behavior in G. But consider S i, the two-node scenario containing the behaviors of nodes i and i+1, where 0 -< i hi(t'). Then h'i(t) > t'. By Lemma 9, i and i+1 are correct in Si hi, so by the agreement assumption ICi+l(Ei+lhi(h'i(t)))- Ci(Eihi(h-i(t)))l < l(q(h'i(t))) - l(p(h'i(t))) - cx. The result is immediate. Let time t" = hk(t' ). Note that t" >_ hi(t' ), for i < k.

Lemma 11: For alt i, 1 _ l(p(t)). Setting t = t", and substituting qh "1 forp, we have the basis step: CI(EI(t")) >_ l(qh'l(t")). Now make the inductive assumption Ci(Ei(t")) >_ l(qh'i(t")) + (i-1)cz, for I < i < k. Since t" >_ hi(t'), from Lemma I0, we know ICi+l(Ei+l(t")) - Ci(Ei(t"))l < l(qh'i(t")) l(ph-i(t")) - or. This implies Ci+l(Ei+l(t")) >_ C~(Ei(t"))- l(qh'i(t")) + l(ph'i(t")) + ct. } Substituting for C~Ei(t")) using the inductive assumption gives us Ci+l(Ei+l(t")) >- l(qh-i(t")) l(qh-i(t")) + l(ph-i(t")) + ict = l(ph'i(t")) + ion. Noting that p = qh "1, we have the result, Ci+l(Ei+l(t") ) >_ l(qh-(i+l)(t")) + iot. Proof of Theorem 8: Lemma 11 implies Ck+l(Ek+l(t")) > l(qh'(k+1)(t")) + kot. Since t" = hk(t'), we have Ck+l(Ek+l(t") ) = Ck+l(Ek+l(hk(r ))) = Ck+l(ek+lhk(t')) > l(qh-(k+l)hk(t')) + ktx = l(p(t')) + kot. But the upper envelope constraint for the scaled scenario Skhk (in which k+l is correct and has hardware clock p(t)) implies that Ck+l(Ek+lhk(t')) < u(q(t')). Thus, l(p(t')) + kcx < u(q(t')). This violates the assumed bound on k, l(p(t' )) + k(x > u(q(t' )). Once again, the general case of IGI _< 3f is a simple extension of this argument. connectivity bound also follows easily, as with the earlier results.

The

7.1. Linear Envelope Synchronization and other Corollaries Linear envelope synchronization, as defined in [DHS], examines the synchronization problem when the clocks and envelope functions arc linear functions (q(t)=rt, p(t)=t, l(O=at+b and u(t)=ct+d). It requires correct logical clocks to remain within a constant of each other, so that the agreement condition is ICi(Et~t)) - Cj(Ej(t))l < a, for all times t, instead of our weaker

169

condition ICi(Ei(t)) - CJEJt))I < art - at - t~, for all times t > t'. Our validity condition is slightly weaker, as well. Thus, the proof of [DHS] shows that logical clocks cannot be synchronized to within a constant; we show that that the synchronization of logical clocks cannot be improved by a constant over the synchronization (art - at) that can be achieved trivially. Thus the following corollary follows immediately from Theorem 8. (Each of the four corollaries below holds for models satisfying the Scaling axiom.)

Corollary 12: Linear envelope synchronization is not possible in inadequate graphs. By choosing specific values for the clock and lower envelope functions, we get the following additional results immediately from Theorem 8. Note that the particular choice of the upper envelope :function does not affect the minimal synchronization possible in inadequate graphs, although the existence of some upper envelope function is necessary to obtain our impossibility proofs.

Corollary 13: If p(t)=t, q(t)=rt, and l(t)=at+b, no devices can synchronize a constant closer than art-at in inadequate graphs. Corollary 14: If p(t)=t, q(t)=t+c and l(t)=at+b, no devices can synchronize a constant closer than ac in inadequate graphs. Corollary 15: If p(t)=t, q(t)=rt and l(O=log2(O, no devices can synchronize a constant closer than log2(r ) in inadequate graphs. In general, the best possible synchronization in inadequate graphs can be achieved without any communication at all. The best that nodes can do is run their logical clocks as slowly as they are permitted, C(E(t)) = l(D(t)).

8. Conclusion Most of the results we have presented were previously known. While our proofs are both simpler than earlier proofs, and apply to more general models, these are not the main contributions. The simplicity and generality are welcome byproducts of our attempt to identify the fundamental issues and assumptions behind a collection of similar results. One important contribution is to elucidate the relationship between the unrestricted or Byzantine failure assumption and inadequate graphs. As is clear from our proofs, this fault assumptio~a permits faulty nodes to mimic executions of disparate network topologies. If the network is inadequate, a covering graph can be constructed so that correct devices cannot distinguish the execution in the original graph from one in the covering graph. A second contribution is related to the generality of our results. Nowhere do we restrict state sets or transitions to be finite, or even to reflect the outcome of effective computations. The

170

inability to solve consensus problems in inadequate graphs has nothing to do with computation per se, but rather with distribution. It is the distinction between local and global state, and the uncertainty introduced by the presence of Byzantine faults, which result in this limitation. Finally, we have identified a small, natural set of assumptions upon which the impossibility results depend. For example, in the case of weak agreement and the firing squad problem, the correcmess conditions are sensitive to the actions of faulty nodes. Instantaneous notification of the detection of fault events would allow one to solve these problems. An assumption that there are minimum delays in discovering and relaying information about faults is sufficient to make these problems unsolvable.

9. References [A]

[B] [BL] [CDDS] [D] [DHS] [DLPSW]

[]:R] ILl [LSPI [MSI [PSL]

D. Angluin, "Local and Global Properties in Networks of Processors," Proc, of the 12th STOC, April 30-May 2, 1980, Los Angeles, CA., pp. 82-93. J. Bums, "A Formal Model for Message Passing Systems," TR-91, Indiana University, September 1980. L Bums, N. Lynch "The Byzantine Firing Squad Problem," submitted for publication. B. Coan, D. Dotev, C. Dwork and L. Stockmeyer "The Distributed Firing Squad Problem," Proc. of the 17th STOC, May 6-8, 1985, Providence R.I. D. Dolev, "The Byzantine Generals Strike Again," Journal of Algorithms, 3, 1982, pp. 14-30. D. Dolev, J. Halpem, H. Strong, "On the Possibility and Impossibility of Achieving Clock Synchronization," Proc. of the 16th STOC, April 30-May 2, 1984, Washington, D.C., pp. 504-510. D. Dolev, N. A. Lynch, S. Pinter, E. Stark and W. Weihl, "Reaching Approximate Agreement in the Presence of Faults," Proc. of the 3rd Annual IEEE Syrup. on Distributed Sofware and Databases, 1983. A. Itai, M. Rodeh, "The Lord of the Ring or Probabilistic Methods for Breaking Symmetry in Distributive Networks," RJ-3110, IBM Research Report, April 1981. L. Lamport, "The Weak Byzantine Generals Problem", JACM, 30, 1983, pp. 668-676. L. Lamport, R. Shostak, M. Pease, "The Byzantine Generals Problem," ACM Trans. on Programming Lang. and Systems 4, 3 (July 1982), 382-401. S. Mahaney, F. Schneider ,"Inexact Agreement: Accuracy, Precision, and Graceful Degradation," Proc. of the 4th Annual ACM Symposium on Principles of Distributed Computing, August 5-7, 1985, Minacki, Ontario. M. Pease, R. Shostak, L. Lamport, "Reaching Agreement in the Presence of Faults," JACM 27:2 1980, 228-234.

AN EFFICIENT, FAULT-TOLERANT PROTOCOL FOR REPLICATED DATA MANAGEMENT

Dale Skeen Teknekron Software Systems Palo Alto, California Amr El Abbadi Computer Science Department University of California, Santa Barbara Flaviu Cristian IBM Almaden Research Center San Jose, California

ABSTRACT A data management protocol for executing transactions on a replicated database is presented. The protocol ensures one-copy serializability, i.e., the concurrent execution of transactions on a replicated database is equivalent to some serial execution of the same transactions on a non-replicated database. The protocol tolerates a large class of failures, including: processor and communication link crashes, partitioning of the communication network, lost messages, and slow responses of processors and communication links. Processor and Link recoveries are also handled. The protocol implements the reading of a replicated object efficiently by reading the nearest available copy of the object. When reads outnumber writes, the protocol performs better than other known protocols.

1. INTRODUCTION The objective of data replication in a distributed database system is to increase data availability and decrease data access time. By data replication, we mean maintaining several physical copies, usually at distinct locations, of a single logical data base object. To make the replication transparent to the user of an object, a replica control protocol is needed to coordinate physical accesses to the copies of a logical data object and to

172

guarantee that they exhibit behavior eqm'valent to that of a single copy object [BGb]. Such a protocol translates a logical writo of a data object x into a set of physical writes on copies of x, and translates a logical read of x into a set of reads on one or more physical copies of x. To increase data availability effe.ctively, a replica control protocol must be tolerant of commonly occurring system component failures. To minimize the overhead caused by replication, the protocol should minimize the number of physical accesses required for implementing one logical access. This paper outlines a replica control protocol that tolerates a large class of failures: processor and communication link failures, partitioning of the communication network, lost messages, and slow responses of processors and communication links. It also handles any number of possibly simultaneous processor and link recoveries. The major strength of the protocol is that it implements the reading of a logical object very efficiently: a read of a logical object, when perrrfiRed, is accomplished by accessing only the nearest available physical copy of the object. In applications where reads outnumber writes, this strategy will reduce the total cost of accessing replicated data objects. There arc two possible approaches to develop a fault-tolerant replica control protocol in such an environment. The first is the status-oblivious approach of which the quorum consensus algorithm [(3] is a well known example. In this protocol a processor executing a logical operation first sends out messages to a/l processors containing copies of the object requesting the execution of the operation. It then waits for a response from a quorum. If as a result of failures, a quorum of processors does not respond, the operation is aborted. The second is the status-dependent approach. The execution of operations depends on the knowledge by each processor of the communication topology, and this knowledge is used to decide on the appropriate set of sites with which to communicate when executing a logical operation. In this paper we use a status dependent approach where each processor maintains a view of the current communication topology. Views arc used to optimize the translation of logical data operations into physical data accesses. Ideally, views should reflect the actual communication topology, but instantaneous detection of failures and recoveries is not possible. In our protocol, a processor's view is an approximation of the set of sites with which it can communicate. Views arc maintained by a sub-protocol of the replica control algorithm. This protocol guarantees that views satisfy a set of well defined properfies. In ~SC], it was proven that these propct~s arc sufficient for the replica control protocol to exhibit to users the behavior of a database where each logical object is implemented by a single copy. The protocol compares favorably with other proposed replica control protocols. It tolerates the same failure classes as majority voting [T] and quorum consensus [G]. It requires fewer accesses to copies, assuming that read requests outnumber write requests and that failure occurrences are rare events. It also tolerates the same failure classes as the "missing write" protocol [ES], but, unlike that protocol, uses a "read-one" rule for reading logical data objects even in the presence of failures. Our protocol is also simpler than the "missing write" protocol In particular, it does not require the extra logging of transaction information that is required by that protocol when failures occur.

173

For a more detailed description of the replica control protocol and for a proof of correctness, the reader is referred to [ESC].

2. FAILURE ASSUMtrrlONS System components (processors, links) can fail in many ways, from occasional processor crashes and lost messages to Byzantine failures, where components may act in arbitrary, even malicious, ways. We consider only those failures that have a reasonable chance of occurring in practical systems and that can be handled by algorithms of moderate complexity and cost. The most general failure classes satisfying these criteria are omission failures and performance failures [CASD,H]. An omission failure occurs when a component does not respond to a service request. Typical examples of such failures include processor crashes, occasional message losses due to transmission errors or overflowing buffers, and communication link crashes. A performance failure occurs when a system component fails to respond to a service request within the time limit specified for the delivery of that service. Message delays caused by overloaded processors and network congestion are examples of performance failures. An important subclass of omission and performance, failures is the class of partition failures. A partition failure divides a system into two or more disjoint sets of processors, where no member of one set can communicate in a timely manner with a member of another set. Our objective is to design a replica control protocol that is tolerant of any number of omission and performance failures.

3. SYSTEM MODEL A distributed system consists of a finite set of processors, P={ 1,2 ..... n}, connected by a commur~cation network. In the absence of failures the network provides the service of routing messages between any two processors. Processors or links may fail, leading to an inability to communicate within reasonable delays. Failed processors and links can recover spontaneously or because of system maintenance. Thus, the system of processors that can communicate with each other is a dynamically evolving system. In the following discussion, we will not be concerned with the details of the. physical interconnection of the processors (e.g. a point-to-point versus a bus-oriented interconnecdon) or with the detailed behavior of the message muting algorithm. Instead, we consider only whether two processors are capable of communicating through messages. We model the current can-communicate relation between processors by a communication graph. The nodes of the graph represent processors, and an undirected edge between two nodes a,b~ P indicates that ff a and b send messages to each other, these are received within a specified time limit. We call a connected component of a communication graph a communication cluster. A communication clique is a communication cluster which is totally connected, that is, there is an edge in the communication graph between every pair of processors in the cluster. We do not assume that the can-communicate relation is

174

transitive. Thus, it is possible that a and b can communicate, and b and c can communicate, but a and c cannot communicate. (Note that if the can-communicate relation is transitive, then all communications clusters are cliques.) In a system where failure occurrences lead quickly to the establishment of new communication routes that avoid the failed system components, communication clusters can be expected to be cliques most of the time. In the absence of failures, a communication graph is a single clique. The crash of a processor p results in a graph that contains two clusters: a trivial cluster consisting of the single node p, and a duster consisting of all other nodes. A partition failure results in a graph containing two or more clusters. For the purpose of adapting to changes in the communication topology, each processor maintains a local "view" of the can-communicate relation. Each processor's view is that processor's current estimate of the set of processors with which it believes that communication is possible. The function view: P --> 2P (where 2p denotes the powerset of P) gives the current view of each processor pc P. A replicated database consists of a set of logical data objects L. Each logical object 1~ L is implemented by a nonempty set of physical data objects (the copies of 1) that are stored at different processors. The copy of 1 stored at processor p is denoted by lp. The function copies: L --->2I, gives for each logical object I the set of processors that possess physical copies of 1. Transactions issue read and write operations on logical objects. A replicated data management protocol is responsible for implementing logical operations (as they occur in transactions) in terms of physical operations on copies. The term event is used to denote a primitive atomic action in the system. A primitive action is an operation that is executed locally on a site, such as the reading and writing of a physical object and the sending and receiving of messages. An execution of a set of transactions is a finite set of events partially ordered by the happens-before relation studied in [L]. We assume that the set of events restricted to a given processor is totally ordered by the happens-before relation. That is to say, if e and f are events occurring at the same processor, then either e happens-before f or f happens-before e. Consequently, the operations executed on a given physical object are totally ordered. An execution is serial if its set of events is totally ordered by the happens-before relation, and if, for every pair of transactions t 1 and t2, either all physical data operations of t 1 happen-before all physical operations of tg, or vice versa. For a replicated data management protocol to be correct, the database system must exhibit the same externally observable behavior as a system executing transactions serially in a nonreplicated database system [TGGL]. This property is known as one-copy serializability [BGb]. One popular approach for designing a replicated data management protocol is to decompose the algorithm into two parts: a replica control protocol that translates each logical operation into one or more physical operations, and a concurrency control protocol that

175

synchronizes the execution of physical operations ~Gb]. The concurrency control protocol ensures that an execution of the translated transactions (in which logical access operations are replaced by physical operations) is scrializable,that is,equivalent to some scriai execution. But the concurrency control protocol does not ensure one-copy serializability,since it knows nothing about logicalobjects. (Itmay, for example, permit two distincttransactionsto update differentcopies of the same logical object in parallel.)The replica control protocol ensures that transactionexecution is one-copy serializable.

4. REPLICA CONTROL Following the decomposition outlined above, we now derive a protocol for correctly managing replicated data in the presence of any number of omission and performance failures. In this section, the emphasis is on formulating the requirements for a replica control protocol and on showing that any implementation satisfying these requirements satisfiesour correctness criteria. In the next section,we describe in some detailone protocol and show that itexhibits the desired properties. Ideally, we would like to design a replica control protocol that can be combined with any concurrency control protocol that ensures scrializability. However, this seems to be difficultto achieve given our performance objectives. Consequently, we will restrictthe class of allowable concurrency control protocols to those ensuring a stronger property known as conflict-preservingscrializability[H]. T w o physical operations conflictif they operate on the same physical object and at least one of them is a write. An execution E of a set of transactions T is conflict-preserving(CP) serializableif there exists an equivalent serialexecution E s of T that preserves the order of execution of conflicting operations (i.e.if opl and oP2 are conflictingphysical operations and opl happens-before oP2 in E, then opl happens-before oP2 in Es) [EGLT,H]. Henceforth in our discussion, we will assume the existence of a concurrency control protocol ensuring that (AI) The execution of any set of transactions (viewed as a set of physical operations) is conflict-preservingscrializable.

Practically speaking, restricting the class of concurrency control protocols to those enforcing CP-serializability is inconsequential, since all published, general-purpose concurrency control protocols are members of this class. This includes two-phase locking [EGLT], optimistic concurrency control [KR], timestamp ordering [BSR], and all distributed protocols surveyed by Bemstein and OoodmRn [BGa]. Our performance objective is to provide cheap read access while offering a high level of data availability. In order to understand better what is attainable, let us first consider a "clean" failure environment in which two simplifying assumptions hold. The first assumption is that the can-communicate relation is transitive: (A2) All communication clusters are cliques.

176

The second assumption (unrealistically)posits that changes in the communication topology (resultingfrom failures and recoveries) are instantlydetected by all affected processors. (A3) The view of each processor contains only the processors adjacent to itin the current communication graph, itselfand no other processors. Thus, from A2 and A3 we can conclude that the views of processors in the same communication cluster are equal and the views of processors in different clusters are disjoint. Given the above assumptions, the following rules can be used to control access to logical objects. W h e n processor p executes a read or a write operation on a logical object l, it firstchecks whether a (possibly weighted) majority of the copies of I reside on processors in itslocal view. If not, it aborts the operation. Otherwise, for a read, itreads the nearest copy which resides on a processor in itsview; and for a write, itwrites all copies on proccssors in itsview. When integrated with an appropriate cluster initialization protocol, which ensures that aLi copies of a logical object accessible in a newly established cluster have the most up-todate value assigned to that object, the above rules can form the basis of a correct replica control protocol. The "majority rule" ensures that only one cluster can access a logical object at a time, and the "read-one/write-all rule" ensures that the copies of an object in a cluster act as a single copy. Together, these rules ensure that all executions are one-copy scrializable. The above rules are simple, intuitive, and ensure a high level of data availability, provided the communication information maintained by the processors is accurate. Unfortunately, the correctness of the rules depends heavily on assumptions A2 and A3. If either is relaxed, non-one-copy-serializable executions can result. Example 1. Figure 1 gives a possible communication graph for three processors when assumption (A2) is relaxed. The graph indicates that processors A and B are no longer able to communicate due to, for example, failures that have occurred in the message routing protocol. Both processors however arc able to communicate with C, and C with them. We thus have: view(A)={A,C}, view(B)f{B,C} and view(C)f{A,B,C}. Let each processor contain a copy of a logical data object x initialized to 0. Assuming that aLi copies are weighted equally, each processor will consider x to be accessible, since each has a majority of the copies in its view. Now, let A and then B execute a transaction that increments x by 1. Based on its own view, processor A reads its local copy of x and updates both its copy and C's copy. Similarly, B reads its local copy of x (which still contains 0) and updates both its copy and C's copy. Observe that after two successive increments, all copies of x contain 1. Clearly, the execution of these transactions is not one-copy serializable.

177

Figure 1. Example 2. Consider an initially partitioned system that undergoes repartitioning as shown in Figure 2. Two processors B and D detect the occurrence of the new partition immediately and update their views. The other two processors (A and C) do not detect it until later. Table 1 shows the intermediate system state after the view updates in B and D and before the view updates in A and C.

Figure 2.

t78

original view

current view ......

A,B A,B C, D C,D

A,B B,C C, D A,D

A B C D

Table I

Assume that, while the views are inconsistent,each processor p executes a transaction tp. Table 2 gives the transaction executed at each processor, and also the data objects stored there.

A

cooies a2,b

B

b2,c

transactions tA: read(b),write(a) ts: read(c),write(b)

C

c2,d

tc: read(d),write(c)

D

d2,a

tD: read(a),write(d)

Table 2

The superscripts on the objects denote "weights". Consider the execution of tA at processor A. Since Bedew(A), A can read its local copy orb. Since A's copy of a has weight 2, A can update it, and, furthermore, A will not attempt to update D's copy since D 4 view(A). Hence, in the execution of the transaction, A accesses itslocal copies only. The executions of transactions is, tc, and tD proceed similarly,with each processor accessing only its local copies. Since no accesses to remote copies arc made, the inconsistency between processor views is not detected. The resulting execution is serializable but not one-copy serializable. As example 1 illustrates,the correctness of the simple replica control protocol critically depends on the property that no two processors with different views be able to access a common set of copies. Example 2 illustratesthat even in a well-behaved communication network, where u-ansitivityof the can-communicate relation is assured, processors can not independently and asynchronously update theirviews. The principal idea in our replica control protocol is to use the majority and the read/write rules mentioned above, but to circumvent the anomalies illustrated in examples 1 and 2 by placing appropriate restrictions on when and how processors may update their views.

179

Toward this goal, we introduce the notion of a virtual partition. Roughly speaidng, a virtual partition is a set of communicating processors that have agreed on a common view and on a common way to test for membership in the partition. For the purposes of transaction processing, only processors that are assigned to the same virtual partition may communicate. Hence, a virtual partition can be considered a type of "abstract" communication cluster where processors join and depart in a disciplined manner. In contrast, in a real communication cluster processors join and depart abruptly (and often inopportunely) because of failures and recoveries. It is desirable, of course, for virtual partitions to approximate the real communication capabilities of a system. The common view of the members in a virtual partition represents a shared estimation of the set of processors with which communication is believed possible. When a processor detects an inconsistency between its view and the can-communicate relation (by not receiving an expected message or by receiving a message from a processor not in its view), it can unilaterally depart from its current virtual partition. Note that the capability for a processor to depart unilaterally is an important capability: since the departing processor may no longer be able to communicate with the other members of its virtual partition, it must be able to depart autonomously, without communicating with any other processor, l~ter departing, the processor can invoke'a protocol to establish a new virtual partition. This protocol, which is part of the replica control protocol, m a t e s a new virtual partition, assigns a set of processors to the partition, and updates those processors' views. ,~aa objective of the protocol is for the new virtual partition to correspond to a maximal set of communicating processors. However, since failures and recoveries can occur during the execution of the view update protocol, it is possible that a virtual partition resulting from a protocol execution only partially achieves this objective. We identify virtual partitions by unique identifiers, and we denote the set of virtual partition identifiers by V. At any time, a processor is assigned to at most one virtual partition. The instantaneous assignment of processors to virtual partitions is given by the partial function vp: P --> V, where vp is not defined for a processor p if p is not assigned to any virtual partition. We use the total function defview: P --+ {true,false} to characterize the domain of vp. That is, defview(p) is true if p is currently assigned to some virtual partition, and is false otherwise. The function members: V -+ 2P yields for each virtual partition v the set of processors that were at some point in time (but not necessarily contemporaneously) assigned to v. In order to ensure that the simple "read-one/write-air' rules achieve one-copy serializability, we require the following properties from any protocol managing processor's views and their assignment to virtual partitions. If p and q are arbitrary processors, the first two properties are

180

(S1) View consistency: view(p)=view(q).

If

defview(p)&defview(q)

and

vp(p)-vp(q),

then

($2) Reflexivity: If defview(p) then p ¢ view(p) Property S 1 states the requirement that processors assigned to the same virtual partition have the same view. With a slight abuse of notation, we let view(v) denote the view common to all members of virtual partition v. Property $2 enforces the requirement that every processor should be able to communicate with itself. From $1 and $2, one can infer that the view of a processor (when defined) is a superset of the processors in its virtual partition, and thereby, a superset of the processors with which it may actually communicate during transaction processing. The final property restricts the way a processor may join a new virtual partition. Let p denote any processor and let v and w denote arbitrary virtual partitions. Let join(p,v) denote the event where p changes its local state to indicate that it is currently assigned to v. Similarly, let depart(p,v) denote the event of p changing its local state to indicate that it is no longer assigned to v. Join and departure events, in addition to physical read and write events, are recorded in the execution of transactions. The third property is ($3) Serializability of virtual partitions: For any execution E produced by the replicated data management protocol, the set of virtual partition identifiers occurring in E can be totally ordered by a relation