FOUNDATIONS OF DEPENDABLE COMPUTING Paradigms for Dependable Applications
THE KLUWER INTERNATIONAL SERIES IN ENGINEERING AND COMPUTER SCIENCE OFFICE OF NAVAL RESEARCH Advanced Book Series Consulting Editor Andre M. van Tilborg Other titles in the series: FOUNDATIONS OF DEPENDABLE COMPUTEVG: Models and Frameworks for Dependable Systems, edited by Gary M. Koob and Clifford G. Lau ISBN: 0-7923-9484-4 FOUNDATIONS OF DEPENDABLE COMPUTING: Implementation, edited by Gary M. Koob and Clifford G. Lau ISBN: 0-7923-9486-0
System
PARALLEL ALGORITHM DERIVATION AND PROGRAM TRANSFORMATION, edited by Robert Paige, John Reif and Ralph Wachter ISBN: 0-7923-9362-7 FOUNDATIONS OF KNOWLEDGE ACQUISITION: Cognitive Models of Complex Learning, edited by Susan Chipman and Alan L. Meyrowitz ISBN: 0-7923-9277-9 FOUNDATIONS OF KNOWLEDGE ACQUISITION: Machine Learning, edited by Alan L. Meyrowitz and Susan Chipman ISBN: 0-7923-9278-7 FOUNDATIONS OF REAL-TIME COMPUTING: Formal Specifications and Methods, edited by Andre M. van Tilborg and Gary M. Koob ISBN: 0-7923-9167-5 FOUNDATIONS OF REAL-TIME COMPUTING: Scheduling and Resource Management, edited by Andre M. van Tilborg and Gary M. Koob ISBN: 0-7923-9166-7
FOUNDATIONS OF DEPENDABLE COMPUTING Paradigms for Dependable Applications
edited by Gary M. Koob Clifford G. Lau Ojfice of Naval Research
KLUWER ACADEMIC PUBLISHERS Boston / Dordrecht / London
Distributors for North America: Kluwer Academic Publishers 101 Philip Drive Assinippi Park Norwell, Massachusetts 02061 USA Distributors for all other countries: Kluwer Academic Publishers Group Distribution Centre Post Office Box 322 3300 AH Dordrecht, THE NETHERLANDS
Library of Congress Cataloging-in-Publication Data A C L P . Catalogue record for this book is available from the Library of Congress.
Copyright ® 1994 by Kluwer Academic Publishers All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photo-copying, recording, or otherwise, without the prior written permission of the publisher, Kluwer Academic Publishers, 101 Philip Drive, Assinippi Park, Norwell, Massachusetts 02061 Printed on acid-free paper. Printed in the United States of America
CONTENTS
Preface Acknowledgements
vii xiii
1 . PROTOCOL-BASED PARADIGMS FOR DISTRIBUTED APPLICATIONS
1
1.1
Adaptive System-Level Diagnosis in Real-Time R.P. Bianchini, Jr. and M, Stahl
3
1.2
Refinement for Fault-Tolerance: An Aircraft Handoff Protocol K. MarzullOy F.B. Schneider, and J. Dehn
1.3
Language Support for Fault-Tolerant Parallel and Distributed Programming /?.£). Schlichtingy D,E. Bakken, and VT. Thomas
2. ALGORITHM-BASED PARADIGMS FOR PARALLEL APPLICATIONS 2.1
2.2
Design and Analysis of Algorithm-Based Fault-Tolerant Multiprocessor Systems S. Yajnik and N.K. Jha Fault-Tolerance and Efficiency in Massively Parallel Algorithms P.C. Kanellakis and A.A. Shvartsman
39
55
79 81 125
VI
3 . DOMAIN-SPECIFIC PARADIGMS FOR REAL-TIME SYSTEMS 3.1
3.2
Use of Imprecise Computation to Enhance Dependability of Real-Time Systems J,W.S, Liu, K-J Lin, R. Bettati, D, Hull and A, Yu
155 157
Analytic Redundancy for Software Fault-Tolerance in Hard Real-Time Systems 183 M. Bodson, J.P. Lehoczky, R. Rajkumar, L. Sha, and /. Stephan
Index
213
PREFACE
Dependability has long been a central concern in the design of space-based and military systems, where survivability for the prescribed mission duration is an essential requirement, and is becoming an increasingly important attribute of government and conmiercial systems where reduced availability may have severe financial consequences or even lead to loss of life. Historically, research in the field of dependable computing has focused on the theory and techniques for preventing hardware and environmentally induced faults through increasing the intrinsic reliability of components and systems (fault avoidance), or surviving such faults through massive redundancy at the hardware level (fault tolerance). Recent advances in hardware, software, and measurement technology coupled with new insights into the nature, scope, and fundamental principles of dependable computing, however, contributed to the creation of a challenging new research agenda in the late eighties aimed at dramatically inaeasing the power, effectiveness, and efficiency of approaches to ensuring dependability in critical systems At the core of this new agenda was a paradigm shift spurred by the recognition that dependabihty is fundamentally an attribute of apphcations and services—not platforms. Research should therefore focus on (1) developing a scientific understanding of the manifestations of faults at the application level in terms of their ultimate impact on the correctness and survivability of the application; (2) innovative, application-sensitive approaches to detecting and mitigating this impact; and (3) hierarchical system support for these new approaches. Such a paradigm shift necessarily entailed a concomitant shift in emphasis away from inefficient, inflexible, hardware-based approaches toward higher level, more efficient and flexible software-based solutions. Consequently, the role of hardware-based mechanisms was redefined to that of providing and implementing the abstractions required to support the higher level software-based mechanisms in an integrated, hierarchical approach to ultradependable system design. This shift was furthermore compatible with an expanded view of "dependabihty," which had evolved to mean "the abiUty of the system to deUver the specified (or expected) service." Such a definition encompasses not only survival of traditional single hardware faults and environmental disturbances but more complex and less-well understood phenomena, as well: Byzantine faults, correlated errors, timing faults, software design and process interaction errors, and—most significantly—the unique issues encountered in real-
Vlll
time systems in which faults and transient overload conditions must be detected and handled under hard deadline andresourceconstraints. As sources of service disruption multiplied and focus shifted to their ultimate effects, traditional frameworks for reasoning about dependability had to be rethought. The classical fault/error/failure model, in which underlying anomalies (faults) give rise to incorrect values (errors), which may ultimately cause incorrect behavior at the output (failures), required extension to capture timing and performance issues. Graceful degradation, a long standing principle codifying performance/dependability trade-offs must be more carefully applied in real-time systems, where individual task requirements supercede general throughput optimization in any assessment. Indeed, embedded real-time systems—often characterized by interaction with physical sensors and actuators—^may possess an inherent ability to tolerate brief periods of incorrect interaction, either in the values exchanged or the timing of those exchanges. Thus, a technical failure of the embedded computer does not necessarily imply a system failure. The challenge of capturing and modehng dependability for such potentially complex requirements is matched by the challenge of successfully exploiting them to devise more intelligent and efficient—as well as more complete— dependability mechanisms. The evolution to a hierarchical, software-dominated approach would not have been possible without several enabling advances in hardware and software technology over the past decade: (1) Advances in VLSI technology and RISC architectures have produced components with more chip real estate available for incorporation of efficient concurrent error detection mechanisms and more on-chip resources permitting software management of fine-grain redundancy; (2) The emergence of practical parallel and distributed computing platforms possessing inherent coarse-grain redundancy of processing and conmiunications resources—also amenable to efficient software-based management by either the system or the application; (3) Advances in algorithms and languages for parallel and distributed computing leading to new insights in and paradigms for problem decomposition, module encapsulation, and module interaction, potentially exploitable in refining redundancy requirements and isolating faults; (4) Advances in distributed operating systems allowing more efficient interprocess communication and more intelligent resource management;
IX
(5) Advances in compiler technology that permit efficient, automatic instrumentation or restructuring of application code, program decomposition, and coarse and fine-grain resource management; and (6) The emergence of fault-injection technology for conducting controlled experiments to determine the system and apphcation-level manifestations of faults and evaluating the effectiveness or performance of fault-tolerance methods. In response to this challenging, new vision for dependable computing research, the advent of the technological opportunities for realizing it, and its potential for addressing critical dependability needs of Naval, Defense, and commercial systems, the Office of Naval Research launched a five-year basic research initiative in 1990 in Ultradependable Multicomputers and Electronic Systems to accelerate and integrate progress in this important discipline. The objective of the initiative is to establish the fundamental principles as well as practical approaches for efficiently incorporating dependability into critical applications running on modem platforms. More specifically, the initiative sought increased effectiveness and efficiency through (1) Intelligent exploitation of the inherent redundancy available in modem parallel and distributed computers and VLSI components; (2) More precise characterization of the sources and manifestations of errors; (3) Exploitation of application semantics at all levels—code, task, algorithm, and domain—to allow optimization of fault-tolerance mechanisms to both application requirements and resource limitations; (4) Hierarchical, integrated software/hardware approaches; and (5) Development of scientific methods for evaluating and comparing candidate approaches. Implementation of this broad mandate as a coherent research program necessitated focusing on a small cross-section of promising application-sensitive paradigms (including language, algorithm, and coordination-based approaches), their required hardware, compiler, and system support, and a few selected modeling and evaluation projects. In scope, the initiative emphasizes dependability primarily with respect to an expanded class of hardware and environment (both physical and operational) faults. Many of the efforts furthermore explicitly address issues of dependability unique to the domain of embedded real-time systems. The success of the initiative and the significance of the research is demonstrated by the ongoing associations that many of our principal investigators have forged with a variety of military, Govemment, and commercial projects whose critical needs are leading to the rapid assimilation of concepts, approaches, and expertise arising from this initiative. Activities influenced to date include the FAA's Advanced Automation System for air traffic control, the Navy's AX project and Next Generation Computing Resources standards program, the Air Force's Center for Dependable Systems, the OSF/1 project, the space station Freedom, the Strategic
Defense Initiative, and research projects at GE, DEC, Tandem, the Naval Surface Warfare Center, and MITRE Corporation. This book series is a compendium of papers sunmiarizing the major results and accomplishments attained under the auspices of the ONR initiative in its first three years. Rather than providing a comprehensive text on dependable computing, the series is intended to capture the breadUi, depth, and impact of recent advances in the field, as reflected through the specific research efforts represented, in the context of the vision articulated here. Each chapter does, however, incorporate appropriate background material and references. In view of the increasing importance and pervasiveness of real-time concerns in critical systems that impact our daily lives—^ranging from multimedia communications to manufacturing to medical instrumentation—the real-time material is woven throughout the series rather than isolated in a single section or volume. The series is partitioned into three volumes, corresponding to the three principal avenues of research identified at the beginning of this preface. While many of the chapters actually address issues at multiple levels, reflecting the comprehensive nature of the associated research project, they have been organized into these volumes on the basis of the primary conceptual contribution of the work. Agha and Sturman, for example, describe a framework (reflective architectures), a paradigm (replicated actors), and a prototype implementation (the Screed language and Broadway runtime system). But because the salient attribute of this work is the use of reflection to dynamically adapt an application to its environment, it is included in the Frameworks volume. Volume I, Models and Frameworks for Dependable Systems, presents two comprehensive frameworks for reasoning about system dependability, thereby estabUshing a context for understanding the roles played by specific approaches presented throughout the series. This volume then explores the range of models and analysis methods necessary to design, validate, and analyze dependable systems. Volume II, Paradigms for Dependable Applications, presents a variety of specific approaches to achieving dependabiUty at the application level. Driven by the higher level fault models of Volume I and buiilt on the lower level abstractions implemented in Volume III, these approaches demonstrate how dependability may be tuned to the requirements of an appUcation, the fault environment, and the characteristics of the target platform. Three classes of paradigms are considered: protocolbased paradigms for distributed applications, algorithm-based paradigms for parallel applications, and approaches to exploiting application semantics in embedded realtime control systems. Volume III, System Implementation, explores the system infrastructure needed to support the various paradigms of Volume II. Approaches to implementing
XI
suppport mechanisms and to incorporating additional appropriate levels of fault detection and fault tolerance at the processor, network, and operating system level are presented. A primary concern at these levels is balancing cost and performance against coverage and overall dependability. As these chapters demonstrate, low overhead, practical solutions are attainable and not necessarily incompatible with performance considerations. The section on innovative compiler support, in particular, demonstrates how the benefits of application specificity may be obtained while reducing hardware cost and run-time overhead. This second volume of the series builds on the modeling foundation established in Volume I by exploring specific paradigms for managing redundancy and faults at the application level through specialized algorithms or protocols. Consistent with the layered view of dependability that characterizes this series, these softwareoriented approaches rely not only on the underlying models of Volume I for their soundness, but on the abstractions of Volume III for their practicality. In distributed systems, general-purpose dependability is often achieved through process replication managed through protocols. The three approaches described in Section 1 vary in purpose and degree of insulation from the application. Bianchini and Stahl explore the nuances of adapting distributed diagnosis algorithms to a real-time environment. Whereas the diagnosis paradigm is largely independent of the application, the authors demonstrate how consideration of the fault environment and scheduling constraints can lead to unanticipated modes of interaction. Marzullo, et al, present the refinement mapping approach for deriving customized dependable protocols for specific applications. The approach is illustrated through an air traffic control example. Finally, in an instantiation of Agha's concept of reflection (Vol. I), Schlichting, et al, consider two classes of language extensions to support enhanced application-specific control over redundancy and recovery management. Parallel systems are characterized by larger degrees and finer granularity of concurrency than distributed systems. In such large-scale systems with frequent interprocess conmiunication, conventional replication approaches are too costly, inefficient, and potentially detrimental to performance. Fortunately, unlike distributed applications which are typically decomposed by function, parallel scientific algorithms often employ data decomposition to assign each processor (running substantially the same program) a sub-domain corresponding, e.g., to a distinct region of physical space. The regular structure of these computations may be exploited through algorithmic transformations to provide low overhead error detection and recovery. Two such approaches are described in Section 2. Yajnik and Jha focus on the data by presenting a graph-theoretic methodology for generating check operations used to detect and locate faults. Kanellakis and Shvartsman exploit the homogenity of typical parallel tasks by allowing work to be dynamically redistributed in the event of failures.
Xll
Although real-time issues are addressed throughout this series, the tight coupling of embedded real-time systems to applications such as process control and the semantics of those applications—characterized by continuously changing physical variables—suggest an opportunity to explore highly effective and efficient dependability mechanisms that recognize potentially relaxed constraints derived from the additional latitude in error sensitivity typical of these ^plications. In Section 3, Liu, et al, present one such approach for managing redundancy and supporting rapid recovery under hard real-time constraints by trading off result quality for computation time. Bodson, et al, present a paradigm for software fault tolerance based on Uie concept of analytical redundancy in which the behavior of a complex control algorithm of uncertain integrity is monitored by a simpler, robust algorithm of similar but less refined functionality. Gary M. Koob Mathematical, Computer and Information Sciences Division Office of Naval Research Clifford G. Lau Electronics Division Office of Naval Research
ACKNOWLEDGEMENTS
The editors regret that, due to circumstances beyond their control, two planned contributions to this series could not be included in the final publications: "Compiler Generated Self-Monitoring Programs for Concurrent Detection of Run-Time Errors," by J.P. Shen and "The Hybrid Fault Effects Model for Dependable Systems," by C.J. Walter, M.M. Hugue, and N. Suri. Both represent significant, innovative contributions to the theoiy and practice of dependable computing and their omission diminishes the overall quality and completeness of these volimies. The editors would also like to gratefully acknowledge the invaluable contributions of the following individuals to the success of the Office of Naval Research initiative in Ultradependable Multicomputers and Electronic Systems and this book series: Joe Chiara, George Gilley, Walt Heimerdinger, Robert Holland, Michelle Hugue, Miroslaw Malek, Tim Monaghan, Richard Scalzo, Jim Smith, Andr6 van Tilborg, and Chuck Weinstock.
This page intentionally blank
SECTION 1
PROTOCOL-BASED PARADIGMS FOR DISTRIBUTED APPLICATIONS
SECTION 1.1
Adaptive System-Level Diagnosis in Real-Time^ Mark E. Stahl^ Ronald P. Bianchini, Jr.^ Distributed real-time systems are subject to stricter fault-tolerance requirements than non-real time systems. This work presents an application of system-level diagnosis to a real-time distributed system as a first step in providing fault-tolerance. An existing algorithm for distributed system-level diagnosis, Adaptive_DSD, is converted to a real-time framework, establishing a deadline for the end-to-end diagnosis latency. Rate monotonic analysis is chosen as the framework for achieving real-time performance. The ADSD algorithm is converted into a set of independent periodic tasks running at each node, and a systematic procedure is used to assign priorities and deadlines to minimize the hard deadline of the diagnosis function. The resulting algorithm, Real-Time Adaptive Distributed System-Level Diagnosis (RT-ADSD), is fully compatible with a real-time environment, where both the processors and the network supportfixed-priorityscheduling. The RT-ADSD algorithm provides a useful first step in adding fault-tolerance to distributed real-time systems by quickly and reliably diagnosis node failures. The key results presented here include a framework for specifying real-time distributed algorithms and a scheduling model for analyzing them that accounts for many requirements of distributed systems, including network I/O, task jitter, and critical sections caused by shared resources.
This research is supported in party by the Office of Naval Research under Grant N00014-91J-1304 and under a National Science Foundation Graduate Research Fellowship. Any opinions, findings, conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the Office of Naval Research or the National Science Foundation. 2. Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA 15213. 3. Associate Professor, Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, PA 15213.
1.1.1 Introduction As distributed systems proliferate, they increasingly become a platform for the implementation of real-time systems. Distributed real-time systems are subject to stricter fault-tolerance requirements than non-real time systems, since a fault can be either a failed resource or a missed deadline. Our approach to distributed fault tolerance is to have the system perform self-diagnosis at the system level. In system-level diagnosis, a network is modeled as a collection of nodes and edges, such that nodes can be faulty or fault-free. Carnegie Mellon University has recently developed the Adaptive_DSD (ADSD) algorithm for performing on-line distributed system-level diagnosis in fully connected networks. The ADSD algorithm allows every fault-free node to diagnose the fault state of every other node in the system. This is the foundation of providing fault tolerance in a distributed system, by quickly and reliably diagnosing node failures [1]. For the diagnosis algorithm to operate in real-time, a deadline is established by which all fault-free nodes in the system are guaranteed to achieve correct diagnosis. This work describes the specification of the Real-Time Adaptive Distributed System-Level Diagnosis (RT-ADSD) algorithm, an implementation of the ADSD algorithm suitable for execution in a real-time environment. The RT-ADSD algorithm provides a hard deadline for the diagnosis latency - the time from a fault event, either a node failure or recovery, until all nodes are aware of the event. The algorithm is fully distributed, executing at all fault-free nodes, an utiUzes adaptive testing to minimize testing overhead. The diagnosis latency deadline is expressed as an end-to-end deadline of a process that is distributed among multiple nodes in a network. Delay is introduced due to computation time at each node and due to transmission time for each message sent between nodes. The approach used in this work is to establish intermediate deadlines for the portion of work performed on each node and link in the network. The end-toend deadline is the sum of the deadlines achieved at each node and Hnk. Other work has addressed the issue of real-time distributed fault tolerance using faulttolerant group membership protocols. Ezhilchelvan and Lemos [2] give a membership protocol for distributed systems that utilizes "cycles" in a synchronous network During a cycle, each processor is given the opportunity to broadcast a message and processes any incoming messages. Practical considerations, such as the length of a cycle or the amount of work performed during each cycle, are not explored. The algorithm's real-time bound, expressed in terms of cycles, is a measure of the time complexity of the algorithm's communication; i.e. the length of the longest sequence of messages that are broadcast before diagnosis is achieved. This work utilizes a different approach to achieving real-time behavior by utilizing rate monotonic analysis (RMA) [3,4, 5] to schedule the tasks. In rate monotonic analysis, there is no overriding cycle that regulates when work is performed nor is a synchronous network required to bound communication time. Rather, each task has its own period and occurs independently of other tasks in the system. Using RMA, the schedulability of each resource (i.e. network link or processor) is independently veri-
fied, and the delay of each resource is independently quantified. RMA provides the framework for the specification of RT-ADSD. By utiHzing RMA, RT-ADSD can be integrated with other tasks being peifonned by the distributed system provided those tasks are also specified using RMA. The design of RT-ADSD differs from conunon approaches to designing real-time systems in fiat, in many real-time systems, the program being designed has some fixed periodic requirements and/or deadlines that must be met. For example, [6] describes a robotic system where a sensor collecting data must be serviced during every collection interval. In contrast, RT-ADSD is formed by coercing the ADSD algorithm into the real-time framework for the purpose of establishing a hard deadline. No a priori period or deadline is given. The techniques used to create and model RT-ADSD are applicable to other distributed algorithms that operate in real-time environments. The models used here encompass many aspects of distributed programs that must be addressed for their real-time behavior to be analyzed, including idle time for server response, conmiunication delay over a real-time network, and task arrival jitter caused by sequentially dependent tasks. This chapter is organized as follows. Section 2 describes the ADSD algorithm. Section 3 develops the programming and scheduling models used by RT-ADSD. It reviews rate-monotonic scheduling theory, along with extensions necessary for RTADSD. Section 4 describes how to specify a task set for RT-ADSD to arrive at a minimal deadline for diagnosis latency. An example specification is given in Section 5. Conclusions are presented in Section 6.
1.1.2 System Level Diagnosis Consider a system of interconnected components, or units, such that each unit can be eithtr faulty or fault-free. The diagnosis problem is that of determining the set of faulty units by utilizing the results of tests performed among the units themselves. Preparata, Metze, and Chien initiated the study of system-level diagnosis in 1967 by presenting necessary conditions for system-level diagnosability [7]. Testing assignments for diagnosable systems were later characterized by Hakimi and /unin [8]. Since those pioneering works, a large body of literature in system-level diagnosis has been generated [9]. Adaptive testing assignments are used to eliminate redundant tests and reduce overhead [10]. In distributed diagnosis [11], each unit in the system forms its own independent diagnosis of the state of every other unit. In 1991, Bianchini and Buskens presented and implemented the first adaptive distributed diagnosis algorithm. Adaptive DSD [1]. The Adaptive^DSD algorithm is used as the foundation for this work in real-time distributed systems. The Adaptive_DSD algorithm is summarized below. The ADSD algorithm has many features that are desirable for a real-time implementation, including a provably minimum number of tests and a highly periodic structure that can be adapted to rate monotonic scheduling theory.
1.1.2.1 Terminology A graph theoretical model of a distributed computer network is utilized [12]. Nodes represent processors; edges represent conmiunication links. Each node has an associated fault state, either/awZry or fault-free, and is assumed to be capable of testing its neighbors. The result of a test is either/aw/ry or fault-free. A fault model is used to relate test results to the fault state of the nodes involved in the test. In the PMC (or permanent) fault model [7], tests performed by fault-free nodes are accurate and tests performed by faulty nodes produce arbitrary results. The testing assignment is the set of all tests performed between nodes. Diagnosis is the process of mapping the set of test results to the set of faulty nodes. In this work, conmiunication links are assumed to be fault-free. 1.1.2.2 The Adaptive^DSD Algorithm The Adaptive_DSD algorithm performs distributed system-level diagnosis in fully connected networks. Adaptive_DSD utilizes an adaptive testing strategy to minimize the testing overhead and achieve distributed diagnosis. Tests are assumed to conform to the PMC fault-model. /* ADAPTIVE_DSD */ /* ExecutedateachnodeUj^,0<x Pj for all i < j . Ties are broken arbitrarily. The response time of a task is the interval between its request and completion. A critical instant is the time when a task has its longest response time. For fixed priority tasks where each task arrives independent of ^ others, a critical instant occurs for task Xm ^ S when it arrives simultaneously with all higher priority tasks. If X^j can meet its deadline during the critical instant, it will always meet its deadline [3].
11 To check the schedulability of Tm, define: ^m - 1
W =
°*™J^Cj\n+C^^Bj/t\ OSfSD„ ;=i I ^; I ^ ^
(EQl)
Given that a critical instant occurs at time t = 0, the quantity ^ J ^ V S ^ ^^^J^ represents the cumulative processor demand from all jobs of higher priority than Tnj, and hence the amount of time X^ can be preempted by higher priority tasks prior to time t. The term B^ represents the longest amount of time that Tm can be blocked beyond the quantity above, e.g. due to priority inversion. For independent tasks, Bm = 0 for all m. Thus Win is the lowest processor utilization that occurs between the start of the critical instant and the deadline of 1^. For a task T^ to complete by its deadline, there must exists some time, t, in 0 < t ^ Dm where the cumulative demand from all higher priority processes and thefirstjob of Tjn is less than t. Equivalently, W^ must be less than or equal to 1. When a non-zero blocking term is present, a task can be delayed for an additional B^^ units during its execution. If W^ is still less than or equal to one with blocking present, the task is schedulable. The entire task set is schedulable if V/, l < f < N , 1V.restRequest = (N+2Fs+l)-CTestRequest
(EQ 19)
^TestResult = (N+2-Fs+l)-CxestRequest + (N+2-F,+l)-CTestResult
(EQ 20)
When CR/WR < CQ/WQ, the priority ordering is reversed; new deadlines are computed in a similar fashion. 4.5.2
Processor Tasks: Priorities and Deadlines
The following analysis assumes RT_ADSD tasks are the only tasks executing on the processor. All tasks in the CPU set share the same period and are assumed to have deadlines shorter than their periods. The execution time of each task, Q, is fixed by the source code and the target platform. Once a priority assignment is given, the deadline for each task can be determined as follows. Assuming the tasks are ordered such that Pi > P2 > •" > Pfl, the shortest deadline that can be met for each task is: Z)i = C i + ^ i
(EQ21)
Z>2 = C14-C2 + 52
(EQ22)
^n = J,l,C,^B„
(EQ23)
30 For a task composed of logical subtasks, the deadline for each subtask T: ^ is found by: ^J-^ = X i : i < ^ i + C j _ , + B j ,
(EQ24)
The deadline for task with priorities lower than j remain as given by Equation 23. Using the expression for diagnosis latency and Theorem 1, a priority assignment can be found for each task in the system. For a task, Tj, that is composed of several subtasks, each subtask has a separate weighting factor Include T^ in priority order by sorting on the term
S i"^= VC^. E;
(EQ25)
l^iJ
rather than Cj/Wj. The priority assignment given by Theorem 1 produces the optimum diagnosis latency when all blocking terms are zero [23]. This result can be extended to show optimality when Bi is constant. However, the blocking quantity varies with the relative ordering of tasks (Equation 3) making the assignment given by Theorem 1 no longer optimum. Instead, Theorem 1 is used to assign an initial priority ordering, then heuristic improvements ar made. No claims are made as to its optimality. The heuristic relies on swapping tasks with adjacent priorities and computing the effective change in diagnosis latency. The change in diagnosis latency when swapping tasks is illustrated with the following example. Assume an initial set of tasks, Tj, T2,..., Tu, where each task is composed of a fixed execution time Q and an order dependent blocking term, B,. Assume the tasks are arranged such that Pj > P2 > '• > PQ. Create a second ordering, X'^, T'2,..., X'^, formed by exchanging the priorities of tj with Tj+i. When 3ie two tasks are exchanged, their blocMng terms are recomputed as B'j. The change in diagnosis latency in moving Tj to T'j+j, as contributed by Tj is: ^-^i'
[C,,,^(B\,,^B,)]
(EQ26)
and the change contributed by Tj+i is: Ai.i = W i , i [ - C i + ( B V - 5 i , i ) ]
(EQ27)
31 When the task Xj is composed of m subtasks, the effect of swapping priorities is more complicated, since there are m blocking terms that can potentially be modified by the swap. If Tj is moved to Xj^.i, the contributed effect on diagnosis latency from Tj is: m
^i = X ^ U [Ci.i + (^•(i.i)j -B\j)]
(EQ28)
j = i
If the Xj+i task is made up of m subtasks, a similar equation for the contributed change in diagnosis latency is derived:
^i.i = X ^(-i)J f- ^i ^ ^^'u - ^(-i)J) ]
(EQ 29)
If Ai + Aj+i < 0, then exchangmg the tasks reduces the overall diagnosis.latency. Using this result, the procedure for assigning priorities to the CPU task set, S*cPU» is: 1) For the tasks in S^cpu that execute more than once per period, (TestNodesMiddle, RequestOutputTlirottle, ReceiveRequest and ReplyOutputThrottle,) combine identical executions of the same task into one task. i.e. Assume one ReceiveRequest task of length (Fg+l) CReceiveRequest2)
Assign the RequestOutputThrottle task the highest priority in the system. Assign the ReplyOutputThrottle task the next highest priority in the system.
3)
Of the remaining tasks in S'cpu without an assigned priority, use the initial priority order provided by Theorem lusing the values for Wj defined by Equation 12. i.e Q/Wi (or [^j^i ^ijl'^lZj^i "^U^ ^^^ ^ ^ ^ ^^^ subtasks) is monotonically increasing with decreasing priority.
6)
Use a variant on bubble-sort to improve diagnosis latency for the non-throttle tasks. For i = n-1 to 1 begin Forj = 1 to i begin Assume priorities of Xj and Xj+i are exchanged. Compute new blocking terms, B'j and B'j+j. Compute Aj and Aj+j If Ai + Ai+i Tp, increase all S^cpu ^ ^ periods to Tp. Label the resulting task sets S^ P-SYSTEM-
Steps 4 through 7 are repeated for each node task set S* p.^ysTEM4)
Combine the system task set at each node, S^p.sysTEM* with the two output throttle tasks from S'cpuAssign the highest system priorities to the output throttle tasks such that higher priority throtdes handle higher priority messages. Assign priorities to the remaining tasks in deadline monotonic order. Call the new task set S p.svsTEM-
5)
Assume that each RT-ADSD output throttle task occurs only once, (i.e., Tp = oo)
Check if the S p.sYSTEM ^^ schedulable. If it is not schedulable, STOP. RT-ADSD cannot be integrated with the system task set. Find the smallest period, T"p, for the RT-ADSD throttle tasks such that the system is schedulable. 6)
If T"P < T'p, set T"P = T'p. Combine all remaining RT-ADSD tasks into S'p. SYSTEM-
Set the period of all RT-ADSD tasks (including the throttle tasks) as T"p. Leaving the throttle task priorities unchanged, assign priorities to the remaining tasks in deadhne monotonic order. For tasks with multiple subtasks, order on the subtask with the smallest deadline. Call the resulting task set S p.svsTEM7)
Check the schedulabiUty of S P-SYSTEMIf the system is not schedulable. Multiply the RT-ADSD throttle task deadlines by a scale factor, OC^pu > L Reassign priorities in a deadline monotonic fashion, if necessary.
34
Set T p = max (smallest Tp allowed by Equation 7, T p). Check the schedulability of the network using the scaled deadlines and period T p Increase OLQ^U and repeat until the network is schedulable. Call thefinalsystem task set S p.sysTEM. After each individua^„node is scheduled, the global test period, T p, is chosen as the largest test period,^T p, Required by any individual node in the sysfem and is used by all nodes. Since T*p > T^, the network remains schedulable. The final value for TQ^^ is given by Equation 6. Tnefinaldiagnosis latency is given by Equation 12. This procedure does not produce an optimal diagnosis latency. It only gives a systematic procedure for integrating RT-ADSD with other task sets. Abetter approach would include RT-ADSD early in the design of a particular application. However, this is beyond the scope of this chapter.
1.1.5 Example An example of the specification procedure is performed with design parameters of N = 4 nodes and assuming a maximum of F^ = 1 faulty nodes. Table 1 lists the execution time and length of the longest critical section, Cj and S, respectively, for each task. The execution time for each task is estimated in microseconds, calculated from the processor speed and the number of lines of source code. Both Cj and Sj are functions of Fg and N. In the table, Uj represents the number of times task Xj occurs in the processor or network task sets [Equations 4,5]. Packet transmission times, Cj, are also given in microseconds and reflect the relative size of the packets, including network overhead. The weighting factors, Wj, are taken from Equations 12 and 16 for node and network tasks, respectively, when N=4.
Task Set S^CPU
Task
"i
Q
TestNodes
-
175
-
16 1
TestNodesBegin
1
125
0
4
TestNodesMiddle
1
15
0
4
TestNodesEnd
1
35
25
8
ReceiveRequest
2
60
14
13
RequestOutputThrottle
2
10
0
10
ReplyOutputThrottle
2
10
0
13
Si
Table 1.1.1: RT-ADSD task sets for N=4, F^ = 1.
Wi
35 Task Set
SCOMM
Task
"i
Q
Si
Wi
Diagnose
1
32
18
1
TestRequest (Q)
7
1.5
n/a
TestResult (R)
7
3.5
n/a
1^" 13
Table 1.1.1: RT-ADSD task sets for N=4, F^ = 1. Network tasks are scheduled first. From Equations 17 and 18, :Q/WQ = 1.05 < CR/WR = 1.88. Hence, to minimize latency, PjestRequest > PxestResuit^ Cr and the deadlines for the network tasks are DjestRequest = 10.5 and DxestResuit = 35 [Equations 19,20]. Next, the node task set is scheduled. The two highest processor priorities are reserved for the RequestOutputThrottle and ReplyOutputThrottle tasks, with the RequestOutputThrottle receiving the highest priority. The remaining priorities are assigned by sorting on (ni-Q)/Wi, producing the following priority assignment: (EQ30)
"ReceiveRequest ^ *^TestNodes ^ "Diagnose
The blocking terms are computed for each task given the above priority ordering [Equation 3] and deadlines are assigned [Equations 23,21]. The node task set at this point is summarized in Table 1. The period of all tasks is equal to the test period, 1389.4 [Equation 7]. The timeout for test responses is 340.5 |Iseconds [Equation 6].
Task Set
Task and Subtasks
Pi
"i
Ci
S^CPU
RequestOutputThrottle
1
2
10
ReplyOutputThrottle
2
2
ReceiveRequest
3
TestNodes
4
Si
Bi
45
1389.4
0
25
10
65
1389.4
0
25
2
60
185
1389.4
14
25
-
-
-
-
-
-
1
125
303
1389.4
0
18
TestNodesMiddle
-
1
15
193
1389.4
0
18
TestNodesEnd
-
1
35
213
1389.4
25
18
Diagnose
5
1
32
367
1389.4
18
0
TestRequest (Q)
1
7
1.5
10.5
1389.4
n/a
n/a
TestResult (R)
2
7
3.5
35.0
1389.4
n/a
n/a
TestNodesBegin
ScoMM
Ti
I>i
Table 1.1.2: Fully specified RT-ADSD task sets for N=4, F^ = 1.
36 Finally, the reordering heuristic developed in Section 4.5.2 is employed to improve diagnosis latency. The priority of adjacent (non-throttle) tasks are exchanged and the effect on the end-to-end delay is computed. Consider the exchange of PjestNodes ^ ^ ^Diagnose- ^^^^ ^ ^ exchange, the blocking term of all three TestNodes subtasks reduces to zero, while the blocking term for TDiacnose increases to 25. From Equations 26 and 27, ATestNodes = 272 and Aoiagnose = -150. ^Latency = ^TestNodes + ^Diagnose = +122
(EQ 31)
Hence, the exchange of X-re^tNodes ^ ^ "^Diagnose ^ ^ s not result in a lower diagnosis latency. Repeating the same calculation with aJl other adjacent task pairs shows that no exchange produces a lower latency. The values given in Table 1 represent the final CPU and Conun task sets. The final diagnosis latency achieved for this system is 8,320 [Equation 12]. 1.1.6 Conclusion This chapter demonstrates one approach to the difficult problem of real-time distributed system design. Rate monotonic analysis provides the basis for the scheduling model. A programming framework is developed within that model that allows the construction of complex distributed algorithms. The framework is expected to be general enough to implement a variety of distributed applications. The application program presented here, RT-ADSD, solves the fault-diagnosis problem for nodes of fully connected networks. The techniques used for specifying RTADSD are expected to prove useful in specifying other non-real-time algorithms within a real-time environment. An important step in the conversion process is the application of bounds on the algorithm's execution and the use of sporadic service to enforce those bounds, rather than utihzing the worst-case execution time. In this way, the scheduled execution behavior more closely resembles the expected execution behavior. By an intelligent choice of bounds, in our case the fault limit, the algorithm's execution behavior was made tractable and still provided useful results. Further research is needed in the area of distributed real-time systems. Improvements to the programming model could be obtained by the use of a hardware-based network sporadic server, thereby eliminating the overhead required for sporadic throttie tasks. Distributed algorithms would also benefit from the development of better scheduling models. The model presented here utilizes a pessimistic assumption that all tasks at a node or link (except those belonging to the same physical thread) arrive independentiy of each other. The timelines in Section 4.3 show that several tasks arrive upon the completion of other tasks. The independence assumption produces longer deadlines and reduced utilization by assimiing tasks can compete for a resource when they may not be able to do so in the implementation. Harbour, Klein and Lehoczky (HKL) presented a model for analyzing the schedulability of fixed priority tasks [6] that may prove useful in such cases. Each task is allowed to vary its priority in a deterministic way throughout its execution, allowing
37
sequences of RT-ADSD tasks that utilize the same resource (i.e. TestNodesBegin followed by RequestOutputThrottle) to be modeled as a single task. This more closely resembles the actual execution tfehavior of RT-ADSD, producing shorter deadlines for each compound task. However, the HKL model does not account for the effects of idle time nor does it support sporadic service, both of which are required to schedule RT-ADSD. HKL captures sequential dependencies between tasks on the same node. A better model would incorporate tasks composed of subtasks that occur sequentially across several different resources and is an open research issue.
References [1] Bianchini, R. R, and Buskens, R. "An Adaptive Distributed System-Level Diagnosis Algorithm and its Implementation." Proceedings of the IEEE 23rd International Symposium on Fault-Tolerant Computing, June 1991, pp. 222-229. [2] Ezhilchelvan, R D. and de Lemos, R. "A Robust Group Membership Algorithm for Distributed Real-Time Systems." Proceedings of IEEE Real-Time Systems Symposium, December 1990, pp. 173-179. [3] Liu, C. L., and Layland, J. W. "Scheduling Algorithms for Multi-Programming in a Hard Real-Tmie Environment." Journal of the Association for Computing Machinery, 20 (1), January 1973, pp. 46-61. [4] Lehoczky, J. R, Sha, L. and Ding, Y. "The Rate-Monotonic Scheduling Algorithm: Exact Characterization and Average Case Behavior." Proceedings of IEEE Real-Time Systems Symposium, 1989, pp. 166-171. [5] Lehoczky, J. R "Fixed Priority Scheduling of Periodic Task Sets with Arbitrary Deadlines." Proceedings of IEEE Real-Time System Symposium, 1990, pp. 201209. [6] Harbour, M. G., Klein, M. H. and Lehoczky, J. P. "Fixed Priority Scheduling of Periodic Tasks with Varying Execution Priority." Proceedings of IEEE RealTime Systems Symposium, 1991. [7] Preparata, F. P., Metze, G. and Chien, R. T. "On the connection Assignment Problem of Diagnosable Systems." IEEE Transactions on Electronic Computing, EC-16 (12), December 1967, pp. 848-854. [8] Hakimi, S. L., and Amin, A. T. "Characterization of Connection Assignment of Diagnosable Systems." IEEE Transactions on Computers, C-23 (1), January 1974, pp. 86-88. [9] Dahbura, A.T. "System-Level Diagnosis: A Perspective for the Third Decade." Concurrent Computation: Algorithms, Architectures, Technologies, Plenum Publishmg Corp., 1988, pp. 411-434.
38 [10] Hakimi, S. L. and Schmeichel, E. F. "An Adaptive Algoritiun for System Level Diagnosis." Journal of Algorithms, 5, June 1984, pp. 526-530. [11] Hosseini, S. H., Kuhl, J. G., and Reddy, S. M. "A Diagnosis Algorithm for Distributed Computing Systems with Dynamic Failure and Repair." IEEE Transactions on Computers, C-33 (3), March 1984, pp. 223-233. [12] Bondy, A. and Murty, U. S. R. Graph Theory and Applications, Elsevier North Holland, Inc., New York, N.Y., 1976. [13] Sha, L., Rajkumar, R., and Lehoczky, J. R "Priority Inheritance Protocols: An Approach to Real-Time Synchronization." IEEE Transactions on Computers, September 1990. [14] Sprunt, B., Sha, L., and Lehoczky, J. P. "Aperiodic Task Scheduling for Hard Real-Time Systems." The Journal of Real-Time Systems, 1,1989, pp. 27-60. [15] Klein, M. H. et al. A Practitioner's Handbook for Real-Time Analysis: Guide to Rate Monotonic Analysis for Real-Time Systems. Kluwer Academic Publishers, NorweU MA, 1993. [16] "Real-Time Communications Network Operating System. RTCN-OS Users's Guide." XXXX-PX2-RTCN edition, IBM Systems Integration Division, Manassas, VA, 1989. [17] Sha, L., and Goodenough, J. B. "Real-Time Scheduling Theory and Ada." IEEE Computer, 23 (4), April 1990, pp. 53-62. [18] Leung, J. and Whitehead, J. "On Complexity of Fixed-Priority Scheduling of Periodic Real-Tune Tasks." Performance Evaluation, 2,1982, pp. 237-250. [19] Klein, M. H., and Ralya, T. "An Analysis of Input/Output Paradigms for RealTime Systems." Tech. Report CMU/SEI-90-TR-19, Software Engineering Institute, July 1990. [20] Rajkumar, R., Sha, L., and Lehoczky, J. P. "Real-Time Synchronization Protocols for Multiprocessors." Proceedings of IEEE Real-Time Systems Symposium, December 1988, pp. 259-269. [21] Golestani, S. J. "Congestion-Free Transmission of Real-Time Traffic in Packet Networks." Proceedings IEEE Infocom '90, June 1990, pp. 527-536. [22] F. Cristian. "Understanding Fault-Tolerant Distributed Systems." Communications of the ACM, 34 (2), February 1991. [23] Smith, W. E. "Various Optimizers for Single Stage Production." Naval Research Logistics Quarterly, 3,1956, pp. 59-66.
SECTION 1.2
Refinement for Fault-Tolerance: An Aircraft Hand-off Protocol Keith MarzuUo^ Fred B. Schneider^, and Jon Dehn^ Abstract Part of the Advanced Automation System (AAS) for air-traffic control is a protocol to permit flight hand-off from one air-traffic controller to another. The protocol must be fault-tolerant and, therefore, is subtle—an ideal candidate for the apphcation of formal methods. This paper describes a formal method for deriving fault-tolerant protocols that is based on refinement and proof outlines. The AAS hand-off protocol was actually derived using this method; that derivation is given.
1.2.1 Introduction The next-generation air traffic control system for the United States is currently being built under contract to the U.S. government by the IBM Federal Systems Company (recently acquired by Loral Corp.). Advanced Automation System (AAS) [1] is a large distributed system that must function correctly, even if hardware components fail. Design errors in AAS software are avoided and eliminated by a host of methods. This paper discusses one of them—the formal derivation of a protocol from its specification—and how it was applied in the AAS protocol for transferring authority to control a flight from one air-traffic controller to another. The flight hand-off protocol we describe is the one actually used in the production AAS system (although the protocol there is programmed in Ada). And, the derivation we give is a description of how the protocol actually was first obtained. The formal methods we use are not particularly esoteric nor sophisticated. The specification of the problem is simple, as is the characterization of hardware failures that it must tolerate. Because the hand-off protocol is short, computer-aided support was not necessary for the derivation. Deriving more complex protocols would certainly benefit from access to a theorem proven ^Department of Computer Science, University of California San Diego, La Jolla, CA 92093. This author is supported in part by the Defense Advanced Research Projects Agency under NASA Ames grant number NAG 2-593, Contract N00140-87-C-8904 and by AFOSR grant number F496209310242. The views, opinions, and findings contained in this report are those of the author and should not be construed as an official Department of Defense position, policy, or decision. ^Department of Computer Science, Cornell University, Ithaca, NY 14853. This author is supported in part by the Office of Naval Research under contract NOOO14-91 -J-1219, AFOSR under proposal 93NM312, the National Science Foundation under Grant CCR-8701103, and DARPA/NSF Grant CCR-9014363. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author and do not reflect the views of these agencies. ^Loral Federal Systems, 9231 Corporate Blvd., Rockville, MD 20850.
40 We proceed as follows. The next section gives a specification of the problem and the assumptions being made about the system. Section 1.2.3 describes the formal method we used. Finally, Section 1.2.4 contains our derivation of the hand-off protocol.
1,2.2 Specification and System Model The air-traffic controller in charge of aflightat any time is determined by the location of the flight at that time. However, the hand-off of the flight from one controller to another is not automatic: some controller must issue a command requesting that the ownership of a flight be transferred from its current owner to a new controller. This message is sent to a process that is executing on behalf of the new controller. It is this process that starts the execution of the hand-off itself. The hand-off protocol has the following requirements: PI: No two controllers own the same flight at the same time. P2: The interval during which no controller owns a flight is brief (approximately one second). P3: A controller that does not own a flight knows which controller does own that flight. The hand-off protocol is implemented on top of AAS system software that implements several strong properties about message delivery and execution time [1]. For our purposes, we simplify the system model somewhat and mention only those properties needed by our hand-off protocol. The system is structured as a set of processes running on a collection of processors interconnected with redundant networks. The services provided by AAS system software include a point-to-point FIFO interprocess communication facility and a name service that allows for location-independent interprocess communication. AAS also supports the notion of a resilient process s comprising a primary process s.p and a backup process s.b. The primary sends messages to the backup so that the backup's state stays consistent with the primary. This allows the backup to take over if the primary fails. A resilient process is used to implement the services needed by an air-traffic controller, including screen management, display of radar information, and processing of flight information. We denote the primary process for a controller C as C.p and its backup process as C.b. If C is the owner of a flight / , then C.p can execute commands and send messages that affect the status of flight / ; C.b, like all backup processes in AAS, only receives and records information from C.p in order to take over if C.p fails. AAS implements a simple failure model for processes [3]: SI: Processes can fail by crashing. A crashed process simply stops executing without otherwise taking any erroneous action.
41 S2: If a primary process crashes, then its backup process detects this and begins executing a user-specified routine. Property S2 is implemented by having SL failure detector service. This service monitors each process and, upon detecting a failure, notifies any interested process. If the hand-off protocol runs only for a brief interval of time, then it is safe to assume that no more than a single failure will occur during execution. So, we assume: 53: In any execution of the hand-off protocol, at most one of the participating processes can crash. 54: Messages in transit can be lost if the sender or receiver of the message crashes. Otherwise, messages are reliably delivered, without corruption, and in a timely fashion. No spurious messages are generated. We also can assume that messages are not lost due to failure of network components such as controllers and repeaters. This is a reasonable assumption because the processors of AAS are interconnected with redundant networks and it is assumed that no more than one of the networks will fail. In any long-running system in which processes can fail, there must be a mechanism for restarting processes and reintegrating them into the system. We ignore such issues here because that functionality is provided by AAS system software. Instead, we assume thatat the beginning of a hand-off from A to 5, all four processes y4.p,A.6,jB.p, 5.6 are operational.
1.2.3 Fault-tolerance and Refinement A protocol is a program that runs on a collection of one or more processors. We indicate that S is executed on processor p by writing: (5)atp
(1.2.1)
Execution of (1.2.1) is the same as skip if p has failed and otherwise is the same as executing 5 as a single, indivisible action. This is exactly the behavior one would expect when trying to execute an atomic action 5 on a fail-stop processor. Sequential composition is indicated by juxtaposition. (5i)atpi
(52>atp2
(1.2.2)
This statement is executed by first executing {S\) atpi and then executing {S2) atp2Notice that execution of (^2) at pi cannot assume that 5i has actually been performed. If Pi fails before execution of {5i) at p\ completes, then the execution of (5i) at p\ is equivalent to skip. Second, observe that an actual implementation of (1.2.2) when p\
42 and p2 are different will require some form of message-exchange in order to enforce the sequencing. Finally, parallel composition is specified by: cobegin (5i) atpi || (^2) atp2 || ••. || (Sn) atpn coend
(1.2.3)
This statement completes when each component {Si) Sitpi has completed. Since some of these components may have been assigned to processors that fail, all that can be said when (1.2.3) completes is that a subset of the Si have been performed. If, however, we also know the maximum number t of failures that can occur while (1.2.3) executes, then at least n — ^ of the Si will be performed.
Proof Outlines We use proof outlines to reason about execution of a protocol. A proof outline is a program that has been annotated with assertions, each of which is enclosed in braces. A precondition appears before each atomic action, and a postcondition appears after each atomic action. Assertions are Boolean formulas involving the program variables. Here is an example of a proof outline. {x = OA?/ = 0} XI : X :x^\ {x =r 1 A y = 0} X2: y:=y+\ In this example, x = 0 /\y = 0,x — l A y ^ O , and x = 1 A 2/ = 1 are assertions. Assertion x = 0 A y = 0 is the precondition of XI, denoted /7r^(Xl), and assertion X = 1 A y = 0 is the postcondition of XI, denoted/?05"r(X2). The postcondition of XI is also the precondition of X2. A proof outline is valid if its assertions are an accurate characterization of the program state as execution proceeds. More precisely, a proof outline is valid if the proof outline invariant /\ {{at{S) ^ pre{S)) A {after{S) ^ s
post{S)))
is not invalidated by execution of the program, where at{S) is a predicate that is true when the program counter is at statement S, and after{S) is a predicate this is true when the program counter is just after statement S. The proof outline above is valid. For example, execution starting in a state where X = l A y = OA after(Xl) is true satisfies the proof outline invariant and, as execution proceeds, the invariant remains true. Notice, our definition of validity allows execution
43 to begin anywhere—even in the middle of the program. Changing post{Xl) (and pre{X2)) to ;r = 1 destroys the validity of the above proof outline. (Start execution in state X = l A y = 23A after{Xl). The proof outline invariant will hold initially but is invalidated by execution of X2.) A simple set of (syntactic) rules can be used to derive valid proof outlines. The first such programming logic was proposed in [2]. The logic that we use is a variant of that one, extended for concurrent programs [4]. Additional extensions are needed to derive a proof outline involving statements like (1.2.1). Here is a rule for (1.2.1); it uses the predicate up{p) to assert that processor p has not failed. Action at Processor:
up{p) not free in A, up{p) not free in B {A] S {B]
{A}{S)2iip{{AyB)A{up{p)^B)} Since execution of (5) at p when p has crashed is equivalent to a skip, one might think that {A} (S) atp {{up{p) ^B)A
i^upip) ^ A)}
(1.2.4)
should be valid if {A} S {B} is. Proof outHne (1.2.4), however, is not valid. Consider an execution that starts in a state satisfying A and suppose p has not crashed. According to the rule's hypothesis, execution of 5 would produce a state satisfying B. If process p then crashed, the state would satisfy ->up{p) A B. Unless B implies A, the postcondition of (1.2.4) no longer holds. The problem with (1.2.4) is that the proof outline invariant is invalidated by a processor failure. The predicate up{p) changing value from true to false causes the proof outline invariant to be falsified. We define a proof outline to be fault-invariant with respect to a class of failures if the proof outline invariant is not falsified by the occurrence of any allowable subset of those failures. For the hand-off protocol, we are concerned with tolerating a single processor failure. We, therefore, are concerned with proof outlines whose proof outline invariants are not falsified when up{p) becomes false for a single processor (provided up{p) is initially true for all processors). Checking that a proof outline is fault-invariant for this class of failures is simple: Fault-Invariance: For each assertion A: (A A f\up{p)) => f\A[up{p')-false] p
p'
where L[x\= e] stands for L with every free occurrence of x replaced by e.
44
1.2A Derivation of the Hand-off Protocol Let CTR{f) be the set of controllers that own flight / . Property PI can then be restated as PV: \CTR{f)\ < 1. Desired is a protocol Xfer(A, B) satisfying {AeCTR{f)APV} Xfev{A,B) {BeCTR{f) APV} such that PV holds throughout the execution of Xfer(A, B). A simple implementation of this protocol would be to use a single variable ctr{f) that contains the identity of the controller of flight / and to change ctr{f) with an assignment statement: {Aectr{f) APV} ctrif):= (ctrif) - {A})U {B} {Bectr{f) API'} This implementation is problematic because the variable ctr{f) must reside at some site. Not only does this lead to a possible performance problem, but it makes determining the owner of / dependent on the availability of this site. Therefore, we represent CTR{f) with a Boolean variable C.ctr{f) at each site C, where CTRif) :
{C\C.ctrif)}.
By doing so, we now require at least two separate actions in order to implement Xfer(i4,B)—one action that changes A.ctr{f) and one action that changes B.ctr(f). Using the Action at Processor Rule, we get: XI :
{A G CTR{f) A PV} {A.ctr{f):= false)atA {{up{A) ^ {{A 0 CTR{f)) A (CTRif) = 0))) A PV}
X2:
{CTRif) = 0} {B.ctrif):= true) atB {iupiB)^iBeCTRif)))APV)}
Note ihaipreiXl) must assert that CTRif) = 0 holds, since otherwise execution of X2 invalidates PI'. The preconditions of XI and X2 are mutually inconsistent, so these statements cannot be executed in parallel. Moreover, X2 cannot be run first because pre{X2),
45 CTR(f) = 0, does not hold in the initial state. Thus, X2 must execute after XL Unfortunately, post{Xl) does not imply pre{X2); if up{A) does not hold, then we cannot assert that CTR{f) = 0 is true. This should not be surprising: if A fails, then it cannot relinquish ownership. One solution for this availability problem is to employ a resilient process. That is, each controller C will have a primary process C.p and a backup process C.b executed on processors that fail independently. Each process has its own copy of C.ctr{f), and these copies will be used to represent C.ctr{f) in a manner that tolerates a single processor failure: r rtr(f\ . / ^-P-^^^(f) o . c f r u ; . | c.b.ctrif)
ifwpCC'-P) if-^up{C.p)
Since we assume that there is at most one failure during execution of the protocol, the above definition never references the variable of a failed process. Replacing references to processor "A" in Statement XI with "A.p" produces the following: Xla :
{AeCTRif)APl'} {A.p.ctr{f):= false) sttA.p {(upiA.p) => {{A 0 CTRif))
A (CTR(f) = 0))) A PI'}
This proof outline is not fault-invariant, however. If A.p were to fail when the precondition holds, then the precondition might not continue to hold. In particular, if A G CTR{f) holds because/4.p.crr(/) is true and A.b.ctr{f) happens to be false, then when A.p fails, A G CTR{f) would not hold. We need to assert that A.p.ctr{f) = A.b.ctr{f) also holds whenever pr^(Xla) does. We express this condition using the following definition: Pr. (up{A.p) A up{A.b)) ^ {A.b.ctr{f) = A.p.ctr{f)) Note that if one of A.p or A.b has failed then A.p.ctr(f) and A.b.ctr{f) need not be equal. Adding Pr to/7r^(X la) gives the following proof outline, which is fault-invariant for a single failure: Xla :
{A G CTR{f) A PV A Pr} {A.p.ctr{f)\— false) at A.p {{up{A.p) => i(A 0 CTRif))
A (CTRif) = 0))) A PI'}
We need more than just Xla to implement XI, however. Xla does not re-establish Pr, which must hold for subsequent ownership transfers. This suggests that A.6.cfr(/) also be updated. Another problem with Xla is ihaX post{Xla) still does not imply pre{X2): if up{A.p) does not hold, then CTR{f) - 0 need not hold.
46 An action whose postcondition implies up{A.b) ^ {^A.b.ctr(f)
A {-^up{A.p) => {CTR{f) = 0))
suffices. By our assumption, up{A.p) V up{A.b) holds, so this postcondition and post{Xl2i) will together allow us to conclude CTR{f) = 0 holds, thereby establishing/7r^(X2). Here is an action that, when executed in a state satisfying/7r^(Xla), terminates with the above assertion holding: Xlb :
{{A e CTR{f)) A PV A Pr} {A.b.ctr{f):= false) atA.6 {up{A,b) => {-nA.bxtr(f) A i^up{A.p) => {CTR{f) = 0)))}
One might think that since XIa and Xlb have the same preconditions they could be run in parallel, and the design of the first half of the protocol would be complete. Unfortunately, we are not done yet. The original protocol specification implicitly restricted permissible ownership transitions. Written as a regular expression, the allowed sequence of states is: {CTR{f) = {A})+ {CTR{f) = 0)* {CTR{f) = {8})+
(1.2.5)
That is, first A owns the flight, then no controller owns the flight for zero or more states, and finally B owns the flight. The proof outline above does not tell us anything about transitions; it only tells that PI' holds throughout (because PV is implied by all assertions). We must strengthen the proof outline to deduce that only correct transitions occur. A regular expression (like the one above) can be represented by a finite state machine that accepts all sentences described by the regular expression. Furthermore, a finite state machine is characterized by a next-state transition function. The following next-state transition function 8AB characterizes the finite state machine for (1.2.5): f {{A},iD,{B}} SAB:{
{ t) if for every set of s or fewer faults the system can detect the faults but may not he ahle to diagnose the faulty processors, whereas in the presence oft or fewer faulty processors, the system has the capability to diagnose the set of faulty processors.
In case of fault diagnosing systems if some processors share in the computation of a data element and if only the shared data element is in error, it becomes impossible to diagnose the fault to a particular processor. The set of processors which have non-disjoint data sets are said to form a processor class. The faults in the system can be identified to a processor class. Henceforth in this chapter, fault diagnosability refers to processor class fault diagnosability.
2.1.2.1
The Graph-Theoretic Model
Banerjee and Abraham [13] proposed a graph-theoretic model for representing and analyzing ABFT systems. We call this model the original model. Later the model was refined by Yajnik and Jha [9] to include the faults in the processors performing the checking operations. This model is called the extended model. This section will give a brief overview of the two models. The Original Model: An ABFT system can be represented by a tripartite graph called the processor-data-check (PDC) graph consisting of sets of processor nodes (P), data nodes (D) and check nodes (C). Processor nodes are linked by edges to data nodes and the data nodes to check nodes. An edge between processor node pi and data node dj implies that pi is involved in the computation of dj. Similarly, an edge between data node dj and check node Ck implies that dj is checked by c^. Therefore, the processor-data part, called the PD graph, of the PDC graph represents the mapping of data elements of the algorithm to the processors. Given such a mapping, the design of an ABFT system under this model, involves constructing the data-check part, called the DC graph, of the PDC graph such that the system has the desired fault tolerance capabilities. The Extended Model: The original model assumes that checks are evaluated on processors which are either outside the system or are fault-free or self-checking. In actual systems this is not the case. The checks are usually
87 evaluated on processors in the multiprocessor system and are as prone to failures as any other processor in the system. Under the extended model used for representing A B F T systems, the processors c o m p u t i n g the checks are considered to be a p a r t of the system and any failures in the check-computing processors are also taken into consideration. T h e model uses two graphs to represent an A B F T system — an extended t r i p a r t i t e PDC graph and a b i p a r t i t e check evaluation (CE) graph. T h e two graphs consist of six types of nodes. I n f o . p r o c e s s o r nodes compute the original unencoded d a t a elements which are represented by i n f o _ d a t a nodes. T h e r e d u n d a n t p a r t of the o u t p u t d a t a is represented by c h e c k _ d a t a nodes. These are c o m p u t e d by c o d e _ p r o c e s s o r nodes. T h e checking operations are represented by c h e c k nodes which are evalu a t e d by c h e c k - C o m p u t i n g processor nodes. T h e check-computing processor nodes can be of the info-processor node type or the code-processor node t y p e . T h e extended PDC graph represents the processor-data-check relationship as in the original model, except t h a t the info_processor nodes are connected to only the info-data nodes and the code.processor nodes are involved in the c o m p u t a t i o n of only the check-data nodes. Although there is sharing of d a t a elements a m o n g the info_processors nodes, the code-processor nodes are not allowed to share the c o m p u t a t i o n of a d a t a element with other processors in the system. Each check has at least one check-data element in its d a t a set. A check computes some function of the d a t a elements in its d a t a set and compares the result with a particular check-data element in its d a t a set. This check-data element is called the check-compare element of the set. T h e CE graph represents the m a p p i n g of check evaluations to the system processors. There is an edge between check node Ci and processor node pj in the graph if processor pj evaluates check €{. We give an example based on the extended model next.
E x a m p l e 2 Consider the fault tolerant multiplication of two 3 x 3 matrices A and B given in E x a m p l e 1. T h e m a t r i x multiplication result is a 4 x 4 m a t r i x C whose info-data elements are d i , fi2)---> ^9- T h e last column a n d the last row of C are the r e d u n d a n t o u t p u t d a t a elements. These seven elements, dio, ciii,..., di6, are the check-data elements. Info_processors p i , P2)---j P9 produce the info-data elements, whereas code_processors pio, P i i , . . . , Pie perform the r e d u n d a n t c o m p u t a t i o n s and produce the check-data elements. Checks c i , C2,..., cy, given next m a k e the system 3-fault detecting [1].
88
". O Info_processor
Real_data p
(~\
X i, ^,0^ p, o — m , , , o ,o—m
,0(a)
Code_processor
AT'
Figure 2.1.2: (a) The original PD graph, (b) The 3-fault detecting PDC graph Checks ci : dio = di + d2 + d^l C3 * di2 — dj -{- ds -\- dg? C5 • ^13 = di -{- d^ -{- dr? C7 • dis = ds -{- de -\- dg?
C2 : d i i = ci4 + ^5 -f- de? C4 : die = di3 -\- cfi4 -h ci] C6 : cfi4 = ^2 + C^5 + ^8?
In this example, dio is the check_compare element of check ci, dn is the check_compare element of check C2, and so on. The original non-fault-tolerant PD graph and the corresponding extended PDC graph are shown in Figure 2.1.2. For the extended model, the mapping of checks to processors needs to be performed before the system can truly be said to be 3-fault detecting. A general method for finding such a mapping is given in Section 2.1.3.3. D
89
2.1.2.2
The Matrix-Based Model
The tripartite PDC graph can be represented by two matrices — PD and DC [17]. The PD matrix has processors as rows and data elements as columns and represents the PD graph. There is a *1' entry in position (i, j) in the matrix if there is an edge from processor pi to data element dj in the PD graph. The DC matrix has data elements as rows and checks as columns and represents the DC graph. There is a ' 1 ' entry in position (i, j ) in the DC matrix if there is an edge from data element di to check Cj in the DC graph. A third matrix called the PC matrix, and the corresponding PC graph, are also used for representing an ABFT system. The PC matrix is obtained by taking the product of the PD and DC matrices. An entry in position (i, j) in the matrix indicates the number of paths from processor pi to check Cj in the PDC graph. In the extended model the information in the CE graph is represented in the matrices in the form of true and invalid entries. An entry (i, j) in the PC matrix is true if check Cj corresponding to the j ^ ^ column is not evaluated on processor pi corresponding to the i*'^ row, else the entry is defined as invalid. The PD matrix for the PDC graph given in Figure 2.1.2 is an identity matrix of order 16. Therefore, the PC matrix is the same as the DC matrix, and is given below.
Pi P2 P3
P4
Ps Pe P7
PC=PDx
DC =
Ps P9 PlO Pll P12
Pis Pl4 Pis PI6
Ci
C2
C3
C4
C5
Ce
Cj
1 1 1 0 0 0 0 0 0 1 0 0 0 0 0 0
0 0 0 1 1 1 0 0 0 0 1 0 0 0 0 0
0 0 0 0 0 0 1 1 1 0 0 1 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1
1 0 0 1 0 0 1 0 0 0 0 0 1 0 0 0
0 1 0 0 1 0 0 1 0 0 0 0 0 1 0 0
0 0 1 0 0 1 0 0 1 0 0 0 0 0 1 0
The matrices defined next are taken from [4, 17], and are useful in the
90 analysis of ABFT systems. Definition 6 ^ PD is defined as the matrix whose rows are formed by adding r different rows of matrix PD, for all possible combinations of r rows, and setting all non-zero entries in the resulting matrix to 1. Definition 7 "^PC is the m,atrix obtained by the product of the matrices and DC.
^PD
A row in matrix'' PD represents a unique set of r processors and the set of *1' entries in the row represent the union of the data sets of these processors. The (i, i ) entry in matrix '"PC represents the number of data elements of the corresponding r processors that are checked by the j ^ ^ column check. For example, row (pi,P2) in the ^PC matrix derived from the PC matrix given earlier will be [2 0 0 0 1 1 0].
2.1.3
Design of ABFT Systems
Each processor in the system may compute one or more data elements. When a processor is faulty any one or more of its data elements can be in error. Therefore, a fault pattern can give rise to several error patterns. Nair and Abraham [20] proposed a hierarchical method of construction to take care of different error patterns. They defined a unit system in which each processor computes only one data element. Given a unit system of desired fault tolerance capability, they proposed the construction of a product system by connecting every data element affected by a processor in the actual system to every check with which the processor is connected in the unit system. The product system will have the desired fault tolerance capabilities provided there is no sharing of data elements among the processors. An alternative procedure proposed by Vinnakota and Jha [6] takes care of sharing of data elements. Their procedure of forming the final system, called the composite system^ is discussed in Section 2.1.3.2.
2.1.3.1
Unit S y s t e m Construction
In a unit system each processor computes only one data element. The system given in Example 1 is a unit system. The presence of a fault in a processor of the
91 unit system implies the presence of an error in the corresponding data element. Therefore, the cardinality of a fault pattern is the same as the cardinality of the resulting error pattern. In such a case, designing 5-fault detecting/diagnosing systems is the same as designing 5-error detecting/diagnosing systems. Therefore, in this section, we will concentrate on error detecting/diagnosing systems. Randomized [8], as well as deterministic [13, 26, 27], techniques have been used in the construction procedures for such systems, though no general deterministic techniques exist for designing ^-error diagnosing systems, except for the simple case of t = 1 [11]. In this section, we discuss both types of techniques.
Randomized Construction The randomized construction approach is based on a simple procedure called RANDGEN which was proposed by Sitaraman and Jha [8]. RANDGEN is very fast and easily parallelizable, and uses unbounded checks. It can produce efficient DC graphs with a wide spectrum of properties by just changing its input parameters. For example, it can produce 5-error detecting DC graphs with asymptotically optimal number of checks and t-error diagnosing DC graphs with asymptotically nearly optimal number of checks. Let D , \D\ — n, be the set of data elements and C, \C\ — 5, be the set of checks. The DC graph is constructed by adding edges between the set of data nodes and the set of check nodes. RANDGEN makes random decisions during the construction of edges, using probability p which is an input parameter to the procedure, as follows. Algorithm
RANDGEN{q,p):
For every pair (u, v), where u ^ D and v ^ C do the Add edge (u^v) to the DC graph with probability p.
following: •
Algorithm RANDGEN considers an A B F T system under the original model. An extension of the algorithm, as given below, was proposed in [10] which gives design procedures for A B F T systems under the extended model. Let Di be the set of m info_data elements di, ^2,..., dm and D2 be the set of q check_data elements d'^ dj,..., d'^. Let C be the set of q checks, ci, C2,..., Cg.
92 Algorithm
Mod.RANDGEN{q,p):
For the construction of the DC graph: • For every pair (^J, c^), 1 < i < 5, add an edge to the DC graph. • For every pair (1^, c^), 1 < i < g, where u E Di U {D2 — {^i}}, add edge (u, Ci) to the DC graph with probability p. For the construction of the CE graph: • For every check c^, choose uniformly but randomly a processor pk from the set of all processors. Add edge {ci^pk) to the CE graph. D The two algorithms consider each pair of nodes independently. Therefore, it is easy to parallelize their implementation. The following theorems show how RANDGEN and Mod^RANDGEN are used in designing error detecting and diagnosing systems. The proofs of the theorems are given in [8] for RANDGEN and in [10] for Mod-RANDGEN. All logarithms, unless specified otherwise, are to the base 2. Error Detecting S y s t e m s We first discuss methods for design of error detecting systems. Theorem 1 Algorithm RANDGEN, using parameter q — 3.8s log n and p — - , produces an s-error detecting DC graph, under the original model, with probability at least 1 — ;;3y* Example 3 Consider a multiprocessor system on which a data set consisting of a matrix of dimension 512 x 512 ( = 262144 data elements) is mapped. Suppose we want to make the system 3-error detecting. We need to have 3.8 x 3 x 1 8 = 205.2 checks in the system. Since the number of checks is an integer, we add 206 checks to the system. With each check we do the following. We consider each data element and add it to the check's data set with probability | . When we are done with all the checks in the system, we get a DC graph which is 3-error detecting, under the original model, with a probability at least 1 — 262^43 ~ 0.999996. As a comparison, the traditional deterministic method requires 1023 checks. However, each check in that case checks much fewer data elements. D
93 T h e o r e m 2 Algorithm
ModJiANDGEN{q,p),
checks, where n = m-\-q,
andp — 7^37^7; produces an s-error
under the extended
model, with probability
using q -
(^^J^^'l[^L,\\ogn detecting
system,
at least 1 — ;^33;.
Error Diagnosing Systems We discuss design of error diagnosing systems next. T h e o r e m 3 For every S C D, | 5 | = 2t — 1, and for every di E D, di ^ S, let there exist a check which is connected to di, but not to any data element of set S. Then the DC graph of the system is t-error diagnosing under the original model. For the extended model, the check has to satisfy an additional constraint that it is not evaluated on a processor computing any element of set S\^di. Using T h e o r e m 3 we can prove the next two theorems. T h e o r e m 4 Algorithm RANDGEN, p — ^, produces a t-error diagnosing probability at least 1 — ^.
using q = {7M^ + 3 . 8 t ) l o g n checks and DC graph, under the original m,odel, with
E x a m p l e 4 Consider the 512 x 512 m a t r i x , with 262144 d a t a elements, once again. Suppose we want to make the system 3-error diagnosing, under the original model. According to T h e o r e m 4, we need (7.8 x 9 + 3.8 x 3) x 18 = 1468.8 checks in t h e system. Therefore, for each of the 1469 checks we do the following. We consider each d a t a element in the system a n d a d d it to the check's d a t a set with probability | . T h e resulting system is 3-error diagnosing with probability at least 1 - 2 6 ^ = 0.999996. D
T h e o r e m 5 Algorithm
ModJlAN
DOE N{q,p),
using q ^
checks, where n = m-\-q, and p = (^^Vtt2\ P''^oduces a t-error under the extended
model, with probability
'^^^^^+1^^)^'^^'^ diagnosing
system,
at least 1 — - .
In each of the above cases if one is not satisfied with the probability with which one can obtain an 5-error detecting or t-error diagnosing system, one can make this probability even closer to 1 by adding a few more checks.
94 A number of researchers have considered systems which have a tion of error detection and diagnosis properties, e.g. a system which detecting/t-error diagnosing. One simple way of doing this would be two systems, one 5-error detecting and the second t-error diagnosing superimpose one system on the other to get the final system. The theorem states that we only need to worry about the case where s >
T h e o r e m 6 Any t-error
diagnosing
system
is also 21-error
combinais 5-error to design and then following 2t [8].
detecting.
Deterministic Construction Several deterministic techniques exist for the construction of error detecting systems. Some of these techniques use bounded checks whereas others use unbounded checks. There exist no general deterministic techniques for the construction of error diagnosing systems, except for the case of 1-error diagnosing systems. The construction techniques given in this section use bounded checks and are implemented under the extended model. The design techniques under the original model are based on similar concepts [27]. The checks used in the unit system construction methods are (gf, 1) checks. Banerjee and Abraham [13] have shown that even if h of the (y, 1) checks in an error detecting system are combined into a single [gh^ h) check, the system has the same error detection capabilities. Also, most traditional coding schemes use (^, 1) checks even for error diagnosing systems. Therefore, all system designs can be confined to systems which have (^, 1) checks. The info-data elements in the original non-fault-tolerant PD graph are distributed among the template unit system and its copies. The method of distributing the info_data among the unit systems is given later in Section 2.1.3.2. In this section, we assume that the distribution has already been done. Consider a template unit PD graph consisting of m processors and m info_data elements. Each info_processor in the unit system is connected to exactly one info_data element. In the extended model the system has to satisfy the constraint that each check should have at least one check_data element in its data set. In such unit systems, the number of info_data elements is equal to the number of info_processor nodes m and the number of check_data elements is equal to the number of checks q. Only one check_data element can be mapped to one code.processor. So the number of code_processors introduced in a unit
95 system is equal to the number of checks. T h e deterministic m e t h o d s for design of error detecting a n d 1-error diagnosing systems, under the extended model are discussed next [9, 11]. Note t h a t the systems constructed by these m e t h o d s are all unit systems, therefore, error detection and diagnosis is s a m e as fault detection a n d diagnosis for these systems. Error D e t e c t i n g S y s t e m s We discuss m e t h o d s for construction of 5-error detecting systems next. T h e o r e m 7 The number ing system
of checks sufficient
under the extended m,odel is
- ^
for constructing
a 1-error
detect-
.
M e t h o d : Construct d a t a sets of each check by taking g — I elements from the info_data set and an unused element from the check_data set. Each d a t a element is present in exactly one check's d a t a set a n d therefore, any single error will be caught by the checks. D
T h e o r e m 8 For g > 2, m > [g — ly, constructing
a 2-error
detecting
system
the nuvnher of checks sufficient
under the extended
m,odel is
^^
for .
Method: 1. Let A = set of info.data elements in the unit system, where |A| = TTI >
(.9-1?. 2. Let B = set of check_data elements in the unit system, where \B\ — q —
3. Arrange set A of info_data elements in a grid of (^ — 1) columns. 4. Construct r o w c h e c k s by taking (^ — 1) info_data elements of a row in A a n d a single unused element from check_data set B, If the last row has less t h a n (^ — 1) elements in it, take elements starting from the t o p of the first column to complete the row check. 5. Construct c o l u m n c h e c k s by taking an unused element from B a n d {g — 1) elements of a column from A starting from where the last r o w / c o l u m n
96 check ended, going to the t o p of the next column if there are no elements left in t h e present column. •
E x a m p l e 5 Let m = 17, 5 r= 2, ^ = 5. T h e n \B\ = Ijzj]
= 9 a n d \A\ = 17.
Let A = [ai, 0 2 , . . . , 017] and B — [61, 62,..., 69]. A^ arranged in a grid with 4 columns, is shown below. cii
a2
CI3
04
dh
^6
0.7
^8
ag ai3 ai7
aio ai4
ciii ai5
ai2 aie
Checks: Row checks r i = [01,^2,03,04,61] ^2 — 7*3 = 7*4 = ^5 =
[05,06,07,08,62] [09, Oio, Oil, O12, 63] [0l3, Oi4, a i 5 , O16, 64] [017,01,05,09, 65]
C o l u m n checks ci = [013,017,02,06,65] C2 = [oiO, Oi4, O3, O7, 67] C3 = [011,015,04,08,63] C4 = [oi2,Oi6, 69]
Total n u m b e r of checks == 5 -|- 4 = 9. D
T h e o r e m 9 For g > 2, m> construction m+
[g — lY,
the number
of a Z-error detecting system
of checks sufficient
for the
under the extended model is -3^-
+
bzil
^-1
Method: 1. Let B be the set of
^-1
check_data elements.
2. Let A b e the set of all info_data elements in the unit system a n d the rest of
-3j-
check-data elements.
97 3. Construct a grid of d a t a elements of A with (gf — 1) columns. 4. Number the elements columnwise, i.e. go down the first column, then the second column and so on. Place the check_data elements present in A at positions ^, 2^, 3^,... in the grid and if the n u m b e r of elements in A is not a multiple of g then place a check_data element at the b o t t o m of the last column. 5. Construct r o w c h e c k s with a row of A a n d an unused element of B, 6. Construct c o l u m n c h e c k s with g elements by going down a column a n d going to the next column if the present column is used u p . •
E x a m p l e 6 Let m = 17, 5 = 3, ^ = 5. Hence, the n u m b e r of checks = 11.
-3^- -f
Let info_data elements — [^i, 02,..., 017] a n d check_data
elements = [61, 62, -.., &11]. T h e n \A\ - m-f- f ^ j
- 22 a n d \B\ -
.
^_^
6. Let A — [ai, 02,..., aiy, 67, h%^.., 611] and B — [61, 62, •••, ^e]- A^ a r r a n g e d in a grid with 4 columns, is shown below.
ax
ae
^2
a?
ail ai2
03
08
^>9
^10
a^
«>8
^13
ai7
h
ag
ai4
fell
as
^10
^15 ^16
Checks
Row Checks 7*1 = [ai, cte, a i l , ^15, fei] 7*2 = [^2, a?, ^12, ^16, ^2] 7*3 = [^3, ^8, ^9, feio, ^3] r4 — [a4,fegj^13, a i 7 , 64] rs == [67, ag, ai4, ^ n , 65]
Column Checks ci — [ai, 02, a s , a4, 67] C2 == [as, tte, a7, ag, 63] C3 = [ag, aio, a n , ai2, 6g] C4 = [ais, ai4, a i s , a i e , 610] C5 = [ai7, ^ n ]
7*6 = [asjCtio,
D
fee]
For the case of 771 < (^ — 1)^, the m e t h o d s for 5 = 2 and 5 = 3 are similar
98 and can be found in [9]. For 5 > 4 a hierarchical construction method is used which is also given in detail in [9]. Error D i a g n o s i n g S y s t e m s A procedure for designing 1-fault diagnosing systems has to consider the fact that a single fault can give rise to several error patterns and hence to several syndromes. For diagnosing single faults present in the system, it is necessary that each syndrome produced by a fault be distinct from the syndromes produced by other single faults in the system. If there is sharing of data elements among processors, the system cannot always locate faults in individual processors [17]. If only a shared data element is found to be in error, all the processors involved in the computation of that data element are declared to be faulty. However, if at least one of the erroneous data elements in the data set of a processor class is not a shared one, then the faulty processor can be pinpointed exactly. The system design uses (^,1) checks. Such a check may be invalidated if there is more than a single erroneous data element in its data set. In order to prevent invalidation of checks in the presence of a single fault in the system, no check's data set should have more than a single data element from any processor's data set. This means that there cannot be any entries greater than *1' in the PC matrix. Let a be the maximum cardinality of a processor class. No two processors in a processor class should be checked by the same check. If this is allowed, then some copy of the template unit system may have two data elements of the same processor being checked by the same check, which might result in check invalidation. Consider a square grid G of size n x n or a rectangular grid of size (n -f 1) x n. Number the rows in the grid as 0, 1,..., n — 1, n, and similarly the columns from 0 to n— 1. The j^^ diagonal is defined as containing the positions (i, (i -f j) mod n), 0 < i < n — 1 , of the grid. T h e o r e m 10 For a < of checks sufficient
T ^ Z W \ {g - I), g > 2, m > {g - 1)^, the
for constructing
tended model is Ifzi]
a 1-error
diagnosing
system
number
under the ex-
+ [ r ^ W ] id " ! ) •
Method: 1. Let A == set of info_data elements in the unit system, where |A| = m >
99
2. Let B = set of check-data elements in the unit system, where \B\ =: q = jzi
+
(^^1)3 {g - !)• Let m = x{g - 1)^ + y, where x and y are
integers, such that x =
/ ^^va
and y < {g — 1)^.
3. Arrange set A of info_data elements in grids Ai, ^2)---) ^x^ of size [g — 1) X (^ — 1) each. Arrange the rest of the y elements in a grid Ax^i of {g~l) columns. Place the info.data elements corresponding to a processor class either along the diagonals in one grid or in different grids. Once a processor class occupies some positions in a diagonal, the unused positions can be used by another processor class. 4. Construct row checks by taking all info_data elements of a row in Ai^ 1 < i < X -h 1, and a single unused element from check-data set B. 5. Construct column checks by taking an unused element from B and all elements of a column from Ai, 1 < i < x -\- 1. D In the above construction procedure, the data elements corresponding to a processor class are placed along the diagonals in a grid or in separate grids because no two diagonal entries, or entries from separate grids can be in the same check's data set.
Example 7 Let m == 14, g = A. Then \B\ = - ^
+ ^ - ^
(gf - 1) = 11 and
\A\ = 14. Let A = [ai,a2) •••)Cti4] and B = [^i, ^2) •••> ^ii]- Suppose processors Pi, p4 and ps computing ai, a^ and ag, respectively, are in one processor class Pi, and processors p2j Pe and p7 computing 02, ae and ar, respectively, are in another processor class P2. Arrange set A in two grids, Ai of size 3 x 3 and A2 of size 2 x 3 . The data elements corresponding to Pi are placed along diagonal 0, and the data elements corresponding to P2 are placed along diagonal 1 in grid Ai, as shown:
O'l ^
as as ar
0.2 ^
a^ a^ ag
^3
^ ae ae ag
A
,
^10
^11
ai3
ai4
A2 :
^12
100 T h e row a n d column checks can then be derived as follows: R o w checks 7*1 = [ai, CL2, a s , bi] ^2 — [^5, ^4, ae, 62] 7*3 = [a?, 09, ag, 63] r4 = [aio, a n , ai2, 64] 7*5 = [cti3, cti4, ^5]
C o l u m n checks ci = [ai, a s , ay, he] C2 = [^2, 04, ag, 67] C3 = [as, ae, ag, 6g] C4 = [aio, aia, 69] C5 = [ a n , ai4, 610] ce - [ai2,^ii] °
T h e m e t h o d of construction for the case when m < (^ — 1)^ is similar to the m e t h o d given in the previous theorem. 2.1.3.2
Composite System
Construction
It is assumed here, as in [6], t h a t the processors in the original non-faulttolerant system can share d a t a elements, b u t each processor produces at least one d a t a element which no other processor in the system affects. These d a t a elements are defined as the distinguishing data elements of the processor. For designing the complete A B F T system, the unit t e m p l a t e system has to be constructed first. From the unit t e m p l a t e system, copies are created and then these systems are combined to form the composite A B F T system using some rules. T h e m e t h o d of construction is an extension of the design process given in [6]. A d a t a element is defined as filled if it has been used in or added to the tripartite graph of the unit system, else it is called unfilled. Given the original non-fault-tolerant P D graph:
1. Construct the PD graph of the t e m p l a t e unit system, where each processor is connected to exactly one of its distinguishing d a t a elements. If more t h a n one distinguishing d a t a element exist, then one can be chosen at r a n d o m . 2. Construct copies of the t e m p l a t e unit system such t h a t all the unfilled d a t a elements in the system are filled. If there are no unfilled d a t a elements left for a processor, then reuse an already-filled element. For details of this p a r t the readers are referred to [6]. 3. Using the m e t h o d s given in Section 2.1.3.1, design the DC g r a p h of the t e m p l a t e unit system such t h a t it has the desired fault tolerance
101 capabilities. For the extended model, first add check_data nodes and code_processor nodes to the template unit PD graph and then design its DC graph. 4. In the copies of the template, add the check nodes with their edges to the data nodes connected in the same way as in the unit template system. However, the check nodes in the copies are numbered differently. For the extended model, add the code_processor nodes, check_data nodes and the check nodes as in the template unit system but rename the check-data nodes and the check nodes. •
Given the template system and its copies, the composite system is formed by superimposing the unit systems on one another, as follows.
1. The data sets of the processor nodes are the same as in the original non-fault-tolerant PD graph. In the extended model, the data set of a code_processor in the composite system is the union of its data sets in the unit systems. 2. While merging the DC graphs of the different unit systems, the set of checks to which a data element is connected is formed by taking the union of the set of checks to which it is connected in each of the unit systems. After this, the checks whose data sets are found to be identical are merged into a single check. In the extended model, checks which have the same data sets (except for the check_compare elements which may be different) are merged into a single check. The corresponding check_compare element is removed from the system if it is not connected to any other check. • The example given next is under the extended model. E x a m p l e 8 Consider the implementation of some algorithm on a parallel system with the PD graph as shown in Figure 2.1.3(a). Suppose we wish to design a 2-fault detecting version of this system. A template system for the PD graph, which is 2-fault detecting (with g — Z)/\s shown in Figure 2.1.3(b). The info_data set of the template system is {^1,^3,^4,^5}. Each member of this set is a distinguishing element. By Theorem 8 we require four checks to detect two faults in the template system. Therefore, we need to add four
102
Figure 2.1.3: (a) The original PD graph, (b) a 2-fault detecting template unit system, (c) a copy of the template system, (d) the composite system. check-data nodes {de, cZr, dsj dg^^ and consequently four code-processor nodes P5) P6) P7 s-nd ps to the graph. A single copy needs to be formed to fill the remaining info-data elements, as shown in Figure 2.1.3(c). The info.data set of the copy is {^2, ds^ ^4, ^5} and the check-data set is {dio, tin, ^^12, ^£13}. The code_processors remain the same. The template system and the copy are now composed to form the final system which is shown in Figure 2.1.3(d). In the final system, check ce is equivalent to check C2 and check cg is equivalent to check C4. So checks CQ and cg are removed. • The following theorem shows that the fault tolerance properties of the unit systems are maintained in the composite system [6]. T h e o r e m 11 If the template system is s-fault detecting/t-fault diagnosing, the composition of the template system and the copies is also s-fault detecting/tfault diagnosing.
103 The unit/composite system construction techniques only ensure that the fault tolerance properties are maintained if the checks are computed on faultfree processors. For the extended model, a third stage of design called check mapping is required. The randomized construction procedure Mod^RANDGEN takes care of the check mapping part during the unit system construction stage, whereas in deterministic construction this has to be done separately. The next section deals with the mapping of checks onto system processors. Since a general deterministic construction procedure is not available for fault diagnosing systems, we will concentrate on check mapping for fault detecting and 1-fault diagnosing systems only. 2.1.3.3
Check Mapping
Given a composite system which is designed to be 5-fault detecting or 1-fault diagnosing assuming that the checks are computed correctly, check evaluations have to be mapped to the processors in such a way as to maintain the fault tolerance capability of the system even when the checks may themselves fail. This section gives methods for performing the check mapping operation for 5-fault detecting systems and 1-fault diagnosing systems. Fault Detecting S y s t e m s If the check_computing processors in an ABFT system are allowed to be faulty, then a fault pattern is detectable if and only if, for every error pattern that the fault generates, there is at least one non-faulty processor evaluating a check which has a single erroneous data element in its data set. All the processors in the system have the capability to perform the checking operations. The following are some definitions and theorems from [17].
Definition 8 A row of^ PC is said to he completely detectable if and only if the fault pattern represented by the row is detectable for all possible error patterns produced by that fault.
T h e o r e m 12 A system is s-fault detecting if and only if the matrices ^PC, for i — 1, 2,...., s, are com,pletely detectable, i.e. all the rows of each of the m,atrices are completely detectable.
104 For a row of'"PC to be detectable, there should be at least one entry in the row which is less than or equal to the error-detectability (h) of the check used, and the corresponding check should be evaluated on a processor which is not in the set of faulty processors defined by the row. This should be true for all the rows of each of the matrices *PC, for i = 1,2,...., s. Since we want to take care of all the possible error patterns that a fault pattern can generate, we consider the PC matrices of each of the unit systems, instead of operating on the PC matrix of the final composite system. The processor set of a check is defined as the set of processors, any one of which can be used to evaluate the check. We need to find the processor sets of each of the checks such that the mapping of checks to processors maintains the fault detectability of the system. Before going into the details of the algorithm used for check mapping, consider the following example. *
E x a m p l e 9 Consider the PD graph of the system given in Figure 2.1.3(a). Suppose we want to make the system 1-fault detecting. The unit systems consist of 4 info_data elements. In order to detect a single fault in the system we add two checks with ^ = 3 (from Theorem 7). The unit systems and the final system are shown in Figure 2.1.4. Construct the PC matrices of the two unit systems, PCi and PC2 and concatenate them as shown in Figure 2.1.4(d) to form matrix PF. In these PC matrices, all the checks in the composite system, not just those present in the unit system, are included, because of the need to concatenate these matrices. Number the rows in PF from 0 to 11. We have to make sure that each row in PF has at least one true *1' entry. In row 0, there is a *1' entry for check ci. In order to make this entry true, check ci should not be evaluated on pi. In row 2, there is a '1' entry corresponding to check C2. If this entry is to be true, check C2 cannot be evaluated on processor ps. This also makes the *1' entry in row 8 true. Similarly, the other rows are dealt with. The following processor sets for the checks satisfy all the above constraints.
set.
Check
Processor Set
C3
{P3, P4, Pe} {Pi, P2, Ps] {P3, P4, Pe}
We can then map a check to any one of the processors in its processor One possibility is to map ci to p4, C2 to ps, and C3 to pe. In general.
105
PC matrix of , unit system 1 \
PC matrix of unit system 2
^^6
V 0 r 0 0 r 0 1* r 0 0 1* 0 0 0 0 0 r 0 r 0 0 0 r
0 0 0 0 0 0
r r 0 0
r 0
0
• Cj cannot be evaluted on p
I
• Cj cannot be evaluted on p
2 3
^ C2 cannot be evaluted on p ^ C cannot be evaluted on p.
4
^ C cannot be evaluted on p
5
^ C cannot be evaluted on p
6
• C cannot be evaluted on p
7 8
^ Cj cannot be evaluted on p already satisfied by 2
9 already satisfied by 3 10 » Cj cannot be evaluted on p II
already satisfied by 5
(d)
Figure 2.1.4: (a) Unit system 1, (b) unit system 2, (c) the composite system, (d) the PF matrix. because of this flexibility in mapping, we can consider the communication delay overhead and/or the processor load while assigning checks to check_computing processors. For each of the processor choices for a check, one can compute the communication delay to transfer the data elements, that the check checks, to the processor, and then choose the processor which will entail the least delay. At the same time, we can try to avoid assigning more than one check to a processor, whenever possible, in order to balance the processor load. • Algorithm for mapping of checks: 1. For each of the unit systems, construct a PC matrix. Tag each row of the
106 matrix with the corresponding processor. Let d be the number of unit systems. Term the PC matrix of the i^^ unit system as P Q , I < i < d. A check in the final system has a column in each of the PC matrices. Throw out the columns corresponding to the checks which are redundant and have been removed from the composite system. For each of the PCi matrices, construct ^PCi, 1 < r < s. Each row in ^PCi is a combination of r rows of PCi. Tag each row in ^PCi with a set of r processors corresponding to the r rows whose combination it is. Initialize the processor sets of each check to all the processors in the system. Columnwise concatenate all the matrices ^PCi^ 1 ^ '^ ^ ^j 1 < i < c2, to form one matrix PF. Arrange the rows in the final matrix PF in an increasing order of the number of *l's and number the rows from 0 onwards. Number the columns from 0 to 5 — 1, where q is the total number of checks in the system. In each row of PF, mark the *!' which appears first in or after the column i modulo g, where i is the row number. For all rows of PF starting from the first do i = 0; throw — 0; While ( throw = 0 & i < g - l ) { if ( the i^^ column entry is *!') and (the set of processors in the row tag p| the i^^ column check's processor set = (f)) then { throw the row out of the matrix; throw = 1;
} i = i + 1;
} if ( throw = 0) then do the following: (a) Let present-check = Marked *1' column. (b) If (processor set of present_check) —(set of processors in the row tag) = <j)^ label the marked *!' as had and mark a *!' in that row which is not labeled bad. Go back to Step 7(a). Else, remove those processors from the present_check's processor set which are also in the row tag of the present row. (c) Throw the present row out of the matrix.
107 8. Map each check to any of the processors in its processor set. Here, we can consider the communication overhead and/or the processor load while assigning the checks to the processors. D
Fault Diagnosing S y s t e m s For a system to be 1-fault diagnosing, no two faulty processors should produce the same syndrome. Nair and Abraham [17] introduced the concept of disagreements to check whether two faulty processors are distinguishable or not. A few definitions and theorems adapted from [17] are presented here.
Definition 9 Rows Ri and R2 of matrix PC are said to have a 0 — 1 disagreement if there is at least one true ^0' in either Ri or R2, such that the other row has a true *1' in the corresponding position, for all possible error combinations caused by the faults corresponding to the two rows. Definition 10 If all pairs of rows of matrix PC have a 0 — 1 disagreement, then PC is said to have a 0 — 1 disagreement with itself. If two rows in a matrix ^PC have a disagreement, it means that the two fault patterns are distinguishable if all the data elements corresponding to those fault patterns are in error. The distinguishability may not hold if only a subset of the data elements corresponding to the fault patterns are in error. Nair and Abraham [17] defined a stronger condition called complete disagreement to ensure full distinguishability of two fault patterns, as follows.
Definition 1 1 - 4 disagreement between two rows is called a complete disagreem,ent, if the disagreement exists for all possible error patterns corresponding to the fault patterns associated with the two rows.
Theorem 13 A fault tolerant system, is 1-fault diagnosing if and only if PC has a complete 0—1 disagreement with itself. If the PC matrices of the unit systems are considered, the information regarding the sharing of data elements is lost. Instead of working with the PC
108 C4 C^ C^
^2
1
0 1 0 0 0
C2 C^ C4 C^ C.
Pi
0 0 0 0
PI
0 1 0 1 0 0
Pi
0 10
1 1 1
Ps
1 0 0 1 0 0
P3
0 0 0
1 1 0
PI
P4
? P6
0 1 1 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0
P4 P5 P6
0 10
11
0 0 1
0 0 0 0 0 1 0
10
0 0 0
9
0 0 1 0 0 0
B
0 0 0 0 0 1
P.
0 0 0 1 0 0
p.
0 0 0 10
(a)
0
(b)
Figure 2.1.5: (a) PC matrix of the template unit system, (b) PC matrix of the copy matrices of the unit system directly, the PC matrices of the units systems are modified to preserve this information. For the info.processor part of the system, take the PD matrix of the original non-fault-tolerant system, but consider only those columns whose corresponding data elements are present in the unit system. The DC matrix has rows corresponding to the data elements in the unit system and columns corresponding to all the checks in the composite system. The columns whose checks are not in the unit system have all *0' entries. In the DC matrix, a check in a unit system copy which is equivalent to a check in the template unit system or another copy, is represented by its corresponding check in the composite system. The augmented PC matrix of the unit system is then obtained by multiplying the PD and the DC matrices. Hereafter, in this section, the PC matrix of a unit system means the augmented PC matrix. The PC matrices of the unit systems in Figure 2.1.3 are given in Figure 2.1.5. T h e o r e m 14 If PC matrices of each of the unit systems have 0—1 disagreem,ent with them,selves and no check in the com,posite system has m,ore than a single data element from a processor's data set, then the PC matrix of the com,posite system, has a complete 0 — 1 disagreement with itself. The PC matrix of the composite system should have a complete 0 — 1 disagreement with itself to ensure 1-fault diagnosability in the system. By Theorem 14, the PC matrices of the unit systems can be handled directly, instead of the PC matrix of the composite system. A PC matrix having a 0 — 1 disagreement with itself means that the ^PC matrix should have at least one *1' in each row. For the 0 — 1 disagreement between two rows Ri and R2 to
109 hold even in the presence of faults in the check_coniputing processors in the system, at least one *1' in the row in ^PC^ corresponding to the (i^i, R2) pair, should be true. The algorithm for mapping of checks is similar in flavor to the algorithm used in the case of fault detecting systems, except that the algorithm considers only the ^PC matrices of the unit systems while performing the check mapping. The details of the algorithm can be found in [11].
2.1.4 An of s or section as well
Analysis of ABFT systems analysis of an ABFT system involves determination of the largest value ^, such that the system is 5-fault detecting or ^-fault diagnosing. This will present methods for analyzing an ABFT system under the original as extended models.
As defined earlier, a row in ^PD corresponds to a unique set of i processors and the entries in the row correspond to the data elements computed by this set of processors. Similarly, a row in *PC corresponds to a set of i processors. A fault pattern of size i can be represented by a row R in the ^PC matrix. If all the data elements in the data set of the fault pattern are erroneous, then an entry ^PC{R^j) represents the number of erroneous data elements in the data set of the check corresponding to the j ^ ^ column of the matrix. In the presence of fault pattern R:
1. U'PC{RJ)
- 0, then the 3^^ bit in the syndrome is a *0'.
2. If 0 < 'PC{RJ) < /i, then the j ^ ^ bit is a 'V and the entry 'PC{RJ) defined as valid. 3. If^PC(Rjj)
is
> h the value of bit j is unpredictable.
4. In the extended model • If the check corresponding to the j ^ ^ column is not evaluated on any processor corresponding to R then ^PC{R^j) is a true entry. • Else, it is an invalid entry and the value of bit j is unpredictable.
no 2.1.4.1
A n a l y s i s o f Fault D e t e c t a b i l i t y
The algorithm for analyzing the fault detectability of an ABFT system under the original model was given by Nair and Abraham [17]. For each fault pattern there may exist more than one error pattern. If the algorithm enumerates all possible error patterns it will be very complex. Nair and Abraham proposed an error collapsing technique so that their algorithm converges much faster and requires less storage. Their algorithm for analyzing detectability is as follows. Algorithm for Fault Detectability: 1. 2 = 1, 5 = 0.
2. Construct ^PC. 3. For every row R in ^PC do the following: (a) If row R has no valid entry, set 5 = i— 1, STOP. Else, go to the next step. (b) Find all values of j such that 0 < 'PC{RJ)
< h.
(c) For all j do the following: • For all rows in the DC matrix, \iDC{k,j) - 1, set ^PD{R, k) = 0 and set the corresponding elements in the k^^ column of the PD matrix to '0'. (d) If at least one row of the new PD matrix has all zeros, row R is completely detectable. Go to Step 3 for the next row. Else, continue. (e) Find the new ^PC matrix by multiplying the new ^PD matrix and the DC matrix, and go back to Step 3(a). D When we analyze the fault detectability of a system under the extended model, the basic algorithm is the same except that instead of only valid entries, valid and true entries have to be considered. A small example of analysis under the original model is given next. Example 10 Consider the PD, DC and *PC, 1 ^ ^ ^ 4, matrices given next. Matrices PC, ^PC and ^PC have *1' entries in all rows for all error patterns. Matrix ^PC does not have any valid entries for error pattern (cfi, ^2,0^3,^^4, ^5). Therefore, the system is only 3-fault detecting.
Ill
^
d2 0 1 0 0
ds 0 1 0 0
Cl
C2
C3
C4
1
0 0 0 1 1
1 0 0 1 0
1 1 0 0 0
Cl
C2
C3
C4
1 1 0 0
0 0 1 1
1 0 1 0
di
PD =
Pi P2
r 1
P3 P4 L
di
DC
PC =
d2
ds d^ ds
Pi P2 P3
P4 1
^
0 0
0 1 0 0
(Pl,P2) (Pl)P3)
^PC =
(Pl,P4) (P2,P3) (P2,P4) (P3,P4)
^PC
^5
0 0 1 0
0 0 0 1
1 0
0J
C2
C3
C4
2 1 1 1 1 0
0 1 1 1 1 2
1 2 1 1 0 1
2 1 1 1 1 0
(Pl)P2,Pa ) (Pl>P2,P4 ) (Pl)P3,P^ )
-^
Cl
C2
i'3
C4
2 2 1 1
1 1 2 2
2 1 2 1
2 2 1 1
Cl
(Pi,P2,P3,P4)
-1
1 1
Cl
(P2, P3,P4 )
*PC =
04
[
C2
2
2
C3
C4
2
2
D
112 2.1.4.2
A n a l y s i s of Fault D i a g n o s a b i l i t y
For a system to be t-fault diagnosing, no two fault p a t t e r n s of size t or less should produce the same syndrome. T w o fault p a t t e r n s Fi and F2 are said to be fully distinguishable if there exist no two error p a t t e r n s Ei and E2 corresponding, respectively, to Fi and F2 such t h a t the syndrome produced by the two error p a t t e r n s are the same. Next we give a few definitions from [17].
D e f i n i t i o n 12 Rows Ri and R2 of a matrix ^PC are said to have a 1 — 0 disagreement if there exists at least one valid entry in row Ri such that the corresponding entry in row R2 is ^0\
D e f i n i t i o n 1 3 Rows Ri and R2 of a matrix ^PC are said to have a 0 — 1 disagreem,ent if there exists at least one valid entry in one row such that the corresponding entry in the other row is a '0\
D e f i n i t i o n 1 4 Matrix ^PC has a 0 — 1 disagreement with itself if every row in the m,atrix has a 0 — 1 disagreement with every other row in the matrix.
D e f i n i t i o n 15 Matrix PC has a 1 — 0 disagreement with matrix ^PC if every row R of PC has a 1 — 0 disagreement with every row in ^PC which does not contain R.
All the above definitions apply to the extended model if the valid entries a n d the *0' entries considered are all true entries. Henceforth, when we refer to disagreement, we mean a complete disagreem,ent. T h e following theorems from [4] form the basis for the fault diagnosability algorithm.
T h e o r e m 15 A system is t-fault diagnosing iff ^PC has a com,plete 0 — 1 disagreement with itself and PC has a complete 1 — 0 disagreement with ^~^PC.
T h e o r e m 16 Let ki be the largest integer such that PC has a complete 1 — 0 disagreem^ent with ^^PC. Let k2 he the largest integer such that ^^PC has a complete 0 — 1 disagreement with itself. The system is then t-fault diagnosahle for t = min{ki + 1, AJ2).
113 Theorem 17 If every check is connected to at least two processors in the PC graph, then t < ta in a t-fault diagnosing system, where ta is the minimum number of checks checking a data element in the system. Let COV be the minimal set of processors which is connected to all the checks in the system. Let tj = |COV|, if COV is the only such set of minimal cardinality. If there is more than one such minimal set, th — \COV\ — 1. Theorem 18 In a t-fault diagnosing system,, t 0 and i^e* > 0, where e represents the vectors corresponding to the edges in the DG [36]. The synthesis method from [5] is as follows. Synthesis Method:
1. Encode the algorithm using a property which remains invariant during the computation. 2. Write the encoded algorithm in single assignment form. 3. Construct a DG for the encoded algorithm from the single assignment form. 4. Project the DG in different directions to get various architectures. 5. Choose the architecture which is optimal in terms of some cost function based on fault detectability, diagnosability, hardware and performance. D
120
r
^31
^32
^33
5^^j
^2,
«i3 «i2 ^ i i - K j — K J — K J — K J T'^ll 1^12 1^13 Y ^23 ^22 « 2 1 - K J K) KJ KJ 1^21 1^22 1^23 1^: ^33 «32 ^ 3 1 - K J
KJ
p31
Zai3 Iai2 I^ii-^^O
< )
1^32
KJ
Ml
(c)
''42
L I^3j ^ 33 ^32 ^ 3 1 - * 0
r'n J^-); o ^ ^-K. 2j ^^23 ^22
^ ^21 - H D
K j
Hr>—
T«23 T«33 K >
< )
1^33
KJ ^-43
]C
KJ
lb^•b^^ fc,2 ^11-^0 «11
HO «21
HO—HO ^31
I«i,
"-44
(d)
Figure 2.1.6: (a) The DG for matrix multiplication (b) projection in the i direction, (c) projection in the k direction, (d) projection in the j direction. Example 13 Consider the multiplication of two 3 x 3 matrices A and B given in Example 1. We encode the matrices using row and column checksums which remain invariant during the multiplication. Matrix A is encoded as A' using column checksums and matrix B is encoded using row checksums as matrix B\ The resultant matrix C : {cij) is a 4 x 4 matrix which has both row and column checksum property. The DG for the matrix multiplication is given in Figure 2.1.6(a). The DG is projected along the i, k and j dimensions and the array of processors obtained are given in Figure 2.1.6(b), (c) and (d), respectively. In the array obtained by projecting the DG in the k direction, each
121 processor computes only one output data element. Therefore, the effect of a fault in a processor is limited to only one data element and it is easy to catch the error. The arrays obtained by projecting the DG in the i, j directions, though better in terms of hardware, have much worse fault tolerance capabilities than the array in Figure 2.1.6(c). Each processor computes more than one data element and, therefore, the chances of check invalidation increase even for the simple case of a single fault. •
2.1.6
Conclusions
This chapter gave a brief review of several aspects of A B F T systems. It first presented an overview of the models used for representing A B F T systems. Various design techniques which use these models were discussed and compared. Methods were given to assess the fault tolerance capabilities of an A B F T system. Two simple diagnosis algorithms were presented. Finally, another approach to designing A B F T systems, called synthesis for fault tolerance, was briefly explained.
References [1] K.H. Huang and J.A. Abraham, ^^Algorithm-based fault tolerance for matrix operationSj^^ IEEE Trans. Com,put.^ vol. C-33, no. 6, pp. 518-528, June 1984. [2] J.Y. Jou and J.A. Abraham, ^^Fault tolerant m^atrix arithmetic and signalprocessing on highly concurrent computing structures^^'' Proc. lEEE^ vol. 74, pp. 732-741, May 1986. [3] J.A. Abraham et al.^ ^^Fault tolerance techniques for systolic Computer, pp. 65-74, July 1987.
arrays,^^ IEEE
[4] B. Vinnakota and N.K. Jha, ^^Diagnosability and diagnosis of algorithmsbased fault tolerant systems,^^ IEEE Trans. Com,put., vol. 42, no. 8, pp. 924-937, Aug. 1993. [5] B. Vinnakota and N.K. Jha, "^ dependence graph-based approach to the design of algorithm,-based fault tolerant system,s,^^ in Proc. Int. Symp. FaultTolerant Comput., Newcastle-upon-Tyne, pp. 122-129, June 1990.
122 [6] B. Vinnakota and N.K. Jha, ^^Design of multiprocessor systems for concurrent error detection and fault diagnosis^^^ in Proc. Int. Symp. Fault-Tolerant Comput.^ Montreal, pp. 504-511, June 1991. [7] B. Vinnakota, ^^Analysis, design and synthesis of algorithm,-based fault tolerant systemsj^^ Ph.D. Thesis, Dept. of Electrical Engg., Princeton University, Oct. 1991. [8] R. Sitaraman and N.K. Jha, ^^Optimal design of checks for error detection and location in fault tolerant multiprocessor systems,^^ IEEE Trans. Comput., vol. 42, no. 7, pp. 780-793, July 1993. [9] S. Yajnik and N.K. Jha, ^^Design of algorithm-based fault tolerant system's with in-system, checks,^^ in Proc. Int. Conf. Parallel Proc, vol. 1, St. Charles, IL, Aug. 1993. 10] S. Yajnik and N.K. Jha, ^^Analysis and randomized design of algorithmbased fault tolerant m.ultiprocessor system,s under the extended graphtheoretic model^''^ in Proc. ISCA Int. Conf. Parallel Dist. System,s, Louisville, KY, Oct. 1993. 11] S. Yajnik and N.K. Jha. ^^Graceful degradation in algorithm-based fault tolerant systems^^^ in Proc. Int. Symp. Circuits & Systems^ London, UK, May 1994. 12] S. Srinivasan and N.K. Jha, ^^Efficient diagnosis in algorithm-based fault tolerant multiprocessor systems^^^ in Proc. Int. Conf. Computer Design, Boston, MA, pp. 592-595, Oct. 1993. 13] P. Banerjee and J.A. Abraham, ^^Bounds on algorithm,-based fault tolerance in multiple processor systems,^^ IEEE Trans. Comput., vol. C-35, no. 4, pp. 296-306, Apr. 1986. 14] P. Banerjee and J.A. Abraham, ^^Concurrent fault diagnosis in m,ultiple processor systems,^^ in Proc. Int. Sym,p. Fault-Tolerant Comput., Vienna, pp. 298-303, June 1986. 15] P. Banerjee, "-A theory for algorithm-based fault tolerance in array processor systems,^^ Ph.D. Thesis, Coordinated Science Laboratory, Univ. of Illinois, Urbana, Dec. 1984. 16] P. Banerjee et a/., ^^Algorithm,-based fault tolerance on a hypercube m,ultiprocessor,'' IEEE Trans. Comput., vol. 39, pp. 1132-1145, Sept. 1990. 17] V.S.S. Nair and J. A. Abraham, "^ m,odel for the analysis of fault tolerant signal processing architectures,^^ in Proc. Int. Tech. Symp. SPIE, San Diego, pp. 246-257, Aug. 1988. 18] V.S.S. Nair and J.A. Abraham, ^^General linear codes for fault-tolerant m,atrix operations on processor arrays,^^ in Proc. Int. Sym,p. Fault-Tolerant Comput., Tokyo, pp. 180-185, June 1988.
123 [19] V.S.S. Nair and J.A. Abraham, " ^ model for the analysis, design and comparison of fauH-iolerant WSI architectures,^^ in Proc. Workshop Wafer Scale Integration, Como, Italy, June 1989. [20] V.S.S. Nair and J. A. Abraham, ^^Hierarchical design and analysis of faulttolerant multiprocessor systems using concurrent error detection,^^ in Proc. Int. Symp. Fault-Tolerant Comput., Newcastle-upon-Tyne, pp. 130-137, June 1990. [21] V.S.S. Nair, ^^Analysis and design of algorithm-based fault tolerant system^s,^^ Ph.D. Thesis, Coordinated Science Laboratory, Univ. of Illinois, Urbana, Aug. 1990. [22] A.L.N. Reddy and P. Banerjee, ^^Algorithm,-based fault tolerance for signal processing applications,^^ IEEE Trans. Comput., vol. 39, pp. 1304-1308, Oct. 1990. [23] V. Balasubramaniam and P. Banerjee, ^^Algorithm-based fault tolerance for signal processing applications on a hypercube multiprocesor,^^ in Proc. 10th Real-time Systems Symp., Santa Monica, CA, pp. 134-143, 1989. [24] V. Balasubramaniam and P. Banerjee, ^^Trade-offs in design of efficient algorithm,-based error detection schem,es for hypercube m,ultiprocessors,^^ IEEE Trans. Software Engg., vol. 16, pp. 183-196, Feb. 1990. [25] V. Balasubramaniam and P. Banerjee, ^^Com,piler assisted synthesis of algorithm-based checking in multiprocessors^^ IEEE Trans. Com,put., vol. 39, no. 4, pp. 436-446, Apr. 1990. [26] D. Gu, D.J. Rosenkrantz and S.S. Ravi, ^^Design and analysis of test schem>es for algorithm-based fault tolerance,^^ in Proc. Int. Symp. FaultTolerant Comput., Newcastle-upon-Tyne, pp. 106-113, June 1990. [27] D.J. Rosenkrantz and S.S. Ravi, ^^Im,proved bounds on algorithm-based fault tolerance,^^ in Proc. Annual Allerton Conf. Comm., Cont. and Comput., Allerton, IL, pp. 388-397, Sept. 1988. [28] D.M. Blough and A. Pelc, ^^Alm,ost certain fault diagnosis through algorithm-based fault tolerance,^^ Tech. Rep. ECE-92-09, Dept. of Electrical and Computer Engg., Univ. of California, Irvine. [29] K.H. Huang, ^^Fault tolerant algorithms for multiple processor systems,^^ Ph.D. Thesis, Coordinated Science Laboratory, Univ. of Illinois, Urbana, Nov. 1983. [30] Y.H. Choi and M. Malek, "A fault tolerant FFT processor,'' IEEE Trans. Comput., vol. 37, no. 5, pp. 617-621, May 1988. [31] J.Y. Jou and J.A. Abraham, ''Fault-tolerant FFT networks,'' IEEE Trans. Com,put., vol. 37, no. 5, pp. 548-561, May 1988.
124 [32] F.T. Luk and H. Park, ^^Fault-tolerant matrix triangularization on systolic arrays'' IEEE Trans. Comput, vol. 37, no. 11, pp. 1434-1438, Nov. 1988. [33] F.T. Luk and H. Park, " ^ n analysis of algorithm-based fault tolerance techniques^'' in Proc. SPIE Adv. Alg. Arch. Signal Proc.^ vol. 696, pp. 222228, Aug. 1986. [34] Y.H. Choi and M. Malek, " ^ fault-tolerant systolic sorter," IEEE Trans. Comput., vol. 37, no. 5, pp. 621-624, May 1988. [35] C.J. Anfinson and F.T. Luk, "-4 linear algebraic model of algorithm,-based fault tolerance," IEEE Trans. Comput., vol. 37, no. 12, pp. 1599-1604, Dec. 1988. [36] S.Y. Kung, VLSI Array Processors, Prentice-Hall, Engelwood Cliffs, NJ, 1988.
SECTION 2.2
Fault-Tolerance and Efficiency in Massively Parallel Algorithms Paris C. Kanellakis^ and Alex A. Shvartsman^
Abstract We present an overview of massively parallel deterministic algorithms which combine high fault-tolerance and efficiency. This desirable combination (called robustness here) is nontrivial, since increasing efficiency implies removing red u n d a n c y whereas increasing fault-tolerance requires adding r e d u n d a n c y to c o m p u t a t i o n s . We study a s p e c t r u m of algorithmic models for which significant robustness is achievable, from static fault, synchronous c o m p u t a t i o n to d y n a m i c fault, asynchronous c o m p u t a t i o n . In addition to fail-stop processor models, we examine a n d deal with arbitrarily initialized memory a n d restricted memory access concurrency. We survey the deterministic upper b o u n d s for the basic Write-All primitive, the lower b o u n d s on its efficiency, and we identify some of the key open questions. We also generalize the robust c o m p u t i n g of functions to relations; this new approach can model a p p r o x i m a t e c o m p u t a t i o n s . We show how to compute a p p r o x i m a t e Write-All optimally. Finally, we synthesize the state-of-the-art in a complexity classification, which extends with fault-tolerance the traditional classification of efficient parallel algorithms.
2.2.1
Introduction
A basic problem of massively parallel c o m p u t i n g is t h a t the unreliability of inexpensive processors a n d their interconnection m a y eliminate any potential efficiency advantage of parallelism. Our research is an investigation of fault models and parallel c o m p u t a t i o n models under which it is possible to achieve algorithmic efficiency (i.e., speed-ups close to linear in the n u m b e r of processors) despite the presence of faults. We would like to note t h a t these models can also ^ Computer Science Department, Brown University, PO Box 1910, Providence, RI 02912, USA. Electronic mail:
[email protected]. This research was supported by ONR grant N00014-91-J-1613. ^Digital Equipm.ent Corporation, Digital Consulting Technology Office, 30 Porter Road, Littleton, MA OI46O, USA. Electronic mail:
[email protected].
126 be used to explore common properties of a broad spectrum of fault-free models, from synchronous parallel to asynchronous distributed computing. Here, our presentation focuses on deterministic algorithms and complexity, as opposed to algorithms that use randomization. There is an intuitive trade-off between reliability and efficiency because reliability usually requires introducing redundancy in the computation in order to detect errors and reassign resources, whereas gaining efficiency by massively parallel computing requires removing redundancy from the computation to fully utilize each processor. Thus, even allowing for some abstraction in the model of parallel computation, it is not obvious that there are any non-trivial fault models that allow near-linear speed-ups. So it was somewhat surprising when in [17] we demonstrated that it is possible to combine efficiency and fault-tolerance for many basic algorithms expressed as concurrent-read concurrent-write parallel (CRCW) random access machines (PRAMS [14]). The [17] fault model allows any pattern of dynamic fail-stop no restart processor errors, as long as one processor remains alive. The fault model was applied to all CRCW PRAMs in [23, 40]. It was extended in [18] to include processor restarts, and in [42] to include arbitrary static m,em,ory faults, i.e., arbitrary memory initialization, and in [16] to include restricted memory access patterns through controlled memory access. Concurrency of reads and writes is an essential feature that accounts for the necessary redundancy so it can be restricted but not eliminated - see [16, 17] for an in-depth discussion of this issue. Also, as shown in [17], it suffices to consider COMMON CRCW PRAMS (all concurrent writes are identical) in which the atomically written words need only contain a constant number of bits. The work we survey makes three key assumptions. Namely that: 1. Failure-inducing adversaries are worst-case for each model and algorithms for coping with them are deterministic. 2. Processors can read and write memory concurrently - except that initial faults can be handled without memory access concurrency. 3. Processor faults do not affect memory - except that initial memory can be contaminated. A central algorithmic primitive in our work is the Write-All operation [17]. Iterated Write-All forms the basis for the algorithm simulation techniques of [23, 40] and for the memory initialization of [42]. Therefore, improved WriteAll solutions lead to improved simulations and memory clearing techniques. The Write-All problem is: using P processors write Is into all locations of an array of size N, where P < N. When P = N this operation captures the
127 c o m p u t a t i o n a l progress t h a t can be naturally accomplished in one time unit by a PRAM. We say that Write-All completes at the global clock tick at which all the processors that have not fail-stopped share the knowledge that I's have been written into all N array locations. Requiring completion of a Write-All algorithm is critical if one wishes to iterate it, as pointed out in [23] which uses a certification bit to separate the various iterations of (Certified) Write-AIL Note t h a t t h e Write-All completes when all processors halt in all algorithms presented here. Under d y n a m i c failures, efficient deterministic solutions to Write-All, i.e., increasing the fault-free 0{N) work by small polylog(iV) factors, are nonobvious. T h e first such solution was algorithm W of [17] which has (to date) the best worst-case work b o u n d 0(N -f- P log^ iV/log log iV) for 1 < P < iV. This b o u n d was first shown in [22] for a different version of the algorithm a n d in [29] the basic argument was a d a p t e d to algorithm W. Let us now describe the contents of this survey, with some pointers to the literature, as well as our new contributions. In Section 2.2.2 we present a synthesis of parallel c o m p u t a t i o n a n d fault models. This synthesis is new and includes most of the models proposed to d a t e . It links the work on fail-stop norestart errors, to fail-stop errors with restarts (both detectable a n d undetectable restarts). T h e detectable restart case has been examined, using a slightly formalism in [8, 18]. T h e undetectable restart case is equivalent to general general model of asynchrony t h a t has received a fair a m o u n t of in the literature. An elegant deterministic solution for Write-All in appeared in [3]. T h e proof in [3] is existential, because it uses a argument. It has recently been m a d e constructive in [33].
different the most attention this case counting
For some i m p o r t a n t early work on asynchronous PRAMs we refer to [9, 10, 15, 22, 23, 30, 32, 34]. In the last three years, randomized asynchronous c o m p u t a tion has been examined in depth in [4, 5, 21]. These analyses involve randomness in a central way. T h e y are mostly a b o u t off-line or oblivious adversaries, which cause faults during the c o m p u t a t i o n b u t pick the times of these faults before the c o m p u t a t i o n . Although, we will not survey this interesting subject here we would like to point-out t h a t one very promising direction involves combining techniques of randomized asynchronous c o m p u t a t i o n with randomized inform a t i o n dispersal [36]. T h e work on fault-tolerant a n d efficient parallel shared memory models has also been applied to distributed message passing models; for example see [1, 11, 12]. In Section 2.2.3 we examine a n array of algorithms for the Write-All problem. These employ a variety of deterministic techniques a n d are extensible to
128 the c o m p u t a t i o n of other functions (see Section 2.2.4). In particular, in Section 2.2.4, we provide new b o u n d s for fault-tolerant a n d efficient c o m p u t a t i o n of parallel prefixes. In Section 2.2.5 we introduce the problem of a p p r o x i m a t e Write-All by computing relations instead of functions. One new contribution t h a t we make is to solve a p p r o x i m a t e Write-All optimally. In Section 2.2.6 we survey the state-of-the-art in lower b o u n d s . In Section 2.2.7 we present a new complexity classification for fault-tolerant algorithms. We close with a discussion of randomized vs deterministic techniques for fault-tolerant a n d efficient parallel c o m p u t a t i o n (see Section 2.2.8).
2.2.2
Fault-tolerant parallel computation models
In the first subsection we detail a hierarchy of fail-stop models of parallel comp u t a t i o n . We then explain the cost measures of available processor steps a n d overhead ratio, which we use to characterize robust algorithms. T h e final three subsections contain comments on variations of the processor, memory, a n d network interconnect parts of our models. 2.2.2.1
Fail-Stop P R A M s
T h e parallel r a n d o m access machine ( P R A M ) of Fortune and Wyllie [14] combines the simplicity of a RAM with the power of parallelism, a n d a wealth of efficient algorithms exist for it; see surveys [13, 20] for the rationale behind this model and the fundamental algorithms. We build our models of fail-stop P R A M S as extensions of the PRAM model. 1. There are Q shared memoTy cells, a n d the input of size iV < Q is stored in the first N cells. Except for the cells holding the input, all other m e m o r y is cleared, i.e., contains zeroes. Each memory cell can store 0 ( l o g i V ) bits. All processors can access shared memory. For convenience we assume they "know" the input size N, i.e., the \ogN bits describing it can be p a r t of their finite state control. For convenience we assume t h a t each processor also has a constant size private memory, t h a t only it can access. 2. There are P < N initial processors with unique identifiers (piDs) in the range 1 , . . . , P . Each processor "knows" its PID and the value of P , i.e., these can be p a r t of its finite s t a t e control. 3. T h e processors t h a t are d a r d PRAM model [14]. an observer outside the event, the processors do
active all execute synchronously as in the stanAlthough processors proceed in synchrony a n d PRAM can associate a "global t i m e " with every not have access to "global t i m e " , i.e., processors
129 can try to keep local clocks by counting their steps and communicating through shared memory but the PRAM does not provide a "global clock". 4. Processors stop without affecting memory. They may also restart, depending on the power of a fault-inducing adversary. In the study of fail-stop PRAMs, we consider four main types of failureinducing adversaries. These form a hierarchy, based on their power. Note that, each adversary is more powerful than the preceding ones and that the last case can be used to simulate fully asynchronous processors [3]. Initial faults: adversary causes processor failures only prior to the start of the computation. Fail-stop failures: adversary causes stop failures of the processors during the computation; there are no restarts. Fail-stop failures, detectable restarts: adversary causes stop failures; subsequently to a failure, the adversary might restart a processor and a restarted processor "knows" of the restart. Fail-stop failures, undetectable restarts: adversary causes stop failures and restarts; a restarted processor does not necessarily "know" of the restart. Except for the initial failures case, the adversaries are dynamic. A major characteristic of these adversary models is that they are worst-case. These have full information about the structure and the dynamic behavior of the algorithms whose execution they interfere with, while being completely unknown to the algorithms. Remark on (un)detectable restarts: One way of realizing detectable restarts is by modifying the finite state control of the PRAM. Each instruction can have two parts, a green and a red part. The green part gets executed under normal conditions. If a processor fails then all memory remains intact, but in the subsequent restart the next instruction red part is executed instead of the green part. For example, the model used in [8, 18] can be realized this way, instead of using "update cycles". The undetectable restarts adversary can also be realized in a similar way by making the algorithm weaker. For undetectable restarts algorithms have to have identical red and green parts. For example, the fully asynchronous model of [3] can be realized this way. • We formalize failures as follows. A failure pattern F is syntactically defined as a set of triples where tag is either failure indicating processor failure, or restart indicating a processor restart, PID is the processor identifier, and t is the time indicating when the processor stops or restarts. This time
130
\piDi
\piDp
^ N M B I N
E T W O
N G
R K
MEM I MEM2
MEMQ\
Figure 2.2.1: An architecture for a fail-stop multiprocessor. is a global time, that could be assigned by an observer (or adversary) outside the machine. The size of the failure pattern F is defined as the cardinality | F | , where \F\< M for some parameter M. The abstract model that we are studying can be realized in the architecture in Fig. 2.2.1. This architecture is more abstract than, e.g., an implementation in terms of hypercubes, but it is simpler to program in. Moreover, various faulttolerant technologies can contribute towards concrete realizations of its components. There are P fail-stop processors [38]. There are Q shared memory cells. These semiconductor memories can be manufactured with built-in fault tolerance using replication and coding techniques [37]. Processors and memory are interconnected via a synchronous network [39]). A combining interconnection network well suited for implementing synchronous concurrent reads and writes is in [24] and can be made more reliable by employing redundancy [2]. In this architecture, when the underlying hardware components are subject to failures within their design parameters, the algorithms we develop work correctly, and within the specified complexity bounds. 2.2.2.2
Measures of Efficiency
We use a generalization of the standard Parallel-time x Processors product to measure work of an algorithm when the number of processors performing work fluctuates due to failures or delays [17, 18]. In the measure we account for the available processor steps and we do not charge for time steps during which a processor was unavailable due to a failure. Definition 2.2.1 Consider a parallel computation with P initial processors that terminates in time r after completing its task on some input data / of size N and in the presence of the fail-stop error pattern F. If Pi{I^ F) < P is the number of processors completing an instruction at step z, then we define 5(7, F, P) as: 5(7, F, P) = E L , Pi{I, F). •
131 Definition 2.2.2 A P-processor PRAM algorithm on any input data 7 of size |/| = N and in the presence of any pattern F of failures of size \F\ < M uses available processor steps S = SN^M.P = niax/^ir{5(7, F, P ) } . • The available steps measure 5 is used in turn to define the notion of algorithm robustness that combines fault tolerance and efficiency: Definition 2.2.3 Let T(N) be the best sequential (RAM) time bound known for iV-size instances of a problem. We say that a parallel algorithm for this problem is a robust parallel algorithm if: for any input / of size N and for any number of initial processors P {1 < P < N) and for any failure pattern F of size at most M with at least one surviving processor (M < N for fail-stop model), the algorithm completes its task with S = SN^M.P < c T(N)\og^ iV, for fixed c, c'. • For arbitrary failures and restarts, the completed work measure 5 depends on the size N of the input / , the number of processors P , and the size of the failure pattern F. The ultimate performance goal is to perform the required computation at a work cost as close as possible to the work performed by the best sequential algorithm known. Unfortunately, this goal is not attainable when an adversary succeeds in causing too many processor failures during a computation. E x a m p l e : Consider a Write-All solution, where it takes a processor one instruction to recover from a failure. If an adversary has a failure pattern F with | P | - n(iV^+^) for e> 0, then work will be ll(iV^+^) regardless of how eflScient the algorithm is otherwise. This illustrates the need for a measure of efficiency that is sensitive to both the size of the input iV, and the size of the failure pattern | P | < M. We thus also introduce the overhead ratio a that amortizes work of the essential work and failures: Definition 2.2.4 A P-processor PRAM algorithm on any input data / of size |/| = iV and in the presence of any pattern F of failures and restarts of size | P | < M has overhead ratio a = (TN,M,P — maxj,F I \IMF\ f *
^
When M = 0{P) as in the case of the stop failures without restarts, S properly describes the algorithm efficiency, and a = 0( ^)^'^). When F can be large relative to N and P with restarts enabled, a better reflects the efficiency of fault-tolerant algorithms. We can generalize the definition of a in Def. 2.2.4 in terms of the ratio ^Ll •'• J , where T{I) is the time complexity of the best known sequential solution for a particular problem.
132 2.2.2.3
Processor issues: survivability
We have chosen to consider only the failure models where the processors do not write any erroneous or maliciously incorrect values to shared memory. While malicious processor behavior is often considered in conjunction with message passing systems, it makes less sense to consider malicious behavior in tightly coupled shared memory systems. This is because even a single faulty processor has the potential of invalidating the results of a computation in unit time, and because in a parallel system all processors are normally "trusted" agents, and so the issues of security are not applicable. The fail-stop model with undetectable restarts and dynamic adversaries is the most general fault model we deal with. It can be viewed as a model of parallel computation with arbitrary asynchrony. Remark on stronger survivability assumption: The default assumption we make is that throughout the computation one processor is fault-free. This assumption can be made stronger, i.e., a constant fraction of the processors are fault-free. We always list the stronger assumption explicitly when used (e.g., in the complexity classification). • Remark on weaker survivability assumption and restarts: For the models with restarts one can use the weaker survivability assumption that at each global clock tick one processor step executes. In [18] this was stated using "update cycles", but it can be stated using our green-red instruction implementation - remark on (un)detectable restarts. • 2.2.2.4
Memory issues: words vs bits and initialization
In our models we assume that logiV-bit word parallel writes are performed atomically in unit time. The algorithms in such models can be modified so that this restriction is relaxed. The sufficient definition of atomicity is: (1) logiV-size words are written using log N bit write cycles, and (2) the adversary can cause arbitrary fail-stop errors either before or after the single hit write cycle of the PRAM, but not during the bit write cycle. The algorithms that assume word atomicity can be mechanically compiled into algorithms that assume only the bit atomicity as stated above. A much more important assumption in many Write-All solutions was the initial state of additional auxiliary memory used (typically of ^{P) size). The basic assumption has been that: The n ( P ) auxiliary shared memory is cleared or initialized to som,e known value.
133 While this is consistent with definitions of PRAM such as [14], it is nevertheless a requirement t h a t fault-tolerant systems ought to be able to do without. Interestingly there is an efficient deterministic procedure t h a t solves the Write-All problem even when the shared memory is contaminated, i.e., contains a r b i t r a r y values. 2.2.2.5
Interconnect issues: concurrency vs redundancy
T h e choice of CRCW (concurrent read, concurrent write) model used here is justified because of a lower bound [17] t h a t shows t h a t the C R E W (concurrent read, exclusive write) model does not a d m i t fault-tolerant efficient algorithms. However we still would like control memory access concurrency. We define measures t h a t gauge the concurrent m e m o r y accesses of a c o m p u t a t i o n . D e f i n i t i o n 2 . 2 . 5 Consider a parallel c o m p u t a t i o n with P initial processors t h a t terminates in time r after completing its task on some input d a t a / of size N in the presence of fail-stop error p a t t e r n F. If at time ^ (1 < i < r ) , P/^ processors perform reads from N^ shared memory locations a n d P^^ processors perform writes to N^^ locations, then we define: (i) the read concurrency (ii) the wmte concurrency
p as: p — PI^F.P
— 5^1=1 (-^i^ "~ -^/^)i ^^^
u a,s: CJ = UJI^F.P = 531=1 (-^t^ ~ -^i^)-
'-'
For a single read from (write to) a particular m e m o r y location, the read (write) concurrency p (CJ) for t h a t location is simply the n u m b e r of readers (writers) minus one. For example, if only one processor reads from (writes to) a location, t h e n p (u) is 0, i.e., no concurrency is involved. Also note t h a t the concurrency measures p and u are cumulative over a c o m p u t a t i o n . For the algorithms in the EREW model, p = u = 0, while for the CREW model, cj = 0. T h u s our measures capture one of the key distinctions a m o n g the EREW, CREW and CRCW memory access disciplines.
2.2.3
Robust parallel assignment and Write-All
2.2.3.1
W r i t e - A l l a n d initial faults
We first consider the weak model of initial (static) faults in which failures can only occur prior to the start of an algorithm. We assume t h a t the size of the Write-All instances is N a n d t h a t we have P processors, P' < P oi which are alive at the beginning of the algorithm. Our EREW a l g o r i t h m E (Fig. 2.2.2) consists of phases E l a n d E2. In phase E l , processors enumerate themselves a n d
134 01 forall processors P I D = 1 . . P parbegin 02 Phase E l : Use non-oblivious parallel prefix to compute rankpjD and P' 03 Phase E2: Set x[{rankpiD — 1) * p - . . . {rankpin * p - ) — 1] to 1 04 parend Figure 2.2.2: A high level view of a l g o r i t h m E. compute the t o t a l number of live processors. T h e details of this non-oblivious counting are in [16]. In phase E2, the processors p a r t i t i o n the i n p u t array so t h a t each processor is responsible for setting to 1 all the entries in its p a r t i t i o n . T h e o r e m 2 . 2 . 1 T h e Write-All problem with initial processor a n d memory faults can be solved in place with S — 0{N + P' log P) on an E R E W P R A M , where 1 < P < iV a n d P — P ' is the number of initial faults. W i t h the result of [7] it can be shown t h a t this algorithm is optimal, without memory access concurrency. 2.2.3.2
D y n a m i c faults and algorithm W
A more sophisticated approach is necessary to obtain an efficient parallel algor i t h m when the failures are dynamically determined by an on-line adversary. Algorithm W of [17] is an efficient fail-stop Write-All solution (Fig. 2.2.3). It uses full binary trees for processor counting, processor allocation, a n d progress measurement. Active processors synchronously iterate t h r o u g h the following four phases: Wl:
Processor enumeration. All the processors traverse b o t t o m - u p the processor enumeration tree. A version of parallel prefix a l g o r i t h m is used resulting in an overestimate of the number of live processors.
W2:
Processor allocation. All the processors traverse the progress measurement tree top-down using a divide-and-conquer approach based on processor enumeration a n d are allocated to un-written input cells.
W3:
Work phase.
W4:
Progress measurement. All the processors traverse b o t t o m - u p the progress tree using a version of parallel prefix a n d c o m p u t e an u n d e r e s t i m a t e of the progress of the algorithm.
Processors work at the leaves reached in phase W 2 .
Algorithm W achieves optimality when parameterized using a progress tree with N/ log N leaves and log N input d a t a associated with each of its leaves. By optimality we m e a n t h a t for a range of processors the work is 0{N). A
135 01 forall processors PID=l..iV parbegin 02 Phase W3: Visit leaves based on FID to work on the input data 03 Phase W4: Traverse the progress tree bottom up to measure progress 04 while the root of the progress tree is not N do 05 Phase W l : Traverse counting tree bottom up to enumerate processors 06 Phase W2: Traverse the progress tree top down to reschedule work 07 Phase W3: Perform rescheduled work on the input data 08 Phase W4: Traverse the progress tree bottom up to measure progress 09 od 10 parend Figure 2.2.3: A high level view of algorithm W,
complete description of the algorithm can be found in [17]. Martel [29] gave a tight analysis of algorithm W, T h e o r e m 2 . 2 . 2 [17, 29] Algorithm W is a robust parallel Write-All algorithm with 5 = 0{N + ^ l o g ^ i V / l o g l o g i V ) , where N is the input array size a n d the initial number of processors P is between 1 and N.
Note t h a t the above b o u n d is tight for algorithm W . This upper b o u n d was first shown in [22] for a different algorithm. T h e d a t a structuring technique [22] might lead to even better b o u n d s for Write-All.
2.2.3.3
D y n a m i c faults, detected restarts, and algorithm V
Algorithm W has efficient work when subjected to a r b i t r a r y failure p a t t e r n s without restarts a n d it can be extended to handle restarts. However, since accurate processor enumeration is impossible if processors can be restarted at any time, the work of the algorithm becomes inefficient even for some simple adversaries. On the other hand, the second phase of a l g o r i t h m W does implement efficient top-down divide-and-conquer processor assignment in O(logiV) time when p e r m a n e n t processor F I D s are used. Therefore we produce a modified version of a l g o r i t h m W, t h a t we call V. To avoid a restatement of the details, the reader is referred to [18]. V uses the optimized algorithm W d a t a structures for progress estimation a n d processor allocation. T h e processors iterate t h r o u g h the following three phases based on the phases W 2 , W 3 a n d W 4 of a l g o r i t h m W:
136 VI: Processors are allocated as in the phase W2, but using the permanent PiDs. This assures load balancing in O(logiV) time. V2: Processors perform work, as in the phase W3, at the leaves they reached in phase VI (there are logJV array elements per leaf). V3: Processors continue from the phase V2 progress tree leaves and update the progress tree bottom up as in phase W4 in O(logiV) time. The model assumes re-synchronization on the instruction level, and a wraparound counter based on the PRAM clock implements synchronization with respect to the phases after detected failures [18]. The work and the overhead ratio of the algorithm are as follows: Theorem 2.2.3 [18] Algorithm V using P < N processors subject to an arbitrary failure and restart pattern F of size M has the work S — 0(iV-|-P log N-\MlogiV), and its overhead ratio is: a — 0(log N). Algorithm V achieves optimality for a non-trivial set of parameters: Corollary 2.2 A Algorithm V with P < N/\o^ N processors subject to an arbitrary failure and restart pattern of size M < N/\ogN has 5 = 0{N). One problem with the above of restarts and a large amount algorithm X of the next section [3] to provide better bounds on 2.2.3.4
approach is that there could be a large number of work. Algorithm V can be combined with or with the asymptotically better algorithm of work.
Dynamic faults, undetected restarts, and algorithm X
When the failures cannot be detected, it is still possible to achieve sub-quadratic upper bound for any dynamic failure/restart pattern. We present Write-All algorithm X with 5 = 0{N • P^""^^) = N • P^'^^. This simple algorithm can be improved to 5 = 0{N • P^) using the method in [3]. We present X for its simplicity and in the next section a (possible) deterministic version of [3]. Algorithm X utilizes a progress tree of size N that is traversed by the processors independently, not in synchronized phases. This reflects the local nature of the processor assignment as opposed to the global assignments used in algorithms V and W. Each processor searches for work in the smallest subtree that has work that needs to be done. It performs the work, and moves to the next subtree.
137 01 02 03 04 05 06 07 08 09 10 11 12 13
forall processors PID=O..P — 1 parbegin Perform initial processor assignment to the leaves of the progress tree while there is still work left in the tree do if subtree rooted at current node u is done t h e n move one level up elseif ^ is a leaf t h e n perform the work at the leaf elseif u is an interior tree node t h e n Let UL and UR be the left and right children of u respectively if the subtrees rooted at UL and UR are done t h e n update u elseif only one is done t h e n go to the one that is not done else move to UL or UR according to FID bit values fifi od parend Figure 2.2.4: A high level view of the algorithm X.
T h e a l g o r i t h m is given in Fig. 2.2A. Initially the P processors are assigned to the leaves of the progress tree (line 02). T h e loop (lines 03-12) consists of a multi-way decision (lines 04-11). If the current node u is marked done, the processor moves u p the tree (line 04). If the processor is at a leaf, it performs work (line 05). If the current node is an u n m a r k e d interior node a n d b o t h of its subtrees are done, the interior node is marked by changing its value from 0 to 1 (line 08). If a single subtree is not done, the processor moves down appropriately (line 09). For the final case (line 10), the processors move down when neither child is done. Here the processor PID is used at d e p t h h of the tree node: based on the value of the h^^ most significant bit of the binary representation of PID, bit 0 will send the processor to the left, a n d bit 1 to the right. T h e performance of algorithm X is characterized as follows: T h e o r e m 2 . 2 . 5 Algorithm X with P processors solves the Write-All problem of size N {P < N) in the fail-stop restartable model with work 5 = 0{N'P^^^^). In addition, there is an adversary t h a t forces algorithm X to perform S = ^ ( i \ r . p i o g f ) work. T h e a l g o r i t h m views undetected restarts as delays, a n d it can be used in the asynchronous model where it has the same work [8]. Algorithm X could also be useful for the case without restarts, even t h o u g h its worst-case performance without restarts is no better t h a n a l g o r i t h m W. O p e n P r o b l e m : A major open problem for the model with undetectable restarts is whether there is robust Write-All solution, i.e., where the work is Npolylog(N). Also, whether there is a solution with a = polylog(N).
138 01 02 03 04 05 06 07 09 10
forall processors PID = 1..VN parbegin Divide the N array elements into VN work groups of VN elements Each processor obtains a private permutation T^pj^ of {1, 2 , . . . , y/N} foTi=l,.VNdo if 7rp^^[t]th group is not finished t h e n perform sequential work on the vN elements of the group and mark the group as finished fi od parend Figure 2.2.5: A high level view of the a l g o r i t h m Y.
2.2.3.5
D y n a m i c faults, undetected restarts, and algorithm Y
A family of randomized Write-All algorithms was presented by Anderson a n d Woll [3]. T h e m a i n technique in these algorithms is abstracted in Fig. 2.2.5. T h e basic a l g o r i t h m in [3] is obtained by r a n d o m l y choosing the p e r m u t a t i o n in line 03. In this case the expected work of the a l g o r i t h m is O(iVlogiV), for P — y/W (assume iV is a square). We propose the following way of determinizing the algorithm (see [19]): Given P = \N^ we choose the smallest prime m such t h a t P < m. Primes are sufficiently dense, so t h a t there is at least one prime between P a n d 2 P , so t h a t the complexity of the algorithms is not distorted when P is not a prime. We then construct the multiplication table for the n u m b e r s 1 , 2 , . . ,m— 1 m o d u l o m. Each row of this table is a p e r m u t a t i o n a n d this structure is a group. Processor with P I D i uses the i t h p e r m u t a t i o n as its schedule. This table need not be pre-computed, as any item can be c o m p u t e d directly by any processor with the knowledge of its P I D , a n d the number of work elements w it has processed thus far as {PID • w) m o d m. C o n j e c t u r e : We conjecture t h a t the worst case work of this deterministic algorithm is no worse t h a n the expected work of the randomized algorithm. Experimental analysis supports the conjecture. Formal analysis can be reduced to the open problem below t h a t contains an interesting group-theoretic aspect of the multi-processor scheduling problem [41]. In order to show t h a t the worst case work of y is 0 ( i V log iV), it is sufficient to show t h a t : Given a prime TTI, consider the group G = ( { l , 2 , . . . , m — !},# (mod m)). The multiplication table for G, when the rows of the table are interpreted as permutations o f { l , . . . , ? n — 1}, i s a group K of order m — 1 (a subgroup of all permutations). Show that, for each left coset of K (with respect to all permutations) the sum of the number of left-to-right maxima of all elements of the coset is 0 ( m log TTI).
139 1 01 forall processors PID=1..P parbegin P processors t dear N locations 02 Clear the initial block of No = Go elements sequentially using P processors 03 i := 0 Iteration counter 04 while Ni < N do 05 Use Write-All solution with data structures of size Ni and Gi^i at the leaves to clear memory of size Ni^i =Ni'Gi^i ; i := i + 06 07 od 08 parend
elements
1
Figure 2.2.6: A high level view of algorithm Z. 2.2.3.6
Bootstrapping and algorithm Z
The Write-All algorithms and simulations (e.g., [17, 22, 23, 40]) or the algorithms that can serve as Write-All solutions (e.g., the algorithms in [9, 32]) invariably assume that a linear portion of shared memory is either cleared or is initialized to known values. Starting with a non-contaminated portion of memory, these algorithms perform their computation by "consuming" the clear memory, and concurrently or subsequently clearing segments of memory needed for future iterations. We define an efficient Write-All solution that requires no clear shared memory [42]. The solution uses a bootstrap approach: In stage 1 all P processors clear an initial segment of No locations in the auxiliary memory. In stage i the P processors clear Ni^i = Ni • Gi^i memory locations using Ni memory locations that were cleared in stage i — 1. Using algorithm W and tuning the parameters Ni and Gi we obtain a solution (algorithm Z, see Fig. 2.2.6) that for any failure pattern F {\F\ < P) has work 0{N -\- ^(lo^io N\^ ) '^^^^^^^ ^"^y initialization assumption. A similar algorithm that inverts the bootstrap procedure can be used to clear the contaminated shared memory if the output must contain only the results of the intended computation. The complexity of algorithm Z~^ is identical to the complexity of algorithm Z. For algorithm simulation and for transformed algorithms, the complexity cost is additive in both cases. 2.2.3.7
Minimizing concurrency: processor priority trees
Among the key lower bound results is the fact that no efficient fault-tolerant CREW PRAM Write-All algorithms exist [17] - if the adversary is dynamic then any P-processor solution for the Write-All problem of size N will have (deterministic) work Q{N ' P), Thus memory access concurrency is necessary to combine efficiency and fault-tolerance. However, while most known solutions
140 for the Write-All problem indeed make heavy use of concurrency, the goal of minimizing concurrent access to shared memory is attainable. We gave a Write-All algorithm in [16] in which we bound the total amount of concurrency used in terms of the number of dynam,ic processor faults of the actual run of the algorithm. When there are no faults our algorithm executes as an EREW PRAM and when there are faults the algorithm differs from EREW in the amount of concurrency proportional to the number of faults. The algorithm is based on a conservative policy: concurrent reads or writes occur only when the presence of failures can be inferred and then concurrency is allowed in proportion to the failures detected. The robust CRCW algorithm WCR/W '^^ [16] is based on algorithm W and it uses processor identifiers to construct m,ergeahle processor priority trees (PPT), which control concurrent access to memory. During the execution, the PPTs are com,pacted and w,erged to remove faulty processors and to determine when concurrent access to memory is warranted. By taking advantage of parallel slackness and by clustering the input data into groups of size logiVlogP, we obtain an algorithm that has a range of optimality and that controls its memory access concurrency: Theorem 2.2.6 Algorithm W C R / W of [16] with input clustering is a robust Write-All algorithm with 5 == 0{N-^P^^^^^^), write concurrency u < | F | , and read concurrency p < 7 | F | logiV, where 1 < P < iV. The basic algorithm can be extended to handle arbitrary initial memory contents [16]. It is also possible to reduce the maximum per step memory access concurrency by polylogarithmic factors by deploying a general pipelining technique. Finally, [16] shows that there is no robust algorithm whose total write concurrency is bounded by |JP|^ for 0 < e < 1.
2.2.4
Computing functions robustly
In this section we will work our way from the simplest to the most complicated functions with robust solutions. 2.2.4.1
Constants, booleans and Write-All
Solving a Write-All problem of size N can be viewed as computing a constant vector function. Constant scalar functions are the simplest possible functions (e.g., simpler than boolean OR and AND). At the same time, it appears
141 that Write-All problem is a more difficult (vector) task than computing scalar boolean functions such as multiple input OR and AND. In the lower bounds discussion we consider a model with memory snapshots^ i.e., processors can read and process the entire shared memory in unit time. For the snapshot model there is a sharp separation between Write-All and boolean functions. Clearly any boolean can be computed in constant time in the snapshot model, while we have a lower bound result for any Write-All solution in the snapshot model requiring work n{Nj^^). Solving a Write-All problem is no more difficult than computing any other vector function, e.g., parallel prefix. In the next subsection we also show that the best (as of this writing) Write-All solution can be used to derive a robust parallel prefix algorithm that has the same work complexity. 2.2.4.2
Parallel prefix and Write-All
Solutions for the Write-All problem can be used as building blocks for custom transformations of efficient parallel algorithms into robust algorithms [17]. Transformations are of interest because in some cases it is possible to improve on the work of oblivious simulation such as [23, 32, 40]. These improvements are most significant for fast algorithms when a full range of processors is used, i.e., when N processors are used to simulate N processors, because in this case parallel slack cannot be taken advantage of. One immediate result that improves on the available general simulations follows from the fact that algorithms V, W and X, by their definition, implement an associative operation on N values. Theorem 2.2.7 Given any associative operation © on integers, and an integer array x[l..iV], it is possible to robustly compute ©£_i x[i] using P fail-stop processors at a cost of a single application of any of the algorithms F , W or X, This saves a full log N factor for all simulations. The savings are also possible for the important prefix sums and pointer doubling algorithms. Efficient parallel algorithms and circuits for computing prefix sums were given by Ladner and Fischer in [26], where the prefix problem, is defined as follows: Given an associative operation ® on a domain X>, and x i , . . . , x^ ^ ^ j compute, for each k^ (1 < k < n) the sum ©^_i Xi. In order to compute the prefix sums of N values using N processors, at least log iV/log log iV parallel steps are required [6, 27], and the known algorithms require at least log N steps. Therefore an oblivious simulation of a known prefix algorithm will require simulating at least logiV steps. When using P = N
142 processors with algorithm W (the most efficient as of this writing Write-All solution) whose work is S^ — Q(iV^^^^^ ^ ) , the work of the simulation will be We can extend Theorem 2.2.7 to show a robust prefix algorithm whose work is the same as that of algorithm W. In the fail-stop model we have the following result that uses as the basis an iterative version of the recursive algorithm of [26]: Theorem 2.2.8 Parallel prefix for N values can be computed using N fail-stop processors using 0{N) clear memory with 5 = ^{^i^^Tw)' A similar approach was also taken by Martel et al. [30] to produce an efficient randomized transformation of the prefix algorithm. 2.2.4.3
List ranking
Another important improvement for the fail-stop case is for the pointer doubling operation that is used in many parallel algorithms. The robust algorithm is implemented using a variation of algorithm W and the standard pointer doubling algorithm. We associate each list element with a progress tree leaf. In the work phase of algorithm W we double pointers and update distances. The log N pointer doubling operations in the work phase make log N/ log log N overall iterations sufficient with each iteration performing the same work S^ as algorithm W. Theorem 2.2.9 There is a robust list ranking algorithm for the fail-stop model with S = Q(J"f^^^5^(iV, P)), where JV is the input list size and S^{N, P) is the complexity of algorithm W for the initial number of processors P : 1 < P \N, C . Dwork observed that the Q[N\ogN) lower bound [22] on solving Write-All using N processors also applies to producing a member of A4 in the presence of failures. It turns out that 0(iV log iV) work is also sufficient to compute a member of the majority relation. Let's parameterize the majority problem in terms of the approximate WriteAll problem by using a quantity e such that 0 < 5 < | , thus we would like to initialize at least (l—e)N array locations to 1. We call this problem the AWA{6), Surprisingly, algorithm W has the desired property: Theorem 2.2.15 Given any constant e such that 0 < e < ^, algorithm W solves the AWA{e) problem with 5 = 0{NlogN) using N processors. If we choose e — 1/2* {k — const) and then iterate this Write-All algorithm log log iV times, the number of unvisited leaves will be Ne^^^^^^^^"^ — iV(logiV)^^8^ = N{logN)-^ r= N/log^N, Thus we can get even closer to solving the Write-All problem: Theorem 2.2.16 For each k = const^ there is a robust AWOl(T—^) algorithm that has work 5 = 0{NlogNloglogN).
2.2.6
Lower bounds
The strongest known lower bound for Write-All was derived by Kedem, Palem, Ragunathan and Spirakis in [22]. Theorem 2.2.17 [22] Given any P-processor CRCW PRAM algorithm for the Write-All piohlem of size iV, an adversary can force fail-stop (no restart) errors that result in iV + Q{PlogN) (where P < N) steps being performed. Recently, Martel and Subramonian [31] have extended the Kedem et al. deterministic lower bound [22] to randomized algorithms against oblivious adversaries. It is open whether this lower bound applies to the static fault case. It was shown in [17] that no optimal solutions for the Write-All problem exist that use the range of processor 1 < P < N even when the processors can take instant memory snapshots^ i.e., processors can read and locally process the
147 entire shared memory at unit cost. T h e lower b o u n d below applies to fail-stop, deterministic or randomized, P R A M S a n d it is the strongest possible b o u n d under the memory snapshots assumption, i.e., there is a m a t c h i n g upper b o u n d . T h e o r e m 2 . 2 . 1 8 [17] Given any iV-processor CRCW P R A M a l g o r i t h m for the T^rzte--4//problem of size iV, an adversary can force fail-stop errors t h a t result in ^{^lo^iJ^i^) steps being performed, even if the processors can read and locally process all shared memory at unit cost. W h e n restarts are introduced, we show the following result t h a t also is the strongest possible result under the snapshot assumption [8]: T h e o r e m 2 . 2 . 1 9 Given any P-processor CRCW PRAM a l g o r i t h m t h a t solves the Write-All problem of size N (P < iV), a n adversary ( t h a t can cause a r b i t r a r y processor failures and restarts) can force the a l g o r i t h m to perform N -h fi(PlogP) work steps. T h e next result shows t h a t CRCW is necessary to achieve efficient solutions to the Write-All problem. In the absence of failures, any P-processor CREW (concurrent read exclusive write) or EREW (exclusive read exclusive write) P R A M can simulate a P-processor CRCW PRAM with only a factor of O ( l o g P ) more parallel work [20]. However a more severe difference exists between CRCW and CREW PRAMS (and thus also EREW P R A M S ) when the processors are subject to failures. T h e o r e m 2 . 2 . 2 0 Given any deterministic or randomized iV-processor CREW PRAM algorithm for the Write-All problem, the adversary can force fail-stop errors t h a t result in fl{N^) steps being performed, even if the processors can read and locally process all shared memory at unit cost. For the CREW PRAMs, Martel a n d S u b r a m o n i a n [31] show a randomized algorithm with expected work of only O(iVlogiV) for P = N,
2.2.7
A Complexity classification
2.2.1 A
Efficient p a r a l l e l c o m p u t a t i o n
Many efficient parallel algorithms can be used to show problem membership in the class A/'C (of polylog time and polynomial n u m b e r of processors [35]). T h e inverse is not necessarily true. This is because the algorithms in MC allow for polynomial inefficiency in work [25] - the algorithms are fast (polylogarithmic
148 time), but the computational agent can be large (polynomial) relative to the size of a problem [35]. A characterization of parallel algorithm efficiency that takes into account both the parallel time and the size of the computational resource is defined by Vitter and Simmons [44] and expanded on by Kruskal et al. [25]. The complexity classes in [25] are defined with respect to the time complexity T{N) of the best sequential algorithm for a problem of size N - this is analogous to the definition of robustness. Each class is characterized in terms of parallel time T{N) and, parallel work T{N) • P{N), We give these class definitions below, but instead of failure-free work, we use the overhead ratio a that for the failure-free case is simply T{N) • P{N)/T{N): Let A be a problem with sequential (RAM) time complexity T{N), A parallel algorithm that solves an iV-size instance of A using P{N) processors in r{N) time belongs to the class: ENC:
if T{N) = \og^^^\T{N))
EP:
if T{N) < T{NY
ANC:
if T{N) = log^(^)(T(iV))
AP:
if T{N) < T{Ny
SNC:
if T{N) = log^^^\T{N))
SP:
if T{N) < T{Ny
2.2.7.2
and a = 0(1).
(const e < 1) and cr = 0(1). and a =
log^^^\T{N)).
(const s < 1) and a =
\og^^^\T{N)).
and a =
T{N)^W,
(const e < 1) and a =
T{N)^(^\
Closures under failures
We now define criteria for evaluating whether algorithm transformation preserves the efficiency of the algorithms for each of the classes above. To use time complexity in comparisons, we need to introduce a measure of time for the fault-tolerant algorithms. In a fault-prone environment, a time metric is meaningful provided that a significant number of processors still are active. Here we use the worst case time provided a linear number of processors are active during the computation. This is our weak survivability assumption. Without this assumption, all one can conclude about the running time is that it is no better than the time of the best sequential algorithm, since the number of active processors might become quite small. We assuming P is a polynomial in N (note that until now we generally assumed P < N). Then l o g P = O(logiV). We now state the definition:
149 II Complexity
1
Class ENC
1 ^^ ANC AP SNC
1
^P
Time with > cP processors
0{r{N)\o^Nl\og\ogN) = o(iogO(0(T(i\r))) = o(T{Ny) = log^(i)(T(N))
=
OiTiNY)
= log^(0(T(iV'))
=
o{T{Ny)
Overhead c 0(log'^(i)iV) >0(1)
Closed T under f ? | No
>0(1)
No
= log^(0(T(iV)) = log^(0(T(i\r)) = T(i\r)^(i)
Yes Yes
= T(iV)^(i)
Yes
1
Yes
1
Table 2.2.1: Closure under the fail-stop transformation ^.
D e f i n i t i o n 2 . 2 . 6 Let Cr^w be a class with parallel time in the complexity class T and parallel work in the complexity class w. We say t h a t Cr^w is closed with respect to a fault-tolerant transformation (p if for any a l g o r i t h m A in Cr^w(1) overhead a of (t>{A) is such t h a t cr • r • P is in ly, a n d (2) when the n u m b e r of active processors at any point of the c o m p u t a t i o n is at least cP for constant c > 0, then the running time t is in r . D In the fail-stop model without restarts, given any a l g o r i t h m A, let C{A) be the fault-tolerant algorithm t h a t can be constructed as either a simulation or a transformation. Using, for example, algorithm W as the basis for transforming non-faulttolerant algorithms, we have the following: (1) T h e multiplicative overhead in work is 0 ( l o g i V ^ / l o g l o g i V ) , a n d so the worst case overhead a is 0 ( l o g i V ^ / l o g l o g i V ) = log ^^^ N a n d the worst case work of the fault-tolerant version C{A) is a • r{N) • P. (2) Algorithm W terminates in Sy,/cP = 0(log^ iV/loglogiV) time when at least cP processors are active, therefore if the parallel time of a l g o r i t h m A is T(iV), then the parallel time of execution for C{A) using at least cP active processors is 0 ( r ( i V ) log'^ JV/log log iV). T h e resulting closure properties of the classes in [25] under our fail-stop t r a n s f o r m a t i o n ' s summarized in Table 2.2.1. In the fail-stop model with detectable restarts, for any a l g o r i t h m A, let p{A) be the fault-tolerant algorithm constructed using any of our techniques. In this model we provide existential closure properties by taking a d v a n t a g e of the existential result by Anderson a n d Woll [3], who showed t h a t for every e: > 0, there exists a deterministic algorithm for P processors t h a t simulates P instructions with 0(P^"*"^) work. Given the a l g o r i t h m [3], we interleave it with algorithm V, for example, so t h a t the overhead a of the combined a l g o r i t h m
150 1 Complexity
1
Class
0{-T(N)-P'^)
ENC
> iog^(i)(T(i\r))
EP
= 0{T(Ny) > log^(0(T(i^))
ANC AP SNC \
Time with > cP processors
SP
=
0{T(Ny)
> log^(i)(T(i\^))
0{T(Ny)
Overhead a 0(log2 N)
>o{i) >o{i) = log^(i)(T(i\r)) = log^(0(T(i\r)) = T(iV)^(0
= r(iv)^(i)
Closed 1 under p? |
No
1
No Unknown | Yes Unknown |
Yes
1
Table 2.2.2: Closure under the restartable fail-stop transformation p. is 0(log^ N). Table 2.2.2 gives the closure properties under the restartable fail-stop transformation. Note t h a t due to the lower b o u n d s for the WriteAll problem, the entries t h a t are marked "No" m e a n non-closure, while the "Unknown" result means t h a t closure is not achieved with the known results.
2.2.8
Discussion: on randomization and approximation
We have presented an overview of the theory of efficient a n d fault-tolerant parallel algorithms. Our focus has been deterministic algorithms, p a r t l y because our work has concentrated on this topic, b u t also because m a n y deterministic techniques exist for the problems of interest. We close our exposition with an observation (by D. Michailidis) t h a t illustrates the power of randomization (vs determinism). As we described above deterministic Write-All solutions require logarithmic time. This is true even for a p p r o x i m a t e Write-All. However: T h e o r e m 2 . 2 . 2 1 T h e a p p r o x i m a t e Write-All problem (AWA) of size N where the number of locations to be written is iV' = aN and the number of surviving processors is a t least (3N^ for some constants 0 < a , / 5 < 1 can be solved probabilistically (error is Monte Carlo) on a CRCW PRAM with 0{N) expected work in 0 ( 1 ) parallel steps. R a n d o m i z a t i o n is an i m p o r t a n t algorithmic tool which has h a d extensive a n d fruitful application to fault-tolerance, e.g., [36]. Probabilistic techniques have played a key role in the analysis of asynchronous parallel c o m p u t i n g - see for example, [4, 5, 9, 10, 15, 22, 23, 2 1 , 30, 32, 34]. Note however, t h a t it is often hard to compare the analytical b o u n d s of deterministic vs randomized algorithms, since much of the randomized analysis is done using a n oblivious adversary a s s u m p t i o n .
151 Randomized algorithms often achieve better practical performance than deterministic ones, even when their analytical bounds are similar. Future developments in asynchronous parallel computation will employ randomization as well as the array of deterministic techniques surveyed here.
Bibliography [1] M. Ajtai, J. Aspnes, C. Dwork, O. Waarts, "The Competitive Analysis of Wait-Free Algorithms and its Application to the Cooperative Collect Problem", manuscript 1993. [2] G. B. Adams III, D. P. Agrawal, H. J. Seigel, "A Survey and Comparison of Fault-tolerant Multistage Interconnection Networks", IEEE Computer, 20, 6, pp. 14-29, 1987. [3] R. Anderson, H. WoU, "Wait-Free Parallel Algorithms for the Union-Find Problem", Proc. of the 23rd ACM Symp. on Theory of Computing, pp. 370-380, 1991. [4] Y. Aumann and M.O. Rabin, "Clock Construction in Fully Asynchronous Parallel Systems and PRAM Simulation", in Proc. of the 33rd IEEE Symposium on Foundations of Computer Science, pp. 147-156, 1992. [5] Y. Aumann, Z.M. Kedem, K.V. Palem, M.O. Rabin, "Highly Efficient Asynchronous Execution of Large-Grained Parallel Programs", in Proc. of the 34th IEEE Symposium on Foundations of Computer Science, pp. 271-280, 1993. [6] P. Beame and J. Hastad, "Optimal bounds for decision problems on the CRCW PRAM," Journal of the ACM, vol. 36, no. 3, pp. 643-670, 1989. [7] P. Beame, M. Kik and M. Kutylowski, "Information broadcasting by Exclusive Read PRAMs", manuscript 1992. [8] J. Buss, P.C. Kanellakis, P. Ragde, A.A. Shvartsman, "Parallel algorithms with processor failures and delays", Brown Univ. TR CS-91-54, August 1991. [9] R. Cole and O. Zajicek, "The APRAM: Incorporating Asynchrony into the PRAM Model," in Proc. of the 1989 ACM Symp. on Parallel Algorithms and Architectures, pp. 170-178, 1989. [10] R. Cole and O. Zajicek, "The Expected Advantage of Asynchrony," in Proc. 2nd ACM Symp. on Parallel Algorithms and Architectures, pp. 85-94, 1990. [11] R. DePrisco, A. Mayer, M. Young, "Time-Optimal Message-Optimal Work performance in the Presence of Faults" manuscript, 1994. [12] C. Dwork, J. Halpern, O. Waarts, "Accomplishing Work in the Presence of Failures" in Proc. 11th ACM Symposium on Principles of Distributed Computing, pp. 91-102, 1992. [13] D. Eppstein and Z. Galil, "Parallel Techniques for Combinatorial Computation", Annual Computer Science Review, 3 (1988), pp. 233-83. [14] S. Fortune and J. Wyllie, "Parallelism in Random Access Machines", Proc. the 10th ACM Symposium on Theory of Computing, pp. 114-118, 1978.
152 [15] P. Gibbons, "A More Practical P R A M Model," in Proc. of the 1989 Symposium on Parallel Algorithm>3 and Architectures, pp. 158-168, 1989.
ACM
[16] P. C. Kanellakis, D . Michailidis, A. A. Shvartsman, "Controlling Memory Access Concurrency in Efficient Fault-Tolerant Parallel Algorithms", 7th Int-l Workshop on Distributed Algorithms, pp. 99-114, 1993. [17] P. C. Kanellakis and A. A. Shvartsman, "Efficient Parallel Algorithms Can Be Made Robust", Distributed Computing, vol. 5, no. 4, pp. 201-217, 1992; prelim, vers, in Proc. of the 8th ACM PODC, pp. 211-222, 1989. [18] P. C. Kanellakis and A. A. Shvartsman, "Efficient Parallel Algorithms On Restartable Fail-Stop Processors", in Proc. of the 10th ACM Symposium on Principles of Distributed Computing, 1991. [19] P. C. Kanellakis and A. A. Shvartsman, "Robust Computing with Fail-Stop Processors", in Proc. of the Second Annual Review and Workshop on Ultradependable Multicomputers, Office of Naval Research, pp. 55-60, 1991. [20] R. M. Karp and V. Ramachandran, "A Survey of Parallel Algorithms for Shared-Memory Machines", in Handbook of Theoretical Computer Science (ed. J. van Leeuwen), vol. 1, North-Holland, 1990. [21] Z. M. Kedem, K. V. Palem, M. O. Rabin, A. Raghunathan, "Efficient Program Transformations for Resilient Parallel Computation via Randomization," in Proc. 24th ACM Symp. on Theory of Comp., pp. 306-318, 1992. [22] Z. M. Kedem, K. V. Palem, A. Raghunathan, and P. Spirakis, "Combining Tentative and Definite Executions for Dependable Parallel Computing," in Proc 23d ACM. Symposium on Theory of Computing, pp. 381-390, 1991. [23] Z. M. Kedem, K. V. Palem, and P. Spirakis, "Efficient Robust Parallel Computations," Proc. 22nd ACM Symp. on Theory of Computing, pp. 138-148, 1990. [24] C. P. Kruskal, L. Rudolph, M. Snir, "Efficient Synchronization on Multiprocessors with Shared Memory," in ACM Trans, on Programming Languages and Systems, vol. 10, no. 4, pp. 579-601 1988. [25] C. P. Kruskal, L. Rudolph, M. Snir, "A Complexity Theory of Efficient Parallel Algorithms," Theoretical Computer Science 7 1 , pp. 95-132, 1990. [26] L. E. Ladner, M. J. Fischer, "Parallel Prefix Computation", Journal ACM, vol. 27, no. 4, pp. 831-838, 1980.
of the
[27] M. Li and Y . Yesha, "New Lower Bounds for Parallel Computation," of the ACM, vol. 36, no. 3, pp. 671-680, 1989.
Journal
[28] A. Lopez-Ortiz, "Algorithm X takes work fi(7ilog^ 7 i / l o g l o g 7 i ) in a synchronous fail-stop (no restart) P R A M " , unpublished manuscript, 1992. [29] C. Martel, personal communication, March, 1991. [30] C. Martel, A. Park, and R. Subramonian, "Work-optimal Asynchronous Algorithms for Shared Memory Parallel Computers," SI AM Journal on Computing, vol. 21, pp. 1070-1099, 1992
This page intentionally blank
153 [31] C. Martel and R. Subramonian, "On the Complexity of Certified Write-All Algorithms", to appear in Journal of Algorithms (a prel. version in the Proc. of the 12th Conference on Foundations of Software Technology and Theoretical Computer Science^ New Delhi, India, December 1992). [32] C. Martel, R. Subramonian, and A. Park, "Asynchronous PRAMs are (Almost) as Good as Synchronous PRAMs," in Proc. 32d IEEE Symposium on Foundations of Computer Science^ pp. 590-599, 1990. [33] J. Naor, R.M. Roth, "Constructions of Permutation Arrays for Ceratin Scheduling Cost Measures", manuscript, 1993. [34] N. Nishimura, "Asynchronous Shared Memory Parallel Computation," in Proc. 3rd ACM Symp. on Parallel Algor. and Architect., pp. 76-84, 1990. [35] N. Pippinger, "On Simultaneous Resource Bounds", in Proc. of 20th IEEE Symposium, on Foundations of Computer Science, pp. 307-311, 1979. [36] M.O. Rabin, "Efficient Dispersal of Information for Security, Load Balancing and Fault Tolerance", J. of ACM, vol. 36, no. 2, pp. 335-348, 1989. [37] D. B. Sarrazin and M. Malek, "Fault-Tolerant Semiconductor Memories", IEEE Computer, vol. 17, no. 8, pp. 49-56, 1984. [38] R. D. Schlichting and F. B. Schneider, "Fail-Stop Processors: an Approach to Designing Fault-tolerant Computing Systems", ACM Transactions on Computer Systems, vol. 1, no. 3, pp. 222-238, 1983. [39] J. T. Schwartz, "Ultracomputers", ACM Transactions on Programming Languages and Systems, vol. 2, no. 4, pp. 484-521, 1980. [40] A. A. Shvartsman, "Achieving Optimal CRCW PRAM Fault-Tolerance", Information Processing Letters, vol. 39, no. 2, pp. 59-66, 1991. [41] A. A. Shvartsman, Fault-Tolerant and Efficient Parallel Computation, Ph.D. dissertation. Brown University, Tech. Rep. CS-92-23, 1992. [42] A. A. Shvartsman, "Efficient Write-All Algorithm for Fail-Stop PRAM Without Initialized Memory", Information Processing Letters, vol. 44, no. 6, pp. 223-231, 1992. [43] R.E. Tarjan, U. Vishkin, "Finding biconnected components and computing tree functions in logarithmic parallel time", in Proc. of the 25th IEEE FOCS, pp. 12-22, 1984. [44] J. S. Vitter, R. A. Simmons, "New Classes for Parallel Complexity: A Study of Unification and Other Complete Problems for P," IEEE Trans. Comput., vol. 35, no. 5, 1986.
This page intentionally blank
SECTION 3
DOMAIN-SPECIFIC PARADIGMS FOR REAL-TIME SYSTEMS
SECTION 3.1
Use of Imprecise Computation to Enhance Dependability of Real-Time Systems Jane W. S. Liu, Kwei-Jay Lin, Riccardo Bettati, David Hull and Albert Yu
Abstract In a system based on the imprecise-computation technique, each time-critical task is designed in such a way that it can produce a usable, approximate result in time whenever a failure or overload prevents it from producing the desired, precise result. This section describes ways to use this technique together with traditional fault-tolerance methods to reduce the costs of providing fault tolerance and enhanced availability. Specifically, an imprecise mechanism for the generation and use of approximate results can be integrated in a natural way with traditional checkpointing and replication mechanisms. Algorithms and process structures for this integration and rules for determining when approximate results can be used in place of the desired results are discussed.
3.1.1 Introduction The imprecise computation technique was proposed as a way to handle transient overloads in real-time systems [1-3]. Here, by real-time system, we mean a computing and conmiunication system in which a significant portion of the tasks have deadlines. The term task refers to a unit of work to be scheduled and executed. A task may be the computation of a control law, the transmission of an operator command, the Jane W. S. Liu, Riccardo Bettati, and David Hull are at the Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801. Kwei-Jay Lin is at the Department of Electrical and Computer Engineering, University of California, Irvine, Irvine, California. Albert Yu is at Hughes Aircraft, Radar Systems Group, Los Angeles, California 90009. This work was partially supported by the U. S. Navy ONR contracts No. NVY N00014 89-J-l 181 and No. NVY N0001489-J-l 146.
158 retrieval of a file, etc. As the result of its execution, each task delivers some data or service. The failure of a time-critical task to deliver its result by its deadline is a timing fault. A real-time system functions correctly only in the absence of timing faults. An example is the TCAS (Traffic Alert and Collision Avoidance System) used in commercial aircraft to alert pilots of potential collisions. The command telling the pilot of the conflict traffic and the necessary evasive action not only must be correct but also must be issued in time. Other examples of real-time systems include flight control and management, intelligent manufacturing, and various monitoring systems. For many real-time applications, we may prefer to have approximate results of a poorer but acceptable quality on a timely basis to late results of the desired quality. For example, it is better for a collision avoidance system to issue a timely warning together with an estimated location of the conflict traffic than a late command specifying the exact evasive action. Other examples are video and voice transmissions. While poor quality images and voices may be tolerable, late frames and long silences are often not. The imprecise computation technique was motivated by this observation and the fact that good approximate results can often be produced with much less processor time than results of the desired quality. By trading the quality of the results for the amount of time and resource required to produce them, a system based on this technique tries to make approximate results of acceptable quality available whenever it cannot produce results of the desired quality in time. Hereafter, we will refer to such a system as an imprecise system and a result of the desired quality a precise result. An imprecise computation is a generalization of an anytime computation, a term used in the AI literature. The focus of the AI community has been on anytime computational algorithms and reasoning about the results produced by them (e.g., [4-6]). On the other hand, the work by the real-time systems conmiunity has been concerned with how to structure the operating and run-time systems to take advantage of the schedulingflexibilityprovided by imprecise computations. In recent years, several models have been developed to characterize the behavior of imprecise computations and to quantify the costs and benefits of the tradeoff between result quality and required processing time for different classes of applications and implementation methods. Many efficient scheduling algorithms that achieve the optimal or suboptimal tradeoff are now available; examples are those described in [7-18]. These algorithms have made the imprecise computation approach to overload handling feasible. The imprecise computation technique is also a natural means for enhancing fault tolerance and graceful degradation of real-time systems. We consider here only hardware faults and transient software faults. Permanent software faults are not considered. To see how availability and fault tolerance can be increased in an imprecise system, we consider tracking and control systems for example. A transient fault may cause a tracking computation to terminate prematurely and produce an approximate result. No recovery action is needed if the result allows the system to maintain track of the targets.
159 Similarly, as long as the approximate result produced by a control law computation is sufficiently accurate for the controlled system to remain stable, the fault that causes the computation to terminate prematurely can be tolerated. In embedded systems, this technique can be used together with the traditional replication and checkpointing techniques (e.g., [19,20]). The result is a reduction of the costs of providing fault tolerance and enhanced availability. This section describes an architecture that integrates the imprecision mechanism for the storage and return of intermediate, approximate results of computations with the fault-tolerant mechanisms that support traditional checkpointing and replication. The Imprecise Computation Server (ICS) system is being built on the Mach operating system to implement this architecture. ICS will make it easy for us to implement imprecise computations and experiment with them. When it is completed, we plan to implement several representative applications using the ICS system in order to evaluate experimentally the effectiveness of the imprecise computation technique. The applications we plan to implement and experiment with include multimedia data transmission, direct digital control and optimal control, and database queries and updates. A reason for choosing these types of applications for in depth examination is that they require different imprecision management rules. These rules govern whether approximate results produced by prematurely terminated tasks are acceptable and what error recovery operations are to be carried out when the results are not acceptable. Following this introduction. Section 3.1.2 gives a brief overview of the different ways to implement imprecise computations, workload models used to characterize them, and algorithms to schedule them. Section 3.1.3 describes an architectural framework for the integration of an imprecision mechanism and a traditional checkpointing mechanism, as well as the ICS system built on this framework. Section 3.1.4 describes algorithms for scheduling replicated imprecise computations and Section 3.1.5 a process structure that supports imprecision and replication-masking. Section 3.1.6 discusses the imprecision management rules for different types of applications. Future extensions are discussed at the end of each section. Section 3.1.7 is a summary.
3.1.2 Imprecise Computation An imprecise system can be represented abstractly by a variation of the precedence graph in Figure 3.1.1. Specifically, this graph represents an imprecise system T as a set of related tasks. Each task is represented by a node. Nodes are depicted by boxes of all shapes in this figure. Tasks may be dependent; data and control dependencies between tasks impose precedence constraints on the order of their execution. There is an edge from a node Ti to a node Tj if the task 7f must be completed before Tj can begin execution.
160
jj^ multiple
optional
Figure 3.1.1: General Imprecise Computation Model Implementation Methods In an imprecise system, some tasks are identified by the progranmier as mandatory. Like tasks in traditional real-time systems, mandatory tasks must be executed and completed by their deadlines in order for the system to function correctly. These tasks are shown as solid boxes in Figure 3.1.1. The progranmier identifies some less important tasks as optional, meaning that these tasks can be skipped (i.e., not executed) without causing intolerable degradation in the output of the system. Optional tasks are represented by dotted, shaded boxes in Figure 3.1.1. The system may skip optional tasks during overload so that mandatory tasks can complete in time. Sieve Method and the 0/1 Constraint For example, in radar signal processing the task that estimates the background noise level in the received signal can be skipped.
161 An old estimate of noise level can be used when a new estimate cannot be produced in time. Therefore this task is optional. Similarly, in aflightmanagement system the step that updates the estimated time of arrival can be skipped when the system is busy dealing with local turbulence and nearby traffic and is, therefore, optional. In general, we call a task whose sole purpose is to produce a result that is at least as precise as the corresponding input or an old output a sieve function. A sieve can be skipped in order to save time and resource. When it is skipped, the result is less precise. Conversely, when a sieve completes in time, it improves the quality of the result. This method for producing approximate results is called the sieve method. There is no benefit gained by completing a sieve in part. Therefore, we want to either execute such an optional task to completion before its deadline or skip it entirely. In this case, we say that the execution of the optional task satisfies the Oil constraint. When tasks have the 0/1 constraint, the scheduler must decide before the execution of each task begins whether to schedule the task entirely or to skip it entirely. Some schedulingflexibilityis thus lost. Monotone Tasks and Milestone Method A task, as well as the underlying computational algorithm, is said to be monotone if the quality of the intermediate result produced by it is non-decreasing as it executes longer. We can logically decompose each monotone task into a mandatory task and an optional task. A monotone task produces a precise result when the entire task completes. An approximate result can be made available by recording the intermediate results produced by the task at appropriate instants of its execution. If the task is terminated before it is completed, the approximate result produced by it at the time of its termination is the best among all the intermediate results produced before its termination. This result is usable as long as its mandatory task is completed. This method for returning approximate results is called the milestone method. Clearly, the milestone method relies on the use of monotone computational algorithms. Such algorithms exist in many application domains, including numerical computation, statistical estimation and prediction, sorting, facsimile transmission [21], video and voice transmission [22,23], and database query processing [24,25]. Monotone tasks can be easily implemented in most existing programming languages; an example illustrating how to implement monotone tasks in Ada 9x can be found in [26]. When tasks are monotone, the decision on which optional task and how much of the optional task to schedule at any time can be made dynamically. Because the scheduler can terminate a task any time after it has produced an acceptable result, scheduling monotone tasks can be done on-line or nearly on-line. Consequently, we have the maximum flexibility in scheduling when imprecise computations are implemented using this method.
162 Multiple-Version Method For some applications, neither the milestone method nor the sieve method can be used. Trading off result quality for processing time is nevertheless possible by using multiple versions of tasks. Using this method, we need to provide at least two versions of each task: the primary version and the alternate version(s). The primary version of each task produces a desired, precise result but has a longer processing time. An alternate version has a shorter processing time but produces an approximate result. During a transient overload, when it is not possible to complete the primary version of every task by its deadline, the system may choose to schedule alternate versions of some tasks. The feasibility and effectiveness of the multiple version method has been demonstrated for both real-time computing and data communication; tools and environments have been developed to support this method [27-33]. A higher storage overhead incurs because of the need for maintaining multiple versions. Performance data shows that there is little advantage to be gained by having more than two versions [30,32]. For scheduling purpose, we model the alternate version of each two-version task as a mandatory task and the primary version as a mandatory task followed by an optional task. The processing time of the mandatory task is the same as that of the alternate version, and the processing time of the optional task is equal to the difference between the processing times of the two versions. The optional task must be either scheduled and completed by its deadline, corresponding to the primary version being scheduled, or skipped entirely, corresponding to the alternate version being scheduled. Consequently, scheduling tasks that have two versions is the same as scheduling tasks with the 0/1 constraint. Error Characteristics and Scheduling Algorithms For the purpose of scheduling, we logically decompose each task 7f into two tasks, the mandatory task Mi followed by the optional task Oi, independent of the method used to implement it. Let n, rui and Oi denote the processing times of 7^, Mi and Oi, respectively. Clearly, rrii -h Oi = Ti. The classical deterministic model is a special case of this imprecise computation model where all tasks are mandatory, that is, Oi = 0 for all i. Similarly, a sieve or an anytime computation is a task that is entirely optional. In general, the error of a task Ti is a function of the processing time of the portion that is executed, as well as the errors in the inputs of the task. Let ai denote the amount of processor time that is assigned to the optional task Oi. All existing scheduling algorithms assume that the inputs to every task have zero error. In other words, the error Ci in the result produced by the task Ti (or simply the error of Ti) depends solely on ai and is given by the error function Ei{(Ti) of the task. Moreover, when (Ti is equal to Oi, that is, when the scheduler allows Ti to execute until it terminates normally, its error Ci is zero.
163
linear processor time Mi
Xi
processor time
(a). Types of error functions
(5) Effects of input error
Figure 3.1.2: Error Characteristics The error in the result of a task typically decreases in discrete steps and is, therefore, discontinuous. Because it is difficult to work with discontinuous functions, we typically approximate them by the continuous error functions. The continuous error functions were found to be good approximations except when the error decreases to zero in very few (e.g., two or three) steps [8]. Figure 3.1.2(a) shows three types of error functions that characterize the general behavior of different monotone computations. When the scheduler works correctly, it allows every mandatory task to complete. Therefore, the value of the error in the range where the amount of assigned processor time is less than the mandatory processing time is unimportant; for the sake of convenience, we let this value be 1. When the exact behavior of error functions is not known, a reasonable choice is a = (oi — (Ti)/oi for all i\ that is, the error of a task is equal to the processing time of its skipped portion. For this reason, most studies on scheduling imprecise computations assume this linear error function. For a given schedule, the total error of the task set T is e = Yll=i '^i^i^ where wi are the weights of the tasks. By choosing the weights, we can account for the different degrees to which the errors in the results produced by the individual tasks impact the overall quality of the result produced by the entire task set. Examples of optimal and suboptimal algorithms that minimize the total error can be found in [8,11,15]. An iterative algorithm, such as Newton's method forfindingroots of polynomials, typically gives us a monotone task whose error decreases faster earlier during its execution. The rate of convergence to zero error decreases as such a task continues to execute. Its error function behaves like the convex function in Figure 3.1.2(a). Many
164 other computations, such as statistical estimation, also have this type of error function. The average error of tasks with convex error functions can be kept small by making the maximum normalized error of all tasks as small as possible. Given a schedule of a set {Ti} of tasks with identical weights, the maximum normalized error of the task set is 6 = max, {{oi — (Ti)/oi}. Polynomial-time algorithms for finding optimal schedules with the smallest maximum normalized error can be found in [13,17]. The error of a monotone task whose error function is concave decreases at a faster rate as it executes longer. The optional part of such a task should be scheduled as much as possible or not scheduled at all. In the limit, the error of an optional task with the 0/1 constraint stays at the maximum value of 1 until the optional task completes. In a schedule that satisfies the 0/1 constraint, the amount of processor time assigned to every optional task is equal to either oi or 0. The general problem of scheduling to meet the 0/1 constraint and timing constraints, while minimizing the total error is NP-complete when the optional tasks have arbitrary processing times. Approximate algorithms with reasonably good worst-case performance can be found in [14]. When the optional tasks have identical processing times and weights, tasks with the 0/1 constraint can be optimally scheduled in 0{n log n) time or 0{'n?) time depending on whether the tasks have identical or different release times. These algorithms can be found in [11]. Future Work in Scheduling Again, all existing algorithms for scheduling imprecise computations assume that tasks have zero input errors. Moreover, the release times and deadlines of individual tasks are given. Both these assumptions are often not valid. The timing constraints of a set of dependent tasks are often end-to-end in nature. As long as the last task(s) in the set completes before its end-to-end deadline, when the individual tasks in the set complete is unimportant. The possibility of postponing the deadlines of some tasks gives an additional dimension in the tradeoff between the quality and the timeliness of the overall result and makes the problem of scheduling imprecise computations more difficult. The result produced by a task may be an input to its inmiediate successors. When this result is imprecise, the input is erroneous. We can account for the effect of errors in the inputs of a task T, by using an error function that is also a function of input errors. Specifically, let the errors in the inputs of a task Ti be denoted by the vector e,. The error in the result produced by Ti whose optional task is assigned cr, units of processor time is given by Ci — Ei{ai,ei), where Ei{ai,ei) is a non-increasing function of (Ti and a non-decreasing function of e,. Figure 3.1.2(b) illustrates the effect of input errors. A task may need to do some additional work to correct its input error, and a poorer input may slow down the rate at which its result converges to the precise one. Consequently, the processing times of both Mi and Oi may become larger as the input
165 error ef increases. When the magnitude of e, becomes larger than some threshold, the processing time o,(ei) of the optional task Oi may become infinite, and the task Ti can never produce a precise result no matter how long it executes. The need to keep the error in an input below this threshold determines when a result produced by each predecessor task is acceptable and how large the processing time of the mandatory part of the predecessor task should be. We are developing scheduling algorithms that take input errors into account and make use of the end-to-end nature of the deadlines when trying to minimize the error in the overall result.
3.1.3 Integration of Imprecision with Checkpointing As stated earlier, the mechanism for the storage and return of the intermediate, approximate results of computations can be easily integrated with the traditional checkpointing mechanism for fault tolerance. This section describes an architecture that allows this integration and the Imprecise Computation Server (ICS) system which implements this architecture. Throughout this section, we confine our attention to transient faults and failures. To make approximate results available in an imprecise system, the intermediate results produced by each task must be recorded as the task progresses toward completion. The application programmer defines the intermediate result variables and the accuracy measure variables to record, as well as the time instants at which their values are recorded. Each set of recorded values can be viewed as a checkpoint. Therefore, we call the mechanism for returning intermediate approximate results user-directed checkpointing. In contrast, the traditional checkpointing mechanism is system-directed. Using user-directed checkpointing to supplement system-directed checkpointing has several obvious benefits. Because the application progranmier can make use of the semantics of the individual computations, it is possible to keep the amount of state information recorded at each user-directed checkpoint small. For example, it suffices to record two consecutive intermediate roots produced by an iterative rootfinding computation or the current sample mean and variance of a simulation run. In the absence of user provided checkpointing routines, the operating system must save the entire address space of each task. Consequently the cost of user-directed checkpointing is usually lower than for system-directed checkpointing. Furthermore, since fault recovery can make use of user-directed checkpoints as well as systemdirected checkpoints, system-directed checkpoints need not be done as often, thus reducing the cost of providing fault tolerance. When a task terminates before it is completed, recovery action is not necessary if the last result returned was sufficiently good. Thus the processing time that would be spent repeating the work between the last checkpoint and the point of failure may be saved.
166 Client
Server
Caller
Callee
Handler
Supervisor Scheduler
Figure 3.1.3: ICS Process Structure An Imprecise System Architecture The process structure that we use for the result-saving purpose is the simple variation of the client-server model shown in Figure 3.1.3. There is a server type for each service provided by an imprecise system. Each server consists of a callee and a supervisor, while each client consists of a caller and a handler. The caller and callee are applicationspecific and are written by the application programmer. The handler and supervisor are part of the underlying system. The supervisor is created at configuration time or at the instantiation time of the server. It may be dedicated to one server or shared by several servers. These details are unimportant for the purpose of this discussion. As in traditional systems, the callee executes whenever it is called by the client. The client calls a server by sending an invocation request to the server's supervisor. Typically, only the client knows how accurate a result must be for it to be acceptable and how many faults each server invoked by it must tolerate. On the other hand, only the scheduler has the information on the demand and the availability of all resources. Both kinds of information are needed to support the decision on what quality of service to provide to each client and which tasks to terminate when an imprecise system recovers from a transient overload. To support this decision, the information regarding the result accuracy required by the client, the minimal level(s) of accuracy achievable by the server, and the required number of tolerated faults is exchanged between the client, the server, and the scheduler at invocation time. We will return shortly to describe the interactions between these system components for the purpose of this information exchange. When the supervisor grants the request, it activates the callee. When the time allocated for the callee's execution expires, the supervisor terminates the callee. The client has the final, precise result when the callee terminates normally. If the callee terminates prematurely, the client has the best approximate result produced by the
167 callee before it terminates. Based on the latest recorded value of the accuracy measure variable, the handler can decide whether this approximate result is acceptable. In addition to aiding the result-saving process, the supervisor also checkpoints the callee and carries out sanity checks on a periodic basis in order to enhance fault tolerance. Because the supervisor and the callee do not communicate during the callee's execution, a fault occurring during the callee's execution does not affect the supervisor. We further assume that the effect of any fault occurring during the callee's execution is confined to it. This simplifying assumption allows us to focus on the interaction between the imprecision and fault-tolerance mechanisms. When a failure is detected before the callee *s mandatory portion completes, recovery is necessary. The supervisor restores the callee to its state at the time of the last checkpoint and resumes its execution from that state. We call such an action a system-directed recovery action. An imprecise system must also support user-directed recovery. For real-time applications, forward recovery is typically more appropriate. Moreover, it can often be done effectively, especially for embedded applications. To carry out forward recovery, the client may want the sequence of approximate results produced by the callee during the course of its execution, not just the last result before its termination. This sequence of approximate results is often the consecutive discrete-time sample values of a continuous-time function. The future values of such a function can be estimated by extrapolating from the available past sample values. Depending on the semantics of the problem, there may be several ways to do this extrapolation to arrive at different acceptable estimates of thefinalresult. For example, the application programmer may provide the client with some routines that implement filters with prediction or pure prediction filters. If the callee terminates prematurely, the client would take the approximate results and apply one of the given routines on them to generate a better approximation of thefinalresult. In this way, the application system can better control of the quality of approximate results. Scheduling Checkpointed Imprecise Tasks From the scheduler's point of view, an imprecise application system T is a set of n (callee/server) tasks. Suppose that the supervisor of the callee Ti takes a checkpoint every k units of time, and that it takes Ci units of processor time to generate a checkpoint. While a checkpoint is generated, a sanity check is made. We assume that sanity checks never fail, and a failure occurring at any time instant is detected by the next sanity check after the instant. The callee task is allowed to fail ki times, {ki is specified by the client in its invocation request, in a manner described later in this section.) Some other recovery measure will be taken if it fails more than ki times. Let pi denote the amount of time the supervisor executes in order to roll back the callee task. A task Ti
168 can tolerate ki faults only when a sufficient amount of processor time is allocated so that its mandatory portion can complete before its deadline even when it must be rolled back ki times. In general, we say that a schedule isfc^-tolerantfor task Ti if the task can recover from ki failures without causing any task in the system T to miss its deadline. A schedule for the system T is k-tolerant, where k denotes the vector (fci, ^ 2 , . . . , kn), if it is fci-tolerant for Ti, ^2-tolerant for T2, and so on. The goal of scheduling checkpointed imprecise tasks in T is to find a k-tolerant schedule that minimizes the total error when all /?i + ^2 + \- kn failures occur. (Here, the error of each callee task Ti is equal the processing time of its skipped optional portion.) To accomplish this goal, the scheduler must allocate at least m / « mj + [mi/li\ Ci -h ki{li -h Cj + pi) units of time to the task when the processing time of its mandatory portion is m,. When tasks are statically bound to processors and tasks on each processor are scheduled as in a uniprocessor system, the algorithm called Algorithm COL in [33], can be used for this purpose. When the tasks are off-line, this algorithm is optimal, in the sense that it can always find afc-tolerantschedule if such a schedule exists and the total error of all the tasks in T is minimized. (A task system is off-line when the release times, deadlines and processing times of all the tasks in it are known before the processor begins to execute any task.) The version described below assumes that the tasks are on-line; the algorithm is optimal only when the system satisfies the feasible mandatory constraint [15]. Similar to the NORA algorithm for scheduling on-line imprecise tasks which are ready for execution at their arrival times [15], Algorithm COL maintains a reservation list of time intervals for all tasks that have been released but are not yet completed. This list is updated each time a new task Ti is released; as a result of this update, an interval (or several intervals) of length equal to rrii' is reserved for it. The reservation list is derived from a feasible schedule of all the unfinished portions of the mandatory tasks. This schedule is generated by backward scheduling according to the latest-ready-time-first rule; a time interval is reserved if some task is scheduled in the interval. The scheduler uses the reservation list as a guide. It never schedules any optional task in a reserved interval. The reservation list is also updated as the execution of each mandatory task Mi progresses to completion. At an update when y units of the mandatory portion have already been completed and z of the ki tolerable failures have occurred, the amount of time reserved for Mi is reduced to rrii — y-\- [(m^ - y)/li\ Ci + {ki — z){li -{- Ci -\- Pi). The reservation for it is eventually deleted when the mandatory portion completes. Algorithm COL differs from the NORA algorithm in the way it schedules mandatory and optional tasks. Algorithms COL maintains an EDF-ordered main task queue and an optional task queue. When a task is released it is put into the main task queue after the reservation list is updated. Thefirsttask in this queue is scheduled for execution. The task executes for as long as there is time before the beginning of the earliest reserved
169 interval. When the task completes or is terminated at its deadline, it is removed from the main task queue and its reservation is deleted from the reservation list. If before its completion or its deadline, the beginning of the earliest reservation interval is reached, the reservation of the task is deleted if it has any reserved time according to the current reservation list. Otherwise, the task is terminated and is removed from the main task queue. If a task is not completed when it is removed from the main task queue, it is placed in the optional task queue, which is also EDF-ordered. When the main task queue is empty, the processor executes the optional task at the head of the the optional task queue, until the completion or the deadline of the task; it is then removed from the optional task queue. The elegance of this algorithm in combination with a checkpointing scheme is that any unused recovery time is automatically made available for the other tasks, allowing their optional portion to execute longer. The performance of this algorithm for scheduling on-line transactions that have bounded-response-time and high-availability requirements has been evaluated and performance data can be found in [33]. The ICS System We are implementing the ICS (Imprecise Computation Server) system based on the architecture described above. ICS runs on top of the Mach operating system and is integrated with the Mach Interface Generator (MIG) [34]. MIG takes an interface description for a service and generates interface code for the client and the server. This interface hides the call to the remote server from the client, making it appear to be a local procedure call. Interfaces To develop an application system using ICS, the application programmer first defines the interface for each service, and writes a MIG specification file that describes the interface. A modified version of MIG, called IMIG, is run to generate the interface code. IMIG adds an argument to each of the interfaces, which is used to exchange ICS-specific information between the caller and the callee. Specifically, it is a pointer to a data structure containing scheduling information; a portion of this structure is shown in Figure 3.1.4. The requiredAccuracy and outputAccuracy parameters measure the accuracy of the result. The deadline and mandatoryTime parameters allow the programmer to specify the timing requirements of the application system. The resources parameter is used for server selection. The toleratedFaults parameter specifies the number of faults that the server must be able to tolerate. This information, together with the value mandatoryTime, is used by the scheduler to determine the amount of processor time to reserve for the mandatory portion. Next, the programmer writes the code that implements the callers and callees in the application system and then compiles and links the code with the ICS library, which
170 struct icslnfo { long requiredAccuracy; struct timeval deadline; long resources;
/* User Inputs.
int serviceNumber; /* System Inputs, struct timeval mandatoryTime; int toleratedFaults; long outputAccuracy;
/* Outputs.
*/
Figure 3.1.4: The icslnfo Structure
ICS-supplied
User-supplied
Client Side icsFindService client interface routine
Server Side icsRegisterService icsService result-saving routine server interface routine checkpoint routine restart routine
Table 3.1.1: ICS Functions contains the supervisor and handler code. Each client or server produces a separate application program. ICS's interface routines fall into two groups: client-side routines and server-side routines. They may be further divided into those that are provided by ICS to be called by the application and those that are written by the programmer to be called by ICS. The routines and their classifications are listed in Table 3.1.1. Details on these routines and their usage can be found in [35]. The client interface routine is the client's interface to a server and is called by the client when it wishes to make use of a service. The server interface routine is the corresponding interface on the server side. When a client interface routine is called, the server interface routine is eventually invoked by a supervisor on the selected server. The checkpoint routine provided by the application programmer is used for both
171 result-saving and checkpointing purposes. If no checkpoint routine is specified, ICS will checkpoint the callee in the traditional manner by saving the callee's entire address space. In this case, ICS will be unable to return an approximate result to the caller if the callee fails; the computation will have to be restarted from the last checkpoint. Service Establishment The programmer starts the application system by starting the application programs for each of the servers and clients in the system. Each server, after performing any necessary initialization, registers itself with its supervisor by calling icsRegisterService, This call informs the supervisor of the name of the service, the resources required by the callee, and the routines to call to perform checkpointing and to restart the callee from a checkpoint. Once the service has been registered, the server calls the icsService routine, which passes control to the supervisor. The supervisor advertises the services that it provides and waits for requests from clients. To make use of a service, a client first calls the icsFindService routine with the name of the service it wishes to call. icsFindService returns a service number. The service number is used by the handler to keep track of all of the servers for that service. When a client calls a server, the caller fills in the icslnfo structure and calls the ICS-supplied client interface routine. This interface routine invokes the handler. The handler uses the scheduling information provided in this structure to select a server and sends a message to the chosen server's supervisor. The supervisor creates a new thread of control for the callee and informs the scheduler of the callee's deadline and processing time requirements. It then starts the callee when the scheduler grants its request to start the callee. After starting the callee, the supervisor sets a timer. When the timer expires, the supervisor calls the callee's checkpoint routine, which instructs the callee to call its result-saving routine. When the callee wants to save an approximate result, it calls its result-saving routine itself. The result-saving routine, which is generated by IMIG, sends the result and the callee's current state to the handler that originated the request. The result-saving routine also resets the timer, thus delaying the next supervisor-invoked checkpoint. If the handler finds that the server has died or has missed its deadline, it can use the state information received at the last checkpoint to decide whether to return the result that has been computed so far to the caller, or to carry out some user-directed recovery action, or to use the saved state to restart the request on another server. Extensions to ICS We have implemented a prototype version of ICS and the design seems reasonable. Once we complete the version described here, there are several extensions we would like to make. One extension is to replicate servers on redundant processors. The following section discusses an approach to use the imprecise computation technique to reduce the overhead of replication. Another approach is
172 to replicate only the mandatory portions. The approximate results produced by the mandatory portions are used to ensure the correctness of the refined results produced by the unreplicated optional portion. We will take this approach because it is more compatible with the ICS architecture. In the near future, we will extend the ICS system to support imprecise service negotiation. When it is not possible for the scheduler to allocate sufficient amounts of processor time or other resources to grant a request, it informs the supervisor of the available amounts. Rather than simply denying the request, the supervisor and client may enter an optional negotiation phase. The willingness of the client to accept a poorer quality result that can be produced with the available resources is determined. The callee is scheduled if the client can use such a result. In this way, the quality of each imprecise computation can be negotiated at invocation time.
3.1.4 Integration of Imprecision and Replication For real-time applications, replication and masking is a more appropriate approach to provide fault tolerance whenever there is more than one processor. This section describesfirstalgorithms for scheduling replicated periodic imprecise tasks and then a process structure that allows replication be incorporated in an imprecise system. Scheduling Replicated Imprecise Taslis We consider here a system T of n independent, replicated periodic tasks. They are to be assigned to identical non-faulty processors and allocated sufficient processor time for them to produce acceptable results in time. The Allocation Problem We say that the system is in mode j when the system contains j non-faulty processors. Each task Ti is replicated. There are c,(j) clones, or copies, of Ti when the system is in mode j . Ci{j) is chosen by the application programmer. The set {ci(j)} is an input of the task assignment module. When invoked, this module allocates a fraction of processor time to (each clone of) each task andfindsan assignment of the clones of all the tasks to the j processors. An assignment in mode j is a valid one if the Ci{j) clones of every task Ti is assigned on a one-clone-per-processor basis to c, (j) non-faulty processors. Moreover, let Xi{j) be the amount of processor time per period that is allocated to each clone of task Ti. Because the mandatory portion of every task must always be completed, Xi{j) must be larger than the processing time rrii of the mandatory task Mi of Ti. Since the optional portion need not be completed, Xi{j) may be less than the processing time Vi of the task as a whole. Let pi be the period of Ti, and Ui{j) = Xi{j)/pi be the fraction of processor time allocated to each clone of T,. We refer to Ui{j) as the allocated
173 utilization, or simply the utilization, of Ti. For an assignment to be valid, the allocated utilization Ui{j) must be such that rui/pi < Ui{j) < ri/pi for all I
