Analyzing and Aiding Decision Processes, 14

ANALYSING AND AIDING DECISION PROCESSES

This Page Intentionally Left Blank

ANALYSING AND AIDING DECISION PROCESSES Editors

Pat rick HUMPHREY S London School of Economics and Political Science, England Ola SVENSON Department of Psychology, University of Stockholm,Sweden Anna VARI Bureau for Systems Analysis, State Office for Technical Development, Hungary Co-editors Tibor ENGLANDER and Janos VECSENYI, Hungary, Willem WAGENAAR, The Netherlands, Detlof VON WINTERFELDT, U.S.A.

1983

NORTH-HOLLAND PUBLISHING COMPANY AMSTERDAM NEW YORK OXFORD

-

Joint edition with Akadhiai Kiad6, Budapest AN rightsreserved. N o part ofthis publication may be reproduced, stored in a retrieval system, or transmitted, in any form or b y any means, electronic, mechanical, photocopying, recording or otherwise, without the prior permission of the copyrighr owner.

ISBN: 0 444 86522 5 @ AkadBmiai I Kiado, Budapest 1983

Publishers:

NORTH HOLLAND PUBLISHING COMPANY AMSTERDAM * NEW YORK * OXFORD Sole distributors f o r the U.S.A.and Canada:

ELSEVIER SCIENCE PUBLISHING COMPANY, INC. 5 2 VANDERBILT AVENUE NEW YORK, N.Y. 10017 PRINTED IN HUNGARY

TABLE OF CONTENTS

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

11

Section 1: SOCIETAL DECISION MAKING

k,

Patrick Humphreys Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Ralph L. Keeney Evaluation of Mortality Risks for Institutional Deci23 sions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . John Latluop and Joanne Linnerooth The Role of Risk Assessment in a Political Decision 39 Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Howard Kunreuther A Multi-Attribute Multi-Party Model of Choice: Descriptive And Prescriptive Considerations . . . . . . . . . 69 Rex V. Brown and Jacob W. Ulvila The Role of Decision Analysis in International Nuclear Safeguards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 David H. Custafson, Robert Peterson, Edward Kopetsky, Rich Van Koningsveld, Ann Macco, Sandra Casper, and Joseph Rossmeissl A Decision Analytic System for Regulating Quality of Care in Nursing Homes: System Design and Evalua105 tion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Section 11: ORGANIZATIONAL DECISION MAKING JBnos Vecsenyi and Detlof von Winterfeldt

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . Oleg I. Larichev Systems Analysis and Decision Making . . . . . . . . . . Andrew R. Lock Applying Decision Analysis in an Organisational Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Detlof von Winterfeldt Pitfalls of Decision Analysis . . . . . . . . . . . . . . . . . Anna VBri and JBnos Vecsenyi Decision Analysis of Industrial R & D Problems. Pitfalls and Lessons . . . . . . . . . . . . . . . . . . . . . . . . .

123 125

145

167

183

Section 111: AIDING TtIE STRUCTURING OF SMALL. SCALE DECISION PROBLEMS Anna Viri Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . Gordon F. Pitz Human Eugineering of Decision Aids . . . . . . . . . . . I-lelmut Jungerniarln, Ingrid von Ulardt, and Lutz tlaus niann The Role of the Coal for Generating Actions . . . . . . Dimiter S. Driankov and Ivan Stantcliev Fuzzy Structural Modelling--An Aid for Decision Making . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Patrick tiumphreys Use of Problerii Structuring Techniques for Option Generation: A Computer Choice Case Study . . . . . . Fred Bronner and Robert de Hoog Non-Expert Use of a Computerized Decision Aid. . . . Richard S. John, Detlof von Winterfeldt, and Ward Edwards The Quality and User Acceptance of Multiattribute Utility Analysis Performed by Computer and Analyst Stuart Wooler and Alma Erlich Interdependence Between Problem Structuring and At tribute Weighting in Transitional Decision Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

199 205

2 23

237

253

28 1

30 1

32 1

Section IV: TRACING DECISION PROCESSES Ola Svenson and Patrick Humphreys Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . Henry Montgomery Decision Rules and the Search for a Dominance Structure: Towards a Process Model of Decision Making Ola Svenson Scaling Evaluative Statements in Verbal Protocols from Decision Processes . . . . . . . . . . . . . . . . . . . . Lennart Sjoberg To Snioke or Not T o Smoke: Conflict or Lack of Differentiation? . . . . . . . . . . . . . . . . . . . . . . . . . Joshua Klayman Analysis of Predecisional Information Search Patterns . Rob Ranyard and Ray Crozier Reasons Given for Risky Judgnwnl and Choice: A Comparison of Three Tasks . . . . . . . . . . . . . . . . . Eduard J . Fidler The Reliability and Validity of Concurrent, Retrospective, and Interpretive Verbal Reports: An Experimental Study . . . . . . . . . . . . . . . . . . . . . . . . . . . Oswald Huber The Information Presented and Actually Processed in a Decision Task . . . . . . . . . . . . . . . . . . . . . . . . . Kobert W. Goldsmith and Nils-Eric Sahlin The Role of Second-Order Probabilities in Decision Making.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marian Smith and William R. Ferrell The Effect of Base Rate on Calibration of Subjective Probability for True -False Questions: Model and Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . Ruma Falk and Don MacGregor The Surprisingness of Coincidences . . . . . . . . . . . .

337

343

371

383

401

41 5

429

44 1

455

469

489

Section V: A SYMPOSIUM ON THE VALIDITY OF STUDIES ON HEURISTICS AND BIASES Willem-Albert Wagenaar Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 505 Ward Edwards Human Cognitive Capabilities, Representativeness, and Ground Rules for Research . . . . . . . . . . . . . . . 507

Baruch Fischhoff Reconstructive Criticism . . . . . . . . . . . . . . . . . . . 515 Lawrence D . Phillips A Theoretical Perspective on Heuristics and Biases in Probabilistic Thinking . . . . . . . . . . . . . . . . . . . . . 525 Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

545

Subject lndex . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

553

PREFACE

The papers in this book are an edited selection from those presented at the Eighth Research Conference on Subjective Probability, Utility and Decision Making, held in Budapest in August 1981. Together they span a wide range of new developments in studies of decision making, the practice of decision analysis and the development of decision-aiding technology. The international, interdisciplinary nature of the work represented here makes it difficult and perhaps unwise to assign papers to categories according to the methodology or approach used, or the area surveyed. Nevertheless, we have arranged the book in five principal sections, according to the different fields of interests our readers may have, and placed in each section those papers which seem to be particularly significant in providing food for thought and new perspectives within that field. The titles we gave to the first four sections are "Societal Decision Making", "Organizational Decision Making", "Aiding the Structuring of Small Scale Decision Problems, and "Tracing Decision Processes". Throughout, the emphasis i s on decision processes and structures and their applications, rather than formal modelling in isolation. We do not see this as a 'bias', but rather a reflection of current developments in research and practice which follow from the understanding of the nature and operation of decision theoretic models gained during the 1970's. Here you will find suggestions on how to bring these models alive and put them to work in a wide range of contexts. The fifth section is of a different nature. It presents papers given at a symposium on the validity of studies on heuristics and biases at the Budapest conference. These papers take stock of the considerable volume of work investigating 'heuristics and biases' in decision making over the past decade, and their implication for theory and practice. The papers give the authors' own viewpoints and are presented here unedited with the hope that they will stimulate discussion among a wider audience.

10

We d o not propose to make any suggestions directing particular readers t o particular sections, as this would run counter t o the spirit of the book and the conference which gave rise to it. However, if IOU would like a general overview of the papers in a section before delving more deeply, you will find this in the introduction at the start of the section. The Eighth Research Conference on Subjective Probability, Utility and Decision Making was part of a biennial series, currently attracting over one hundred leading practitioners. Participation in these conferences is open to anyone working in relevant areas, and the conference is advertized through the mailing of information t o active individuals and institutions. It is a European conference, and while participation is encouraged from all countries in the world, its principal aim is t o stimulate the exchange of ideas and development of research and practice throughout Europe, east and west. Conferences have been held in Hamburg (1969), Amsterdam (1 970), London ( 1 971 ), Rome ( 1 973), Darmstadt ( 1 975), Warsaw ( 1 977), Gothenburg ( 1 979) and Budapest ( 1 981 ). The Ninth conference will be held in Groningen, the Netherlands. in 1983. Procedures for setting up and running each conference fall under the responsibility of an organizing committee chosen at the closing session of the previous conference. The conference is not affiliated t o or sponsored by any institution or organization, being supported each time by its participants and by national research agencies of the host country, and we consider that such independence has been a cornerstone in maintaining its vitality and acceptance across the complete range of European countries. Edited versions of the proceedings of most of'the conferences in the series have been published in books, or in Actu Psychologicu.* Keviewing earlier publications in the series, one can see how foundations laid earlier come t o fruition and in turn set the scene for new developments in the field. We have compiled this volume with the hope that it will play its part in this process. Patrick Humphreys Ola Svenson Anna Vari

* D. Wendt and C.A.J. Vlek (eds.), 1975. Utility, Probability and Human Decision Making, Amsterdam: Reidel (Proceedings o f the 4th Conference). H . Jungerniann andG. de Zeeuw (eds.), 19ll.DecisionMaking and C h a w e in Humon Affairs. Amsterdam: Reidel (Proceedings of the 5th Conference). L. Sjoberg, T. Tyszka and J.A. Wise (eds.), 1983. Decision Anulysis and Decision Processes. Lund: Doxa (Proceedings of the 6th Conference). L.R. Beach, P.C. Humphreys, 0. Svenson and W.A. Wagenaar (eds.), 1980.Exploring Human Decision Making. Amsterdam: North Holland (Proceedings of the 7th Conference); also published as Acta Psychologica, 45, 1980 (complete volume).

ACKNOWLEDGEMENTS

The editors gratefully acknowledge the assistance o f the following persons who acted as referees of contributions t o this volume. Berndt Brehmer (Sweden) Vaclav Brichaced (Czechoslovakia) Klara Farago (Hungary) Baruch Fischhoff (U. K.) Robert Goldsmith (Sweden) Robin Hogarth (U. S. A.) Robert de Hoog (The Netherlands) Richard John (U. S . A.) Howard Kunreuther (Austria) Oleg Lariclicv ( [ I . S. S . R.) Sarah Liclitenstein (U. S. A.) Lola Lopcs (LI. S. A.) Jan Meisrier ( U K.) Henry Montgomery (Sweden) Gerard do Zeeuw (The Netherlands)

Czeslaw Nosal (Poland) Lawrence Phillips (U. K.) Gordon Pitz (U. S. A,) Rob Ranyard (U. K.) Dieter Schroder (West Germany) Zur Shapira (Israel) Lennart Sjoberg (Belgium) David Spiegelhalter (U. K.) Ivan Stantchev (Bulgaria) Tadeusz Tyszka (Poland) Charles Vlek (The Netherlands) Stephen Watson (U. K.) Ayleen Wisudha (U. K.) Stuart Wooler (U. K.)

This Page Intentionally Left Blank

Section I

SOCIETAL DECISION MAKING

This Page Intentionally Left Blank

IN TROD UCTlON Patrick HUMPHREYS

Decision theory, as a theory of individual choice has been largely concerned with the delineation of act -event sequences leading to consequences, the evaluation of those consequences, computing the expected value (or expected utility) of courses of action immediately available through principles which enable the uncertainty with which consequences are linked to immediate acts t o be taken explicitly into account. The papers in tlus section describe models and procedures which encompass these ideas, but demonstrate how they in themselves map out only a small part of the issues which must be considered in studying decision making within a social context either descriptively or prescriptively. I t is not sufficient t o 'socialise' the study of decision processes by grafting ideas about communication, equity or bargaining onto models of personal decision making. The models explored here demonstrate how decision-theoretic models of the problem need to be refocussed, with goals concerning 'good' decision processes which are qualitatively quite different from those conventionally adopted in studying individual choice. Ralph Keeney sets the scene for this refocussing in considering the evaluation of mortality risks. Estimating fatalities consequent upon a socially controversial course of action, like siting a new energy plant, has often been proposed as an objective measure of the 'risk' involved in the enterprise. The "objectivity" is commonly thought to reside in the possibility of separating probabilistic estimates of magnitude of risk (number of fatalities expected) from severity of risk (the value of each life lost through such fatalities). However, the concept of "expected fatalities" h d e s the distribution of those fatalities in the population at risk. Keeney considers the two extremes which characterize 'real life' situations in various mixtures: loss of an identifiable life, where a single expected fatality refers t o one known person, and loss of a statistical life, where a

16

P.Humphreys

single expected fatality refers to a I/n chance of loss of life for each member of a population of n people. Keeney shows that there is no reason to assume the value of a statistical and an identifiable life to be the same. Arguments may be adduced for distinct preferences for avoiding either type of fatality over the other. Moreover, valuation of a statistical life as greater than an identifiable life implies a preference for risk equity : spreading the risk equally as possible over the population, whereas valuation of an identifiable life as greater than a statistical life goes with a preference for catastrophe avoidance. It is not possible to find a valuation function which meets both preferences, one has to decide on the tradeoffs one makes between them in deciding how to value lives which may be lost in a particular situation. This analysis reveals two crucial flaws in risk assessments which try to avoid the subjectivity inherent in value judgements by describing risk solely in terms of probabilities of fatalities. First, without adequate information concerning the expected distribution of these fatalities, one cannot place a value on the lives in any unique way. Secondly, the values assigned depend of necessity on the relative preference for risk equity versus catastrophe avoidance of the assessor. The authorsof the other papers in this section start from an acknowledgement of the essential subjectivity of risk assessment and focus on modelling the context of this subjectivity as an essential pre-requisite to any prescriptive analysis. John Lathrop and Joanne Linnerooth examine the role of risk assessments-as represented in risk studies commissioned by interested parties within a political decision process: siting of a Liquid Energy Gas plant in California. They show how the content of a risk assessment is largely determined by the intended use of the study in the current round of political debate on the decision. This varies the need to model the decision process sequentially, as the same assessment may be used in different ways i n different rounds. Lathrop and Linnerooth describe how a risk assessment, containing “worst case” scenarios, introduced in one round as part of a local government’s environmental impact report can be taken up in subsequent rounds by pressure groups from among the population shown at risk in these scenarios. The formulation of the problem may change between rounds, as sequential constraints operate. Lathrop and Linnerooth describe by round four in a LEG siting decision this can result in the formulation being constrained to the question “is the single site under consideration safe?”, despite changes in the wider problem context (fall in energy demand) which are left unconsidered as they now lie outside the problem formulation.

Section 1 : INTRODUCTION

17

In this context, what should be the role of risk assessments within the decision making process? Lathrop and Linnerooth critically examine the implications ot' the idea commonly found in literature on prescriptive models of decision making that the goal should be to concentrate on improving risk assessment reports (e.g., by establishing independently funded public research bodies t o provide the analysts) or on improving their interpretation ( e g , by including technical experts among the judges). They suggest that a more realistic aim would be to improve the role that risk assessments actually play in the decision process by treating them as evidence, rather than attempting t o promote them as fact. Some desirable features are indicated for rules of evidence:Assessments covering different sites (policies) s h o d d be comparable, so that alternafhes, rather than assessments can be judged; they should discuss the modelling of uncertainties and tradeoffs "about which there cannot be any objectivity"; they should address the assessments of risk as perceived by those at risk, not relying on pseudo-objective measures like "expected fatalities" when the political process is primarily sensitive t o potential for catastrophe. In line with the change from "facts" to evidence, the advocacy role of experts should be recognized, and treated as a productive rather than a silent element in making public policy. In line with these ideas, Howard Kunreuther describes a niultiattribute multiparty (MAMP) model for structuring the social decision making processes which pass through a number of rounds and involve various parties with different concerns. Kunreuther shows how each round in the process can be characterised by a unique problem formulation, involving a particular set of interested parties. Interpreting the mpdel involves the development of a party/concern matrix, studying the impact of exogenous events, and analysing the sequential constraints imposed on subsequent rounds by the outcome of the current round. Kunreuther shows how the MAMP model may be used to draw lessons from case studies of liquid energy gas siting decisions. Specifically, (i) there was little articulation of value judgements by different parties in the process, who would state objectives, but were reluctant t o provide the importance weight which would reveal their value structures; (ii) the constraints guiding the decision process were unstable, consistent with the view that policies are determined through a process where each of the interested parties attempts t o modify the rules of the game in a way that will serve t o maximize the likelihood o f the attainment of their own goals and objectives; (iii) parties exploited the fact the siting of sophisticated technologies was "not well understood scientificallyff as a licence t o concentrate on whatever measures of risk they wished. Moving from descriptive t o prescriptive analysis, Kunreuther notes

2

18

P. Humphreys

that within the MAMP model there is no reason why one cannot focus on how well different decision making procedures (rather than outcomes) score with respect t o a well defined set of objectives. One can use MultiAttribute Utility theory t o examine how well a procedure scores on attributes like: does each party have an opportunity t o voice its own position; were a wide enough set of attributes considered t o feel a choice was actually made? Given that decision processes may be evaluated in this way, Kunreuther discusses how GERT (Graphic Evaluation and Review Technique) may be used t o investigate how they might be improved. Questions which can be addressed through the use of GEKT include: ”What is the likely impact on the decision process if some of the existing constraints are relaxed; what would the impact be if certain parties were given power they currently d o not have; what would happen if there was a change in the way alternatives sites were introduced into the picture?” The MAMP niodel helps us t o understand how decision analysis may be employed in providing inputs to decision making, conceived as an essentially political process. This orientation is essential when interested groups or parties t o the decision are in conflict, where the goals of a decision analysis should not extend t o trying to organize the decision process itself, as in such cases there is no formally ’equitable’ way of doing this from all parties’ points of view. In contrast are c a m where decision analyses are commissioned as a way of promoting goals shared by society, excepting a few minority of renegade groups or individuals whose actions or goals are generally viewed as going against consensus interests. Here the context is different from that addressed by the MAMP model and usually encompasses an agency constituted as the guardian of society’s interest and granted appropriate executive powers. The final pair of papers in this section describe decision analyses having the goal of improving the regulatory activities of two such agencies in rather different fields. Rex Brown and Jacob Ulvila describe the role of decision analysis in international nuclear safeguards, examining the issues that need t o be considered in improving the efficiency of the inspection activities of the International Atonuc Energy Agency (IAEA). David Gustafson and his colleagues describe the development and evaluation of a decision analytic system for regulating the quality of care in nursing homes in the state of Wisconsin. While the overall goal is the same in each of these two cases, the very different contexts means that their interpretation in practical terms are rather different, and the contrast between the two studies illustrates the diversity of potential applications of decision analysis in prescriptive social decision making.

Section 1: INTRODUCTION

19

In the IAEA decision analysis the goal was to allocate inspection activities in a way that would minimize the diversion of nuclear material from peaceful use; in the nursing home decision analysis the aim was t o allocate inspection activities in a way that would maximize the quality of care. In both cases time and scope of inspection activities were an overall constraint as inspection resources were severely limited. In the IAEA case further constraints resulted from the agreements with the participating nation states on which the acceptance of the legitimacy of the agency's activities rested. For example, the IAEA is constrained not to allocate its resources in a way that discriminates between states (hence indicating its view that particular states are more likely t o contemplate diversion). No such constraint existed in the Wisconsin case, where nursing homes must undergo annual inspection, backed up by the possibility of state enforcement action. Here one task of the initial analysis was t o develop a screening instrument which would indicate those homes of suspected inferior quality, so that r~zorctinze could then be spent in these homes in subsequent inspection activities. In the initial phase of the analysis described by Brown and Ulvila, the idea of developing a screening instrument based upon judges' ratings is replaced by a diversion path analysis operating under the constraint of only analysing info, mation which can be "uncontroversially and objectively" documented. This constraint rules out a classical decision analytic approach which would typically involve modelling the likelihood of different types of diversion in a particular context; how a diverter would behave, who the diverter might be. But a fornial analysis of these issues would force the innate logic out into the open and thereby open t o objection. Brown and Ulvila discuss ways of overcoming these problems. While, t o avoid objections, the likelihood of a diversion along a path must be given equal weight in every context, one can vary the number of paths objectively identified in any particular context. Paths may also be weighted by their attractiveness, with thcse weights operating within the analysis as relative probabilities. Rather than make any attempt t o identify "diverted', one can use "objective" surrogates for the seriousness of a diversion path: classifying it according t o the type of nuclear material that may be diverted along it. Brown and Ulvila describe how these possibilities form the basis for an analysis, in which diversion paths are defined by the material involved, its location and the method of concealing its removal. This information is determined for any particular facility from analysis of design information, operating and accounting procedures, and inspection histories. Determining a diversion path does not indicate that it will be used, hence the probability that it is in use must also be determined

2*

20

P. Humphreys

"objectively", This is done by determining 'anomalies', unusual and observable conditions that might occur in the event of a diversion through the path concerned. What sort of decision aids should be developed on the basis of the types of analysis Gustafson et al., and Brown and Ulvila describe? Both groups of authors agree that these should involve (i) developing decision rules for when to initiate a special inspection of a facility and when t o proceed from one level of response open t o the regulating activity to a more severe one; (ii) how best t o allocate inspection resources a t a facility and between facilities. In the nursing home case, the rules for initiating special inspection are: if the home passes the initial (screening) review, its survey is complete; if only 3 few minor problems are found, surveyors would seek t o correct them; if many deficiencies or a few substantial problems are revealed, a complete survey (i.e., special inspection) would be implemented concerning the full set of 1,500 regulations for nursing homes, legal investigations, special review teams collecting data which might be useful in court proceedings. Choice of consequent executive action is not based o n the overall evaluative score for a home from the analysis, but rather involves the srlec.tion of the one intervention that would most likely lead to a change in the way care was delivered. In the case of nuclear safeguards, results of inspections are also not expressed as evaluation numbers, but for 3 different reason: to d o so would require weighting the importance of different levels of detection and. given the context, there are difficulties for the regulatory agency in analysing such weights explicitly. Rather, Brown and Ulvila suggest that analysis of anomalies be translated into detection probabilities for diversion paths. In the current version the single 'detection' response has been used corresponding t o the observation of an anomaly. If the anomaly remains unresolved, inspection activity may be stepped up, the relevant state missions may be consulted, or ultimately a report may be made t o the U.N. Security Council. A future possibility would be t o assess explicitly the detection probabilities t o be associated with different levels of IAEA response. Regarding allocation of inspection resources, Gustafson et al. stress thrt their goal was t o maximize the potential for contingent regulation : identifying and discharging problems in agencies delivering good care, while concentrating on those agencies which need assistance. Gustafson et al. describe how their Quality Assurance Process (QAP) inspection procedures (comprising multi-attribute utility based screening, together with interviews of 10% of residents chosen statistically) was better able t o meet these goals than the old regulatory process, since it led t o more time

Section I: INTRODUCTION

21

being spent in poor quality homes, action being taken on more problems, improvement in the quality of care, and actions being more frequently "consistent with effective change agents". In the aid proposed for use by the IAEA, the fundamental goal in allocating inspection resources is one of deterrence by risk of detection, To this end a decision analytic model was used to develop a prioritized list of'inspection activities, with priority depending on the value of the activity and its cost in terms of inspection time. Brown and Ulvila describe how value can be indexed through an aggregate measure involving explicit consideration of the probability of detecting a diversion, the amount and type of material diverted, the technical complexity specific vulnerabilities of the system and the timeliness of the detection. Determining the optimal relation between priority of an inspcction activity and including it in the inspection regime at any facility was a much easier task in the case of ensuring quality of care in nursing homes than in ensuring nuclear safeguards. In each case the inspection procedures, as instruments of social policy, must of necessity be made public. In Gustafson et al.'s QAP procedure it was not considered that this would degrade the efficiency of the procedures. On the contrary, knowledge of the procedures was expected t o aid the majority of nursing homes in their own attempts to improve the quality of care they offered. Public knowledge of the IAEA's prioritization of inspection activities might, however, aid potential diverters in their activities, and such aid would run precisely counter t o IAEA's goals. Any time-limited inspection strategy is sure t o leave some diversion possibilities uncovered, and fixed prioritization serves to identify those uncovered paths on which the diverter might then focus his attention. Brown and Ulvila describe how in this context it is not possible to perform the sort of analysis that Kunreuther recommends in his multiattribute multiparty model, since it is not permitted to model a potential diverter (i.e., one of the parties in the social decision process) explicitly, so no party-concern matrix can be obtained and 'solved' for an ideal inspection strategy. Instead they recommend partial randomization as a source of deterrence: Inspection activities covering those paths of low technical complexity and easy t o use nuclear activity should be covered in every inspection, together with inspection activities selected on a random basis to provide some coverage of paths that are more complex or involve hard-to-use material. In general, the papers in this section demonstrate how decision analyses for societal decision making must be sensitive t o the social and political context in which they are employed. They also illustrate ways of modelling that context that aid the design and use of decision analyses in

22

P. Humphreys

meeting goals held by social consensus across parties to the decision. In many cases these goals involve the use of analyses to optimise procedures rather than outcomes. Conversely, in assessing the status of decision theory, we find here evidence that, while retaining characteristic emphasis on the assessment of probabilities of events and the utilities of consequences, the scope of this theory now extends well beyond models predicated on individual or corporate goals, providing the basis for both descriptive and prescriptive analysis of situations of social concern.

EVALUATION OF MORTALITY RISKS FOR INSTITUTIONAL DECISIONS' Ralph L. KEENEY Woodward-Clyde Consultants, San Francisco, California, U.S.A.'

Abstract This paper investigates the implications of various value judgments 3n the evaluation of public mortality risks. Using utility functions to quantify values indicates the mutual inconsistency of three reasonable goals: minimize the expected number of fatalities, promote equitable distribution of public risk, and a preference for catastrophe avoidance. Utility analyses and some other approaches for assisting with decision malung involving public risks are appraised from an institutional or governmental perspective.

Introduction An important aspect of many institutional decision problems involves possible health and safety risks to members of the public. Decisions involving the licensing of drugs, medical research and service programs, use of existing and future technologies, road safety, and military programs can each have a significant effect on public risks. These risks may involve possible fatdlitks, sicknesses, and injuries, as well as psychological effects. For appraising alternatives in such decision contexts, the decision makers need to address public risks in addition to other economic, social, and environmental consequences. This paper addresses issues that an institution 'This work was conducted under a subcontract with Decision Research, a branch of Perceptronics, for Oak Ridge National Laboratory under ORNL subcontract No. 7656. It was performed for the 11,s.Nuclear Regulatory Commission under NRC Interagency Agreement 40-550-75. fliis papr is a revised version of a presentation given at the Conference on the Value of Life sponsored by the Geneva Association, Geneva, March 30-April 1, 1981. The comments of Detlof von Winterfeldt were very helpful in the revision. 2The address of WoodwardClyde Consultants is: Three Embarcadero Center, Suite 700, San Francisco, California 94111, U S A .

24

R.L. Keeney

should consider in order to evaluate responsibly the public risks of alternatives. Our focus is on the evaluation of mortality risks, those risks which could end the life of individuals. It must be understood that a period of pain or morbidity resulting in the fatality is perhaps worse than an immediate fatality. However, the difference is the undesirability of the pain or morbidity, which should be explicitly included as an additional factor in an evaluation process. In order to explicitly evaluate mortality risks, an index of the relative desirability (or undesirability) of any circumstance characterizing the specific risks is needed. Our task is to structure potential indices. The process will necessarily involve value judgements. Hence, we proceed by postulating various reasonable value judgements and then derive their implications for the resulting index. Examining the implications of different basic values results in some interesting insights about evaluating possible fatalities. The paper is outlined as follows. The first section characterizes the evaluation of mortality risks from an institutional decision making perspective. The next section introduces three common approaches: insurance, willingness-to-pay, and court awards for examining mortality risks from that perspective. As an alternative, we propose utility analysis. In the third section, the restrictions that various value assumptions place on the evaluation of potential fatalities are examined, and the contradictions between these assumptions are highlighted. Final sections consider the extension of the approach to multiple causes of risk, discuss implementation of utility analysis, and present a brief summary and conclusions.

Structuring the Problem Over the past decade, several approaches have been suggested for evaluating decisions that involve potential fatalities (e.g., Acton, 1973; Jones-Lee, 1974; Howard, 1979; Keeney, 1980a). Linnerooth (1975, 1979) and Zeckhauser (1975) provide excellent surveys of much of this literature, which carries the label "value of life". One conclusion is that there is certainly no consensus on an appropriate approach for evaluating lives. A major reason for this lack of consensus is the complexity of the problem. But perhaps a more important explanation is the fact that there are several different problems categorized under the rubric "value of life". For instance, these problems might be characterized along the following dimensions:

EVALUATION OF MORTALITY RISKS

25

1 . prescriptive versus descriptive, 2. evaluation before or after fatalities occur, 3 . whether many individuals or one individual is at risk, 4. whether contemplated actions affect the risks, 5. who is doing the evaluating. Regardless of the problem addressed, value judgments must be made. However, there is a choice of whether these value judgments should be made implicitly or explicitly. Also, the value judgments appropriate for one characterization of the ”value of life” problem may not be appropriate for other characterizations. Our orientation is prescriptive throughout this paper, since the ultimate concern is to identify appropriate actionsgiven the risks and the basic value judgments articulated for the problem. We will be particularly interested in examining the desirability of spending funds to reduce (directly or indirectly) the mortality risks. Hence, we will require an index of costs, as well as the risks, as arguments in an analysis. By an institutional decision making perspective we mean that an institution provides the value judgments necessary to appraise various risks as well as the valbe tradeoffs between the risks and costs. There are two general circumstances where this model seems particularly appropriate. The most common is when individuals in the government evaluate programs t o spend public funds t o reduce mortality risks to members of society. The second circumstance is when individuals in an institution spend institutional funds to reduce the risks t o members of that institution. An example is expending industry funds to make equipment safer to reduce the risks to workers. These circumstances are to be contrasted with those where individuals may appraise risks involving only themselves. We wish t o evaluate mortality risks before any fatalities have occurred. Also, there will be numerous individuals potentially a t risk. Hence, our problem can be characterized by a mortality risk vector p = (pl ,pz ... ...,p,) where pi is is the risk to individual i and n is the total number of individuals at risk. Specifically, pi will be the probability that individual i will be a fatality in the time period of concern due to the cause of concern. This probability is independent of all the other probabilities in the mortality risk vector. To describe circumstances involving risks to individuals where the risk of one person is related to others, lotteries over mortality risk vectors must be utilized. An appropriate manner to measure the desirabillty of lotteries is a utility function assessed in accordance with assumptions postulated alternatively by von Neumann and Morgenstern (1947) or Savage (1945). The cost that one might be willing to expend to reduce mortality risks will be denoted c. Thus, a possible consequence characterizing a

26

R.L. Keeney

situation would be the vector (p; c) = (pI , pz , . . . , Pn; c). The basic problem for evaluating mortality risks is then to determine an appropriate utility function u which assigns the utility u(p;c) to each consequence (p;c). Based on the assumptions of utility theory, consequences with higher utilities should be preferred to those with lower utilities, and when uncertainty is involved, situations correspondmg to higher expected utilities should be preferred. Given this characterization of the problem, consequences which were indifferent to each other would have the same utility. Specifically, suppose we searched for 3 consequence of the form (0, . . . , O;c*) that was indifferent t o a consequence (1,0, . . . 0;O). Then became the first consequence involves no risks and a cost of c*, and the second consequence involves a risk of I to one individual and no costs, one could define c* to be the ”value of life”. One might assume that if there were a unique measure of the value of life, then the value of each life would be equal and the value of 2 lives lost would be equal t o 2 times the value of 1 life lost. This would imply that consequences (l,O, . . . O;c*) and (O,l, . . . , O;c*) are indifferent and that consequences ( l , l , O , . . . 0;O) and (0, . . . ,0;2c*) are indfferent. However, these are not reasonable assump tions for many decision problems. Our problem formulation, which does not make such simplifying assumptions, can characterize many different ”value of life” problems using the utility function u(p;c). Problems are distinguished by whose lives are at risk, whose funds might be expended to reduce these risks, and whose values are utilized t o evaluate the consequences. One situation is where the individual at risk is the same individual whose funds and value judgments are utilized. Another situation occurs when the risks are to one group, the costs are borne by another group, and the evaluation is done by only one of the two parties. A third situation is where the risks are to one set of individuals, the costs are to a second party or organization, and the value judgments are made by an individual or individuals presumably representing both parties. This latter situation is what we have referred to as the institutional perspective.

Insurance, Willingness-to-Pay, and Court Awards Approaches utilizing insurance, willingness-to-pay schemes, and court awards are among those suggested for evaluating mortahty risks to individuals (see, for example, Zeckhauser, 1975). As we wd1 see, each of these schemes seems to have serious shortcomings for prescribing an

EVALUATION OF MORTALITY RISKS

21

appropriate institutional evaluation of mortality risks. That is not to say that the schemes are inappropriate for other uses. It is simply the case that there are widely different “value of life” problems.

Insurance It has been proposed that if an individual has an insurance policy of x dollars, then he or she implicitly values life at x dollars. Given that, the leap has been sometimes made t o suggest that institutions should utilize the same value in examining options affecting risks to the insured. From the prescriptive institutional viewpoint, there are many shortcomings to this approach. First, decisions of whether or not to insure and how much insurance are not necessarily made with one’s best judgments. Because one buys an insurance policy does not imply that he or she should have bought that policy, so an institution may not want to follow such individual a3ions. Second, the existence of an insurance policy usually does not affect the mortality risks t o an individual. Consequently, the value tradeoff between mortality risks and costs is not considered. I would assume that a majority of the individuals with life insurance amount x would not gladly give up their lives for a return of slightly greater than x dollars.

Willingness-to-Pay With the willingness-to-pay approach, individuals are asked how much they would be willing t o pay to reduce a particular mortality risk by an amount E . If the response is y dollars, this might then be extrapolated linearly to imply that y/e dollars is an appropriate value of life for the respondent. One shortcoming often found in the assessment procedure is that the implications of the initial responses about willingness-to-pay are not investigated. Experience indicates that it is very difficult to respond consistently to questions involving very small changes in one’s mortality risks. An additional problem is to find an appropriate manner to aggregate individual responses which is necessary to obtain any institutional evaluation of mortality risks to several individuals. Because the responses about willingness-to-pay concern funds of the individual at risk and the institutional problem concerns funds of some other entity, a direct aggregation of the individual responses may be inappropriate.

R.L. Keeney

28

Court Awards A third approach suggested for evaluating the value of a life involves court awards. In some Circumstances after an individual has lost his or her life, the court awards the heirs a dollar amount z as restitution from a party responsible for the fatality. The manner in which the court selects an amount z could result from several factors and is often not clearly articulated. There seems to be no clear justification for utilizing such a value for prescriptive evaluation. Whereas insurance and willingness-to-pay involve situation where the individual at risk made value judgments about his or her own costs and risks, the judge has more of an institutional perspective with court awards. However, court awards involve circumstances where lives are evaluated after a fatality has resulted, so no contemplated action can affect the risks. Because we wish t o evaluate circumstances where action can influence risks to individuals, the value of a life based on court awards may not be appropriate. Also, a court award is usually based on a specific individual’s circumstance and would necessarily need to be extrapolated in some manner to deal with the institutional problem involving risks to several individuals. A reasonable manner for this extrapolation does not obviously present itself.

Evaluating Mortality Risks Using Utility Analysis Utility analysis uses a utility function for evaluating mortality risks. In t h s section, several suggestions for an appropriate structure of such an institutional utility function are discussed. The implications of various basic value judgments wdl be examined. Rather than use a formal theorem-and-proof format, we will simply state some of the key ideas as observations. When appropriate, details wdl be referenced to other works. The observations in this section are appropriate for any separable utility function (u(p;c) = f [uR(p),uc(c)], where f is any function, U R is the utility function for mortality risks, and uc is the utility function for costs. However, for simplicity, we will assume that an appropriate utility function for risks and costs is U(P;c> = UR(P) 4-hW(c),

(1)

where X is a scaling constant to assure that uR and are consistently scaled. For convenience, we will set the origin of all the utility functions by

EVALUATION OF MORTALITY RISKS

29

and establish a scale for each of the utility functions by u(l,O, . . . , O ; O )

= U,(l,O,.

. . ,O)

=

w ( 1 ) = -1

where c is evaluated in millions of dollars.

Identifiable versus Statisticul Fatalities Let us now examine the concept of value of life introduced earlier. Specifically, we suggested that the c* defined by ( l , O , . . ., 0;O) indifferent to (0, . . . , O ; c * ) can be interpreted as the value of life. However, the indifference of ( l l n , . . . , l/n;O) and ( 0 , . . . , 0;c’) gives us another possible value of life. In the first situation, we know that individual 1 will be a fatality and there will be no costs or that no individual will be a fatality and there will be c* costs. Hence, the life at risk is clearly identifiable; it is individual 1’s life. Such a circumstance is sometimes referred to as an identifiable fatality, since the individual is identifiable. Hence, it might be more appropriate to refer to c* as the value of an identifiable life. T o interpret the second pair of indifferent consequences, recall that there are n individuals at risk. Since each of these individuals has a I/n chance of being a fatality i n the first consequence, the expected number of fatalities resulting from the risks is I . Hence, we could say that 1 expected fatality and no costs is indifferent t o 0 expected fatalities and the cost of c’. However, at the time the evaluation must be made, it is clear that each individual has an equal chance of being a fatality and it is not possible to identify who will be fatalities (in fact, there could be zero, one, or more than one fatality). Such a circumstance is sometimes referred t o in the literature as a statistical fatality, so c’ could be referred t o as the value of a statistical life. Many people feel that the risk ( l , O , . . , , 0) is worse than ( l / n , l / n , . . , l / n ) because the former consequence is so unevenly distributed and at least the risk in the latter case is ”somewhat equitable”. If this is the case, the value c‘ should be less than the value c*. In essentially all situations involving an iristitutional evaluation of mortality risks, we would expect to be dealing with only small risks to for eacb individual. These risks should certainly be much less than each person and would much more likely be in the range of lo4 and smaller. However, we could certainly have situations where 20 percent of the individuals had a iisk of 5/n each and the other 80 percent had a 0 risk. The expected fatalities corresponding to this risk would also be 1 although the risk in some sense would not be as equitable as the situation where each individual had a l/n risk. Consequently, although this

R.L. Keeney

30

latter risk situation might be characterized as having 1 statistical fatality, it may be appropriate to value this statistical fatality less than the former case. Observation 1 . There is no unique definition for the term value of life. Aside from the lack of a clear definition, use of the term "value of life" can be misinterpreted to imply that the value of 2 lives is equal t o twice the value of 1 life. This may be the case but it requires additional value judgments. Furthermore, for decision making, we are interested in evaluating the risks prior to possible fatalities. Of course we are interested in risks because of the possible resulting fatalities, but it might be appropriate to refer t o the value of a specific risk or of a specific risky situation when considering decisions involving mortality risks rather than to the value of lives.

The Utility Function uR To determine the utility function u R , we need to consider utility functions over each component pi and the integration of these utility functions. Concerning the former problem, it may be reasonable to assume that an organization should evaluate an individual's risks as the individual would want t o evaluate them. Assuming that the individual wants t o minimize his or her risks, such a utility function should be linear in pi. This linearity condition also follows from a consistency argument which assumes that the relative utility of mortality risk vectors must equal the expected utility of the implied set of fatalities (see Keeney, 1 9 8 0 ~ ) Given . the individual had a different value structure, we would need t o use a nonlinear utility function ui (pi). In this paper, we concentrate on the evaluation of simultaneous individual risks so we will employ the linear case for ui for simplicity. Similar developments are possible for nonlinear cases. We also make the assumption that risks to each individual should be evaluated identically. Furthermore, it may be appropriate to evaluate risks to any subset of individuals in the same manner regardless of fixed risks to those individuals not in the subset. In technical terms, these assumptions are equivalent t o the following.

Basic Model Assumptions. The attributes measured by the individual risks are utility independent, all subsets of attributes are preferentially independent, and the utility function for each of these single attributes is linear (namely, minus pi for individual i).

EVALUATION OF MORTALITY RISKS

31

As shown in Keeney (198Oc), the resulting risk utility function uR must either be the additive form

or the multiplicative form

where k is a scaling constant. We will now examine further implications on the form of uR given the assumptions above for each of three different objectives: that we want to minimize the expected number of fatalities, that we prefer equitable distributions of risk, and that we wish to avoid catastrophes. As we will see, these desires are mutually inconsistent.

Minimizing Expected Fatalities

Given specific funds to reduce the risks to individuals, a reasonable guideline might be to minimize the resulting total expected fatalities to save as many lives as possible. Observation 2. Given the basic model assumptions and a preference to minimize the expected number of lives lost, the utility function for mortality risks must be the additive form (4).

This observation almost directly follows from the fact that the sum of the pi's is equal to the expected number of fatalities with any mortality risk vector. If we use expected fatalities as our criterion, the additive utility function must be appropriate. lf we define f to be the number of fatalities resulting from any particular risk situation, a utility function UF for fatalities consistent with the additive UR is UF(f) = -f.

(6)

An Equitable Distribution of Risk We will define consequence (pI , . . . ,pi, . . . , pj . . . , pn), for any i and j , to be a more equitable distribution of risk than (pl, . . . , p i + € , . . . , p j - ~ , . . . , pn) if the difference between pi and pj is less than the difference

R.L. Keeney

32

between pi + f and pj- e . Note that all Ph, h # i, j are held fixed in the definition. This definition merely says that given all risks but t w o are fixed, the better balanced these two are, the more equitable the risk. A reasonable assumption might be that a more equitable distribution of risk is preferred t o a less equitable distribution.

Observation 3. Given the basic model assumptions and a preference for a more equitable distribution of risk, the utility function for mortality risks must be the multiplicative form (5) where 0 > k > 1 . ~~

As shown in Keeney (198Oc), the utility function for fatalities consistent with this preference for risk equity is

uF(f)

=

- l / k [(l

+ k)'

-

11

(7)

which is risk prone for the allowable values of parameter k. With a risk prone utility function for fatalities, a lottery with T expected fatalities would be preferred to ffatalities for sure. In simple terms, one would gamble in an attempt t o save more than the average possible fatalities. Thus, a risk prone utility function would not always result in evaluating alternatives in the manner t o save the most expected fatalities. It follows that t o have the original risks more equitably spread among the individuals at risk, one must be willing t o allow a few more expected fatalities. Whenever there is a preference t o better achieve an aspect of risk other than the fatalities themselves, it is necessary t o give up something in terms of fatalities.

Catastrophe A voidance A number of papers, such a Ferreira and Slesin (1976) and Slovic et al. (1 977), suggest that a small probability of a catastrophic loss of life is worse than a larger probability o f a smaller loss of life, given the expected number of fatalities are the same for each case. To be precise, let us say that one prefers catastrophe avoidance if a probability n of having f fatalities is preferred t o a probability 71' of having f fatalities for any f less than f' given that nf = n ' f . Given this assumption, the following is proven using results in Keeney (1980b, 1 9 8 0 ~ ) .

Observation 4. Given the basic model assumptions and a preference for catastrophe avoidance, the utility function for mortality risk must be the multiplicative form (5) where k > 0.

EVALUATION OF MORTALITY RISKS

33

The corresponding utility function for fatalities is (7) where k > 0. As can easily be demonstrated, this utility function is risk averse, which 5 consistent with a preference for T fatalities for sure to any lottery with f expected fatalities. With this preference to avoid a large loss of life (i.e., a catastrophe), one must be willing to accept more expected fatalities.

h Statistical versus Identifiable Fatalities Revisited Combining the cost utility function in (1) with the utility functions for mortality risks in observations 2 through 4 leads to the following interesting result.

Observation 5 . Given the basic model assumptions and ( l ) , the value of an identifiable life equals the value of a statistical life if and only if there is a preference to minimize expected fatalities, the value of an identifiable life is greater than the value of a statistical life if and only if there is a preference for risk equity, and the value of a statistical life is greater than the value of an identifiable life if and only if there is a preference for catastrophe avoidance. Many individuals might feel that more funds should be expended to eliminate an identifiable risk (1,0, . . . ,0) than a statistical risk (l/n, l/n, . . . , l/n). It follows that the preference for risk equity and the risk prone utility function for fatalities are appropriate, given the basic model assumptions.

Evaluating Circumstances with Multiple Causes of Risk In the literature (see, for instance, Starr, 1965); and Slovic et al., 1977), one finds that the undesirability of various risks as perceived by the public seems t o be dependent on many factors. Such factors include whether the risks are voluntary or involuntary, whether they are associated with catastrophic accidents or not, and whether the risks result in immediate or delayed fatalities. To the extent that these factors matter, the evaluation of risks needs t o take them into account. However, it seems that if one is willing to utilize an individual’s relative values for evaluating risks to themselves, the problem might appear differently.

Observation 6. Concerning risks to oneself or loved ones, many individuals do not seem to differentiate between different causes of risk.

34

R.L. Keeney

Although I have not conducted any formal tests to support this observation, I have questioned a number of individuals t o lend some insight about its appropriateness. For example, it has often been suggested that more funds should be expended t o reduce risks from possible nuclear accidents than to reduce risks of the same magnitude from other types of accidents. In this regard I've asked many individuals whether they would prefer option A or option B. Option A is a 50-SO lottery involving an individual in a nuclear accident or not on the following day. If involved in the nuclear accident, painless death immediately results. If not involved, life goes on as usual. Option B involves a S l % chance of being in an ;iutomobile accident and a 49% chance of not being involved on the followriig day. If involved in the automobile accident, the individual will immediately die in a painless manner. If not involved in the accident, life will go on as usual. Almost every individual to whom I have posed this question prefers option A for themselves or for their loved ones. If the likelihood of the automobile accident in option B is reduced to 49%, all of the individuals prefer option B. This is a rather strong indication that an individual is indifferent t o dying in a nuclear or an automobile accident, even thought the nuclear accident has all the "undesirable characteristics" such as involuntary risk associated with it. 1 have posed the same types of questions t o several individuals involving circumstances other than nuclear accidents, where one of the types of death might be categorized "a good kind" and the other type as "a bad kind". In all cases, the results seem to be the same, implying that the individual concerned wishes t o maximize his or her likelihood of surviving past tomorrow unharmed. Thus, if an institution wishes to utilize an individual's own values for evaluating personal risk, it may be appropriate to evaluate risks in the same manner regardless of the cause or circumstances associated with that mortality risk. Typically, individuals face risks from multiple independent sources. Thus, if an individual has risks ql from source 1, q2 from source 2 , . . . , and risks q,, from source in, we might say that the individual risk profile is [ q l . 42 . . . , q m ] . With the independence assumption, the total risk t o the individual is then m

q=l -n(lj=1

By utilizing the same types of questions as discussed above involving multiple risks, it seems as if many individuals prefer circumstances which maximize their likelihood of unharmed survival.

EVALUATION OF MORTALITY RISKS

35

Observation 7. If an individual wishes to maximize his or her likelihood of survival, risk profiles with a smaller total risk should be preferred. If an institution cared to adopt such a preference for evaluating risks from several different sources, it should evaluate alternatives to reduce the overall mortality risks to an individual rather than to reduce risks from specific causes. If government programs involving safety to citizens were consistent with such a policy, funds should be expended based on the numerical risks only and not on the "type" of risk.

Implementation of Utility Analysis A general comment that is sometimes heard about "value of life" literature is "Does anybody ever expect to use that?" It's a reasonable question. And from a prescriptive viewpoint, the answer is most certainly yes. However, one must be careful about the meaning of "use". Numerous programs are aimed at reducing mortality risks (or causing mortality risks as an undesirable effect associated with a perceived beneficial action). T o responsibly evaluate decisions on such programs, it is essential to evaluate the various mortality risk vectors. The evaluation might not be conducted in such a formal manner, but one must have some concept of what these vectors will be and make value judgments to appraise them. The choice is whether to make the assessments of the vectors and value judgments formally or informally. Given the complexity of the problen,, I think it is appropriate to utilize both procedures. The formal procedure is utilized as a device to assist one in understanding and examining various value structures and their appropriateness for institutional policy regarding mortality risks. The purpose of the models and the methodology is to provide critical insights and guidance for action. They should not be a substitute for the decision makers. Use of the models requires the careful consideration of several value judgments. First, one must consider the general structure of the problem. There are implicit value judgments built into this structure as they are built into any problem structure. Second, one needs to appraise the value judgments necessary t o structure the utility function u. For illustration in this paper, we assumed by (1) that the utility function for mortality risks and costs was additive. This assumption could easily be relaxed and all the ideas in the paper would still be valid. The assumptions leading to functional forms for the mortality risk utility function need to be appraised for any specific use. Finally, even if a particular functional form is found to be 3*

36

R.L. Keeney

appropriate, parameters for that utility function must be assessed. These would require the decision makers t o make a few specific value judgments t o identify pairs of indifferent consequences. For these difficult judgments, it is best to ask redundant questions from different perspectives. Inconsistencies will no doubt be observed. However, this is part of the motivation for the process, t o identify the inconsistencies and bring about an internal reconciliation after decision makers understand where and why these inconsistencies occurred. In Keeney ( 1 980a), the assessment of sotne parameters for utility functions involving public risks is illustrated.

Summary and Conclusion There are many approaches which have been suggested for evaluating risks t o life. One of the explanations for the large number of approaches is the fact that there are a large number of distinct generic problems involving risks t o life. It seems as if different approaches t o evaluating such risks would be appropriate for the different circumstances. To date, many approaches involve a "value of life" derived from situations where decisions were not necessarily made in a consistent manner or from situations where only risks t o single individuals were traded off against costs. These were the basic building blocks for models involving risks t o several individuals. These models based on insurance, willingness-to-pay, and court awards seem inappropriate for de terniining prescriptive policy for institutional evaluation of mortality risks. As an alternative, we structure the risk problem using explicit value judgments and examine their implications on the evaluation of mortality risks. I t is evident from observations 2 through 4 that three seemingly reasonable assumptions lead to three incompatible utility functions for mortality risks. Namely, preference t o minimize the expected number of fatalities leads t o the linear utility function for fatalities, preference for risk equity leads t o a risk prone utility function for fatalities, and preference for catastrophe avoidance leads t o a risk averse utility function for fatalities. The important implication of these results is that three desirable objectives are t o some degree mutually inconsistent. One might ask t o what extent is this inconsistency due t o the formalization of the problem using lotteries over mortality risk vectors (p, , pz , . . . , p,) or to the specific definitions of equity and catastrophe avoidance. Unless one is willing t o conclude that this formulation and these definitions are all completely irrelevant to the problem of evaluating mortality risks, the inconsistency holds. If additional attributes are deemed appropriate

EVALUAHON OF MORTALITY RISKS

37

for characterizing mortality risks or if additional concepts of equity and catastrophe avoidance seem reasonable, the degree (but not the fact) of the inconsistency may diminish. The implications of this paper provide some interesting insights, but in no way can be considered a solution to the general problem. The insights, however, do indicate the value of pursuing similar research to determine reasonable institutional evaluation structures. The search for such evaluation schemes is currently being pursued actively by U.S. governmental organizations, such as the Advisory Committee on Reactor Safeguards (1980). The arguments for a systematic approach t o assist with the evaluation of mortality risks are compelling (see, for instance, Okrent, 1980). With better allocation of the available funds, we are likely to be able t o reduce the risks to the public at large in a manner felt t o be appropriate and responsive to the problem.

References Acton. J . P., 1973. Evaluating a public program t o save lives: The case of heart attacks. Santa Monica. California: Rand Corporation Report R-950-RC. Advisory Committee on Reactor Safeguards, 1980. An approach to quantitative safety goals for nuclear power plants. Washington, D.C.: U. S. Nuclear Regulatory Commission, NUREG -0739. Ferreira, J., Jr. and L. Slesin, 1976. Observations on the social impact of large accidents. Cambridge, Massachusetts: Operations Research Center, Massachusetts Institute of Technology, Technical Report No. 122. Howard, R. A., 1979. Life and death decision analysis. 1n:Lawrence Symposium on Systems and Decision Sciences. North Hollywood, California: Western Periodicals. Jones-Lee, M., 1974. The value of changes in the probability of death or injury. Journal of Political Economy, 99, 835-49. Keeney, R. L., 1980a.Evaluating alternatives involving potential fatalities. Operations Research, 28, 188-205. Keeney, R. L., 1980b. Equity and public risk. Operations Research, 28, 527-534. Keeney, R. L., 1980c. Utility functions for equity and public risk. Management Science, 26, 345 -35 3. Linnerooth, J., 1975. A critique of recent modeling efforts t o determine the value of human life. Laxenburg, Austria: International Institute for Applied Systems Analysis, Research Memorandum RM-75 -67. Linnerooth, J., 1979. The value of human life: A review of the model. Economic Inquiry, 17, 52-74. Okrent, D., 1980. Comment on societal risk. Science, 208, 372-375. Savage, L. J., 1954. The Foundations of Statistics. New York: Wiley.

38

R.L. Keeney

Slovic, P., B. Fischhoff, and S. Lichtenstein, 1977. Risk assessment: Basic issues. In: R. W . Kates (ed.), Managing Technological Hazards: Research Needs and Opportunities. Boulder, Colorado: Institute of Behavioral Science, University of Colorado. Stan, C., 1969. Social benefits versus technological risk. Science, 165, 1232-1238. von Neumann J . and 0. Morgenstern, 1947. Theory of Games and Economic Behavior. Princeton, New Jersey: Princeton University Press, 2nd ed. Zeckhauser, R., 1975. Procedures for evaluating lives. Public Policy, 23, 419--464.

THE ROLE OF RISK ASSESSMENT IN A POLITICAL DECISION PROCESS' John LATHROP and Joanne LINNEROOTH' International Institute f o r Applied System Analysis, Laxenburg. Austria

Abstract In this paper, we examine the role risk assessments played in a political decision process: the siting of an LNG facility on the California coast. We find that the political process, where the decisions are made sequentially, bears little resemblance to the analyst's perspective, where objectives are traded off under conditions of uncertainty. A detailed comparison of three risk assessments used in this sequential process reveals that there are many degrees of freedom left to the analyst's judgment, and the results of an assessment may be determined as much by this judgment as by the site and technology considered. In addition, the effectiveness of a risk assessment is shown t o depend not only on its analytic rigor, but on the persuasiveness of its presentation. In order to improve the use of risk assessments in setting public policies, we suggest that rules of evidence, or standards to which risk assessments must adhere in order to be admissible evidence, be considered.

1. Introduction

How did that get there? This is a question that might come to one's lips when driving along a beautiful section of the California coastline, spoiled, suddenly, by a number of large storage tanks. The analytically-minded person might suppose that this "place" has become a "site" only after an elaborate screening process, where careful tradeoffs have been made between the likes of spoiling his view and other socioeconomic-technical concerns. The politically-minded person, alternatively, might suppose that organizational thinking and political game-playing were more important factors in the siting decision. The research reported in this paper is supported by the Bundesministerium fur Forschung und Technologie, F.R.G., contract No. 321/7591/RGB 8001. While support for this work is gratefully acknowledged, the views expressed are the authors' own and are not necessarily shared by the sponsor. We wish to thank Patrick Humphreys, Howard Kunreuther, and James Vaupel for helpful commentson earlier drafts. The author's names are listed in alphabetical order.

40

J. Lathrop and J. Linnerooth

We shall begin this paper by contrasting these two Weltansrhauungen of the siting problem. The analyst's single decision maker who balances the welfare and concerns of those affected by his actions does not coincide with the reality of many conflicting parties who interact in a process that resolves the large problem sequentially, where early-on decisions tend to constrain the alternatives open for the next decision, and so on. The sequential and adversary nature of the process are important factors determining the role that formal analyses can play in influencing the decision outcome. In this paper, we will demonstrate the ways in which risk analyses have been used in a controversial siting issue, the siting of an LNG terminal in California. The conflicting and contradictory results of these studies, we will suggest, is a predictable and important element of the political debate. Not unlike many other areas of scientific investigation, it is difficult, if not impossible, to arrive at indisputable scientific truths especially where the data are scarce and subjective. Yet, because the risk studies are highly quantitative, imitating in some sense technical, engineering studies, they generate false expectations regarding the conclusiveness of the results. While it would be desirable for risk assessments to provide conclusive and indisputable facts, this is an ideal that can never be fully achieved. For this reason, risk analyses should be regarded as introducing necessarily ambiguous evidence into the policy process. Viewing the results of a study as "evidence" instead of "facts" offers a more realistic perspective for improving the uses of these studies, and for improving the studies themselves. The intent of this paper is to describe the results, interpretations, and uses of three risk studies prepared during the course of the attempted siting of an LNG terminal in Oxnard, California. The decision process is briefly presented in Section 11, and the three studies are described in the context of this process in Section 111. In the final section, we draw some tentative conclusions regarding an improved role of technical analyses in aiding or improving siting decisions.

11. Siting an LNG Terminal in California

A . Nature of the Risk Methane, or natural gas, becomes a liquid when cooled to -163 O C , with a density more than 600 times that of its gaseous phase. Liquefied natural gas (LNG) can be economically transported over long ocean distances; the economies of scale lead t o large ships (e.g., 130,000 m3 LNG) and large onshore storage tanks (e.g., 77,500 m3 LNG each) for a base load opera-

RISK ASSESSMENT IN A POLITICAL DECISION PROCESS

41

tion such as the one proposed for California. In the event of a ship or terminal accident, a significant amount of LNG could be spilled, whch would "boil of? into a methane cloud possibly covering a sizeable area before igniting and burning. Since there exist almost no data concerning large LNG spills, and since the dispersion characteristics of methane clouds are poorly understood, there is a great deal of uncertainty involved in predicting accident consequences. Yet, the present state of knowledge indicates that at some very low probability an LNG accident could result in a cloud covering several miles before igniting. Depending on the population density of the area covered by the cloud, the possibility exists, albeit at a low probability, for a catastrophic accident.

B. The Siting Problem from the Perspective of the Decision Analyst The general purpose of a risk assessment is to estimate the probability and consequences of the range of possible accidents. It is important to distinguish a risk analysis, which is intended to provide specific data on the safety of the operation, from a decision analysis, the intent of which is to aid an interested party in making a choice or taking a stand. A risk analysis supplies, but does not evaluate, one piece of the information needed to make an informed decision on the selection of a site. The decision analyst might view the problem as consisting of two decisions: whether or not to import LNG and if so, where to site the plant. Both decisions involve different tradeoffs. The importation of LNG would reduce the risk of a shortage of natural gas and would improve air quality (due to an increased use of a clean-burning fuel). Yet these benefits would come at a financial cost (LNG is an expensive form of natural gas), an environmental cost (a large facility on the coast), and a cost in terms of population safety. Siting the plant at a remote and beautiful part of the coast reduces the population risk relative to siting the plant in a port, but increases environmental degradation and financial cost. Because of the uncertainty surrounding estimates of population risk, as well as estimates of the risk of a shortage in natural gas, in a decision-analytic sense the "whether" and "where" decisions involve the trading off under uncertainty of natural gas shortage risk, air quality, environmental degradation, financial cost, and population risk. From this perspective, estimates of population risk take their place among the myriad of uncertain inputs to the decision. Yet, it may come as no surprise that the actual political siting process bears little resemblance

42

J . Lathrop and J . Linnerooth

to the decision-analytic framework just described. As we will show in this section, the sequential nature of the process and the multiple parties give risk analyses a special status.

C. A Description of the Decision Process

In the late 1960s, faced with projections of decreasing natural gas supplies and increasing need, California gas utilities began to seek additional supplies. In 1974, Western LNG Terminal Company (Western), which was formed t o represent the LNG interests of the gas utilities, applied for approval of three LNC import sites on the California coast: Point Conception (PC),located on a remote and attractive part of the coast; Oxnard (X),a port city; and Los Angeles (LA), a large harbor metropolis. The LNC would be shipped from Southern Alaska, Alaska’s North Slope, and Indonesia. As of this writing, Point Conception, the one site remaining under active consideration, is still pending approval. This section describes the procedures, decisions, and events of this lengthy process (for a more complete review see Linnerooth, 1980; and Lathrop, 1980). Though much preliminary work had been done by the California utilities in negotiating a contract with Indonesia and in preselecting possible sites, for our purposes the process began with Western’s application for approval of the three sites. This marked the beginning of the fourround process as shown in Table 1, where each round can be characterized by a problem formulation as perceived by most if not all of the interested par tie^,^ by an event (proposal, request, etc.) initiating the discussions, and by a decision(s) or nondecision concluding the round (for a more detailed description of this characterization see Kunreuther et al., 1981 ). A routine process for approving industrial facilities existed at the time of Western’s applications. This siting procedure was, however, coniplex, involving three levels of government. The Federal Power Commission was responsible for assessing national need as well as environmental impact; the local authorities were required to grant the various licenses for land use, access, and so forth, and the California Coastal Commission (CCC) was mandated to give the final approval for any facility on the California coastline. As the application progressed through the approval channels, it became increasingly apparent that these routine procedures, especially on the local level, were ill suited to handle a large-scale facility having the potential for a catastrophic accident. The mismatch between 3This does not preclude the possibility that some parties might object to this formulation and challenge it during the course of the debate.

RISK ASSESSMENT IN A POLITICAL DECISION PROCESS

43

Table 1. Summary of Rounds in the California LNG Siting Case

ROUND I

DA TE

Problem Formulation:

Should the proposed sites be approved? That is: Does California need LNG, and if so, which, if any, of the proposed sites is appropriate?

Initiating Event :

Applicant files for the approval of three sites.

September 1974 (34 months)

Conclusion:

Applicant perceives that no site is approvable without a long delay.

July 1977

ROUND 2 Problem Formulation:

How should need for LNG be determined? If need is established, how should an LNG facility be sited?

Initiating Event:

Applicant and others put pressure on the State Legislature to facilitate LNG siting.

July 1977 (2 months)

Conclusion:

A new siting process is established

September 1977

that assumes a need for LNG, which is designed to accelerate LNG terminal siting.

ROUND 3 Problem Formulation:

Which site is appropriate?

Initiating Event:

Applicant files for approval of Point Conception site.

October 1977 (10 months)

Conclusion:

Site approved conditional on consideration of additional seismic risk data.

July 1978

ROUND 4 Problem Formulation:

Is Point Conception seismically safe?

Initiating Event:

Regulatory agencies set up procedures to consider additional seismic risk data.

Conclusion:

(Round still in progress)

1980

44

J. Latlirop and J . Linnerooth

the scale of the project and the proccdures designed t o approve it was aggravated by the novelty of the technology. The risks were ill defined, the experts were not in agreement on the possible consequences of a spill, and there were iio standard operating procedures or regulations for the technology. From the point of view of formal risk assessments, the lirst round of the California siting process was the most interesting. To support its applications t o the Federal Power Commission, Western was required t o submit an analysis of the safety of the facilities and their operations. For this purpose, it contracted with a consulting firm (Science Applications, lnc. /SAI/). As required by State Law, the local governments involved were required to submit an Environmental Impact Report (EIR); of most interest to us here was the Oxnard study which was also submitted by a consulting firm (Socio-Economic Systems /SES/). As will be shown in later sections, the Oxnard study incited a great deal o f public resistance t o the terminal plans. Finally, the Federal Power Commission was required t o carry out an in-house Environmental Impact Statement (EIS). Though the approval appeared t o be a routine matter, the lowprobability consequences of this large-scale operation complicated the process considerably, resulting in the stalemate (at last as perceived by Western) concluding Round One. The apparent significance of the risks of the planned facilities appear surprising in view of the low estimates of these risks assessed by Western and the FPC. But the picture began t o cloud when the FPC staff noted an earthquake fault near the Los Angles site, when several worst-case scenarios for Oxnard were published in the SES report, and with the growing resistance on the part of the CCC ant! Santa Barbara County which held approval authority for the Point Conception site. With the possibility of not receiving approval for any one of the three sites, Western turned t o the California State Legislature for help, initiating Round Two of the process. Western perceived a better chance in changing the siting process in its favor than in fighting the multipleapproval, standard process, and successfully brought pressure to bear on the State Legislature (along with business and labor) t o pass the 1977 California LNG Siting Act (Senate Bill 1081). This legislation concluded Round Two (see Table 1) which was effectively a "problem bounding" round, or a round for the purpose of narrowing the bounds of the problem t o a proportion that could be handled by the relevant institutions. The Siting Act removed the decision authority from the local agencies and the CCC and vested sole state licensing authority with the California Public Utilities Commission (CPUC), a regulator with a history of sympathy for utility capacity expansion. The act also gave the CCC the role of ranking

RISK ASSESSMENT I N A POLITICAL DECISION PROCESS

4s

alternative sites, including the applied for site, but that ranking was not binding upon the CPUC. Finally, the act required that the site be remote and onshore. The applicant's decision t o reapply for the Point Conception site under the new process initiated the third round of political negotiations; this round was formulated more narrowly than those preceding it. Essentially, the only question open for the political agenda was "which site is appropriate?" While the CCC ranked Point Conception third out of its four top-ranked sites, the CPUC selected this site for conditional approval on the grounds that the higher-ranked sites would involve excessive delays as the applicant would have t o draw up new plans. The CPUC approval was conditional on analysis o f wind and wave conditions, archeological data, and, most importantly, seismic risk. At the federal level, where both the Oxnard and Point Conception sites remained "alive", a reorganization had replaced the FPC with two agencies: the Economic Regulatory Administration (ERA) in charge of import approval and the Federal Energy Regulatory Commission (FERC) in charge of site approval. The ERA approved the Indonesia import project. The FERC staff, which carried out detailed risk studies, favored Oxnard, but the Commissioners approved Point Conception t o avoid a confrontation with the State which had legislated against the nonreniote Oxnard site. Conditional approvals on the part of the three mandated decision makers, the CPUC, the ERA, and the FERC, did not, however, resolve the siting issue. Opponents of the project petitioned a federal appellate court for a stay in the proceedings on the grounds that not all seismic risk evidence had been considered. The court has remanded the case back t o the FERC. That ruling, and the subsequent procedures t o investigate seismic risk set u p by the relevant agencies, has initiated the fourth round of discussions (see Table 1). This round is tightly formulated as a technical risk issue. The single question open for discussion o n the political agenda is whether Point Conception is seismically safe.

D . Siting Decisions as Public Policy An examination of the California siting process reveals that siting an LNG terminal is not so much a decision, in the usual sense of individual goaldirected behavior, as a policy-malung process. Majone (1981) gives three main reasons that differentiate private decisions from public policies: First, in the public domain actions must be justified with seemingly objective arguments. Second, policies, unlike individual decisions, need t o gain a consensus in order to be viable. Finally, public choices are not

46

J. Lathrop and J. Linnerooth

made by only one person. A consensus within andlor beyond an organization can be reached only with convincing and institutionally appropriate arguments. Because of this need on the part of an interested party, whether a citizens’ group or a licensing agency, t o justify its stand on the siting issue, risk assessments showing the plant t o be safe or not safe are especially useful. It is also important t o recognize that organizations often deal with current issues, not so much for their sake alone, but for their longer term implications for the institutions. Western, for instance, may have pursued a change in siting procedures (S. 1081), not so much because it perceived Oxnard to be blocked by local opposition, but because it recognized the longer term benefits of shifting the power away from the local communities by a one-stop licensing procedure. The California State Legislature did not set a policy for remote siting as an analyst would prescribe, that is, malung explicit the tradeoffs, but compromised instead among the proand the anti-oxnard interests. The Federal Energy Regulatory Commission also had t o consider longer term implications of its siting policy as was evident in its strategy not t o pursue its preference for Oxnard and thus provoke a Federal-State confrontation. Another feature of the political siting process that frustrates attempts on the part of the decision analyst t o set out clearly the tradeoffs involved is the sequential nature of the decisions. In California, resolution of the question whether a site was needed necessarily preceded the site selection phase? which in turn will precede the licensing process. Because of time and cost considerations, a decision on one level is often binding in that it cannot easily be reopened for political debate. Thus the process becomes tied or locked into certain courses of action. The responsible agencies have little alternative but t o consider increasingly narrow aspects of the problem. As a case in point, during the seven-year course of the California proceedings, the need for imported natural gas in the State diminished greatly.’ Instead of reexamining this need, the process is presently locked into a commitment for an import facility. All efforts are now directed toward pursuing the narrower problem of seismic risk a t Point Conception. A perhaps more important case in terms of the long-run implications is the remote siting constraint set by the California Legislature which has effectively ruled out the possibility of trading off population risk for other ecocomic or environmental benefits. 41n the first round, the questions of need and site were considered simultaneously. This, however, did not lead to a decision on site. In the second round, the State Legislature effectively resolved the need question. 5 Gas prices were deregulated during this time which increased the domestic supply of natural gas.

41

RISK ASSESSMENT IN A POLITICAL DECISION PROCESS

111. The Oxnard Risk Assessments

During the course of events in the California LNG terminal siting debate, there were seven major risk assessments carried out for the three prospective sites: Los Angeles, Oxnard and Point Conception. To understand the role these assessments played in the process, as well as in the outcome of the debate, it is instructive to review their content and use. An important point of this paper is to demonstrate that the content of such a study is largely determined by the use of the study in the political debate. It is only with an understanding of the latter that recommendations can be made for improving the former. For the sake of brevity, and with no loss in generality, we will limit our discussion to the early studies concerning the Oxnard site. These studies, the Science Applications, lnc., risk assessment (SAI, 1975), the Federal Power Commission risk assessment (FPC, 1976), and the SocioEconomic Systems risk assessment (SES, 1976) will be discussed in turn.

A . A n Overview 1. Science Applications, Inc., Risk Assessment

As part of its case for the Federal Power Commission, the applicant commissioned. a consulting firm, Science Applications, lnc. (SAI), to do a risk assessment of its proposed Oxnard terminal. This assessment was elaborate, involving calculations of probabtlities of vessel accidents, tank ruptures, LNG spill sizes, methane cloud dispersion and ignition, and the resulting fatalities. The computer model developed for cloud hspersion was deemed one of the two best in a Coast Guard review of several models (Havens, 1977). Ship collision calculations also involved a computer model, calibrated to statistics from several harbors. The SAI results were presented in the form of several different indices of risk. Individual annual probabilities of fatality due to the terminal were presented in the format of iso-probability contour maps of the site (see Figure 1). Those probabilities ranged from a maximum of 1.5-lo-' near the terminal to less than lo-'' beyond three miles for the most conservative (risk-overstating) set of assumptions. Other contour maps were presented for less conservative assumptions. The maximum individual probability of LNG fatality was compared to other risks. The individual probability of dying in a fire generally was reported as 220 times greater; the maximum probability of having a plane fall on a person in the site vicinity was reported as 10 times greater than the LNG risk. b

48

J. Lathrop and J. Linnerooth

0

1

2

--SeJ. (km)

Figure I . Iso-Probability Contour Map for Oxnard, Most Conservative Assumptions. Source: SAI (1972)

Annual probabilities of catastrophes were also presented, including lo-' for a 2,000 to 10,000 fatality year, and 1.4 lo-", or "one chance in 710 septendecillion", for the maximum catastrophe: 1 13,OO fatalities. The study concluded that LNG risks at the Oxnard site were "extremely low". The results of the SAI study seem to have been persuasive at the FPC hearing. The FPC decision of July, 1977, cited all the various numbers mentioned above and a few more, noted the conservative assumptions, pointed out that no party disputed the findings, and found that the Oxnard site involved levels of risk sufficiently low for FPC approval. While the SAI ztudy seems to have been effective in its intended role before the FPC', as events transpired, it did not have any bearing on the siting process, as shortly after the FPC decision a federal reorganization abolished the FPC and set up a new approval procedure.

2 . Federal Power Commission Staff Risk Assessment The staff of the FPC also carried out a risk assessment as part of the Environmental Impact Statement (EIS) to be presented to the Commission in the July, 1977, hearing. This assessment generally used less elaborate models and less resources than SAI in reaching its conclusions. The logic of the report can be stated quite simply: All significant risks were seen as arising from ship accidents. While this is plausible on technical grounds, the assessment did not defend this assumption with analysis. Those accidents were assumed to happen at least as far from shore as the

RISK ASSESSMENT IN A POLITICAL DECISION PROCESS

49

end of the 6,000 ft (1.8 km) trestle of the Oxnard facility. Since the FPC staff determined that the maximum travel of the flammable vapor cloud and maximum distance of significant fire radiation effects were both less than 6,000 feet, the risk was deemed to be "negligible". The FPC assessment results included risk measures for the Point Conception and Los Angeles sites. In all three cases, risk was measured by two indices: annual expected fatalities and annual individual probability of LNG fatality. However, for the reasons discussed above, no numbers were given for those indices for Oxnard, only the abbreviation for "negligible". The report concluded that ship transport to the Oxnard site "constitute[s] an acceptable risk to the public". As with the SAI study, the results of the FPC staff assessment seem to have been accepted and persuasive at the FPC hearing. The decision of July, 1977, cites both the FPC and SAI results in support of its conclusions already discussed.

3. Socio-Economic Systems Risk Assessment As part of its Environmental Impact Report, the city of Oxnard commissioned a consulting firm, Socio-Economic Systems, Inc. (SES), to carry out a risk assessment of the LNG terminal. This assessment took a broader look at the problem than the previous two assessments. Rather than characterize the risk solely in probabilistic terms, the report presented 26 "population risk scenario;, with maps of the Oxnard area with shaded maximum plume areas or fire radiation zones superimposed, for each of several wind directions, spill sizes, etc. (see Figure 2). Each scenario named a "population risk", in fact the number of people covered by the maximum plume or fire zone, which ranged from 0 to 70,000. These scenarios could be described as maximum credible accidents (though SES did not do so). They were not accompanied by any estimates of their probabilities, which would have been quite low. In the section immediately following the scenarios, the SES report presented a more probabilistic analysis, which in fact combined the most conservative numbers and assumptions from the SAI and FPC studies as well as a Coast Guard study. In tabulating these, the report pointed out wide differences in numbers used in different studies. For example, the FPC used a probability of ship collision more than 5,600 times larger than the one used by SAI. The number of expected fatalities per year computed by SES in this way was 5.74, or 380 times larger than the SAI estimate. These numbers (SES and SAI estimates) were compared with expected fatalities from other hazards. While the SAI report estimates 4

50

J. Lathrop and J . Linnerooth

-

a a

LEGEND 0

..... f: ...... .:,:.. . . .

= = = = =

500PEOPLE CENSUS TRACT NUMBER HYPOTHETICAL VAPOR CLOUD PATH POSSIBLE IGNITION SOURCE PROBABLE IGNITION SOURCE IDWELLINGS) = SHIPCOURSE

g'

*

--

Figure 2. SES Population Risk Scenario for Oxnard. Source: SES (1976)

RISK ASSESSMENT IN A POLITICAL DECISION PROCESS

51

LNG to have 7 times more expected fatalities than a hypothetical Oxnard nuclear plant, the SES report estimates LNG to have 2,900 times more expected fatalities. The SES report plotted annual probabilities of catastrophes against the numbers of fatalities involved, for both its and the SAI estimates, and compared these plots with other types of hazards (see Figure 3). Once

FIGURE 39

RISK COMPARISON, LNG PROJECT

AND OTHER NATURAL AND MAN-MADE HAZARDS

...... ............ ...... ...... ...... ............ ...... .....

Lagand RANGE OF UNCERTAINTY, OXNARO LNG FACILITY RISK ASSESSMENTS

'US. Nudaar Rquletory Commmon. "Ramor Salaty Study" Wuh.1400. October 1075

'.SCianca Applications. Inc. "LNG T a m i n d 11111: A-ant lor O x m d , Cxlifornia." O w m b r r 1976

Figure 3. Probabilities vs. Size of Catastrophes. Source: SES (1976)

4*

52

J. Lathrop and J. Linnerooth

again, the SAl estimates for LNG were higher than the numbers for a nuclear plant, while the SES estimates were much higher still. In marked contrast to the other two assessments, the SES study concluded that in view of the problems of estimating risks with very little experience base ff. and the differences in risk estimates between reports, it is not now possible to state confidently that the proposed facility poses a ’low probability’ of a high consequence accidentf’. As it happened, the SES report was never directly used because the California LNG siting process was changed by new legislation in 1977. which ruled out non-remote sites such as Oxnard. However, the SES report may have been influential in indirect ways. The population risk scenarios, which allowed local residents t o see a deadly methane plume covering their own homes, in Ahern’s (1980) words “electrified opposition to the terminal”. In addition, the generally cautious tone of the report may have increased the sense of caution and dampened support for the terminal in the City Council. The report seems to have increased opposition to the terminal, opposition which led eventually to the remote siting provision of the 1977 legislation.6

B. Comparing the Assessments In this section, key features of each of the three assessments are selected for comparison. Table 2 shows these features in summary form.

1 . Use

As indicated in Table 2 , each assessment was used differently in the siting process. The SAI study was used to defend Western’s application. In the other two cases, the assessments could be seen as having advisory roles: the FPC study was used by the staff to advise the commissioners, and the SES study was part of an environmental impact report (EIR) with the purpose of providing a data base for all parties to the process.

As late as 1980 one of the authors was told by a state legislative aide that the statc could not site a plant that could kill 40,000 people. The fatalities described without an associated probability coincides (in format and number) with one of the scenarios in the SES report.

RISK ASSESSMENT IN A POLITICAL DECISION PROCESS

53

Table 2. Summary Comparison of Risk Assessments SAI

FPC Staff

SES

use:

support applicant stand to regulator

upport staff stanc t o commission

support SES stand to all parties

Scope:

thorough, but own models only

mly ship accidcnt at end of trestle

composite of several studies

Assessor

Example Parameters:' probability of ship accident

5.6

cloud distance 25,000 m 3

*

1U'

3.1cr2

1.2 k m

23 km

3

1U6

2 km

*

Example Results:' maximum individual probability (pi) catastrophe size: prob.

1.5

"negligible" 4,000:1 lp (note2)

2,000-1 0,000 J F 8 11 3 , 0 0 0 : 1 ~ s 7

expected fatalities

Formats:

. 1u7

' )max plume maps

iso-pi contour maps (see Figure 1 )

(see Figurc 2) *)catastrophe size vs probability (comparative) (see Figure 3)

(besides tables)

Conchsion :

Effect:

'

risks ... are extremely low." I'...

FPC persuaded to approve

5.74

"negligible"

,015

"

... risks .. arc

... negligible."

" ... an acceptable risk to the public."

FPC persuaded t o approve

... it is not now possible t o state confidently that (it) poses a 'low probability' of a highonsequence accident." "

increased opposition dccreased support

Several differing qualifying conditions apply t o each number, so data are only appropriate for very rough comparisons. All probabilities and expected values are annual. This figure was scaled off of a low-resolution figure, and so is quite approximate.

54

J . Lathrop and J. Linnerooth

2 . Scope

The analysts for each assessment adopted widely varying assumptions, problem scopes, which explain many of the fundamental differences in the results and effects of the assessments. The SAI report was quite thorough, but used primarily in-house computer models for the important calculations concerning ship accidents and flammable cloud travel distances. While those models were impressive, by neglecting to acknowledge the existence of experts and models with conflicting results, the SAI report over-stated the confidence with which their results should be accepted. The FPC report considered only one type of accident: ship accidents at the end of the trestle. There were reasons for believing that other accidents would not add significantly to the risk. However, because the staff did not carry out calculations to prove that contention, that important narrowing of the problem is defended only by the analysts’judgment. Other parties to the process are then apt to suspect that the risk is understated. The SES study combined widely varying results of several other studies. In doing 90, the analysts were able to make abundantly clear the extent of uncertainty in expert opinions on LNG safety.

3 . Parameters As shown in Table 2 the critical parameters used in the risk calculations differed by from one to four orders of magnitude among the assessments. This makes clear the large uncertainties in elements of the assessments, uncertainties which contributed to the grossly differing conclusions and effects of the assessments. Clearly, such uncertainties should be reflected in the assessment results. Yet, the uncertainties were not addressed in the SAI report, and while the FPC report acknowledged them, they were not presented as qualifying factors for the results. (A later report on Oxnard by the Federal Energy Regulatory Commission, successor t o the FPC, took some of the uncertainties into account.) If each risk assessment was in fact a probabilistic representation of the available technical knowledge concerning LNG risk, as may have reasonably been assumed by a reader, then each assessment should have acknowledged the existence of conflicting opinions about physical processes important t o LNG risk. This is an especially important matter for low-probability catastrophic risks such as LNG, since the very low reported probabilities of catastrophe could be meaningless given conflicting expert opinions. For example, how meaning ful is a probability of of a catastrophe if there is a high probability

RISK ASSESSMENT IN A POLITICAL DECISION PROCESS

55

that the cloud dispersion model on which that lo-’ was based is so erroneous that the actual catastrophe probability is more than (see Mandl and Lathrop, 1981).

4. Results and Formats It is instructive to ask what is being assessed by a risk assessment. As shown in Table 2, not only do the assessments differ by up to two orders of magnitude on the measurements of risk, they differ more fundamentally on how risk is characterized, that is, the dimensions used to describe risk and the formats used to present the results. The maximum plume maps of the SES report (essentially maximum credible accidents) were terrifying in comparison to the more reassuring message of the other two reports. Underlying the problem of selecting an appropriate format to communicate the results of a risk assessment is the fact that there is no objective risk associated with a novel and complex technology. Beyond the problems of relying on subjective probabilities where no frequentistic data base is available, risk itself is a many-dimensioned concept, with no apparent consensus on how those dimensions should be combined into a (1980) point out, social risk measure. In fact, as von Winterfeldt and RIOS groups may differ in their values and beliefs about technological risks in ways related to more general value orientation. It would seem, then, that it is important to define clearly the assessed risk. Yet, there is an odd lack of precision in many risk assessments as to just what it is that is being assessed. A survey of eighteen risk assessments found that only four included an explicit definition of risk, and those four differed greatly (Mandl and Lathrop, 1981). It appears that risk assessments are commissioned wilhout any specification of what it is that is to be assessed. Operating in that vacuum, each risk assessment team sets out to characterize risk in whatever way it sees fit. Not only do risk assessments differ on what they are assessing, but also risk assessments of an identical physical plant, compared on the same measure, are often quite dissimilar. Compare, for example, estimates of estimated annual expected fatalities for the Oxnard facility. The SAI report estimated ,015;the SES report estimated 5.74, or 380 times the SAI number (which puts into question the two and three significant figures of the assessments). Underlying this lack of agreement is the fact that these probabilistic assessments are not entirely probabilistic. Gaps in the probabilistic models are filled by assumptions that often are not probabilistic in the sense of certainty-equivalent representations of incomplete knowledge. As a case in point, consider estimates of the probability

56

J . Lathrop and J . Linnerooth

of ignition at the spill site in the event of an LNG ship collision. This estimate was not generated from a computer model, but was set by expert judgment. The number selected was not an expected value, modal value, or certainty equivalent, but was typically a conservative value, i.e., a number that would overstate the risk with some unstated confidence. Such an assessment is more defensible; the final measure is not apt to understate the risk, since all gaps in the model are filled by risk-overstating numbers. But with this procedure the expected fatalities measure is not, in fact, expected fatalities, but some odd mix of maximum and expected fatalities. In order t o calculate actual expected fatalities, the uncertainty in underlying assumptions must be explicitly modeled by assessing a probability distribution over the various sets of assumptions that could be used in the analysis, and taking the expectation of analysis results over that probability distribution. That approach would not only yield more meaningful expectations, but would also provide the basis for analyses t o assess the value of research to improve the data base used for risk assessment. The conservatism described above has three important effects. First, the range of possible conservative assumptions contributes t o differences among assessments. I n this respect, the SES study simply compared the SAI, the FPC, and the U.S. Coast Guard studies, and adopted the most conservative assumptions from each to reach its 5.74 expected fatalities, 380 times the S A I estimate. Second, conservatism blurs the distinction between probabilistic and non-probabilistic measures. The population risk scenarios of the SES study, which are similar t o maximum credible accidents, seem non-probabilistic. Yet, since each was generated by making many conservative assumptions, they are not in fact qualitatively different from the seemingly more probabilistic expected fatality measures. Finally, by ruling out the possibility of modeling the uncertainty (usually by modeling uncertain quantity as several levels of conservativeness), the possibility of using sensitivity analyses to examine the importance or redundancy of improving the modeling is also ruled out. In this way, a major decision aiding possibility of decision analysis is precluded.

5. Conclusions One of the most striking contrasts aniong the assessments is found by comparing their verbal conclusions, as summarized in Table 2 . It appears that the SES study could hardly be describing the same site and technology as the SAI and FPC studies. Yet with the exception of the FPC "acceptable risk" statement, in some sense the conclusions in Table 2 are correct and

RISK ASSESSMENT IN A POLITICAL DECISION PROCESS

57

even consistent. Because the SAI and FPC statements are prefaced by "it is the opinion of . . . that . . and there is n o description of how confident the analysts are of the results, they do not conflict with the phrasing of the SES report, "it is not . . . possible to state confidently . . .". But, of course, confidence in results is an essential issue here, as has been discussed in preceding paragraphs. The "acceptable risk" opinion of the FPC staff raises the issue of how broad the mandate of the risk assessors should be. The other assessments only assess the level of risk, or the "facts" of the situation, and not the acceptability of the risk which naturally brings values into the assessment in a more direct way. Several authors have made the point that the acceptability of a risk is an essentially political question, beyond the legitimate mandate of technical risk assessors (see, e.g., Fischhoff et al., 1981). .f'

6 . Effect As shown in Table 2 , there were two markedly different effects produced

by the three assessments, persuading authorities of the safety of the terminal and increasing public opposition t o the terminal. The opposition following publication of the SES study could be attributed to its larger measures of risk. However, according to a key participant in the process (Ahern, 1980), the more important contribution was made by the maximum plume map formats used by SES and the great deal of uncertainty acknowledged in the SES conclusions.

7. Other Factors

Perhaps one of the most important factors in all three of the risk assessments is the highly conditional nature of the results. Yet, without exception, this factor is not clearly presented as a qualifier in their conclusions. For instance, the assessments do not take sabotage or terrorist action into account. According to the FPC decision, SAI maintained that such risks cannot be quantified. Yet the appropriate response to that problem is either to use direct subjective judgment (as was used for ignition probabilities) or to make clear in the presentation of the results that such events are omitted. To do neither has the effect of understating, in contrast to the conservatism discussed earlier, the risks of the terminal.

58

J. Lathrop and J. Linnerooth

8. In Sum In reviewing the differences among the assessments, it becomes clear that there is a large degree of flexibility or freedom left t o engineering and analytic judgment. Among those degrees of freedom are: decisions concerning how t o characterize risk, what formats to use for presentation, what gaps to fill with assumptions of what degree of conservatism, which of several conflicting models t o use, how to portray the degree of confidence in the results, and what contingencies simply t o leave out of the analysis. This leeway explains the differences among the three risk assessments examined here. It can push the risk measurement results in any direction. Very conservative assumptions can drive it up; omissions of inconvenient aspects such as terrorism can drive it down; clear presentations of expert disagreements can decrease the confidence in the results; particular formats can feature more or less salient aspects of the risk; and so on. This flexibility on the part of the analyst can have a major effect on the results of the analysis, over- or understating the risk over such a large range that the final result may havc more t o d o with the predilections of the analyst than with any physical characteristics of the site or technology. Yet, risk assessments are generally viewed more in the realm of scientifically-determined facts than of subjectively-determined evidence, which serves only t o fog discussions on the safety of a terminal. This dual nature of subjective, probabilistic analyses, and the implications for improving the policy process, is the topic of the following sections.

C. Policy Context

In this section, we would like t o turn t o examining the assessments within the context of the policy process. Particularly, we will describe the timing of the assessments in relation to how the problems were defined on the political agenda, the purpose of the assessments in relation to these problems, and the uses of the assessments. I n the following section, we will reexaniinc the content of the reports from the perspective of their role in the policy process.

1 . Timing of the Risk Assessments The Oxnard risk assessments examined here were carried out in an early stage of the process. The problem (Round One) was defined in vague terms. Both questions. whether an LNG terminal is needed and which, if any, of

RISK ASSESSMENT IN A POLITICAL DEClSlON PROCHSS

59

the proposed sites is acceptable, were yet t o be resolved. The process, not locked into having a terminal or necessarily approving any of the sites, did not have a clearly defined direction. Many possible issues could have served as a focus or "handle" for the arguments, pro and con, any site. Since the assessments were commissioned during this unstructured and vague stage of the debate, it is not surprising that "risk" became an important focus for the discussions.'

2 . Purpose of the Risk Assessments A general purpose of each of the assessments was t o establish the accept-

ability or nonacceptability of the Oxnard terminal from the standpoint of the safety risks it would impose on the surrounding population. From an analyst's point of view, it in sttiking that the question of safety was not viewed in terms of the benefits of the facility (cost-benefit framework) or in terms of the safety of alternative energy sources. This threshold concept of safety (Is it safe enough irrespective of benefits or alternatives?) is typical of debates on the assessment of new and novel technologies, and is not surprising when seen in relation t o the political decision procedures. The assessments were not msde as an input t o a wholistic analysis, such as that described in Section 11, where tradeoffs are explicit and all alternatives are evaluated, but rather as support for an argument supporting or opposing less significant, incremental decisions a t a particular point in time. The sequential nature of the decision procedures, as clearly demonstrated by the increasing concreteness of the problem formulations through the four rounds of discussions in California, limits the possibilities for comprehensive analyses. 'The risk studies were carried out, not as an input to a broad energy siting analysis in California, but to support a more narrowly defined problem (Should site x or site y be approved?). Since Round One in California was not defined in these narrow terms (the question of whether the terminal was needed was yet t o be resolved), the analyses were ill-suited t o address fully the issues o n the table. In some sense, then, analyses designed t o address the question of safety were prematurely introduced into a process that had not resolved higher-order 7The significance of the timing of these studies is especially evident when viewed in a cross-national context. In the F.R.G., for example, the risks to the surrounding population from a proposed LNG terminal were first considered after the site, but before the design of the terminai had been approved (Atz, 1981). In the U.K., a risk assessment will be carried out for an export terminal planned in Scotland only after the terminal has been built (Macgill, 1981). It is not surprising that the safety of the plant has played a larger role in the US.than in the F.R.G. and the U.K.

60

J . Lathrop and J . Linnerooth

questions of energy policy. Though they served to focus the debate on the safety question, they could not offer (nor were they intended to offer) a panacea for the resolution of the siting question. It is not surprising, then, that Round One ended i n a stalemate. The second round, where the State Legislature took center stage, narrowed the problem (by resolving the question whether California needed a site) to a proportion more receptive of technical risk studies. Whether the site was safe, and not whether it was needed, became the critical question.

3. The Uses Made of the Risk Assessments

It might be useful to distinguish between those risk assessments commissioned (or carried out in-house) to advise the client (or advise the agency) on a course of action (pre-decision) and those commissioned t o support or rationalize a client's actions or intended actions (post-decision). The latter should not be viewed as falling outside the routine business of policy analysis. As Majone (1981) points out, the policy-making process is driven by the clash of opposing arguments and for this reason policy makers need retrospective (post-decision) analysis, including 'rationalization' as much as they need prospective (or predecision) analysis" (p. 17). Though the preand post-decision distinction is useful, the distinction was not always in the case of Oxnard a clear one. The reason is that there were, as would be expected in the dynamic process described above, several audiences over time for each of the three studies. The SAI study, as commissioned by Western, can be fairly unambiguously classified as a post-decision rationalization; the expressed purpose was to defend Western's decision to site at Oxnard at hearings held on the question by the FPC, which was in charge of approving the application. The SES report, alternatively, was commissioned for the purpose of advising the Oxnard City Council of the accident risks for the Oxnard site. As could have been anticipated, its audience expanded in time to include local interest groups, the Sierra Club, and eventually the State Legislature, and the purpose of the report was transformed from advising to rationalizing arguments against Oxnard as a suitable site. The case of the FPC is the most ambiguous. Though the report was prepared to advise the staff in t:king a position on Oxnard, it was carried out in full knowledge that it would be used to rationalize the staff's chosen position at the Commission hearings. An important point here is that all of the analyses were used, if not immediately, at some time to support a stand taken by one or more of the parties. Their functions were not in any case limited to that of solely

RISK ASSESSMENT IN A POLITICAL DECISION PROCESS

61

advisory. Each was intended to go beyond that of planning a course of action (or stand) on the part of the client to that of defending the client's position in a public setting (usually a legalistic hearing). For this reason, it was important that each report not only inform the client but make its case as persuasively as possible.

D.77ie Role of Risk Assessments The role of riskassessments can be viewed from two opposing perspectives. Because the assessments are quantitative, imitating the physical sciences, they are seen as objective facts. Yet, as we have shown in this paper, the large uncertainties involved necessarily push the evidence out of the realm of "facts" and into the realm of "opinion". This dual nature of formal risk studies has fogged discussions on their role in the policy process. I . Facts vs. Judgment

The possibility that the form or content of scientific knowledge, as distinct from its incidence or reception, might in some way be socially determined, has recently been put forth by sociologists of science. Indeed, the sociology of science as developed in Europe since the early 1970s has challenged the positivist view of science (for a review of this literature, see Mulkay, 1980). Though several authors have discussed the possible "pitfalls" of analysis, whereby values on the part of the analyst color his "methodologies" and results and whereby heuristics introduce biases into his work, Wynne (1981) reminds us of the importance of recognizing these biases as part and parcel of science and not eradicable lapses from proper rational scientific analysis. There is a pervasive myth about the nature of science which supports this false approach t o the question of "analytic bias". The tendency in the literature is to regard bias or mistakes as individual and isolated in origin, which suggests that ideal objective scientific knowledge can be attained in professional practice and as an input to policy issues. .. This gives a fundamentally misleading and politically damaging picture of the role of expertise, and may make us part of the problems we analyze (pp. 1-2). A recognition of the inevitability of intertwining facts and values leads us to examine critically recent notions of the desirability of separating information from judgment. The widely-held view of the scientist

62

J. Lathrop and J . Linnerooth

producing "objective" information and keeping "facts" and "values" in separate, airtight containers, clearly reflects the acceptance of the "standard view", of science. An opposing view, as suggested in the above quote by Wynne, is t o recognize that there is a n important element of subjective judgment in all scientific experiments, which is especially apparent in policy analysis. There is a clear need for judgment in all steps of an analysis: (a) setting or defining the problem; (b) collecting the data; (c) choosing the tools and methods of analysis; (d) presenting the evidence and formulating the arguments; and (e) drawing conclusions, coniniunicating and implementing the results. According t o Vaupel (1981), since nearly all statistics used in policy analysis not only summarize a body of data, but also imply a policy thrust, statistical analysis for policy making is fundamentally different from statistical analysis for descriptive scientific research. As has been argued by Ravetz (1971) and Majone ( l 9 8 0 ) , policy analysis is inore craft than science. Seen in this light, it would be naive to suppose that a level of risk can be estimated and accepted as fact. Scientific truths are not proved but are the product of a process of general acceptance in the scientific community. Since rational methods for discovering the scientific information t o guide policy are often inconclusive, Majone (1981) points out that often non-rational (not t o be confused with irrational) methods are used, including bargaining, voting, delegation, material incentives, and procedures. Majone suggests further that persuasion is perhaps the most important of these non-rational methods. As Vaupel(l981) has illustrated in the case of the US. Environmental Protection Agency, the importance o f using the analyst's rational methods may lie as much in justifying the EPA's stxidards and regulations for the Federal Register and the courts as for liclping the staff t o set them. In this role as justifying or rationalizing policy positions, the analyst will naturally choose the statistics and methods that present a convincing case.

2 . The Advocacy Role of Risk Assessments A review of the three Oxnard risk assessments has revealed striking differences in their "scientitic" content: the relative conservativeness of the assumptions, the completeness of the analysis, the characterization o f safety or "risk", and the formats for presenting the results. A review of the policy process has revealed that the assessments were done to persuade either the client (advisory role) or a decision-making body (rationalization) of the safety or nonsafety of the Oxnard terminal. The SAI study was intended t o rationalize t o the FPC Western's choice of Oxnard; the

RISK ASSESSMENT IN A POLITICAL DECISION PROCESS

63

FPC study was intended t o advise the FPC staff and then t o persuade the commissioners of the merits of the staff's position; the SES study was intended t o persuade the client, the Oxnard City Council, of the analyst's own reservations of the safety of the Oxnard terminal. It is clear from the nature of the problem, indeed from policy analysis in general (and maybe all "scientific" investigations), that there are many competent and respectable ways of analyzing the problem. NO one set of assumptions is best; no analysis can be complete, no assessments are "free" of judgment. In fact, the assessments express opinion, an opinion in support or rejection of a policy argument. As in any area of uncertainty, it would be expected that there is a range of opinions. Since the assessments are ultimately intended t o support arguments in the policy process it would be expected that these opinions are expressed as persuasively as possible. Seen in this light, it is not surprising that SAI found the risks of Oxnard t o be "extremely low", that the FPC found those risks t o be "negligible", and that SES, having concluded that there are large uncertainties involved, presented some of their results in the form of worst-case scenarios. It is also not surprising, in view of the incentives t o present these results persuasively, that the reports did not elaborate in any detail on the uncertainty of the results. Working in a client-oriented environment, the analysts chose the format and presentation that made the best case for their and/or their client's position. As pointed out by Sjoberg (1980), risk analyses often increase polarization of the arguments rather than produce a consensus.

IV. Summary and Recommendations A . Summary

In this paper we have examined the role risk assessment played in a political decision process: the siting of an LNG facility on the California coast. From an analytic perspective the siting problem was one of trading off several objectives under uncertainty: environmental quality, cost, gas supply interruption risk, and population risk. Yet the political decision process studied here bore no resemblance to such a rational structure: several interested parties with conflicting goals and different short-hand long-term agendas came together in a series of structured and unstructured debates. That set of debates generated a sequential decision process, where bounds on the overall siting problem were successively narrowed as parts of the problem were resolved. An essential element of those debates was

64

J. Lathrop and J. Linnerooth

risk. With few exceptions, some measure of risk was used in support of each party’s stand. Yet, risk is a multidimensional concept; party stands were often supported by diverse definitions of risk, based on all manner of ways of combining dimensions into a risk measure, of assessing each dimension, and of portraying the results. We examined and compared three risk assessments that were produced in the course of the California LNC siting process. That comparison established that beyond the basic differences in definitions and measurements of risk, there are many other degrees of freedom in a risk assessment which are left to the analysts’ judgment. Those factors can affect the results of the assessment in major ways, determining the level of risk measured over a very broad range (perhaps two orders of magnitude), the degree of confidence ascribed to the results, and the salience of the risk. As a consequence, the results of a risk assessment may be determined as much by the judgments of the analysts as they are by the site and technology considered. But risk assessments should not be considered as andyses existing in a vacuum. An understanding of the political context within which the assessments operate is vital to the development of improvements in those assessments. The purpose of a risk assessment in the political process we studied was not only t o assess risk, but to support one side or another in debates concerning the acceptability of a risk, where those debates affected incremental decisions in a sequential process. For this reason, the timing of a risk assessment can be crucial t o its nature and effectiveness, and its effectiveness depends not only on its analytic rigor, but on its persuasiveness.

B, A Suggestion for Improving Risk Assessments. Rules ofEvidence From some normative point of view, the ideal role of a risk assessment is to make such technical information as can be mustered available to the political decision process in such a way that the information is most effectively used. Yet our comparison of risk assessments found that the assessment results depended as much on analysts’judgments and presentation formats as they did on technical aspects of the site and technology. An examination of the role actually played by the assessments found that they were designed as much to persuade as to inform. Procedural reforms to improve the content and presentation of risk assessments so that the users have a clear idea of the technical failure possibilities, along with the uncertainty involved, could take several direc-

RISK ASSESSMENT IN A POLITICAL DECISION PROCESS

65

tions. First, the analysts could be changed, e.g., independent publiclyfunded research bodies could be set up to work in a less adverse atmosphere. Alternatively, the users could be changed, e g , the present hearing system where the judges are not trained in quantitative methods could be reformed to include technical experts. Along this same line, a Technical Advisory Board (see Ackerman et al., 1974) could be set up which would critique the content of technical reports for the benefit of those using them. In the opinion of the authors, a more useful and implementable strategy for improving the content of risk assessments would involve not changing the analysts or the users, but changing the rules under which the analysts operate. Given the necessarily subjective content of risk assessments, it appears appropriate that they be received in the same manner as other more obviously subjective evidence in hearings and procedures. It follows that a promising strategy for improving risk assessments lies in the development of rules of evidence, or standards that assessments must meet in order to be used in a hearing or accepted as part of an Environmental Impact Report. The notion of introducing rules of evidence as a possibly useful procedural reform is based upon the belief on the part of the authors that the advocacy role of analysts, with the existing incentives for them to produce a persuasive case for their results, can be a productive element in malung public policy. The presentation of polar views in a policy debate, by highlighting conflicting approaches to the problem and uncovering opposing evidence, can be more informative than a single analysis with whatever attempts are made to reduce the biases discussed in this paper. However, and this point cannot be overemphasized, the introduction of competing expertise to the policy process is useful only if those in the position of judging the merits of the evidence are able to recognize the assumptions and methodologies underlying the differences in t h s evidence. In other words, rather than proposing the impossible - that only objective reports concerning the risks of a terminal be allowed as impacts to siting decisions - we propose instead that more subjective and argumentative evidence be admissible but only under the condition that those elements of the analysis which are left to the analysts’ judgment are clarified, to the extent possible, in the presentation of the results.

1. Rules of Evidence: Desirable Features

A desirable feature of a risk assessment is that it communicate an appropriate degree of confidence in its results, and that it make clear the limitations of the analysis and current technical knowledge. If risk 5

J. Lathrop and 3. Linnerooth

66

assessments are used as arguments, one on each side of a perhaps many-sided debate, it would be desirable to those judging their merits that they be as comparable as possible. The debate can then focus on comparing aspects of the alternatives themselves, as opposed to unwittingly comparing aspects of the assessments and presentations of the various sides. In this way, the interested parties can discuss differences in the modeling of uncertainties and the tradeoffs involved, about which there cannot be any "objectivity" (see Humphreys, 1982). When the relevant physical processes are poorly understood, widely different sets of assumptions can be defended so that two different analyses can deliver two different results, both following correctly from the assumptions adopted. While that murce of difference w n o t be eliminated, the other sources of differences listed in Section 1II.B. can be minimized by procedural standards for risk assessments, or rules of evidence such as those suggested below.

2 . One Suggested Set of Rules of Evidence

a. Clearly define the "risk" being assessed. As was stressed before, risk is a manydimensioned concept that is characterized in different ways for different people. Differing characterizations of risk may be such an intrinsic part of the political debate that any consensus on risk definiton may not be feasible, or more to the point, such a consensus may be more difficult to achieve than the resolution of the risk management debate itself. At the very least, then, risk assessments should clearly state what aspects of risk are being assessed, so that differences among assessments due to differing risk measures are recognized as such. As a case in point, it hardly helps to measure only expected fatalities when the political process is sensitive to a concern for the potential for catastrophe. While different political processes may be sensitive to different aspects of risk, one set of measures that take into account sensitivity to catastrophe can be adopted from Keeney er al. (1979) which include the following: (1) Expected fatalities: allows cost/benefit and some value of life calculations to be made. ( 2 ) Individual probabilities of fatality: allows comparison with nondecision "benchmarks" of individual risk, such as smoking, driving a car, etc. (3) Individual probabilities of fatality (for members of groups): when grouped by occupation, neighborhood, or activity (recreation, living), allows consideration of equity.

RISK ASSESSMENT IN A POLITICAL DECISION PKOCFSS

-

61

Risk of multiple fatalities: allows incorporation of sensitivity t o catastrophe. In addition, the analyst should be sensitive t o the many other dimensions of the risk issue that are of political concern. Who are the people at risk? Are they young or old? How frightened are they of the risk situation? Are they voluntarily exposed? Are they familiar with the risk? And this list could continue (see Otway and von Winterfeldt, 1980). (4)

b. Be clear on error bounds of results. Those bounds should include disagreements among experts. This requirement could have the effect of reducing differences among assessments, by forcing all assessments t o take into account a similar data base and the set of relevant experts or models, as opposed t o a single expert or model for each assessment. c. Model uncertainty explicitly. If the assumptions made are clearly stated along with the results, debates over differences between assessments can often be converted into more meaningful debates on assumptions. For instance, if the assessment assumes “no terrorist actions”, that should be clearly stated along with the results. d. Wherever possible, risk measures should be stated in relative terms, relative t o an actual, agreed-upon alternative. Many problems could be mitigated by measuring relative risk, as opposed to absolute risk. But this suggestion would require an agreement on a particular alternative to the applicant’s project when the debate may be due to two different ideas of what that alternative should be. However, where possible, t h s requirement would lead t o risk assessment results that are more easily incorporated into the actual political process.

References Ackerman, B., S. Adterman, J . Sawyer, Jr., and D . Henderson, 1974. The Uncerrain Search for Environmental Quality. London: Collier Mamillan. Ahern, W . , 1980. California meets the LNG terminal. Coastal Zone Management Journal, 7, 185-221. Atz, H . , 1981. Decision-making in LNG terminal siting:Wilhelmshaven,F.R.G. Draft Report. Laxenburg, Austria: IIASA. Fischhoff, B., S . Lichtenstein, P. Slovic, S . Derby, and R . Keeney, 1981. Acceptable Risk: New York. Cambridge University Press.

68

J. Lathrop and J. Linnerooth

FPC, 1976. Pacific-Indonesia project, final environmental impact statement. Bureau of Natural Gas, lederal Power Commission Staff. Federal Energy Regulatory Commission, Washington, D.C., December. FPC, 1977. Initial decision on importation of liquefied natural gas from Indonesia. Federal Power Commission. Fedcral Energy Regulatory Commission, Warhington, D.C., July. Havens, J., 1977. Predictability of LNG vapor dispcrsion from catastrophic spills onto water: An assessment. U.S. Coast Guard,Washington, D.C. Humphreys, P., 1982. Value structure underlying risk assessments. In: H . Kunrellthcr (cd.), Risk: A Seminar Series. Lawnburg, Austria: IIASA. Keeney, R., R. Kulkarni, and K . Nair, 1979. A risk analysis of a n LNG terminal, Omqa, 7, 191-205. Kurireuther, H., J . W. Lathrop, and J. Linnerooth, 1981. A descriptive model of choice for siting facilities: The case of the California LNG terminal. IIASA Working Paper, WP-81-106.iLaxenburg. Austria: IIASA. Lathrop, J.W., 1980. The role of risk assessment in facility siting: An example from California. WP-80- 150. Laxenburg, Austria: IIASA. Linnerooth, J . , 1980. A short history of the California LNG terminal. WP-80-155. Laxenburg, Austria: IIASA. Macgill, S. M., 1981. lkcision making on LNG terminal siting: Mossmorran-Braefoot Bay, United Kingdom. Draft Report. Laxenburg, Austria: IIASA. Majone, C . , 1980. An anatomy of pitfalls. 1n:G. Majone and E. Quade (eds.),Pitfu//s of Analysis. IIASA Series. New York: Wiley. Majone, G. The uses of policy deemphasis and analysis. Draft. Laxenburg, Austria: IIASA (in press). Mandl, C. and J . W. Lathrop, 1981. Assessment and comparison of liquefied energy gas terminal risk. IIASA Working Paper, WP-81-98. Laxenburg: IIASA. Mulkay, M., 1980. Science and rhe Sociology of Science. London: Allen and Unwin. Otway, H. and D. von Winterfeldt, 1981. Beyond acceptable risk: O n the social acceptability of technologies. Policy Science (forthcoming). Ravetz, H., 197 1 . Scientific Knowledge and its Social Pfohlems. Harmondsworth, Middle se x : Penguin University Books. SAI, 1975. LNG terminal risk assessment study for Oxnard,California. La J o b , California: Science Applications, Inc., December. SES, 1976. Environmental impact report for the proposed Oxnard LNG facilities. Draft EIR Appendix B: Safety. Los Angeles, California: Socio-Economic Systems, Inc., September. Sjoberg, L., 1980. The risks of risk analysis. Acta Psychologica, 45, 301--321. Vaupel, J . W., 1981a. On statistical insinuation and implicational honesty. Unpublished paper, Duke University and International Institute for Applied System Analysis. July. Vaupel, J . W., 1981b. Analytic perspective on setting environmental standards. Draft report prepared for the Office of Air Quality Planning and Standards. U.S. Environmental Protection A g e n q , August. von Winterfeldt. D. and M. Rios, 1980. Conflicts about nuclear power safety: A decision theoretic approach. Los Angeles: Social Science Research Institute, University of Southern California. Wynne, B., 1981. Institutional mythologies and dual societies in the management of risk. Presented at IIASA Summer Study o n Risk. To appear in proceedings. Laxenburg, Austria: IIASA.

A MULTI-ATTRIBUTE MULTI-PARTY MODEL OF CHOICE: DESCRIPTIVE AND PRESCRIPTIVE CONSIDERATIONS Howard KUNREUTHER’ Internutional Institute o l A pplied Systeins Analysis, I.axerzhirrg, Austria

Abstract Societal decision making for siting facilities, such as liquefied natural gas (LNG) terminals, have two primary features which make these problems difficult to structure analytically. First, the decision affects many interested parties each with their own objectives, attributes, data base and constraints. Secondly, there is the absence of a detailed statistical data base on the safety and environmental risks associated with a proposed project. This paper describes a multi-attribute multi-party model (MAMP) which has been developed at IIASA for structuring the process for siting LNG terminals in four different countries. The decision making process in each of these countries can be characterized as a sequence of decisions, subject to change over time, which are influenced by exogenous factors and legislation. The MAMP model separates this process into a series of rounds, each of which is characterized by a unique problem formulation and an interaction phase. The model is illustrated using detailed data on the California siting decision obtained from published and unpublished material as well as personal interviews. The final section of the paper discusses empirical lessons which follow from the MAMP model and proposes areas for future research on prescriptive analysis.

’

The research report in this paper is supported by the Bundesministerium fur Forschung und Technologic, F.R.G., contract No. 321/7591/RGB 8001. While support for this work isgratefully acknowledged, the views expressed are the author’s and not necessarily shared by the sponsor. This paper is part of a larger project a group of us at IIASA are undertaking with respect to siting decisions of Liquefied Natural Gas facilities. The ideas presented here reflect helpful discussions with my IIASA colleagues-John Lathrop, Joanne Linnerooth and Nino Majone-as well as with David Bell and Louis Miller. Helpful comments on an earlier version of this paper were provided by Patrick Humphreys and Detlof von Winterfeldt.

70

ti. Kunreuther

1. Introduction

Society has become increasingly concerned with the question as to how one evaluates the siting of technologically sophisticated projects which provide social benefits over a wide region but also may impose significant costs on certain groups. The recent debates on the future of nuclear power plants as a source of energy throughout the world highlights this point. A less publicized set of decisions is the siting of liquefied energy gas terminals in different parts of the world, the particular technology which serves as an illustrative example in this paper. There are two primary features associated with these proposed projects which make them particularly difficult t o structure analytically. First, the decision affects many different individuals and groups in society rather than being confined to the normal relationship of a private niarket transaction such as when a consumer purchases food or an appliance from a store or firm. In the siting decision, each interested party has its own objectives, attributes, data base and constraints (Keeney, 1980). A second feature of these problems is the absence of a detailed statistical data base on the variety of different risks associated with either investing or not investing in a particular project. For example, if the construction of a nuclear power plant or LNG terminal is approved, then cnvironrnental and safety risks are created. By not building the project there are economic risks with respect to the future cost of energy to residences and businesses. Each interested party is thus likely t o provide different estimates of the uncertainties and consequences of these risks. Hence it is particularly difficult t o utilize what Majone and Quade ( 1 980) call statistic'il rules of evidence t o settle these differences. The puipose of this paper is t o propose a framework for investigating societal p o b l e m s which have the above two characteristics. Section I 1 provides a set of concepts which are relevant for characterizing the decision making process. In Section 111 these concepts are integrated into a descriptive model of choice, a multi-attribute multi-party model (MAMP) which has been developed a t IlASA for structuring the process for siting LNG terminals in four different countries: the F.R.G., the U.K., The Netherlands, and the U S . (Kunreuther, Lathrop, and Linnerooth, 1981). The model will be illustrated using the California siting case. The MAMP model enables the policy analyst to understand more fully the dynamics of the political process and the relevant constraints which impact on final actions. It thus promises t o be a useful tool for improving decision malung for societal problems where there are conflicts between the parties. Section IV discusses lessons which follow from the descriptive analysis and proposes areas for future research on prescriptive analysis.

MULTI-ATTRIBUTE MULTI-PARTY MODEL OF CHOICE

71

11. Relevant Concepts for a Descriptive Model of Choice

Different Interested Parties Facility siting debates vary in detail but there are a well defined set of stakeholders who can be classified into one of four general groups depicted in Figure 1 . Let us briefly look at each of these interested parties in turn to better understand why potential conflict is likely to result when a specific project is proposed.

Pot en t t a l The applicant

sources of

Local residents

conf l t c t

Public interest

groups Figure 1 . Relevant Interested Parties in Facility Siting Decisions

The Applicant Firms or developers who support the construction and operation of a facility have concluded that despite future uncertainties, the expected profits associated with the project exceed the potential costs. Their position is likely to be based on economic factors, although they may also be concerned with the safety risk.

Local Residents Residents in a community that has been proposed as a possible site will have differing views of the situation. Those who own the property where the project is to be constructed have to determine whether the price the

12

H . Kunreuther

developer offers them is attractive enough. If the developer has eminent domain power (e.g., a public utility) then these residents may be concerned that a court will not award them a fair price for their property (O’Hare, 1977). Others in the community may focus on the reduced property taxes or increased employment that a facility is likely t o bring and hence favor the action. A third group may be concerned with the increased safety risk created by the facility and oppose the project.

Cover nmen t Agencies State and federal government agencies normally play the role of referee or arbiter in the decision making process, even though in many cases they have an interest in a particular outcome. Their regulatory actions, which are often constrained by legislation, influence the nature and distribution of the public’s preferences and provide advantages to some interests relative to others (Jackson and Kunreuther, 1981).

Public Interest Groups Recently we have seen the rise of very intense public interest groups. These organizations generally represent the interests and preference of one component of the public. For example, the membership of the Sierra Club is concerned with the effects that the siting of any new facility will have on the environment. Wilson (1975) and Mitchell (1979) have pointed out that those attracted to such organizations have strong, particular interests which dictate the agenda of the organization and influence the type of information that is collected and processed. It should be clear from these brief descriptions that there is considerable room for potential conflict between groups once a specific site is proposed as an option. The relative influence of each of the parties in the process will depend on their composition as well as how well-defined their objectives are. Olson (1971) postulates that each person in a group allocates time and energy in proportion t o the expected benefits he or she receives. If t h s assumption is true, then it is less likely that individuals will devote more effort to supporting a group’s cause as the size of the group decreases and the amount at stake for each individual increases.

MULTI-ATI'KIBUTE MULTI-PARTY MODEL OF CHOICE

13

Sequential Decision Process

Another feature of the facility siting problem is that the process is characterized by sequential decisions. March (1 978) notes that individuals and groups simplify a large problem into small subproblems because of the difficulty they have in assimilating all alternatives and information. Often constraints due t o legislation and legal considerations dictate the order in which certain actions must be taken. If the process is sequential in nature then the setting of an agenda is likely t o play a role in determining the final outcome as well as the length of time it takes t o reach it. There is strong empirical evdence from the field as well as from laboratory experiments (Levine and Plot;, 1977) that the order in which different subproblems are considered frequently lead t o different outcomes for the same larger problem. There are two principal reasons for this. Once a particular decision has been made on a subproblem this serves as a constraint for the next subproblem. If the order of the subproblems is reversed then there would likely be a different set of choices to consider. Secondly, each subproblem involves a different set of interested parties who bring with them their own set of data t o bolster their cause. The timing of the release of this information may have an effect on later actions. For example, citizens' groups normally enter the scene with respect t o siting problems only when their own community is being considered as a possible candidate. The data on the risks associated with siting would be released at a slower rate (but perhaps with greater emphasis and more political impact) if only one site was considered at a time than if all potential sites were evaluated sin~ultaneously.

Hole of Exogenous Events Another important concept, which relates to the uncertainty of infornia tion on probabilities and losses, is the importance of exogenous events in influencing the decision process. Random events, such as disasters, often play a critical role in triggering specific actions to "prevent" future crises and call attention to the dangers associated with a particular technology. The small data base for judging the frequency of low probability events, coupled with systematic biases of individuals in dealing with concepts of chance and uncertainty, increases the importance of a salient event in the decision making process. Tversky and Kahneman (1974) describe this phenomenon under the heading of availability, whereby one

74

H.Kunreuthcr

judges the frequency of an event by the ease with which one can retrieve it from memory. Fischhoff, Iichtenstein , and Slovic (in press) summarize their recent experimental studies on perceived risks by cataloguing the nature of individual estimates on the probability of occurrence and consequence of different types of hazards. One of their principal conclusions is that these estimates are labile and likely to change over time because of salient events which are highlighted by mass media coverage. In a similar spirit, March and Olsen (1976) suggest that random events and their timing play a role in many organizational decisions because of the ambiguity o f many situations and the limited attention that can be given t o any particular problem by the interested parties unless it is perceived as being critical. They provide empirical evidence to support their theory using empirical studies of organizations in Denmark, Noru.ay, and the United States. In another context, Holling (1981) has pointed o u t how specific crises in the short-run can lead to changes in policies with respect to environmental and ecological problems ( e g 3the suppression of the spruce budworm after it had destroyed forest in Canada). Kunreuther and Lathrop (in press) describe with specific examples how exogenous events triggered new coalitions and new legislation regarding LNG siting decisions in the United States. One reason for the importance of exogenous events, such as crises and disasters, in triggering societal interest in a specific problem is that it is easily understood evidence of trouble. Walker (1977) stresses the importance of this factor in setting the discretionary agenda of the U S . Congress or a government agency. T o support these points, Wa!'.er presents empirical evidence o n the passage of safety legislation in the U S . Numerous examples of this process are also provided by Lawless (1977) through a series of case histories of problems involving the impact of technology on society. He points out that frequently: new information of an "alarming nature is announced and is given rapid and widespread visibility by means of modern mass communications media. Almost overnight the case can become a subject of discussion and concern t o much of the populace, and generate strong pressures t o evaluate and remedy the problem as rapidly as possible. (P. 16) In the case of decisions such as the siting of facilities, exogenous events such as an LNG explosion or an oil spill may be sufficiently graphic and affect enough people t o cause a reversal of earlier decisions, inject other alternatives into the process and change the relative strength of

MULTI-ATTRIBUTE MULTI-PARTY MODEL OF CHOICE

-

15

parties interested in the decision outcome. The mass media may play a critical role in focusing on these specific events and in many cases exaggerating their importance.

111. A Multi-Attribute Multi-Party Model of Choice

The above concepts are now incorporated into a model of sequential decision making for large-scale projects such as facility siting. The approach, which has been influenced by the work of Braybrooke (1974), focuses on more than one attribute and involves many interested parties. Hence we have called it the Multi-Attribute Multi-Party (MAMP) model? The MAMP model will be described using an illustrative example: the siting of a liquefied natural gas (LNG) terminal in California. The relevant data for developing the scenarios described below were obtained from published documents as well as personal interviews with key interested parties. For inore detailed discussions see Lathrop (1981) and Linnerooth ( I 980). It i s useful t o provide a brief background on the nature of the siting problem. LNG is a potential source of energy which requires a fairly complicated technological process that has the potential, albeit with very low probability, of creating severe losses. To import LNG the gas has t o be converted t o liquid form at about 11600 the volume. It is shipped in specially constructed tankers and received a t a terminal where it undergoes regasification and is then distributed. The entire system (i.e., the liquefaction facility, the LNG tanker and the receiving terminal and regasification facility) can cost more than $ 1 billion t o construct (Office of Technology Assessment. 1977).

Elements of the Model Figure 2 provides a schematic diagram of the MAMP model. The decision process can be separated into different rounds which are labeled by capital letters, A, B . . . A round is simply a convenient device t o illustrate a change in the focus of discussions either because (1) a key decision was taken (or a stalemate reached due to conflicts among parties) or (2) a change occurred in the context of the discussions due to an exogenous event, entrance of a new party or new evidence to the debate. A round is 2For a more detailed description of the MAMP model see Kunreuther, Lathrop and Linnerooth (1981).

76

H. Kunreuther

initiated by a formal or informal request by one or more of the interested parties. In California Round A began in September 1974 when the applicant filed for approval of three sites on the California Coast Point Conception, Oxnard, and Los hgeles--to receive gas from Indonesia. No matter how a round is initiated it is characterized by a unique problem formulation which is presented in the form of a set of alternatives. There can be several decisions made in any round or there can simply be discussions of issues which revolve around the proposed set of alternatives. By definition the activities during any given round are based on the same set of alternatives. Each alternative is characterized by a set of attributes which may be viewed differently by each of the interested parties. In Round A the alternatives were whether one or more of the proposed sites for an LNG terminal was acceptable. There were four primary attributes used for the ensuing debate among the parties. The need for LNG and the risk of an interruption in the supply of natural gas were arguments for supporting the location of a terminal in at least one of the three proposed sites. Environmental and land use considerations suggested a non-remote site (Los Angeles or Oxnard) while the risks to the population argued for siting the terminal in a remote area (Point Conception). Finally, concerns about earthquake risk brought about opposition to the Los Angeles site, which was found to be crossed by a significant fault. There were several interested parties in Round A which can be referenced t o the four groups depicted in Figure 1. The applicant for the terminal was Western LNG Terminal Associates, a special company set up t o represent the LNG siting interests of the three gas distribution utilities: Southern California Gas Company, Pacific Gas and Electric, and El Paso Natural Gas Company. With respect to local residents, each of the city councils evaluated the proposed terminal in their jurisdiction by looking at the tax revenues and jobs it promised t o provide. These positive features had to be weighed against the negative impacts that the facility might have on land use and risk t o the population. With respect t o government agencies, the Federal Energy Regulatory Commission (FERC) determines whether a proposed LNG project is in the public interest and should be allowed and the California Coastal Commission (CCC) has the responsibility of protecting the California coastline. Finally, the public interest groups, represented by the Sierra Club and local citizens’ groups, were primarily concerned with safety and environmental issues. Each of the interested parties states its preference over the different alternatives and constructs arguments to defend its preference by focusing on different attributes. This descriptive view of the process

MULTI-ATTRIBUTE MULTI-PARTY MODEL OF CHOICE

I

Alternatives in round

77

1

I Relevant interested parties

' Preferences by parties

m 0) 0 L

a c .0 _ + u

P

aI + c -

Use of attributes to defend preferences

Nature of conflicts between parties

e Decision in round

Conclusion of round

Is there a feasible solution or no solution possible

No

Figure 2. Multi-Attribute Multi-Party Model (MAMP) of Choice

H. Kunreuther

78

should be distinguished from the concept of elicitation of preferences as used by psychologists and decision analysts. During this interaction phase certain decisions are made. In the case of Round A in California two key decisions were taken. First, the CCC favored Point Conception over the non-remote sites due l o concerns over the safety risk t o population. Second, the FERC disapproved of the Port of Los Angeles because a recently discovered earthquake fault increased the seismic risk above an acceptable threshold. Round A was concluded with a potential stalemate perceived by the gas industry. We have summarized the elements of Round A in Table 1 . Table 1 . Elements of Round A Problem Formulation:

Should the proposed sites be approved? That is: Does California need LNG, and if so, which, if any, of the proposed sites is appropriate?

Initiating Event:

Applicant tiles for approval of three sites.

Alternatives:

Site at Point Conception: Site a t Oxnartl: Site at Los Angeles: Site at any combination of:

A' A' A3 A' , A', A'

Interaction:

Involved Parties

Attributes Used as Arguments

Applicant *FERC

XI

PI p2 *ccc p3 *City Council5 P, Sierra Club P, Local Citizens P,

XI

x3 x2 X1

xs

X1

XS

Xl

x5

Key Decisions: CCC concerns over population risk implies that A' is preferred over t h e other two sites. 2. FERC would not approve A' because the seismic risk is greater than a prescribed acceptable level. I.

Conclusion: Applicant perceives a stalemate, i.e., that no site is approvable without long delay.

*Interested party with responsibility for dccision(s).

The siting process in California can be characterized by four rounds (A . . . D) as shown in Table 2 . Round B resulted in the passage of the LNG Siting Act of 1977 which was designed to break the stalemate at the end

MULTI-ATTRIBUTE MULTI-PARTY MODEL O F CHOICE

79

Table 2 . Summary of Rounds in California LNG Siting Case

DA TE

ROUND A Problem Formulation:

Should the proposed sites be approved? That is: Does California need LNG, and if so, which if any, of the proposed sites is appropriate?

Initiating Event:

Applicant files for approval of three sites.

Septcmber 1974 (34 months)

Conclusion :

Applicant perceives that no site is approvable without long delay

July 1977

ROUND B Problem Formulation :

How should need for LNG be determined? If need is established, how should an LNG facility be sited?

Initiating Event:

Applicant and others put pressure on state legislature to facilitate LNG siting.

Conclusion :

New siting process set up that essentially assumes a need for LNG, and is designed to acceleratc LNC tcrminal siting.

July 1977

(2 months) September 1977

ROUND C Problem Formulation:

Which site should be approved?

Initiating Event:

Applicant files for approval of Point Conception site

October 1977 (10 months)

Conclusion :

Site approved conditional on consideration of additional seismic risk data.

July 1978

ROUND D Problem Formulation:

Is Point Conception seismically safe?

Initiating Event:

Regulatory agencies set up procedures to consider additional seismic risk data.

Conclusion:

(Round still in progress)

80

H.Kunreuther

of Round A. Its principal feature was that the CCC nominates and ranks potential sites for an LNG terminal in addition to those which the Western LNG Terminal Associates applies for. The California Public Utilities Commission, the principal state body involved in power plant issues, selects a site from the CCC list, not necessarily the top ranked site. In Round C which occurred during the summer of 1977 thc CCC ranked four sites (Camp Pendleton, Rattlesnake Canyon, Point Conception, and Deer Canyon) in that order and the CPUC chose Point Conception, conditional on it being a seismically safe location. Kound D is still in progress with the FERC and CPUC examiningseismic data which will determine whether Point Conception is seismically safe. Whether an LNG terminal will ever be sited at Point Conception is an open question since the enthusiasm of the applicant for an LNG terminal has now waned considerably. In addition, there are two sets of wealthy landholders owning adjacent tracts of land to Point Conception: the Hollister and Bixby Kanches. These landholders are attempting to do everything in their legal power to prevent the siting process at Point Conception and so far have managed to stall any action.

Interpretation o f the Model The MAMP decision process in California reflects the basic concepts which were outlined in Section 11. As indicated by the scenario of the four rounds, there were different interested parties who interacted with each other at each stage of the process. There were three broad categories of concern which are relevant to this problem: risk aspects, economic aspects, and environmental aspects. Each of these concerns can be described by a set of attributes. Table 3 depicts an interested partylconcern matrix showing the main attributes considered by each of the relevant groups over the seven year period based on detailed interviews with key personnel (see Lathrop, 1981). It is clear from this table that each of the parties brought to the debate their own special interests. The applicant’s primary concerns are earning profits for shareholders and delivering gas reliably to consumers. Hence the emphasis on the need for gas, profit considerations, and price of gas as the relevant factors. The federal and state government agencies concerns were specified by legislation; local governments conipared the economic benefits with environmental and safety factors. Public interest groups, like the Sierra Club and local citizens groups, focused their attention on the environmental aspects and safety risks associated with the project.

MULTI-ATTRIBUTE MULTI-PARTY MODEL OF CHOICE

81

Table 3. PartylConcern Matrix Parties Applicant Utility

0

Interest GPS

0

0

0

0

0

0

.

Sierra Local Club GP

0

0

0

ENVIRONMENTAL Air quality Land use RISK Population Earthquake

State Gov’t

- - -Local Gov’t

FERC CCC CPUC LEG

Concerns ECONOMIC Need for gas Profit considerations Price of gas Local economic

Federa1

0

0

0

0

0

0

0

0

.

0

0

0

0

0

0

.

0

0

0

0

0

0

0

The case also illustrates the importance of a small but powerful interested party- the Oxnard citizens group-in influencing legislative actions. Until the publication in 1976 of a worst case scenario associated with a proposed $300 million terminal in Oxnard, there was almost unanimous agreement by all stakeholders that this community would be an ideal site for an LNG terminal. At the time even the Sierra Club was in favor of this location (they changed their feelings about Oxnard in 1977). A worse case scenario indicated that a spill of 125,000 cubic meters of LNG from all five tanks on a tanker would cause a vapor cloud which would affect 50,000 people. Residents could look on a map t o determine whether the cloud covered one’s own house (Ahern, 1980). No estimate of a probability was attached t o this scenario. The graphic depiction of these consequences generated a public reaction by a small group organized by concerned citizens of Ventura County. The California legislature was influenced by this public reaction. One legislative staff member stressed that it was not possible t o allow a site that would lead t o a large number of deaths in a ~ a t a s t r o p h e Hence .~ new siting regulations were passed stating that no more than an average of 10 people per square mile could be within one mile of the terminal and no more than 60 within four miles of the terminal. The President’s National 3This comment was made to John Lathrop in an interview in Sacramento, California, in July 1980, regarding the siting process of an LNG terminal.

6

82

14. Kunreuther

Energy Plan incorporated similar population guidelines which effectively ruled out any high density areas as candidates for an LNG terminal. Interestingly enough the risk assessment used by the citizens’ group at Oxnard was only one of three conimissioned by different interested parties for this site each of which produced different estimates and conclusions (Lathrop and Linnerooth, this volume). The sequential decision process is self-explanatory based on the four rounds depicted in Table 2 . This process may facilitate decisions at each stage by limiting the number of parties but it can have negative l o n g range consequences. For example, the need for imported natural gas has greatly diminished in California but the possihility of siting a terminal is still alive. Point Conception has been deemed an acceptable site subject to a seismic risk study. Due to the nature of the siting process, the only way this site would be unacceptable is if the seismic risk was found t o be too high. Rather than stating that California may not need LNG, the relevant interested parties have preferred to delay the findings of the seismic risk studies (Lathrop, 1981). Another example of the long-range negative effects of the sequential constraints is the case of a supply interruption risk. Initially, the applicant proposed three separate sites t o minimize the risk of California having a shortage of natural gas. When the decision process eliminated two of the three proposed terminals, Western Associates proposed the construction of a large facility at Point Conception capable of producing a throughput of 58,000 1n3 LNG/day, equivalent in energy flow t o roughly 15 modern nuclear reactor units (Mandl and Lathrop, 1981). By concentrating the facilities at one port the supply interruption risk will now likely be increased rather than decreased, if Point Conception is approved and actually utilized. Finally, turning to the role of’exogenous events in California there is one incident which had an impact on the decision making process. In December 1976 the Los Angeles City Council voted t o allow work to begin on an LNG terminal in San Pedro Bay. The following day an explosion ripped the oil tanker Sansinea in bs Angeles harbor, leaving 9 dead and 50 injured. A week later the City Council commissioned a study as to the relative safety of the proposed site. They later approved the terminal. This explosion, although it had nothing t o d o with liquefied natural gas, alerted many Californians t o the potential dangers o f LNG. On a more general level, two disasters in other parts o f the country illustrate the importance that exogenous events have had on the decision process with respect t o LNG siting and regulations. In 1973 an LNG tank in Staten Island, New York, exploded and the roof collapsed burying 40 workers. There was n o LNC in the tank but it

MULTI-ATTRIBUTE MULTI-PARTY MODEL OF CHOICE

83

had seeped through the insulation and caused a huge fire. A result of this explosion was the increased concern with the dangers of LNG by Staten Island residents. The neighborhood organization which was formed a year before the accident, attracted considerable attention and interest because of the media coverage of the tank explosion. In the context of the MAMP model a new interested party played a key role because of an exogenous event. What may have been a foregone decision regarding the location of an LNG tank in Staten Island became problematical (Davis, 1979). The worst LNG accident occurred in 1944 when the storage tank operated by the East Ohio Gas Company in Cleveland ruptured, spilling LNG on adjacent streets and sewers. The liquid evaporated, the gas ignited and exploded, resulting in I28 deaths, 300 injuries and approximately $7 million in property damage. An investigation of this accident indicated that the tank failed because it was constructed of 3.5% nickel steel, which becomes brittle when it comes in contact with the extreme cold of LNG. All plants are now built with 9% nickel steel, aluminium or concrete and the storage tanks are surrounded by dikes capable of containing the contents of the tank if a rupture occurs (Davis, 1979). This example illustrates the impact of a particular incident on new regulations, which otherwise may not have been passed.

IV. Improving the Decision Process: Prescriptive Analysis The siting process for LNG terminals in California has provided a graphic description of the conflicts which exist between different interested parties, each of whom have their own goals and objectives. The partylconcern matrix depicts the different attributes used to defend positions, the MAMP model reveals the dynamics of the decision process and the relevant constraints which determined the outcomes at the end of each of the different rounds.

Lessons from the MAMP Model A retrospective view of the situation through the eyes of the MAMP model provides the following insights which may have relevance for prescription. 1. There is little articulation of v a h e judgments by the different parties. Each of the groups has a set of objectives and related attributes which they are willing t o articulate but there has been n o statement by anyone as to the importance weights assigned t o the different attributes in the 6*

84

H. Kunreuther

problem. This observation coincides with Ward Edwards’ experience in attempting to use multi-attribute utility analysis in evaluating alternative school desegregation plans submitted by external groups t o the Los Angeles School Board. He has noted that the interested parties in a societal decision problem are unlikely to reveal their value structures because this information would then be public and they would be accountable for numerical judgments (Edwards, 1981). For this reason it will be difficult to utilize this technique as a way of determining preferences between alternatives. Humphreys and McFadden ( 1980) have had similar problems using their MAUD interactive computer model. Constraints guiding the decision process are not stable but may change over time as new information is injected into the process by one or more interested parties. An interesting example is the present concern that seismic risk is a potential problem for siting a facility at Point Conception, even though this risk had not surfaced in earlier discussions of the feasibility of the site. Another illustration is the ability of the Oxnard citizens’ group to influence new legislation on siting criteria by focusing on the number of deaths froma catastrophic accident rather than on the extremely low probability of such a disaster actually occurring. These examples illustrate the point made by Majone (in press) that actual policies are determined through a process where each of the interested parties attempts to modify rules of the game which constrain them from achieving their goals and objectives. This may further exacerbate the problem of eliciting the value structure of the different interested parties. The siting of sophisticated technologies is a process that is not well understood scientifically so that there are no measures of risk which can be pinpointed using statistical analysis. Hence each of the interested parties has an opportunity to focus on different measures to support their position. The conflicting risk assessments for evaluating the safety of an I.NG terminal in proposed sites has been well documented by Mandl and Lathrop (1981) and Lathrop and Linnerooth (this volume) for the four IlASA case studies. Each of several different interested parties commissioned a special risk study and used the results for their own purposes. Given these observations what can be done to improve the situation? One of the most important aspects of the MAMP desciiptive model is that it enables the policy analyst t o focus on the actual siting process and to evaluate its success on the basis of several different dimensions. The standard analytic tools such as multi-attribute utility analysis or cost/benefit analysis have normally focused on outcomes rather than process. There is

MULTI-ATTRIBUTE MULTI-PARTY MODEL OF CHOICE

85

no reason why one cannot focus on how well different procedures score with respect to a well-defined set of objectives. The first step in undertaking this type of analysis would be to specify the relative importance of different attributes one would like a process to satisfy. One of these attributes might be related to how well the final choice performs with respect to resource allocation, but there is also likely to be a set of attributes which reflect the way different interested parties feel about the process as well as the outcome. For example, did each interested party have an opportunity to voice its position? Were a wide enough set of alternatives considered so that the parties felt that a choice was actually being made? These factors may be important in some type of cultural settings but less relevant in others. The policy analyst can also point out that a more elaborate process takes time, another dimension t o be considered in the evaluation procedure. By articulating the types of tradeoffs which have to be made in choosing one type of procedure over another, the analyst can provide guidance to policy makers as to what decision process they may want to consider in the future. Future Research Needs The Use of GERT The MAMP model also may be a useful tool for analyzing how alternative procedures are likely to fare for a given problem context. In reality, the decisions made in any round are probabilistic with the chances of different outcomes determined by the partylconcern matrix and the procedures which one employs. One way to modify the MAMP model to incorporate these elements of uncertainty is to employ the concepts of another technique-GERT (Graphical Evaluation and Review Technique).- to structure the process. GERT is a combination of network theory, probability theory and simulation, and was developed by Alan Pritsker (1 966) to analyze the terminal countdown on an Apollo space ~ y s t e m The . ~ basic features of GERT in the context of the California siting decision appears in Kunreuther (1 98 1). The use of GERT to structure the key questions and activities depicted in the MAMP model provides a vehicle for prescriptive analysis. It enables the policy analyst to develop alternative scenarios and likely out4For an excellent description o f the modeling features and capabilities of GERT including its application in real world problems see Moore and Clayton (1976).

86

H.Kunreuther

comes by changing the nature of the decision process. Future research using this model could address the following types o f questions: What is the likely impact on the decision process if some of the existing constraints are relaxed? For example, suppose that experts were explicitly brought into the picture t o attempt to arrive at consensual judgments regarding specific risk and that the interested parties have t o abide by their findings. What impact would this have on the likely outcomes? What would the impact be if certain parties were given power which they currently do not have? For example, suppose that a specific regulatory agency was given full authority t o rank and approve a specific site in California. What difference would this make on the scenario and final outcome? What would happen if there was a change in the way alternative sites were introduced into the picture? For example, suppose the gas companies decided t o propose only one site a t a time for locating a terminal. How would this affect the interaction between different interested parties and the alternative outcomes? In this type of scenario one would first have t o determine the order of the sites t o be introduced and the relevant nodes and activities should a particular site be approved or deemed infeasible.

Efficiency and Equity Tradeoffs The MAMP model can provide insight as to when political considerations are likely t o foreclose certain outcomes which may have desirable economic fwtiires. For example, a particular scenario may reveal that a community is likely to be opposed to a given site and will fight hard t o stop its approval because they feel that the increased risks which they must bear are t o o high. If this project is socially beneficial, then it may be useful t o investigate policy tools for compensating interested parties who feel they will suffer from the project. Insurance may provide a way of protecting potential victims against potential property losses and physical injury. Today there is limited insurance protection against large-scale accidents such as a catastrophic accident of an LNG terminal. A General Accounting Office report (1978) concluded that under present liability arrangements injured parties could not be fully compensated for a serious accident. Some of the research questions which could be appropriately addressed in future problem-focused st d i e s are :

MULTI-ATTRIBUTE MULTI-PARTY MODEL OF CHOICE

81

Which of the interested parties is liable in the event that a specific disaster occurs after a project has been sited? 0 What types of enforcement procedures can be evoked to assure that contract provisions are satisfied ex posf? 0 Are there historical lessons which shed light on the role of insurance as a tool for providing financial protection to potential victims? With respect to more direct consequences of siting a new facility, O’Hare (1977) has proposeda compensation system where there was opposition to proposed sites from certain interested parties, such as residents of the area, who felt that they would suffer losses in property values and would have to bear safety and environmental risks. The essence of his proposal is that each community determines a minimum level of per capita compensation so that it is willing to make a legal commitment to having the project in their backyard if the compensation is paid. Whether or not some type of compensation scheme is a useful policy prescription depends on the specifics of the situation. In this connection, it would be interesting to ask: What type of payments wotild have been required to appease the citizens of Oxnard so that an LNG terminal could have been located there? What would the Sierra Club require in payments so that they would support a site which might have adverse environment effects? These questions can only be answered in a real world problem context. They do reflect an increasing concern of economists and lawyers in dealing with windfalls or wipeouts from specific actions which involve the public sector. Hagman and Misczynski (1978), in their comprehensive study of the subject, believe that windfalls should be partially recaptured t o help compensate for wipeouts. They propose a number of alternative mechanisms for ameliorating this problem ranging from special assessments to development permits. These types of policy instruments could also be investigated in the context of specific siting problems. After all is said and done the final outcome is likely to represent some type of balance between the political constraints and economic criteria. As Wildavsky (1 98 1 ) has pointed out: The criterion of choice in politics and markets in not being right or correct as in solving a puzzle, but agreement based on interaction among partially opposed interests. (p. 133) The MAMP model will not tell any politician how one should deal with the equitylefficiency dilemma but at least it uncovers some of the specific causes of these conflicts. How one actually improves the process is a challenge for the future.

88

H. Kunreuther

References Ahern, W., 1980. California meets the LNG-terminal. CoastaI Zone Managemenr Journal, 7 , 185-221. Braybrooke, D., 1974, Traffic Congestion Goes Throwh the Issue Machine. London : Routledge and Kegan Paul. Davis, L.N., 1979. Frozen Fire (Friends of the Earth). Edwards. W., 1981. Reflections on and criticisms of a highly political multi-attitude utility analysis. In.: L. Cobb. and R. Thrall (eds.), Mathematical Frontiers of the Social and Policy Sciences, Boulder, Colorado. Westview Press, 157 -189. Fischhoff, B., P. Slovic, and S. Lichtenstein. Lay foibles and expert tales in judgments about risk. In: T. O’Riordan and R. K. Turner (eds), Progress in Resource Management and Environmental Planning, Vol. 3. Chichester: Wiley (in press). General Accounting Office, 1978. Need to improve regulatory review process for liquefied natural gas imports. Report to the Congress, ID-78-17, Washington, D. C., July. Hagman, D. and D. Misczynski, 1978. Windfalls for Wipeours. Chichago: American Society of Planning Officials. Holling, C. B., 1981. Resilience in the unforgiving society. Working Paper R-24, March, Vancouver: Institute of Resource Ecology. Humphreys, P.C. and W. McFadden, 1980. Experiences with MAUD: Aiding decision structuring versus bootstrapping the decision maker. Acta Psychologica, 45,51-70. Jackson, J. and H. Kunreuther, 1981. Low probability events and determining acceptable risk: The case of nuclear regulation. Professional Paper PP-8 1-7, May. Laxenburg, Austria: IIASA. Keeney, R., 1980. Siting Energy Facilities. New York:Academic Press. Kunreuther. H., 1981. A multi-attribute multi-party model of choice: Descriptive and prescriptive considerations. IIASA Working Paper, WP-81-123. Kunreuther, H. and J. Lathrop. Siting hazardous facilities: Lessons from LNG Risk Analysis (in press). Kunreuther, H.,J. Lathrop, and J . Linnerooth, 1982. A descriptive model of choice siting facilities. Behavioral Science, 27 ( 3 ) . Lathrop, J., 1981. Decisionmaking on LNG terminal siting: California, U S A . Draft Report IIASA. Laxenburg, Austria: IIASA. Lathrop, J. and J. Linnerooth. The role of risk assessment in a political decision process. In this volume, 39-68. Lawless, J., 1917. Technology and Social Shock. New Brunswick, New Jersey: Rutgers University Press. Levine, ME., and C.R. Plott, 1977. Agenda influence and its implications. Virginia Law Review, 6 3 (4). Linnerooth, J., 1980. A short history of the California LNG terminal. W - 8 0 - 1 5 5 . Laxenburg. Austria: IIASA. Majone, N. and E. Quade (eds.), 1980. Pitfolk of Analysis. Laxenburg: IIASA, Wiley. Majone, N. The uses of policy analysis. Draft: Laxenburg, Austria: IIASA (in press). Mandl, C. and J. Lathrop, 1981. Assessment and comparison of liquefied energy gas terminal risk. IIASA Working Paper, WP-81-98. Laxenburg, Austria: IIASA.

MULTI-ATTRIBUTE MULTI-PARTY MODEL OF CHOICE

89

March, J. G., 1978. Bounded rationality, ambiguity and the engineering of choice. Bell Journal of Economics, 9, 587-608. March, J. and J. Olsen, 1976. Ambiguity and Choice in Organizations. Bergen, Norway: Universtetsforlaget. Mitchell, R.C., 1979. National environmental lobbies and the apparent illogic of collective action. In: C. Russell (ed.), Applying Public Choice Theory: What are the Prospects. Washington, D. C.: Resources for the Future. Moore, L. and E. Clayton, 1976. GERT Modeling and Simulation. New York: Petrocelli. Office of Technology Assessment (OTA), 1977. Transportation of Liquefied Natural Gas. Washington, D. C.: Office of Technology Assessment. O’Hare, M., 1977. Not o n my block you don’t: Facility siting and the strategic importance of compensation. Public Policy, 25, 409-58. Olson, M., 1971. The Logicof Collective Action. Cambridge, Mass.: Harvard. Pritsker, A., 1966. Graphical Evaluation and Review Technique. Santa Monica, CA: The RAND Corporation. Tversky, A. and D. Kahneman, 1974. Judgment under uncertainty: Heuristics and biases.Science, 18.5, 1124--31. Walker, J., 1977. Setting the agenda in the U S . Senate: A theory of problem selection. British Journal of Political Science, 7 , 423-445. Wildavsky, A., 1981. Rationality in writing: Linear and curvilinear. Journalof Public Policy, I , 125-140. Wilson, J.Q.. 1975. Political Organization. New York: Basic Books.

This Page Intentionally Left Blank

THE ROLE OF DECISION ANALYSIS IN INTERNATIONAL NUCLEAR SAFEGUARDS* Rex V. BROWN and Jacob W. ULVILA Decision Science Consortiuni. Irlc.

Abstract This paper discusses how the tools of personalist decision analysis (PDA) are being used to help the International Atomic Energy Agency to safeguard nuclear material against diversion from peaceful uses. PDA has two main roles: as a framework for reporting Agency effectiveness (notably the probability of its detecting diversion); and as a management aid (for example, in the allocation of limited inspection sources). A political requirement that no member state be discriminated against imposes interesting constraints on permissible analyses.

Institutional Background The International Atomic Energy Agency (IAEA) is responsible for assuring the international community that nuclear material “under safeguards” is not diverted from peaceful use (IAEA, 1973; IAEA, 1976; IAEA, 1978). The conduct of inspections, and therefore their effectiveness, is constrained by agreements with the states concerned, by the desire of these to protect their econornic and other interests, by the need to respect state sovereignty, and by a very severe rationing of’personal and other agency resources. One constraint is that the Agency should not allocate its resources in a way that discriminates between states, e.g., an allocation which implies that one state is more likely than another to contemplate diversion. This means that any direction of resources on a non-uniform basis must be based on the objective inspection information the IAEA has at hand. In addition, allocation between facilities might be based on technical characteristics, like the type of facility, the fuel involved, the nature of the fuel cycle pattern in the state concerned, the quality of the state’s accounting *The opinions and assertions contained in this paper are the private ones of the authors and do not reflect any policy of the International Atomic Energy Agency or the United States Government. This paper was supported by the U. S . Arms Control and Disarmament Agency under contract Number ACONCl 10.

92

R. Brown and J.W. Ulvila

system in terms of accuracy and transparency, and the quality of the facility measurement system. Specific characteristics may include: whether all nuclear materials in the state are under safeguards; the history of anomaly observation at that facility or in that state; the quality of the facility measurement system as reflected in the level of material unaccounted for (MUF); the documentation of the measurement quality control system. This safeguards function of the Agency supports the more general objective of non-proliferation, shared by countries party to the Non-Proliferation Treaty and other agreements (IAEA, 1968; IAEA, 1972). (There are those who do not belong t o the Treaty but nonetheless either support or purport to support non-proliferation objectives.) Together with other measures taken by states, singly or in concert, Agency safeguards help both t o deter diversion and mitigate its effects if it does occur. The Agency has two administrative objectives requiring analytic he1p : -- Reporting measures of Agency Luccess to member states, primarily through periodic Safeguards Implementation Reports (SIRS). - Managing the Agency's limited inspection resources, for example, allocating an inspector's time to activities, facilities, and time periods. As we know from analogies in financial and management accounting, the measures used for reporting may not be identical t o those used for decision making. Reporting measures may be limited to what can be uncontroversially and objectively documented, whereas decision making (management) measures can incorporate whatever the decision maker judges to be relevant. Of course, some reasonable correspondence between the two must exist if the Agency is to serve the member states effectively.

Analytic Strategy Decision analysis is a prescriptive modeling technique, which incorporates quantitative measures of human judgment. For this reason, we will refer to it as "personalist decision analysis" (PDA), to distinguish it from other apDroaches to analyzing decisions. In principle, PDA has a role to play in supporting both objectives of the Agency noted above. It can support the reporting function by providing a systematic, non-arbitrary way of addressing significant measures of performance, like detection probability. Without personalist decision analysis, one is largely limited to purely informal subjective evaluations or to purely objective evaluations based on judgment free measurements, like the number of inspection visits. It can support the management function

DECISION ANALYSIS IN INTERNATIONAL NUCLEAR SAFEGUARDS

93

by enhancing the quality of decisions through more efficient use of available data and expertise; and by making decisions easier to defend within and without the Agency. Decision Science Consortium, Inc. is now in the second year of helping the Agency to adapt PDA approaches for both reporting and management purposes (Ulvila and Brown, 1981). Reporting Aids A general purpose aggregate measure of the effectiveness of Agency Safeguards has been developed and is summarized in the Appendix. It will shortly be disseminated in an Agency report (IAEA, in preparation). Although designed primarily as a reporting measure, its primary application to date has been in the context of a resource allocation decision presented in the next main section, where its properties are discussed. The head of the Department of Safeguards is also interested in using probability of detecting a diversion as a primary evaluation measure, both for decision and reporting purposes (Gruemm, 1980). It is easy t o understand and communicate, and is clearly a key feature of the Agency's success. However, it must logically take into account judgments about which types of diversions are most likely, in addition to how well each type of diversion is detected. It is perfectly acceptable to assess the latter explicitly since it is largely a technical matter-which is not to say it is easy to do. However, to assess the likelihood of different types of diversion requires taking some view of how a diverter would behave, and this in turn depends on who the diverter might be. Agency staff believe it is politically unacceptable t o address such issues explicitly, for example, for directly imputing behavior to a potential diverter. A classical decision-analytic approach to this problem forces the innate logic out into the open, and thereby open to objection. The overall (marginal) probability of detection given any diversion is computed as the weighted average of the conditional detection probabilities for all diversion "paths", with weights proportional to path probabilities. It is making the latter explicit which is liable to raise sensitive issues. Two approaches have been explored to get around this problem. One of them, which is the one currently favored by Agency staff, is to assign equal weight, and implicitly equal probability, to all diversion paths. On the face of it, this is objective. In fact, it depends critically on how diversion paths are distinguished. For example, splitting one path into two variants will double the weight for the original path. Implicit relative probabilities could be inobtrusively incorporated by partitioning the more

94

R. Brown and J.W. Ulvila

probable scenarios more finely than the expected ones. Without this, one may end up with a very unrealistic detection probability (unless the probable and improbable scenarios have much the same conditional probability of detection). A second approach is to weight paths by some measure of "attractiveness", say, the technical complexity of a diversion path, which will approximate relative probabilities. This approach is incorporated into the aggregate measure presented in the Appendix, though that measure attempts to capture more concerns than the probability of detecting a diversion. Professional integrity aside, any seriously unrealistic overall detection probability is liable to be shown up by the unfolding of events. The Agency would be embarrassed if diversions ultimately come to light and it turns out that the frequency with which they were detected proves at variance with the reported probability. (On the other hand, if the sample is small enough, as we must hope it will be in the case of nonproliferation, it will not be very diagnostic, and therefore not very embarrassing.) Overall detection probability may not be a sufficient measure of detection performance. It will not reflect the relative importance of detecting different diversion paths. Presumably, the Agency would like to report the highest probability of detection for those potential diversions involving material which may be used directly in fabricating nuclear explosives. Detection probabilities could be reported for each level of diversion seriousness, but there may be some political sensitivity if "seriousness" depends on who the diverter is. Such factors again are handled loosely enough to be politically acceptable in the "aggregate measure", by the use of "objective" surrogates for seriousness, such as types of nuclear material. Another refinement (not yet developed) would be to assess several detection probabilities for different levels of Agency response. In our current analyses, a single "detection" response has been used: the observation of an anomaly by an inspector. In fact, this weak form of detection may lead to stronger responses if the anomaly remains unresolved: inspection activity at a facility may be increased; the Director General may consult informally with state missions concerned; the unresolved anomaly may ultimately be reported to the U.N. Security Council. (Derived responses, say by individual member states, may be the actual instruments of deterrence and mitigation.) Any attempt to collapse such probabilities into a single evaluation number would require weighting the importance of different levels of detection. This would require getting into mitigation and deterrence effects which the Agency may not care t o analyze explicitly.

DECISION ANALYSIS IN INTERNATIONAL NUCLEAR SAFEGUARDS

95

If response probabilities absent diversion are also incorporated to capture the risk of false alarm, the cost of false alarm would need t o be balanced against the value of detection.

Decision Aids A specific decision aid has been proposed for allocating inspection resources within a facility and is discussed in some detail in the next section (Shea ef d.,1981; Ulvila, 1980). Any general measure-like the probability of detection-will inevitably represent only part of the considerations relevant to any particular decision. Even if "public" utility to the international community is fully captured, there will inevitably be more parochial "private" Agency considerations to take into account. A "tough" policy for inspectors to follow-up on observed anomalies may be best for effective detection; but it may be ruled out on the grounds that it is intrusive, impairs relations with states accepting safeguards and might even lead to broken agreements-a severe embarrassment for the Agency and a reduction in the effectiveness of the nonproliferation effort. Multiattributed decision analysis may be used, but the need for discretion in disclosing private utility may dictate handling that informally. Moreover, the appropriate way to measure performance (completely or incompletely) depends critically on the decision. Primary analytic and data-gathering effort should be devoted to what this decision may effect. When allocating inspection resources inside a light water reactor (LWR), one will only want to analyze diversion paths relevant in an LWR. When allocating resources across states, one will want to pay careful attention to political implications that are irrelevant to allocations within a state. A common error by beginning decision analysts is to develop a general purpose measuring procedure for a system (like the Agency) and expect it to be usable as it stands for any particular decision. The most promising decision to analyze W i l l be those: where the stakes are high (heavy cost for a mistake); where the options are clear cut, but their comparison is perplexing; where human judgment is unavoidable, but others need to be persuaded of the rationality of the decision; and where substantially the same decision is made on a recurring basis. Immediately promising candidates appear to be: - Developing decision rules for when to initiate a special inspection, and more generally, when to proceed from one Agency response level to the next (IAEA, 1972; IAEA, 1973). - Allocating inspection resources at a facility (see next section), between facilities, and between states.

R. Brown and J.W.Ulvila

96

Personalist decision analysis and inference techniques appear to have an important potential role to play at IAEA. They may be able to ease the inherent difficulty of determinations the Agency must make, which have unavoidable judgmental components. They can also make those determinations more transparent to the international constituency served by the Agency. On the other hand, t h s very transparency may invite challenge in politically charged situations.

A Decision Aid for Allocating Inspection Resources Noting that the IAEA's personnel resources are insufficient to meet all commitments, allocation of these resources is necessary. The IAEA is investigating the use of decision analysis to aid in the allocation of an inspector's time across activities w i t h n a routine inspection of a facility (IAEA, in preparation; Shea et al., 1981; Ulvila, 1980). The approach uses a decision-analytic model to develop a prioritized list of inspection activities. The general approach is illustrated in Figure 1 . Prioritization is based on the value of the activity to the IAEA and its cost in terms of inspection time. tnspectton Cumulative activities hours Inspection Character activities isttcs I A 6 C

...

Value

~

Cost(hours)

-

"/C

_ -

-

-

_ _ -

A

B C

2 ?

0??:

%z

To;,

25

c3 A

25

Cumulative hours

Fipure 1. ApproaLh to Prioritizing Inspection Procedures

Safeguards Objective Inspection activities have value to the extent that they contribute to the Agency's safeguards objective which is stated as follows: "The objective of IAEA safeguards is the timely detection of diversion of significant quantities of nuclear material from peaceful nuclear activities to the manufacture of nuclear weapons or of other nuclear explosive devices or for purposes unknown, and deterrence of such diversion by the risk of early detection" (IAEA, 1972). This objective is composed of five significant elements, each of

DECISION ANALYSIS IN INTERNATIONAL NUCLEAR SAFEGUARDS

91

which is important in the development of safeguards. These elements are: timely detection, detection of diversion, significant quantities, from peaceful purposes, and deterrence of diversion by the risk of detection. Each element is the subject of a continuing international discussion, but some guidelines have been agreed to and they serve to guide the current safeguards activity (IAEA, 1980). Timeliness is currently addressed by a guideline that the maximum time that may elapse between a diversion and detection by IAEA safeguards should "correspond in order of magnitude to the time required to convert the material into the metallic components of a nuclear explosive device". This conversion time is different for different types of material-ranging from several days for plutonium metal to about a year for low enriched uranium. Since it is highly unlikely that an inspector would catch a diverter in the act of diverting nuclear material, a method had to be devised to detect diversion indirectly. The method currently under consideration makes use of three activities. First, a diversion path analysis is performed that identifies different methods of removing nuclear material from a facility and concealing the removal. A diversion path is defined by the material involved (e.g., spent fuel), its location (e.g., the spent fuel pond), and the method of concealing its removal (e .g., by falsifying accounting records). A diversion path is not limited to the physical route of removal. Diversion paths for a facility are determined from an analysis of design information, operating procedures, the state's system of accounting and control, and inspection histories (where available). Next, anomalies that would be generated by a diversion through the path are determined. Anomalies are "unusual observable conditions that might occur in the event of a diversion". For example, a diversion that was concealed by falsifying records at the facility would produce an anomaly of inconsistent accounting records. Then, inspection activities that would detect the anomalies are defined. For example, checking and comparing account balances would detect an inconsistency in accounting records. A significant quantity is "the approximate quantity of nuclear material in respect of which, taking into account any conversion process involved, the possibility of manufacturing a nuclear explosive device cannot be excluded". Current guidelines provide specific estimates of significant quantities for different types of nuclear material. These range from about 8 kilograms for plutonium to about 20 tons for thorium. Peaceful uses of nuclear materials occur at a variety of nuclear installations under IAEA safeguards. Categories of facilities subject to safeguards include power reactors, research reactors, conversion plants, fuel fabrication plants, reprocessing plants, enrichment plants, separate storage facilities and other locations (such as transit stores). 7

98

R. Brown and J.W. Ulvila

Deterrence by the risk of detection assumes two things: that the negative consequences of detection will act to deter a potential diverter and that a probability of detection will also provide a deterrent effect. This is the deterrence provided by IAEA’s international safeguards. It contrasts with deterrence provided by physical protection of facilities, which is part of a state’s national safeguards.

Aggregafe Measure of Safeguards Effectiveness In order to make managerial decisions affecting safeguards-such as evaluation and resource allocation-it is desirable to have an operational measure of the safeguards objective. For purposes of helping to allocate an inspector’s time among possible inspection activities during the inspection of a facility, the IAEA is considering the use of the aggregate measure presented in the appendix. This measure has been proposed by the lAEA staff and a group of international consultants invited by the IAEA to assist in formalizing the safeguards assessment methodology. The measure incorporates six explicit considerations related to the safeguards objective (IAEA, in preparation): (a) the probability of detecting a diversion; (b) the amount of nuclear material diverted; (c) the type of nuclear material; (d) the technical complexity involved in removing the material and concealing its removal; (e) specific vulnerabilities of the system; and (9 the timeliness of detection. The probability of detecting a diversion is related to the objective of deterrence through the risk of detection. The specific probability measure used is the probability that an inspection activity would detect the presence of an anomaly (which is the easiest probability to assess). Estimated probabilities are high for simple activities (e.g., that an item count will detect that a fuel assembly is missing from the spent fuel pond) and lower for more difficult activities (e.g., that a non-destructive assay will detect the substitution of material in a spent fuel assembly). Value increases as probability increases, but not linearly. l t was assessed that there is considerable deterrence value in achieving even a low probability of detection. This is reflected in the measure by raising the probability to the power of 0 . 3 . This assessment implies, for example, that half of the value of covering a diversion path is achieved with a coverage probability of 0.1. This further represents an assessment that it is more important to cover a large number of paths with low probabilities than a small number of paths with high probabilities.

DECISION ANALYSIS IN INTERNATIONAL NUCLEAR SAFEGUARDS

99

The amount of material diverted is important in relation to the objective of detecting diversion of a "significant quantity of material". This is reflected in the method by defining diversion paths with one significant quantity of material and assessing an activity's probability of detecting an anomaly associated with the diversion of a single significant quantity. If the same activities were performed, probabilities of detection would generally be higher for larger quantities of material and lower for smaller quantities. The type of nuclear material associated with the diversion path and the technical complexity of a diversion and concealment are important because they bear on the probability that a diverter would choose one path over another. To an extent, use of material type and technical complexity is a way to estimate the relative probability of paths without considering characteristics of a diverter explicitly. The aggregate measure, reflecting assessments of the international consultants, ascribes more value to the coverage of paths that involve easy-to-use material and simple concealments than to those that involve difficult-to-use material and complex concealments. This is consistent with the view that a potential diverter is less attracted to a path containing a hard-to-use material like lowenriched uranium than to a path containing an easy-to-use material like plutonium metal. Likewise, a diverter is less attracted to difficult concealment strategies (e.g., those requiring spent fuel dummies with correct gamma signatures to deceive a non-destructive assay) than to easy ones (e.g., those that involve concealment by falsification of accounting records). Specific vulnerabilities of the safeguard system are considered in relation to the objective of deterrence through risk of detection. A diverter is likely to be attracted to a diversion path that is known not to be covered, which is represented by a path with zero probability of detection. The aggregate measure encourages coverage of all paths with some probability by raising value more quickly for increasing the probability of detection of paths that are not covered than for paths that have some probability of detection (by raising the probability to the power of 0.3). More importantly, the procedure suggested for implementing the method includes a recommendation for randomizing the choice of some inspection activities to provide some probability of coverage for every path during each inspection (see below). The timeliness of detection is related to the goal of "the timely detection of diversion". Timeliness is reflected in the aggregate measure through a factor that reduces value if inspection activities are not performed often enough to detect a diversion within one conversion time.

7*

R. Brown and J.W.Ulvila

100

Prioritization Procedure

The prioritization procedure begins with a calculation of the value of the aggregate measure for each inspection activity. Next, an estimate is made of the time required to perform each activity. With this information, an allocation of time can be determined by selecting activities that provide the most value per hour of inspection time. That is, activities are prioritized on the basis of their value-to-cost ratios. Figure 2 displays the cumulative value of inspection activities versus inspection time for a prioritized list. This example is for a pressurized water reactor. The figure shows, for instance, that an inspector can provide coverage that is about 80% effective by performing the first 7 of the possible 17 activities. These seven activities require only about 30%of the total time.

I

00-

10

20

-~

30

I

I

I

40

50

60

Inspection hours

Figure 2. Valuc of Inspection Activities vs. Hours of Inspection Time

With a limited amount of inspection time, an efficient allocation would choose activities in order of priority that could be completed in the available time. However, such a strategy is certain to leave some diversion possibilities uncovered if there is insufficient time to cover all paths. Furthermore, since the method is likely to be public knowledge, a potential divsrter could find out which paths would be left uncovered. The diverter might then focus attention on diversion through those paths. This problem might be addressed analytically as a two-person game between the IAEA and a diverter. This would involve constructing utility functions for the IAEA and the diverter over outcomes of different diversion strategies and solving the game using a concept like equilibrium.

DECISION ANALYSIS IN INTERNATIONAL NUCLEAR SAFEGUARDS

101

However, the analysis is not permitted to model a potential diverter explicitly. (In addition, it is not clear that game theoretic concepts of solutions-which provide advice to both sides jointly--would be of much use for providing advice to IAEA individually .) A modified procedure that uses efficiency information but provides some coverage of all paths is under consideration. In this procedure, some activities would be designated to be performed during every inspection (e.g., the first seven activities in Figure 2). These activities would provide coverage of a large fraction of the total diversion paths, especially those that are of low technical complexity and contain easy-to-use nuclear material. The remaining inspection time would be devoted to activities selected on a random basis from those remaining. These activities would typically cover few paths that are more complex or involve hard-to-use nuclear material. This procedure provides maximum coverage of the most desirable paths and some risk of detection for all paths.

Appendix :Aggregate Measure Definition The following aggregate measure has been proposed for consideration in evaluating the effectiveness of an inspection regime at providing coverage of a facility (IAEA, in preparation). Technical terms included in t h s definition are further defined in IAEA (1980).

where VF,R is the aggregate measure for facility F and inspection regime R; i is the index for nuclear material type and technical complexity level class; F is the facility; is the importance weight for class i (see below); Wi is the number of diversion paths in class i; ni is the index for diversion paths; j t’ is an index for time (expressed in physical inventory verifications, PIVs);

102

R. Brown and J.W. Ulvila

T' f(t') 1 2.4 t T m "TI Pijt

TI' J

is the time period covering the four most recent PIVs; is the weighting function for time (see below); is a normalization factor over T'; is an index for time (expressed in conversion times, CTs); is the material balance period (excluding the earlier PIV); is the fractional value for multiple coverage of a path during T; is the number of CTs for class i in time period T; is the probability that the inspection activities that provide coverage for time t would detect an anomaly associated with a diversion along path j in class i if such an anomaly exists; and is the set of coverage times in T, for path j , excluding the one containing Max pijt T

I )

Importance Weight

Values, in the form of importance weights, were assigned by the group of international consultants to the relative importance of providing coverage of different types and forms of materials for three levels of technical complexity. These values, on a 0 to 100 scale, are as follows (Shea et al., 1981): Technical Complexity Level A

B

C (most complex)

100

58

24

Pu; HEU; U-233 separated from fission products

82

47

15

Pu; HEU; U-233 in spent fuel

45

20

6

LEU; NU; DU; Th

25

12

4

Material Type

Pu; HEU; U-233 metal

Weighting Function for Time

The weighting function for time, qt'), is the product of an importance weight and a coverage factor for the period. A proposed importance weight function is:

DEClSlON ANALYSIS Ih INTERNATIONAL NUCLEAR SAFEGUARDS

1 0.2

2 0.4

1 0.25

2 0.5

Time Period (in PIVs) Importance

3 0.8

103

most recent period 4 1 .o.

A proposed coverage function is:

Time Period (in PIVs) Coverage

most recent coverage 3 1.00.

References Gruemm, H., 1980. Designing IAEA safeguards approaches. Nuclear Materials Management. INMM Proceedings, 14-22. International Atomic Energy Agency, 1968. The Agency’s safeguards system (1965, as provisionally extended in 1966 and 1968). INFCIRC/66/Rev. 2. Vienna: Author. International Atomic Energy Agency, 1972. Structure and content of agreements between the Agency and states required in connection with the treaty on the nonproliferation of nuclear weapons. INFCIRC1153. Vienna: Author., International Atomic Energy Agency, 1973. Statute. Vienna: Author. International Atomic Energy Agency, 1976. Short history of non-proliferation. Vienna: Author. International Atomic Energy Agency, 1978. Non-proliferation and international safeguards. Vienna: Author. International Atomic Energy Agency, 1980. IAEA safeguards glossary. IAEA/SGIINFIl . Vienna: Author. International Atomic Energy Agency, in preparation. Safeguards assessments: Uses, concept, procedures. IAEA/SG/INF/x.Vienna: Author. Shea, T.E., E.W. Brach, and J.W.Ulvila, 1981. Allocation of inspection resourczs. Rocekdings of the 3rd Annual Symposium on Safeguards and Nuclear Material Management. Ulvila, J.W., 1980. A Method for Determining the Best Use of Limited Time Available for the Inspection of a Nuclear Facility. Falls Church, Va: Decision Science Consortium, Inc. Uhrila, J.W. and Brown, R.V., 1981. Development of Decision Analysis Aids forNonProliferation Safeguards. Falls Church, Va: Decision Science Consortium, Inc.

This Page Intentionally Left Blank

A DECISION ANALYTIC SYSTEM FOR REGULATING QUALlTY OF CARE IN NURSING HOMES: SYSTEM DESIGN AND EVALUATION David H. GUSTAFSON University of Wisconsin Center f o r Health Systems Research and Analysis Wisconsin Health Care Review, Inc.

Robert PETERSON, Edward KOPETSKY, Rich Van KONINGSVELD, Ann MACCO, Sandra CASPER Wisconsin Health Care Review, Inc.

and Joseph ROSSMEISSL University of Wisconsin Center for Health Systems Research and Analysis

Abstract In an era of deregulation and diminishing resources, it is essential that effective methods of program evaluation be identified and implemented. This paper describes a methodology for program evaluation which was applied to monitoring the quality of care in nursing homes. While one component of this process lnvolves a new sampling scheme for review of resident populations, this paper deals primarily with the development and testing of a multiattribute utility model of nursing home quality. Results indicate that the method presented here has several advantages including less time spent applying the method and a higher number of more severe problems detected.

Introduction Decision analysis has begun to demonstrate its potential as an evaluation methodology. Edwards et al. (1975) presented a decision theoretic approach to program evaluation which has since been used to evaluate the Community Anti-Crime Program of the U.S. Law Enforcement Assistance Agency (Snapper and Seaver, 1980). Von Winterfeldt (1978) has used decision analysis to evaluate the environmental effects of the North Sea oil drilling programs. Gustafson et ul. (1981b) have constructed utility models to measure the seventy of illness. That severity measure is being used to evaluate the effectiveness of emergency medical services in the U S . (see Gustafson et al., 1981d).

106

D.H. Gustafson el al.

The study reported here differs from previous research in that decision analysis itself is the intervention to be evaluated. This research uses decision analysis as a fundamental part of a new regulatory system and evaluates its effectiveness in monitoring the quality of care in nursing homes. This new regulatory process is now being field-tested in Wisconsin and is being used, with variations, in New York and Massachusetts. A study by the State of Wisconsin’s Medicaid Management Study Team (MMST) (see Gustafson er al., 1981a) found significant problems with the existing approach to monitoring quality of care i n nursing homes. First, the regulatory process was very expensive. Each of the State’s 500 nursing homes was required to undergo an annual survey monitoring compliance with over 1500 regulations. Each survey took a four-person team several days to complete. Second, since each nursing home was surveyed every year, substantial resources were invested in assessments of both good and deficient homes. Few regulatory resources remained for actual promotion of changes in quality of care. The MMST study, in fact, showed that enforcement action was taken against only 2@hof nursing homes found in violation in 1977. Finally, the existing quality assurance process required an assessment of each nursing home patient’s level of care to determine, in part, whether or not the needs of the residents were being met. This review required an average of 20 days of professional staff time to complete. Moreover, its focus on medical records meant that nursing home staff was rarely interviewed and residents seldom examined. This exacting review process seemed to have little impact on nursing home care. So, while the process of regulating nursing home quality seemed ponderous and ineffective, there was a general consensus among professionals and lay persons alike that it was easy to tell a good home from a bad home. The first smell on entering a facility, the presence and treatment of bedsores, the daily variety of the food service menu, the attitudes of personnel toward residents, etc. could clearly identify the good home or the bad home. Aware that an expensive, time-consuming, harassing regulatory process was not working and that a common wisdom regarding good nursing homes versus bad apparently existed, the Medicaid Management Study Team proposed a new approach to monitoring quality of care in nursing homes. That approach was based on the principles of decision analysis and statistical quality control. Briefly, the statistical quality control contribution was the design of a sampling scheme in whch a sample of residents was reviewed rather than an entire population. The sampling scheme required that 10%of the

REGULATING QUALITY OF CARE

107

nursing home's population (or a minimum of 10 residents) be included in the survey of the home. Upon arrival at the facility, the survey team would assign a number to each paiient in the population and select for review those patients whose number appeared on a list of random numbers generated earlier by computer. Under this new design, each resident in the sample was examined, staff were interviewed, and medical records were examined; whereas only record reviews had been conducted previously. Criteria for satisfactory performance were established. If the home failed to satisfy these criteria in the sampling study, then all its residents were reviewed. The idea was to focus the review on patient care arid to leave the good nursing homes alone as much as possible. This paper will focus on the decision analytic aspects cf this new quality assurance process. The same basic idea of a contingent regulatory approach is embodied in the decision analytic component of the quality review. Whereas the statistical quality control component focuses on patient (or resident) review, the decision analytic component focuses on facility review. The decision analytic component is a multi-attribute utility model of nursing home quality. That model and the evaluation of its use in regulating nursing home quality are at the focus of this report.

A MAU Model of Nursing Home Quality The process of developing a multi-attribute utility (MAU) model of nursing home quality began with the assembly of two panels of respected theoreticians and practitioners in the field of long-term care. The Integrative Group Process (see Custafson ef al., 1981c) was used to develop that model. 'The lntegrative Group Process (IGP) is based on the assumption that the type of group process employed can affect the quality of the judgments elicited from the group (see McNamara, 1978). In the IGP approach, groups carry out two basic activities. First, in "qualitative modeling, groups identify the factors and measures to be used in evaluating quality of care. Second, in "quantitative estimation", groups estimate the strength of the relationships that those factors have with quality. The rather sparse literature (see Nemiroff and King, 1975; Hall and Watson, 1970; Gustafson et al., 1980; and Nemiroff ef al., 1976) suggests that effective qualitative modeling processes and quantitative estimation group process should include basic characteristics. These guidelines formed the basis of the ICP: (1) Group members should individually formulate their own models prior to group discussion.

108

D.H.Gustafson et al.

Group discussion should be preceded by informal interaction allowing each group member to establish "legitimacy" in the group. ( 3 ) The group should have an initial "straw man" model on which to focus attention in early stages of group discussion. (4) Time should be available to interpret the ideas of each participant into a formal conceptual model. (5) The modeling procedure should be divided into discrete phases to provide a task structure for the group. (6) A group facilitator should be used to channel discussion in ways that keep the groups task-oriented.

(2)

Index Development Strategy Participating theoretical and applied long-term care experts were convened in two panels. Panels were comprised of gerontologists, therapists, nurses, social workers, and nursing home administrators from universities, state government, and the long-term care industry. Potential physician-panelists were nominated by recognized opinion leaders in the long-term care field. Those receiving several nominations were invited to participate with a panel. Each panelist agreeing to serve was interviewed at length by telephone by the panel facilitator. First, the nature of the project and the use to which the Quality of Care Index would be put were reviewed. The panelist was then asked to identify the important indicators of nursing home quality (indicators for which data would be likely to be recorded) and to provide an example of a poor quality home and a good quality home for each criterion. The panelist was asked to list as few indicators as possible in order to focus attention on a small set of critical variables. (The more that marginally relevant variables are included, the lugher the likelihood of their spurious effect on the final model.) Responses to the telephone interview were examined to identify indicators and concepts frequently mentioned by all panelists. From these a "tentative" quality model was prepared. The first meeting of the group was for the purpose of qualitative modeling. In this meeting, panelists reviewed the tentative model, modified that model, and reached a consensus on a final set of variables from which the quality of care index would be constructed. They also specified operational definitions for use in measuring these variables during observations of the nursing home. The panel reconvened for quantitative estimation the next morning. At that time, the panel reviewed the model they had developed, provided

REGULATING QUALITY OF CARE

109

utility estimates, and made weight assessments for the MAU model. By the end of the session, sufficient quantitative data had been collected to develop additive MAU models for the group as a whole and for each individual in the panel. The IGP produced a list of factors, weights on those factors, and operational definitions for use in measuring the factors. A second panel was then convened to review and refine the model. The model was placed into a format that could be used by surveyors to screen nursing homes (see Figure 1). The idea was that the MAU model could be used to guide a brief review of a nursing home. If the home passed the review, then its survey was complete. If only a few minor problem areas were found, then the surveyors would seek to correct those problems. If the screening process found many deficiencies or a few substantial problems, a complete survey could be implemented using the full array of 1,500 regulations mentioned earlier, legal investigators, and special review teams to collect data that would be useful in court proceedings. In the original regulatory method, surveyors were limited t o citing violations (an action that leads to fines). In the new method, surveyors were to select the one intervention that would most likely lead to a change in the way care was delivered. The intervention could be a violation citation, but it could also be consultation, training, political pressure, etc. It should be noted that a score for each nursing home was not calculated as a result of this process. The MAU model as described above served only as the basis for this new quality assurance process.

D.H.Gustafson el a/.

110 Quality Assurance Project FACILITY ASSESSMENT

FACILITY _ _ ._ -- EVALUATOR __ O A T € -__ / O l O K RESIDENT CONOITION COMMENTS Personal appearance lgroomed /not) -__ ___.___ _ _ ~ !-Odor (no problemlproblem) Clothing I appropriate / inappropriate, cleanlnot) Mood Ihawvlaluml (oDen/afraid to talk1 ... - Awareness Ialert /drugged / disor iented - no activity ) -Physical condition ldecubiti. catheters, geri chairs /good ) -Behavior IappropriateInot) -Appropriate safety devices I used / n o t )

1

I

11 !

~

t+

~~

J 1

~

CARE MANAGEMENT

COMMENTS

-Plan of care (goals appropriate /not) ( t o t a l needs /medical needs only) -Resident involved in own care (yeslnegligible) -Records system (meaningful /routine1 (self-helpful /only to meet regs) -Restorative care (geared to needs / not 1

I

I

I

utilizedInot)

-____

-

-Staff role in planning. evaluation (periodic reevaluation / n o G -Evidence of communication among interdisciplinary stuffs (yes /no)

-

-

RESIDENT IMPORTANCE COMMENTS -Between - resident communication Ipromote interactions, voluntary gatherings/ inhibit. prevent Interactions) -Variety of activities (many in-and-outdoor I T V . bingo only) -Residents' lifestyles and conditions matched to activities Icommunity. facility. bedside /no effort to individualize) - Focility altivity space I adequate / inadequate I --__ -Religious I ties to home churches,-bedside /no services) -___ -Ties to community ( y e s l n o ) -~ -__ -Volunteers I a c t i v e I I i t t ~ r o l e )(trained /untrained1 -Ties to family lagressive attempt to involve family / n o )-___ -Residentouncil ( resident-run /staff r u n / no council) -Staff knows residents and provides some continuity of lifestyle lyeslno) - R e c e n t s ' rights respected lyes/no) -Resgent roam assignments (appropriate /not) -tiandling of problem residents ( discharge quickly / keep) __

t3gure I . M A U Model as Adapted for Facility Review

REGULATING Q U A L I l Y OF CARE

111

A Pilot Test The MAU model was pilot tested prior t o implementation. Five teams of two qualified people each were gathered t o pilot-test the model. Nine nursing homes-six proprietary homes, t w o church-affiliated homes, and one municipal home - agreed t o open their doors t o these teams of judges. The pilot-test judges were paired so that each of the five teams consisted of one 'activities person' and one nurse. One person on each team also had experience in nursing home administration. The teams first met with the administrator of the nursing home who answered questions raised by the team members. The teams next toured the facility. During this time the screening instrument served as a guide for investigation. After the tour, each team member individually completed his or her screening questionnaire and shared it with the other team member. When consensus did not exist and questions on the instrument could not be answered, the team members collected the necessary data by returning to the appropriate places in the nursing home. Discussion was encouraged between members of the same team; however, discussion was prohibited among teams t o ensure that later comparisons of survey results would be valid. Part of the final survey day was spent suggesting revisions in the evaluation procedure. In addition, each team provided a general assessment of the relative quality of care provided by each of the homes. This general assessment was performed in addition t o the assessment done with the screening instrument. Later, all the judges from all the teams were given a list of the nine homes visited in the pilot test. Each judge individually ranked the homes in the order of the general quality of care delivered and assigned 100 t o the "best" home and scores between 0 and 100 t o the remaining homes. These ratings were assigned so as to reflect the relative quality of those other homes. For instance, a home receiving a score of 99 could be assumed t o be about twice as "good" as a home receiving a score of 50 in this rating process. The judges were not permitted t o discuss their scores with each other a t any time. The ratings from both judges in each team were then averaged t o give a team score t o each home. This average general assessment was used later t o test the strength of agreement among the evaluating teams. Three different measures of effectiveness were used in the pilot test of the model: (1) General team assessments of nursing home quality were correlated with the values assigned by the MAU model. Strong agreement

I12

D.H. Gustafson el al.

would indicate that the screening model replicated the judgments of a team of respected surveyors. (2) Screening instrument ratings were correlated across teams. High correlations would suggest that the model was being reliably applied by several different groups. ( 3 ) Screening model ratings were correlated with the number of deficiencies cited in the home during the most recent standard or "old method" survey. Since the screening process had been developed as an alternative to what was considered to be an ineffective system, this correlation was expected to be poor. The results of the pilot test, as summarized below, were encouraging: (1) It appeared that the average general assessments could be correlated with the screening instrument ratings and that the assessments could be used as one standard of comparison for the screening approach. (2) The screening process was reliable. When a composite model rating (obtained by averaging the scoresgiven by all teams to each indicator in the screening model) was compared to the scores given by each team, the correlation averaged 0.78. (3) The correlation between the composite screening rating and the number of deficiencies in the most recent State survey was 0.53 (significant at the 0.10 level). When the number of deficiencies was compared to the average general assessment, the correlation was 0.1 1,

A Field T e s t S t u d y Design

Based on the results of this pilot study and through the efforts of officials in Wisconsin's state government, the United States Government granted permission for Wisconsin to field-test the new Quality Assurance Process (QAP) in 320 nursing homes. The homes were randomly assigned to control groups and experimental groups according to a three-part design. 1 . Rural. One group of 120 nursing homes in a rural area of Wisconsin was assigned to a four-cell design. Thirty homes were to be reviewed using the full QAP process (both MAU model and statistical quality control system). Thirty homes were reviewed by the original regulatory process. Thirty homes used the MAU model and the original resident review. Thirty homes used the original facility review and the new statistical quality control system (sampling). A picture of this design follows:

I

Facility Review

Patient Review

MAU

Original

Sampling

30

30

Original

30

30

v

The same State survey teams used all four methods. So, in one home, they would use MAU and sampling. In another home, the same team would use the original methods of facility and patient review. 2 . Urban. A group of 40 nursing homes in a major urban area was selected for the study. These homes were randomly assigned to two cells. Surveyors used the original patient and facility review process for homes in one cell. In the other cell, the new patient samplinglfacility review MAU model was used (see Figure 1). One team was assigned to do "new-methodA homes only. One team did only "original method" homes. 3 . Suburban. An additional 160 nursing homes in suburban areas of Wisconsin were assigned to two cells: the original patient and facility review method and the new samplindMAU approach. Eighty of the homes were randomly assigned. The other 80 homes were geographically located in two separate areas of the state. These two areas and the homes in them were very similar in terms of socioeconomic and demographic character. Survey teams were assigned to apply one of the methods (either original or new MAUlsampling).

Hypotheses Five hypotheses guided the Quality Assurance Process (QAP) field-test. The QAP was to: (1) Lead to a reallocation of time such that more time would be spent in poor quality homes and less time spent in good homes. ( 2 ) Identify and take action on more problems than the old method. (3) Correct problems and prevent recidivism more effectively than the old method. (4) Lead to an improvement in the quality of care in the nursing home to a greater extent than would the original method. (5) Lead to actions that would tend to be consistent with effective change agents more frequently than the old method.

8

114

D.H.Gustafson e t a / .

Measure of Effectiveness I . Time Allocation. A time study of the surveyors applying the QAP method and those applying the original method was conducted. Each day for a four-week period, surveyors completed a self-ieport of their activities for the day. All surveyors in the state completed this self-report o n which they indicated the types of activities as well as the name of the nursing home in which the activities were applied. The activities were subdivided into three categories: annual visit, interim visit, and office activities. Annual visit referred to the time when the MAU/sampling method or the original method was applied t o assess quality of care. The interim visit was used for consultation, training, surprise surveys, etc. Office-related work should be self-explanatory. The research staff had collected quality of care ratings Tor each nursing home in the state by polling state surveyors, obtaining the advice of nursing home association directors, and eliciting the opinions of knowledgeable people in state government. This iterative process led to a classification of homes into good, medium, and bad groups. Finally, data were collected on-site describing how much time surveyors spent in the annual survey visit t o the nursing home. These data were used t o convert percenttime information into personnel hours. We investigated the amount of time spent in a home, the change in that time as a function of the home's quality, and allocation of time t o various activities. 2 . Problem Detection. A random sample of 60 nursing homes was chosen t o examine the effectiveness of the original method versus the MAU/sampling approach. The sample included 30 homes using each method, split evenly among urban, rural, and suburban areas of the state. Records were examined to determine the number o f problems identified in each home. Those problems could result in a citation being issued or in another action, such as training, taking place. If a citation were given, it would be classified as A (most serious), B (serious), C (not a threat to patient well-being), or F (violation of federal but not state statutes). Each problem (even if n o citation were given) was classified by research staff according to what federal or state code was violated. Five panels of qualified people were convened t o rate the relative seriousness of each problem identified. Inter-rater reliability studies of this rating process were conducted by including between 10 and 35 duplicate problems in the packetsrated b y each panelist. The correlations among the panelists indicating inter-rater reliability were high, ranging from 0.78 to 0.98. Two measures of effectiveness were developed for this study: ( I ) The number of problems detected by the original method versus the MAUlsampling process.

REGULATING QUALITY OF CARE

115

The severity of problems detected by the original method versus the MAU/sampling process. 3 . Recidivism. Obviously, the objective of a quality assurance program is not only to identify a problem but also to correct that problem. The recidivism measure of effectiveness is intended to examine the extent to which problems identified by the two quality assurance methods continue t o exist when the nursing home survey is repeated one year later. Data on recidivism will eventually be presented in two ways: (1) Recidivism rate-the relative frequency with which a problem occurs. (2) Severity/Recidivism rate--each problem that reoccurs will be weighted by the severity of that problem. Reoccurrence of a problem was determined by having a team of nursespecialists review records of two years of visits to the sample of 60 nursing homes mentioned in the previous section. Problems (or types of problems) that occurred in both years were identified. Severity ratings of problems from the first year’s visit were obtained as described in the previous sec. tion. Those severity scores were used to weight the reoccurring problems. Data on recidivism rate will be reported here, data on the severity weighted recidivism rate are not yet available. 4. Quality of Care. The most important change to observe is a change in the overall quality of care delivered by a nursing home. T o this end, an instrument for measuring quality of care has been developed that will be applied t o a sample of 130 homes in the study. The reason a separate index was developed for use in this effectiveness measure was that an independent assessment of changes in quality of care was needed. A standard of comparison was required against which the new Quality Assurance Process could be evaluated. The Quality Instrument was designed t o be applied i n separate visits to the nursiiig home both before and after the new QAP was applied t o the home. The Quality Instrument is built on the same conceptual model used t o develop the MAUlsampling method. However, the instrument is quite different in its methods of measurement. First, it is intended to yield an overall measure of quality but is not intended to identify the source of specific quality problems. Second, the instrument is designed t o be applied in one day rather than the two or t h e e days required by the MAUlsampling method. 1 Third, the Quality Instrument embodies a set of rigid guidelines for measurement designed to ensure inter-rater reliability. (This is currently being tested.) The variables in the model are combined using a MAU format. (2)

8*

D.H. Giistafson et al.

116

Results 1 . Time Allocation. Table 1 is indicative o f the results of this study. The figures presented are for two nursing homes each with 100 beds. the average size of a nursing home in Wisconsin. The data compare the original method and MAU/sanipling method across a “good’’ quality homes and a “poor” quality horrie. The data are further divided into annual and interim visits, as well as office work. The data suggest that the total time required t o conduct a survey is smaller in both “good” and “poor“ homes when the MAU/sampling approach is employed. Moreover, the MAU/sampling approach leads to a much more impressive shift in time allocation based on quality of the home. These results suggest the MAU/sampling method may be less expensive, yet more sensitive t o the quality of the home. Both were intended results of the new Quality Assurance Process.

’

Original Method

MAU/Sampling

I 12.5

Annual visit

12.5

Intcnm V i u t 6.2

2.9

10.1

Related to Home

I p . 8

16.8

26)

Bad Home

26

I .2

1.4

Office Work

Good Homc

Good Home 2.1

Bad Home

3.8

Good Home 4.6

Good Home Bad Home

I

Bad Home

31

.d

32.1

1 Total Time

2. Problem Detection. Table 2 represents the comparison of the number of problems detected using the two methods. The results suggest the MAU/sampling method results in fewer ”C” level (patient well-being not threatened) citations and about the same number of “B” or “A“ violations (threats to patient well-being). MAU/sampling also issues fewer federal violation notices. However, the MAUlsarnpling approach identified

REGULATING QUALITY OF CARE

117

and treated a greater number of problems handled in ways other than citations ("C" level problems where consultation or training was used in place of citations; problems not related to codes, etc.).

20

40 60 Increasing Severity Score

80

100

Figure 2. Cumulative Distribution of Severity Ratings by Method (Provider Panel)

MAU/Sampling N = 30 homes

Original N = 30 homes

I_ -Noncode problems -1 _ Violation

F-level violations

-

C-level Type

C-level violations A&B-level violations Totals

62 (19.4%) 32 (10.0%) 9

(2.8%)

I

89 (25.8%) 15 (4.3%) 146 (42.3%)

202 (63.1%) 15

(4.7%)

320 (100.0%)

14 (4.06%) 345*

I

D.H. Gustafson er al.

118

Figure 2 plots the cumulative distribution of seventy for the problems identified. The results indicate a superiority of the MAU/sampling approach over the original approach to quality assurance. These results suggest that, as intended, the MAU/sampling approach is identifying more problems and more severe problems. Moreover, the MAU/sampling approach is dealing with less serious problems in more flexible ways by using citations, consultation, and training rather than forcing intervenors to rely almost solely on citation' procedures. 3 . Problein Resolution. Results of this study are still incomplete. However, current results are shown in Table 3. Problems are listed along the rows according to their disposition in Year 1. If the problem was identified again in Year 2 and dealt with in any way, that problem was classified as a repeat. So, for instance, of the 68 citations issued by the MAU/sampling method in Year 1, five were identified as still being present in some form one year later. The results suggest that recidivism rates are about the same overall in the MAUlsanipling and original methods. However, when one examines the source of recidivism, we see a much lower recidivism rate among cited problems in the MAUlsampling method. It is assumed the cited problems have higher severity and we therefore conclude that the MAU/sampling method deals with more severe problems in a more efficient manner.

r

Citations Issued

5/68

Citable Problems Dealt with in Other Ways

(7.4%)

14/92 (15.2%)

NonCitable Problems Identified

4/35 (11.4%)

Total

231188 (12.2%) A

231148 (15.5%)

Oll

(0%)

0114

(0%)

231163 (14.1%)

.

Conclusion and Recommendations In an era of deregulation, we run the risk of abdicating our responsibility to monitor the quality of services provided by regulated industries while pursuing the laudable cause of reducing red tape. Decision analysis offers

REGULATING QUALITY OF CARE

119

the prospect of an alternative (contingent regulation) in which a screening process can identify and quickly discharge problems in the agencies delivering good care while concentrating attention on those agencies that need assistance. The results of this study suggest that this prospect is very much alive. The quality assurance method described here seems to have several advantages over other more cumbersome and expensive approaches. Time allocation dies seem to be contingent upon the quality of the nursing home. More problems are being identified and corrected. Yet, the methods used to deal with these problems are more selectively chosen. The current study is not completed. Data are still needed on a number of dimensions including impact on quality of care. However, the results are promising indeed.

References Edwards, W., M. Guttentag, and K. Snapper, 1975. Effective evaluation: A decision theoretic approach. I n : E. Streuning and M. Guttentag (eds.), Handbook of Evaluation Research, VolJ. Beverly Hills, CA: Sage Publications, Inc. Gustafson, D.H., C. Fiss, J. Fryback, P. Smelser, and M. Hiles, 1981. Quality of care in nursing homes: The new Wisconsin Evaluation System. The Journal of Long-Term Care Administration, Y (2). Gustafson, D.H., D. Fryback, and J. Rose, 1981. Severity index methodology development research project. Final Report of Grant No. HS 02621, funded by the U.S. National Center for Health Services Research. Gustafson, D.H., D. Fryback, J. Rose et al., 1981. A decision theoretic methodology for severity index development. Working Paper. Center for Health Systems Research and Analysis, University of Wisconsin-Madison. Gustafson, D.H., P. Jotwani, and G . Huber, 1981. The effects of disclosure as a conflict reduction technique on planning and priority setting. Working Paper. Center for Health Systems Research and Analysis, University of WisconsinMadison. Gustafson, D.H., G. Juedes, J. Rose, D. Detmer et al., 1981. A model to measure and explain the effectiveness of emergency medical services. Working Paper. Center for Health Systems Research and Analysis, University of Wisconsin-Madison. Hall, J. and W. Watson, 1970. The effects of a normative intervention on group decision making performance. Human Relations. 23, 299. McNamara, DE., 1978. The effects of structure, disclosure, closure, and mathematical form of the decision on the small group decision making process. Unpublished doctoral dissertation, University of Wisconsin-Madison. Nemiroff, P.M. and D.C. King. 1975. Group decision making performance as influenced by consensus and self-orientation. Human Relations, 28, 1. Nemiroff, P., W. Passmore, and D. Ford, 1976. The effects of two formative structural interactions on established and ad hoc groups: Implications for improving decision making effectiveness. Decision Sciences, 7,841.

120

D.H. Gustafson et al.

Snapper, K . and D. Seaver, 1980. The use of evaluation models for decision making: Applications to a community anti-crime program. Technical Report 80-1. Decision Science Consortium, Inc. von Winterfeldt, D., 1978. A decision theoretic model for standard setting and regulation. Research memorandum RM-78-7. Laxenburg, Austria: International Institute for Applied Systems Analysis.

Section I1 ORGANIZATIONAL DECISION MAKING

This Page Intentionally Left Blank

INTRODUCTION Janos VECSENYI and Detlof von WLNTERFELDT

Decision analysis is based on the theory of individual decision making and many of its applied techmques are derived from or related t o psychological scaling methods. It is therefore not surprising that decision analysts often experience some stress when they apply decision analysis outside of the domain of individual decision making, e.g., to group or organisational decision problems. The four papers of this chapter share a concern about the difficulties in applying decision analysis to organisational problems, as well as d hope that through an understanding of the underlying organisational issues and an adaptation of analysis, many of these problems can be overcome. Larichev presents a critical survey of systems analysis for organisational decision making. Systems analysis enlarged the domain of the earlier versions of operations research by including "softer '' and ill-structured problems, by allowing subjective judgments to enter the analysis, and by focusing on evaluation and decision making. The problems that systems analysis faces in organisational decision problems are, therefore, as Larichev points out, not very different from those of decision analysis. They include limitations of numerical trade-off analysis, problems in measurement of "qualitative" variables, problems in cases of group decisions and organisational conflicts, and difficulties in implementing the results of analysis effectively in organisational decisions. Larichev argues that if systems and decision analyses are to increase their range of applications they have to pay greater attention to issues of human factors affecting decision making, and to the role of groups and to the nature of organisational decision processes. This wdl most likely involve the development of procedures for problem analysis able to model and compare the decision maker's changing policies on objectives and the means of their accomplishment.

124

I. Vecsenyi and D.von Wintcrfeldt

Many of these points are emphasized in Lock’s paper which stresses explicitly the institutional obstacles t o carrying out and implementing decision analysis. Lock argues that many characteristics of organisational decision processes contradict the traditional decision analytic “ideology”. For example, decision makers are hard t o identify, and decision points are often vague and ill-defined events in a continuous organisational decision process. Goals are seldom stated explicity, but rather emerge through the formation of coalitions among individuals. In contrast to decision analytic principles, organisational goals are often held ambiguous, sometimes fuzzy, and almost always adaptive and subject to change. And finally, analysts, rather than being neutral enforcers of rationality, are likely to become immersed in organisational power games. Lock concludes his discussion with several concrete suggestions how analysts can raise their consciousness about organisational issues and develop-largely clinical-skills in coping with them. The third paper in this chapter discusses a number of specific pitfalls of decision analysis in organisational settings. Von Winterfeldt summarised the experience of several decision analysts arid the failures, problems, or other difficulties they encountered in applications of decision analysis. These pitfalls range from hidden agendas by the clients of the analysis, over wrong problem formulations, t o gaming in probability and utility elicitation, and failures of implementation. These experiences provide a background against which lessons are drawn for improving decision analysis. For example, suggestions are for developing a client-analyst relationship, maintaining a flexible definition of the problem, recognizing and overcoming institutional obstacles, etc. in the final paper Vari and Vecsenyi describe their experiences in three applications of decision analysis t o research and development problems. The pitfalls included being restricted by the decision maker to work only on part of the problem, existence of dependent and overlapping attributes, problems with means-ends relationships in value trees, and the rejection of SEU type considerations by some managers.

SYSTEMS ANALYSIS AND DECISION MAKING Oleg I. LARICHEV Institute for Systems Studies, Moscow, U.S.S.R.

Abstract According to current definitions, systems analysis is a combination of procedures and analytical methods used for the study of ill-structured problems. The concept of systems analysis is broader than that of decision making, including also procedures of problem investigation known as a "systems approach". The systems approach is a train of logical stages: definition of a goal or a set of goals; identification of alternative ways of goal achievement; construction of the model presenting the interdependence of goals, means and parameters of the system;determination of the dccision rule for selecting the preferred alternative. Thus, we may define systems analysis as a combination of the general framework of the systems approach with decision making tools. The last stage is in fact that commonly known as decision making. However, in earlier versions of systems analysis only a "cost-effectiveness" criterion was usually applied in the decision rule. The paper examines the basic features of contemporary systems analysis methodology and its difference from operations research. The capabilities and limitations of systems analysis are also analyzed. The requirements for decision making methods appropriate for use with ill-structured problems are defined and an approach to the development of methods in line with these requirements is proposed. Ways of improving systems analysis methodology in the light of this approach are discussed.

Introduction During the last two decades there have appeared a number of branches of research somehow or other connected with the problem of choice of the best decision alternative out of a set of potential options. Along with the traditional topics of our conference-decision malung-we could also mention such branches as artificial intelligence, cybernetics, systems analysis and a host of others. Systems analysis is probably the most popular of

126

0.1.Larichev

all thanks to a number of large-scale problems having been solved through its application. It is defined by Enthoven (1969) as a "reasonable approach to decision making correctly defined as quantitative common sense". One can easily spot the similarity of systems analysis and decision making: both deal with processes of complex human decision. A question arises: Does it not come to just a terminological difference? The question is far more important than this as both areas of research are undergoing continuous change and, as we shall presently see, there is a set of common problems connected with their future development. There are also other reasons for making such a comparison. As we shall see further, the potential of decision making methods can only be increased by the skillful use of problem analysis (that is, by the means of systems analysis). Then again, it is useful for specialists in decision making to look as if from the side at the characteristic features of their field of activity. From my point of view the best way to do it is to compare decision making with systems analysis, not least of all because for many people the frontier between them is rat her diffuse.

Systems Analysis Nowadays directly opposing attitudes toward systems analysis are often found: either a strong belief in the omnipotence of t h s new approach capable of solving, at last, complex and large-scale problems, or accusations about the use of fashionable terminology failing to produce any specific recommendations. There are numerous books and articles illustrating the effective application of systems analysis in the U.S. Department of Defense in the sixties and equally, books and articles describing the dismal failure of the approach in the civil departments of the United States administration. What then is systems analysis and what problems is it designed t o cope with? By the end of the 1960s the necessity was ripe for the application of some special analytic techniques to large-scale problems of organizational system management. The methods were designed to introduce consistency and logic into the problem solution. Also,successful application of operations research techniques (Wagner, 1969) resulted in attempts t o extend techniques of scientific analysis to problems involving both quantitative and qualitative, though sometimes insufficiently defined, factors (the former are charasteristic of operations research). Undoubtedly, there are

SYSTEMS ANALYSIS AND DECISION MAKING

127

common features in the operations research and systems analysis approaches to problemstudies. In the frst place, both these approaches are supposed to involve five stages or logical elements (Quade, 1969). These are: 1. to identify the objective or a set of objectives; 2. to find alternative ways of identifying the obejctive(s); 3. to determine the resources required for each system employed; 4. to build a mathematical (under the operations research approach) or a logical (most often under the systems analysis approach) model, i.e., a chain of dependences between the objectives; alternative ways of their accomplishment, environment and resources; 5. to define the choice criterion to be used in identifying the preferential alternative. It is worth noting that these stages are by no means inherent exclusively to operations research or systems analysis. They are also related to systems engineering (Shinners, 1967), decision making (Young, 1966), recommendations to inventors (Altshuler, 1973) and the like, briefly, to cases implying a consistent approach to investigaton of complex problems. Various “systems” approaches are integrated primarily by a set of successive logical stages whichcan be referred to as a general systems approach pattern (Figure 1).

Identify goals and resources

---)

Identify problem solution alternatives

Analytically compare the .-alternatives

Choose the most preferable a l t e r native

i

And what is the difference between the systems approaches to problem solutions? It is in the methods of analytical comparison of alternative decision options. Thus, for example, the operations research approach involves a set of quantitative mathematical programming techniques, network techniques, etc. From the very outset the systems analysis approach was associated with methods of comparison of decision alternatives subsumed under the terms of ’’cost-effectiveness” (Heuston and Ogawa, 1966) and ”cost-benefit” (Enthoven, 1969), with cost-effectiveness being the basic term.

0.1. Larichev

128

Systems analysis was fist applied to military engineering problems (Quade, 1964). An example of such a problem is given in Figure 2 .

/

.5I

100

300 500 %mber of missiles

Cost and effective-

ness synthesis

___)_

Recomrnendat Ions

al U 0

E

f u 0

100

300

500

Number o f missiles

Figure 2. Cost-Effectiveness Modelling of a Military Problem

The model consists of two portions: the cost model and the effectiveness model. The models are utilized for selection of a weapons system comprising a specific number of missiles. The cost model constitutes a total cost dependence on the number of missiles, and the effectiveness model constitutes a dependence of target-destruction probabfiity on the number of missiles. Both models may be treated in this case as objective: they are built on the basis of factual data and reliable statistics. The output parameters of the models are not integrated, however, through a given dependence, instead an executive’s judgement is employed: it is he or she who sets the marginal cost and the required effectiveness. Executives often utilize the cost-effectiveness ratio, but it is recommended simultaneously that attention should be paid t o their absolute values. In thls case cost and effectiveness are the decision evaluation criteria, hence the problem becomes a multicriteria one. The tradeoff between the two criteria values is chosen by the decision maker. It is worth mentioning that cost and effectiveness models were developed in a similar way to the ones that were traditionally invoked with operations research techniques. It was assumed in the majority of the initial applications of systems analysis that the analyst was in the position to identify objectively the major characteristics of the problem and reflect them in the model. The subjective element emerges only in the process of synthesizing the cost and effectiveness

SYSTEMS ANALYSIS AND DECISION MAKING

129

measurements. The orientation of systems analysis towards problems involving both quantitative and certain qualitative factors is well reflected in the popular definition of systems analysis offered by Enthoven (1969) who characterized it as an explicit, monitored, self-corrected and objective process. It is interesting that Quade, one of the founders of systems analysis, distinguished quantitative rather than qualitatve difference between operational research and systems analysis (6.Quade, 1969). However, representing the problem in a way analogous to that presented in Figure 2 is not a central requirement for systems analysis. Systems analysis w a s intended for solution of what Simon defines as ill-structured or mixed problems containing both quantitative and qualitative elements, where the qualitative, scarcely conceived and ill-defined facets of the problems tend to dominate. In the cases where qualitative and ill-defined facets prevail it is hardly possible to build an objective quantitative model. To exemplify a t y p i d ill-structured problem, let us turn to the problem of R & D project choice. One of the standard representations of dependences between parameters utilized in the solution of such problems (Dean, 1968) looks like: CommerResearch Profitcid Annual success X sales ability = X success pro baindicator probavolume bility bility R & D cost

Product X unit price

Steady sales

period (years)

+ Engineering expenses + Marketing expenses

The "profitability indicator", expressing the project value, does indeed depend on the listed factors. Yet it depends on many other variables which are missing from this formula, as for instance the qualifications of potential project executants. The lund of dependences between the variables contained in the formula is not objectively defined; it is just clear that some of them increase the project value while the others reduce it. It is not for nothmg that there is a host of such dependences and there are no objective reasons for singling out any one of them. The given model just reflects the belief of an organization executive that the project choice must be exercised on the presented dependences. Should there be no data necessary for construction of an objective model conceived as the only correct model, its parameters become the decision alternative evaluation criteria. The model itself, however, can be built only on the basis of the subjective preferences of the decision maker. The successful applications of systems analysis within the framework of the famous PPB system (Novick, 1965) can obviously be attributed 9

130

0.1. Larichev

primarily to the successful application of the cost-effectiveness technique to tactical military-engineering problems. As far as the broader class of strategic problem is concerned, there is no basis for the construction of an objective cost-effectiveness model. Nevertheless, attempts at adapting the cost-effectiveness technique to such problems have been and are being made. As is well known, the success of the PPB system resulted in a much broader application of systems analysis. Various planning and decision problems could be found within the range of its application. The uselessness of the method of constructing "objective" cost-effectiveness models was one of the major reasons for the failure of the PPB system in civil institutions, as was convincingly illustrated in the interesting book by Hoos (1 972). Besides, practical experience of application of systems analysis suggests that the general pattern of systems approach is far from versatile. The "magic" of the logical stage sequence of Figure 1 disappears when it comes to complex practical, "wicked" problems (Rittel and Webber, 1972) often leading to trivial recommendations like: "formulate the goals-define subgoals-identify optional means of goal accomplishment", etc. It was found that objectives are significantly affected by the means, that problem setting can be affected by a successful guess concerning the way of its solution and the achievement of the result by the possibility of re-structuring the organizational system (Gershuny, 1978). New and rather significant difficulties were encountered in the course of the application of systems analysis to problems involving several persons or active groups affecting the decision. It is quite clear that equally influential proponents of conflicting standpoints can paralyze any possibility of reaching effective decisions. The objective truth is somehow convincing. And as the decision rules are inevitably subjective, a question arises as to whose preferences must provide the basis for decisions. Systems analysis, oriented at the preferences of a single decision maker, failed to answer thls question. Practical experience of systems analysis application revealed also that the very process of application is an art. It is an art to set a problem correctly "in the solvable manner", to undertake a creative approach in the search of decision alternatives and to select appropriate scientific aids for the analysis. This art is required both of the executive responsible for prbblem solution and of the systems analyst. Successful application of systems analysis depends on a variety of factors. The very criteria for success may be objective and subjective. An experienced analyst can, with a certain degree of success, solve the problem just through a simple discussion of its aspects with many of the contributors to the decision. In this case he acts as a diagnostician (or

SYSTEMS ANALYSIS A N D DECISION MAKING

131

psychoanalyst, cf. Fischhoff, 1981.) Thus, it becomes clear how important the art of problem analysis is. It should be noted that as far back as in the sixties it was understood that the purpose of systems analysis is not a simple extension of operations research methodology to problems featuring a number of qualitative factors, that is, ill-structured problems. Thus, Schlesinger pointed out in 1963 that there are no inner reasons why subjective evaluation problems cannot be treated within systems analysis. He believed that "it was necessary to identify the area of systems analysis application as a problem where the conflict between the numerous incommensurable objectives is solved through judgement". Thus, there was some agreement that systems analysis can be utilized as a tool for problem investigation on the basis of subjective judgement. However, actual successful application of systems analysis to this end was greatly limited by the methods employed for the comparison of alternatives. These remained based on cost-effectiveness techniques which were not suitable for this purpose. The experience of systems analysis applications reviewed so far suggests the following methodological conclusions: First, each technique employed for alternative comparison is not versatile and suits a specific area of application. The cost-effectiveness technique, in its generally known version, is ill-suited for solving strategic choice problems with heterogeneous criteria. It is simply nor suitable for the solution of ill-structured problems typically encountered in social systems (education, health care, environmental protection, and the like). Second, the alternative comparison technique employed affects the overall general systems approach pattern, influencing the content of its individual stages. A unified whole appears as a result. Due to this, the general success or failure of the analytical approach to choice between the alternatives strongly depends on the particular alternative evaluation technique which is employed.

Current Trends in Systems Analysis The current state-of-the-art of systems analysis makes a complex picture. Problems continue to be studied (and sometimes successfidly) with objective cost and effectiveness models analogous to those used in the initial applications of systems analysis. As the boundaries between the various types of problems are fuzzy, there are continuing attempts to build "objective" models for the problems where inadequate objective informa-

9*

132

0.1.Larichev

tion can be supplemented only through subjective judgement. Very often, owing to the lack of any data source, the consulting analyst "patches the holes" on the basis of his own knowledge of relationships between the system parameters. In complex models, information provided in this lund of way affects the final results in an unpredictable manner. The models developed reflect to a great extent the beliefs of their creators that the world is arranged in some particular way. Sometimes the qualitative dependences between the parameters in the model are quite explicit, but it proves difficult (or impossible) to determine the exact quantity of each of these dependences. Hence the consultant "fills the gap", and in so doing also strongly affects the result. The resulting pseudo-objective models are often then found unac ceptable by the decision makers, as they are not based on the executives' experience, intuition, and preferences. As a result the model builders often do not exert any influence on decision making. Though well-known and popular definitions of systems analysis emphasize its direct orientation toward decision making (Quade, 1964, 1969; Schlesinger, 1963), the same term systems analysis has in recent years often been associated with the development of pre-fabricated, "context-free" mathematical models with a view toward creating "stores" of models for potential use in decision making. The understanding of the fact that application of systems analysis represents a combination of art and scientific analysis was used in the service of the "separation" of the analytical aid from art, together with the separation of their further study. This directior. of research has resulted in the emergence of a great number of mathematical models while there is much less data concerning anythmg to do with their application. We believe that practical problems possess characteristic features whch can be reflected only in a model built specially for the problem. This is because in the very structure of a correct model all interconnections must reflect the subjective preferences of a decision maker. Many of the models (e.g., the so-called global models; Meadows and Meadows, 1973) contain a lot of assumptions and premises of their creators intermixed with certain objective dependences. Hence application of such models in decision making is simply dangerous. The experience gained in unsuccessful applications of systems analysis to problems with a subjective structure and a list of parameters brought about two direction of research. The first one, "policy analysis" is concerned with the solution of public policy problems. Research conducted along these lines places a strong emphasis on the art of problem analysis, on problems of the organizational mechanism for decision

SYSTEMS ANALYSIS AND DECISION MAKING

133

implementation, etc. The article by Rittel and Webber (1972) is an example of interesting research conducted in this field. The second direction is concerned with the systematization of systems analysis applications and typical errors and miscalculations. Here it is worth mentioning Quade and Majone's book, "Pitfalls of Analysis" (1980), recently published through the International Institute for Applied Systems Analysis. The book attempts t o systematize unsuccessful approaches to problem analysis, model formulation, consultant-decision maker interrelationships, and so on. For all this, extremely little attention has been paid to improving the methods and procedures for analytical comparison of decision alternatives. As usual, one may encounter in the literature descriptions of cases applying "cost-effectiveness" techniques t o such problems as storage of radio active waste and construction of atomic power stations, though these problems undoubtedly involve a variety of subjective and objective factors. An impression is gained that the authors of these papers have overlooked popular critical articles and books (e.g., J. R. Hoos, 1972; 1. Hoos, 1974) and well known results of the theory of collective choice (Arrow, 1963). In actuality, the major research on methods of comparing complex decision alternatives is being conducted at present by decision making experts, and not by systems analysts.

System Analysis and Decision Making: Parallels

Let us compare some characteristics of systems analysis and decision making. Actual Content Judging by the actual content of research conducted under the terms of decision making and systems analysis one can detect the following differences : 1. Systems analysis is generally of a prescriptive (or normative) nature; i.e., it looks for an answer to the question as to how decisions should be made. In decision making one can identify three branches of research : (a)

descriptive research into decision making processes (see the recent overviews by Slovic er al. 1977; and Einhorn and Hogarth, 1982);

0.1. Larichev

134

(b)

development of normative or prescriptive techniques for decision making (see, for example, Keeney and Raiffa, 1976; multi-criteria techniques are the most popular at present); (c) mathematical research into the problems of choice (see, for example, Fishburn, 1970). It is evident that the range of research directions in decision making is much wider than in systems analysis, though here too the normative techniques in use serve to justify the existence of the entire area as a unified whole (without them we would have had some mathematical and some psychological research). 2 . The class of ill-structured systems subjected to systems analysis features both quantitative and qualitative characteristics and encompasses the business activities of man: decision making in organizational system. Most frequently these are large-scale decisions in large organization. The class of problems subjected to decision making analyses is much wider: it encompasses both problems possessing qualitative and quantitative factors and problems possessing only qualitative factors; ill-structured and non- structured problems. Decision makmg analyses have been concerned with a wide range of business decisions: from planning of astronomical (Boichenko et. al., 1977) or space (Dyer and Miles, 1976) research to consumer choice (Bettman, 1971). Also decision making covers personal, individual problems such as job choice (Cerds et al., 1979) and family planning (Jungermann, 1980). Interrelation

Systems analysis is largely oriented towards procedures of organizational decision malung while decision making is primanly oriented at the techniques for the comparison of decision alternatives. In this connection, ill-structured problem (quantitative-qualitative problems) can well be solved through the application of a systems approach, utilizing appropriate techniques from those developed for decision making analyses. The "cost-effectiveness" technique is not, in our opinion, a vital part of systems analysis, and so it can be replaced with some other for the technique for the comparison of alternatives. What is characteristic of systems analysis, in our opinion, is (a) its orientation towards ill-structured problems; (b) its utilization of a general systems approach pattern; (c) its use of a decision alternative comparison technique which allows the consideration of both qualitative and quantitative factors; (d) its normative, prescriptive nature. If we consider systems analysis from this standpoint, we see that a portion of decision making techniques makes up one of its component parts. Actually, when we look beyond the terminology

SYSTEMS ANALYSIS AND DECISION MAKING

135

employed, we find that we are familiar with such version of systems analysis. It is worth pointing out in this connection that systems analysis and decision making share conimon problems in the development of advanced techniques for comparing decision alternatives relative t o ill-stuctured problems containing qualitative and quantitative factors. The development of these techniques is of paramount importance for systems analysis as the technique employed to compare alternatives determines to a great extent the potential practical application of systems analysis. (We have seen how this has already been the case with systems analysis employing the cost-effectiveness technique ). Subjective and Objective Models As was shown above, systems analysis aims at replacing "objective" models (inherited from operations research) with subjective models better suiting the nature of ill-structured problems. On the other hand, papers on decision making have always recognized the central role of the subject, i.e., the decision maker, and the development of decision making as a branch of research connected with the problem of choice has obviously been strongly influenced by expected utility theory, based on measurement of subjective values.

Validation of Practical Need

To validate the practical need for systems analysis the following advantages are often mentioned (Quade, 1964; Majone and Quade, 1980; Optner, 1965): - systematization of complex decision making processes, - more extensive information support of the decision maker, - services of experienced consultant-analyst, - quantitative comparison of decision alternatives. Most often it comes down t o the efficient utilization of the systems approach (Churchman, 1968, 1980). When validating the need for decision making analyses, the same factors are often mentioned. Beside these there is, in our opinion, yet another significant basis for validation, this time relating to the expansion of the actual capabilities of the decision maker in perceiving complex, multi-attributed information. A skillful and experienced decision maker develops in due course his own approach to choice problems, his own policy. However, in complex decision problems involving a variety of factors this policy is but a tradeoff between the complexity of a problem and the capabilities of the decision maker. These capabilities are limited and the

136

0.1. Larichev

major constraints depend not so much on the individual traits of the decision maker as on the general characteristics of the human data processing system. Thus, the limited capacity of man's short-term memory makes one utilize various methods of data grouping (Cranovskaya, 1974), and different heuristics (Russo and Rosen, 1975). Frequently a person knows the key factors to be considered in the process of decision making but employs them inconsistently by using simplified rules. The flexibility of man, h s ability to adapt information to his potential, can lead to some undesirable consequences: compression of information can result in contradicting and erroneous estimates. For example, Tversky (1 972) has shown convincingly that by utilizing a seemingly absolutely jcstified rule for choosing between alternatives involving the neglect of insignificant differences in evaluations by criteria, one finds oneself in a "trap of contradictions", A more reasonable application of decision making techniques can however contribute to the consideration of many factors, to conscious tradeoffs being made between contradicting criteria and to the reduction of contradictions. All t h l s increases the actual capacity of the decision maker in processing complex, multi-attributed information.

Dissatisfaction with the Current Stateof-theart and Techniques It is characteristic that both systems analysis and decision making have been criticized for their limited capacities. There are by now a lot c f papers supporting the assertion that there is dissatisfaction with systems analysis in its traditional form (i.e., a general systems approach pattern incorporating the employment of cost-effectiveness techniques). This dissatisfaction ranges from complete negation (Hoos, 1972) t o consenting to its application "for want of something better" (Fischhof, 1977). It is worth noting that both proponents and opponents of systems analysis believe that it is inseparable from the cost-effectivenesstechnique. Actually, the major share of criticism concerns the application of the cost-effec tiveness technique in situations where it is inadequate (Shuman, 1976). There has also been some sharp criticism concerning various groups of decision making techniques. There is a multitude of techniques, generally subsumed under these five groups (Larichev, 1979): 1. Axiomatic (MAW; e.g., Keeney and Raiffa, 1979) 2. Direct methods for estimating probability and utility in decision analysis (Raiffa, 1968) and for estimating weighted sums of criteria (e.g., Edwards, 1977).

SYSTEMS ANALYSIS AND DECISION MAKING

137

3 . Tradeoff techniques (e.g., Indifference curves, McCrimmon and Siu, 1975; additive difference model, Tversky, 1972). 4. Threshold comparability techniques such as ELECTRE (Roy, 1968). 5. Man -machine procedures in multi-criteria linear programming (Larichev and Polyakov, 1980). Despite this abundance of methods it is rarely the case that a method actually influences the decisions made through its use. Though in formal terms the axiomatic methods are the best validated ones, the critique of such formal validation (von Winterfeldt, 1975; Rivett, 1977; Fischer, 1979) deserves consideration. With respect to practical application, the weighted sum of criteria estimates technique ranks high in use among the others (Edwards, 1977; n e e , 1971). although its sole advantage is the simplicity of using it. Hence, we cannot simply employ a decision making technique (or a group of techniques like one of the above) in systems analysis, declaring it to be “the most suitable tool for comparing decision alternatives inill-structured (qualitativequantitative) systems”. Choosing the appropriate technique is a complex case indeed as the areas of systems analysis and decision making are in a state of continuing development and there are a number of common problems affecting their future progress.

General Methodological Difficulties Problems of Measurement Both in systems analysis and decision making the decision rules employed (i.e., the models for comparing decision alternatives) are developed on the basis of data received from the decision maker and experts. Hence the methods used to obtain the data are of great, sometimes decisive importance. Some 5-10 years ago it was considered possible to obtain reliable information from man in almost any form desired. Recent research has shed new light on the actual characteristics of man’s performance under particular methods of data acquisition. Thus, Tversky and Kahneman (1974) analyzed the ability of man to provide probabilistic data. Slovic, Fischhoff and Lichtenstein (1977) studied the phenomena of overconfidence and retrospective confidence (Fischhoff and Beyth, 1975). A number of papers (e.g., Russo and Rosen, 1975; Marshak, 1968) indicate that man makes inconsistent, often contradicting holistic judgements of the worth of alternatives made up of partial estimates on multiple criteria.

I38

0.1.Larichev

The problem of assigning appropriate weights to multiple criteria is also difficult for man (Slovic and MacPhillamy, 1974). In comparing multi-criteria decision alternatives man employs heuristics leading to contradictions (Russo and Dosher, 1976; Tversky, 1972; Montgomery, 1977). According to some information, people make systematic mistakes in identifying preferences in lotteries (Dolbear and Lave, 1967). This would introduce errors when determining utilities by lottery methods (eg., Raiffa, 1968). Systematization of the results attained may be found in the overviews (by Slovic et al., 1977; Einhorn and Hogarth, 1981; Aschenbrenner, 1979;and Lanchev, 19lu)). Given all these results, what do they indicate? First of all, the methods for decision making techniques should be revised. If the methods are criticized for failing to consider the real constraints and informational capabilities of men, then it is necessary to take tlus criticism into account at the method development stage. For example, a significant problem encountered when measuring qualitative characteristics inherent to ill-sructured problems is the impossibility of using the quantitative measurement scales employed in current methods in any reliable way. Unfortunately, there are at present no reliable methods of quantitative measurement of the multiple subjective factors involved in decision quality evaluation like the organizational image, the scientific level of problem solution, the technological decision irnpct on society, etc. In measuring these factors we can employ only ordinal scales, with a verbal definition of each grade of quality on the scale. The same position that we face here probably existed for physical variables, such as heat, length and the like, up until the time when quantitative measurement techniques were devised (Carnap, 1965). We shall see whether quantitative measures will appear for "qualitative" factors like those listed above, but currently we have none. Accordingly, it is much better to use reliable ordinal scales than unreliable ratio or interval scales. Along with qualitative factors, ill-structured problems involve also the quantitative factors (i.e., those on which man usually makes quantitative evaluations). Generally speaking, it is desirable that, in describing factors, the measurement techniques employ a language which is amenable to the decision maker and experts who will potentially participate in their use (Larichev, 1979). From my point of view this point is very important. Using adequate language in forming the problem description can roughly be counted upon to increase the trust of a decision maker and an expert in the results of the analysis. Any transformations of information within the model without proper utilization of the decision maker's preferences harm this trust.

SYSTEMS ANALYSIS AND DECISION MAKING

139

Development of Decision Rules for IllStructured Problems The problem of measurement is closely associated with the problem of decision rules integrating quantitative and qualitative factors on the basis of the decision maker’s preferences into a unified rule for the evaluation of alternatives. The problems of identification of preferences are closely linked with the psychology of decision making. As a general case, questions concerning the ability of man to provide the type of reliable information required for the development of a decision rule are the subject matter of specialized research. The word “reliable” implies here the following three characteristics: (1) recurrence in subsequent interviews; ( 2 ) transitivity; (3) the possibility of formulating complex rules for estimating decision alternatives (Larichev ef al., 1980). Hypotheses concerning ways of obtaining the data must be proposed and checked in model experiments (Larichev, 1979). Something has already been achieved to this end. It is known, for example, that the application of several heuristic rules substantially reduces the number of errors made when comparing multi-criteria systems (Montgomery, 1977). It is also known that man can be taught to evaluate subjective probabilities of events better (Lichtenstein el al., 1977j. Also, given a certain number of criteria and binary estimates, man can classify multi-dimensional alternatives on ordinal scales (Larichev et a/., 1980). It is well-known that, using verbal evaluations on ordinally scaled criteria, man can compare the effects of decreasing estimates along the scales of two criteria while the estimates on all other criteria under estimation are set to create the image of the best (or worst) decision (Larichev, 1979). This method lies behind the ZAPROS technique (Zuev et al., 1979). Of particular significance for ill-structured problems is how to integrate qualitative and quantitative factors within a model. These problems also typically involve objective dependences between a number of factors and some objective models. The integration of all this on the basis of subjective preferences into a general model for evaluation of the decision alternatives implies a host of complex methodological problems. Must consideration be given first t o qualitative factors, followed by the quantitative ones (Emelyanov and Larichev, 1976)? How, for example, does one compare the cost (represented on an uninterrupted quantitative scale) with undesirable spinoffs (represented on qualitative scales)? It should be pointed out that any simplifications introduced into problem solutions in the face of such difficulties will significantly affect the confidence of the decision maker in the results of the analysis.

140

0.1.Larichev

Collective Decision Making Both systems analysis and decision making face the problem of collective decisions, the problem of viewpoints held by different groups influencing the decisions. Despite the need here for practical methods, we find that results taken from purely theoretical models prevail in decision making practice (see Emelyanov and Nappelbaum, 1978, for an overview). From the pragmatic point of view it is clear that there is a need for procedures for the discussion and harmonization of opinions contributing to a collective decision. It is clear, though, that no procedures will be helpful in cases where there are well thought through and entrenched contradicting policies (e.g., an environmentalist versus a proponent of industrial development). We strongly believe that a thorough consideration should be given to the problems which lie in between those of personal and collective decision making and its implementation. In contrast to problems characterized as individual decision making, however, here an individual decision maker is guided by the criterion of maximum satisfaction of interests of those persons (or groups) who have an interest in the problem solution. Should there be a decision option satisfying all of the participants then the preferences of the decision maker do not affect the outcome. Illustration of the impossibility of attaining such a result influences the participants and reveals the true position of the decision maker. As was correctly noted in Thurow (1969), such a demonstration is very important. Moving further on, the executive compares the individual utilities from his own point of view but tries not t o be autocratic and tries to find a decision which is close to being "natural". An overview of a number of works concerning this problem can be found in Rosenman (1978), an example of one of the papers is that by Larichev and Kozhukharov (1979).

Rational Procedures of Roblem Analysis Modem systems analysis is in need of additional research aimed at the development of procedures allowing the generalization of results and discoveries of talented analysts concerning problem setting and ways of implementation. This is equally true for decision making as it is rarely the case that a real-world problem consists simply of the comparison of decision alternatives. Usually, this is preceded by extensive work on the part of the consulting analysts. And should they succeed in achieving a situation allowing the application of a decision making technique, it would be safe to say that the problem is 70 per cent solved.

SYSTEMS ANALYSlS AND DEClSlON MAKINcl

141

What must the real procedures of problem analysis be like? Most probably these must be iterative procedures for comparing the changing objectives (and means of their accomplishment) which account for the major lines of the decision maker’s policy. Certainly, the art of their application will play the central role but a certain framework will appear permitting a less skillful executive (and/or consultant) to perform on a iugher level.

Conclusion The last 20 to 30 years have witnessed the emergence of decision making techniques for the analysis of problems that have traditionally been considered as unyielding to rational research. The methods of problem analysis are far from perfect and do not take into account the complex nuances of human decision making. The process of improving analysis techniques involves periods of excessive belief in the versatility of particular methods of problem solution (e.g., mathematical models), followed by periods of absolute scepticism concerning the efficiency of the said techniques. The truth is somewhere in the middle: there are groups of problems which are procurable with adequate analysis techniques but the majority of real problems still lack such tools for their solution. The current stage of development of systems analysis and decision malung dictates the necessity of considering the human factors which affect decision making. To the forefront come psychological and sociological factors. The problems of cognition and the study of decision makers’ policies and the role of active groups acquire an ever greater significance. We believe that the future of systems analysis depends on the successful solution of these problems: either it will flash and then vanish as the next ”fashonable” trend, or gain strength, integrating knowledge from a variety of scientific disciplines with the art of problem analysis.

References Altshuler, G. S., 1973. The Algorithm of Invention (in Russian). Moscow:Rabochi. Arrow, K. J., 1963. Social Choice and Individual Values. New York: Wiley. Aschenbrenner, M., 1979. Komplexes Wahlverhalten als Problem der Informationsverarbeitung. Universitat Mannheim, Sonderforschungsbereich 24. Bettman, J . R., 1971. A graph theory approach to comparing consumer information processing models. Management Science, 18. 114- 128.

142

0.1. Larichev

Boichenko, V. S., 0. I. Larichev, V. A. Minin, and H. M. Moshkovich, 1977. The method of the development of scientific and technological policy for the planning of fundamental research. In: V. A. Trapeznikov (ed.), The Management of Research, Development and Introduction of New Technology (in Russian). Moscow: Economiks. Carnap, R., 1965. Philosophical Foundations of Physics. Ncw York: Basic Books. Churchman, C. W., 1968. The Systems Approach. Delta Books. Churchman, C. W., 1980. The Systems Approach and Its Enemies. New York: Basic Books Dean, B. V., 1968. Evaluating, Selecting and Controlling R & D Projects. New York: American Management Association. Dolbear, F. T. and L. B. Lave, 1967. Inconsistent ;behavior in lottery choice experiments. Behavioral Science, 12 (1). Dyer, J. C. and R. F. Miles, Jr., 1976. An actual application of collective choice theory for the Mariner JupiterlSaturn 1977 Project. Operations Research, 27. Edwards, W., 1977. How to use Multi-Attribute Utility Measurement for social decision making. IEEE Transactions on Systems, Man and Cybernetics, SMC- 7 (5). Einhorn, H. J. and R. M. Hogarth, 1981. Behavioral decision theory: Processes of judgement and choice, Annual Review of Psychology, 32. Emelyanov, S. V. and E. L. Nappelbaum, 1978. The Principles of Rational Collective Choice (in Russian). Moscow: VINITI. Emelyanov, S. V. and 0. I. Larichev, 1976. A multicriteria approach to applled R & D planning: The case of qualitative criteria. Problems of Control and Informtion Theory, 5- 6, 385-399. Enthoven, A., 1969. The systems analysis approach. In: Program Budgeting and Benefit-Cost Analysis. Pacific Palisades, California: Goodyear Publishing Co. Fischer, G. W., 1979. Utility models for multiple objective decisions: Do they accurately represent human preferences? Decision Sciences, 10, 45 1-479. Fischhoff, B., 1977. Cost-benefit analysis and the art of motorcycle maintenance. Policy Sciences, 8, 177-202. Fischhoff, B., 1981. Decision analysis: Clinical art or cllnical science? In:L. Sjoberg, T. Tyszka and J. A. Wise (eds.), Decision Analyses and Decision Processes. Lund: Doxa Fischhoff, 8. and R. Beyth, 1975. I knew it would happen-remembered probabilities of oncefuture things. Organizational Behavior and Human Performance, 13. Fishburn, P. C., 1970. Utility Theory for Decision Making. New York: WiIey. Cerds, U., K . Aschenbrenner, S. Ieromin., E. Kroh-Puschl, and M. Zaus, 1979. Probiemorientiertes Entscheidungsverhalten in Entscheidungssituationen mit Mehrfacher Zieisetzung. In: H. Ueckert and D. Rhenius (eds.), Komplexe menschliche Informatwnsverarbeitung. Bern: Huber. Gershuny, J. I., 1978. Policymaking rationality: A formulation. Policy Sciences, 9., 295-316. Granovskaya, R. M., 1974. Perception and Memory Models (in Russian). Leningrad: Nauka. Heuston, M. C. and G . Ogawa, 1966. Observations on the theoretic basisof costeffectiveness. Operations Research, 14., (2). Hoos, I. 1974. Can systems analysis solve social problems? Datamation, June, 82-92. Hoos, J. R., 1972. System Analysis in Public Policy. Los Angeles: University of California Press.

SYSTEMS ANALYSIS AND DECISION MAKING

143

Jungermann, H., 1980. Speculations about decision theoretic aids for personal decision making. Acta Psychologica, 45, 7-34. Keeney, R. and H. Raiffa, 1916. Decisions with Multiple Objectives: Preferences and Value Trudeoffs. New York: Wiley. Klee, A. J., 1971. The role of decision models in the evaluation of competlng environmental health alternatives. Management Science, 18 (2). Larichev, 0. I., 1979. Science and Art of Decision Making (in Russian). Moscow: Nauka Larichev, 0. I., 1980. Process tracing for the problems of estimation and choice of multicriteria alternatives in decision making. In: S. Emehnov and 0. Larichev (eds.), Descriptive Studies of Multicriteria Decision Making Processes (in Russian). Moscow: VINITI. Larichev, 0. 1. and A. N. Kozhukharov, 1979. Multiple criteria assignment problem: Combining the collective criterion with individual preferences. Mathematique et Science Humines, 68, 63- 77. Larichev, 0. I., V. S. Boichenko. H. M. Moshkovich, and L. P. Sheptalova, 1980. Modelling multiattribute information processing strategies in a binary decision task. Organizational Behavior and Human Performance, 26. 218-29 1. Larichev, 0. I. and 0. A. Polyakov, 1980. Interactive procedures for solution of multicriterial mathematical programming problems (in Russian). Economika i Matematicheskije Methodi, 1. Lichtenstein, S., B. Fischhoff, and L. D. Philhps, 1977. Calibration of probabilities: The state of the art. In: H. Jungermann and G. de Zeeuw (eds.), Decision Making and Change in Human Affairs. Amsterdam: Reidel. McCrimmon, K. R. and J. K. Siu, 1914. Making trade-offs. Decision Science, 5. Majone, G. and E. S. Quade (eds.), 1980. Pitfalls ofAnalysis. New York: Wiley. Marshak, J ., 1968. Decision making: Economic aspects. International Encyclopaedia of the SocialSciences. New York: Crowell, Colier, Macmillan, vol. 4. Meadows, D. L. and D. H. Meadows (eds.), 197 3. Toward Global Equilibrium. Wright Allen Press. Montgomery, H. A., 1977. A study of intransitive preferences using a think aloud procedure. In: H. Jungermann and G. de Zeeuw (eds.), Decision Making and Change in Human Affairs. Dordrecht: Reidel, 1965. Novick, D. (ed.), 1965. Program Analysis and the Federal BudRet. Cambridge, Mass.: Harvard University Press. Optner, S. L., 1965. Systems Analysis for Business and Industrial Problem Solving. Englewood Cliffs, N.J.: Prentice Hall. Quade, E. S. (ed.), 1964. Analysis for Militav Decisions. Chicago: Rand McNally and co. Quade, E. S., 1969. Systems Analysis Techniques for Planning-Programming -Budgeting: Systems Approach to Management. Chicago. Raiffa, H , 1968. Decision Analysis: Introductory Lectures on Choices under Uncertainty. New York: Addison-Wesley. Rittel, H. and M. Webber, 1972. Dilemmas in a general theory of planning. Policy Sciences, 4. Rivett, P., 1917. The dog that did not bark. Engineering Economics, 2 (4). Rosenman, M. I., 1978. Methods of collective decision making with participation of the decision maker. In: S. Emelyanov (ed.), Multicriteria Choice in Solving Ill-Structured Problems (in Russian). Moscow, VINITI, 38-48.

I44

0.1. Larichev

Roy, B., 1968. Classement er choix en presence de points de w e multiples (la methode ELECTRE). Revue Franfais d e s Informatiques et Recherches Operationales, 2 ( 8 ) . Russo, J. E. and 9. A. Dosher, 1976. An Information processing analysis of blnary choice. Technical Report, CarnegbMellon University. Russo, J. E. and L. D. Rosen, 1975. A n eyc fixation analysis of multi-alternative choice. Memory and Cognition, 38 267--276. Schksinger, J . R., 1963. Quantitative and national security. World Polirics, 1.5 ( 2 ) . Shinners, S. M., 1967. Technique for System Engineering. New York: McCraw Hill. Shuman, J., 1976. Mathematlcal model bullding and public policy: The game some bureaucrats play. Technological Forecasting and Social Change, 9. Slovlc. P . , B. Fischhoff, and S . Lichtenstein, 1977. Behavloral decision theory. Annual Review of Psychology, 28. Slovic, P. and D. MacPhillamy, 1974. Diniensional commensurability and cue utilization in comparative judgment, Organizational Behavior and Human Performance, 1 1 . Thurow, L. C., 1980. The ZeraSurn Society. New York: Baslc Books. Tversky, A., 1972. Choice by elimination. Journal of Mathematical Psychology, 76

(I). Tversky, A. and D. Kahneman, 1974. Judgment under uncertalnty: Heuristlcs and biases. Science, 18.5. von Winterfeldt, D., 1975. An overview, integration and evaluation of utility theory for decision analysis. Technlcal Report 75-9. Los Angeles: Social Science Research Instltute, Unlversity of Southern Californla. Wagner, H. M., 1969. Principles of Operations Research. New Jersey: Prentice Hall. Young, S., 1966. Management: A Systems Analysis. Glenview, Illinois: Scott, Foresman and Co. Zuev, Ju. A., 0. I . Larichev, V . A. Filippov, and Ju. V. Chuev, 1979. The problems of R & D project estimations (In Russian). Vestnik AcedemiNauk, 8.

APPLYING DECISION ANALYSIS IN AN ORGANISATIONAL CONTEXT* Andrew R LOCK Kingston Poly lechnic, England

Abstract The structures and political processes of client organisations present a number of potential pitfalls for decision analysis applications. Above the purely operational level, decisions appear within organisations as disruptions and imply changes in the relative power and status of participants. Uncertainty about such changes is a source of resistance to the analysis process. The balance of individual goals within overall organisational goals results from the internal power structures and political processes, and the nature of the dominant coalition. The presence of an 'executive' coalition with the power to take explicit decisions and to implement them markedly increases the likelihood of acceptance of 'rational' choice models such as decision analysis. In other situations, analysis takes place within a more complex political framework and has to be seen more as a contribution to a broader organisational decision-making process. In the light of this, c h i c a l strategies are proposed for the decision analysis process Particular emphasis is placed on the nature of the 'contract' before entering the organisation, and on problem definition and structuring.

Introduction As Bross (1953) points out, decision consdtancy has distinctly dubious antecedents. Decision analysts play similar roles to those played in the past by medicine men, soothsayers, prophets, wise men, sorcerers and even witches-and the field has always been vulnerable to charlatanry. It would be comforting to believe that resistances encountered in practical applications were due to the scepticism engendered by these predecessors, and

* The author would like to thank the editors and two anonymous referees for their extremely useful suggestions, and t o express gratitude t o the participants in the Pitfalls in Decision Analysis symposium in Budapest, notably Rex Brown, Ward Edwards, Dave Gustafson, Larry Phillips and Detlof von Winterfeldt for providing many of the ideas which stimulated the revisions of this paper. 10

146

A.R. Lock

that a few demonstrations of the effectiveness of' newer decision-making technologies would serve to quell unbelievers. Apart from the fact that some of this technology is cumbersome, testing the credulity of subjects and clients, I argue here that much of the decision analysis seed will be cast on stony ground, without an awareness of the organisational environment and inherently political process of applied decision analysis. The decision analysis paradigm has generally met most success in organisational applications at the operational level. Despite the fact that practitioners have felt that the technology would be better applied to the less-structured problems of strategic directions for the organisation, applications in this area have not met with conspicuous success. Part of the problem is that it has rarely been admitted that resistance was encountered or that the recommendations were not taken up. Such adnussions are revealed betwsen decision analysts but have to date received little or no attention in the main stream of decision analytic literature. Exceptions are Fischhoff s (1977) comparison between decision analysis and psychoanalysis and Phillips' (1980) discussion of the relationship between organisational structure and the ability t o cope with uncertainty, both of which stimulated many of the thoughts that have gone into this paper. Interestingly, similar concerns about the breadth of applications and the acceptability of recommendations are being expressed in the O.R. and Management Science literature (Ackoff, 1979a and b; Eilon, 1980). As one studies problems at higher levels in organisations, the range of options and the intervention process itself have organisational implications, which links analysis to political processes within the organisation. If the objective of a study is acceptance of its prescriptions, analysis strategies have to include a deliberate consideration of the structure and processes of the organisation involved. The goal of this paper is to explore the issues of power and political decision-making processes witlun organisation and the extent to which these relate to decision analysis and its implicit ideology. Specific pitfalls are discussed and suggestions are made for approaches to the analysis process whch might increase the probability of acceptance and successful implementation of the ensuing recommendations. Controversy seems inevitable. Some of the views expressed in this paper might be dismissed as being mainly applicable to a conventional conservative U.K. corporate environment. It is true that the decision analysis paradigm has been found to be less successful in Europe than in North America. However, many of the pitfalls mentioned have been encountered in U.S. applications. Given the small number of decision makers and the disparate problems and situations with which analysts have

DECISION ANALYSIS IN ORGANISATIONAL CONTEXT

147

worked, any development of a theoretical framework for the organisational context for decision analysis has to be based on an eclectic selection of material from many areas.

Decision Analysis and Ideology Decision analysts often see their work as disinterested and, in helping decision makers, as a value-free embryonic science, despite the many demonstrations that science does reflect a particular Weltanschauung. Fischhoff (1 977), for example, has neatly identified the rationale for much of this work with the ideological biases of therapeutic interventions in the field of psychotherapy. The belief that analysis is de facto a valuable intervention is at the heart of the decision analyst's credo, and underlies the concept of intellectual technology promulgated by Bell (1979) and echoed by Phillips (1 980). A structured approach t o decision making reflects, among other things, both a desire for orderliness, and a belief that an orderly solution exists. Thus can sometimes lead to analysis becoming a form of ritual that has to be gone through in making a decision. In some such cases, the impression is left that decision analysis becomes a form of (unwitting) casuistry for salving clients' consciences over morally difficult decisions. Under decision analysis lies a model of rational choice. Its roots are explicitly normative and are based on the general proposition that there exists a set of clearly definable goals or objectives which can be modelled in terms of well-defined preferences. Findings that SEU models can be poor predictors of actual individual choice behaviour within an organisational context (Simon, 1978, 1979), have not led decision analysts to question whether they are a good basis for improving decision making. Consequently 'better' decisions are often defined in the analyst's terms rather than those of the client. As well as carrying with them the normative content implicit in the paradigm, analysts frequently enter decision situations with preconceived ideas about the structure of problems and the preference structures of the individuals involved. Particularly in business organisations, verbal statements of goals and the structures elicited are influenced by external norms perceived by the subjects and/or the analyst. As an example, discounting is a frequently used technique to aggregate money or utility flows in different time periods; Lock (1982) reports the results of preference analyses for one corporate decision maker whose relative weights for money flows increased up to the third year out and only declined slowly thereafter. 10*

148

A . K . Lock

T h s does not mean that analysts should avoid making any personal inputs. The passive role is rarely an option. Decision analysts should be aware of their own goals and be prepared to question whether the assumptions and structure being used are consistent with the broad perspective of the client.

Decisions, Procedural Rules and Organisational Upheavals

In discussing organisational decision processes it is convenient to distinguish between one-off decisions and the repeated application of procedural rules. The latter may be laid down by executive fiat or established custom. They represent ways of responding to situations or events that can be categorised in terms of those that have previously been encountered. There is an analogy here with the procedural scripts that individuals apply in repetitive choice situations (Abelson, 1976). Which procedure to use is not always easily determined, and is sometimes the subject of a procedure in itself, or has a stochastic component. The degree of development of such rules is largely dependent on the external environment and the type of problems that the organisation encounters. Organisations, or sections of organisations, in stable environments tend to have well-developed if not ossified procedural rules, and are olten referred to as mechanistic (Burns and Stalker, 1961). As organisations become larger a higher level of programming, in the form of standardised procedures and control systems, takes place (Pugh and Hickson, 1976), although, at the top of large organisations decision problems tend to become less structured, because of the lengthening of the time span of the decision maker as organisational size increases (Jaques, 1976). Within the organisation, decisions appear as events or episodes, representing discontinuities in the organisational continuum. The more of these discontinuities that a section of an organisation and its members experience, the less likely they are to perceive them as major upheavals and the better they will be at handling uncertainty (Phillips, 1980). Thus, the best opportunities for decision analysis applications are offered by lateral or matrix structures, based on a task or project oriented culture, which develop as a response to the need for flexibility in a changing environment. In other types of organisational structure, the climate for decision analysis is less favourable. In vertical-hierarchical organisations, which are the dominant form in stable situations, decisions are perceived as significant upheavals and there is a low ability to cope with uncertainty. In power cultures, where power is concentrated at the centre, or person cultures,

DECISION ANALYSIS IN ORGANISATIONAL CONTEXT

149

where power is diffused according to expertise (Handy, 1981), decisionmaking is highly politicked and there is a tendency to reject explicit choice models.

Contrasting Perspectives

For most practising decision analysts, even internal consultants, the frequency with which they change problems or situations makes it difficult to appreciate the position of people in an organisation whch has undergone relatively few upheavals, particularly as they do not experience the longer term consequences. Organisational upheavals imply changes in relationships and relative power positions. The more deeply established these are, the greater the obstacles in the path of change and those who appear to be harbingers of change. The insecurity that one frequently observes in this type of situation reflects primarily the uncertainties relating to the organisational consequences as opposed t o the uncertainties that analysts are usually concerned with measuring.The uncertainties that engender insecurity concern organisational positions, relative power, interpersonal relations and interactions with the exterior environment. As a newcomer to an organisation, the analyst lacks a perspective on the particular organisational culture. Although it is quite easy to list the factors that influence the type of culture-history and ownership, size, technology, goals and objectives, environment, and people (Handy, 198 1)-it is rather harder for an outsider t o gain a full understanding in a short time. Things are further complicated by the differences in culture between groups-the dominant coalition and affected groups lower down the organisation, for instance. Outsiders frequently find that the most established parts of the culture are taken for granted and therefore not mentioned and others may be taboo subjects.

Power in Organisations Pettigrew (1973) observed that analysis of organisations as political systems had not been widespread. Indeed it has seemed that power in organisations has been a subject that people did not wish t o address. There is a distinction between power and authority in organisations. Authority is voluntarily obeyed being seen as a legitimate contribution to the functioning of the organisation which the recipient of orders has usually joined voluntarily. Organisation charts map the national authority structure. Against h s is the concept of power whereby individuals or coalitions

150

A.R. Lock

are in a position to change positively or negatively the achievement of the goals of the organisation or others within it. It may not be exercised directly but represent an unspoken internalised constraint upon decisionmakers. The power of organisational actors is determined by their ability in performing their organisational role and the importance of that role (Pfeffer, 1981). There are a number of general forms and sources of power in organisations: Power over resources. Clearly, possession of resources that others want and need confers bargaining power. This extends beyond the obvious examples of money and slack resources t o the provision of jobs and career advancement of subordinates. Coping with uncertainty. Individuals who are able to cope well with uncertainty, even to the point of being able to repair machines that are crucial to the functioning of a system, acquire power. Mintzberg (1978) suggests that consultants are hired t o absorb uncertainty for the organisation. Being irreplaceable. As technological specialisation increases, the more likely it is that certain actors possess skills or knowledge that renders them irreplaceable. Strikes by key computer personnel, which prevented the government collecting large amounts of tax revenue, enabled unions in the U.K. Civil Service to maintain pressure on the government for some long time in 1981 at relatively little cost. Similar dependence patterns are to be found increasingly in organisations. Controlling the decision process This flows from control of information about alternatives, and will need little amplification for decision theorists. It can also stem from control over agendas (Plott and Levine, 1978). Power of consensus The degree of internal cohesion and shared world view markedly affects the influence a sub-group can have o n the organisation and its credibility and consequent influence with other actors. Possession of political skills. It is unlikely that an individual can exercise power purely by the possession of political skdls, though that may have been the means by which he or she achieved a position with other sources of power. Political skills determine the effectiveness with w h c h other sources of power are used. They become paramount in organisations where the other sources of power for individuals and coalitions result in a fairly even balance.

DECISION ANALYSIS IN ORGANISATIONAL CONTEXT

151

The concept of power and its sources are important in analysing situations particularly those which are approached as a newcomer. Few participants in decision processes involving outside consultants are remote from the political processes in organisations and so that derived accounts of relative power are usually slanted and freqiieotly very sketchy. Resistance t o bringing power issues into the open is often encountered (Pettigrew, 1973). Thus much of the analysis of the political processes in the organisation has t o be done by direct observation and inference rather than relying on participants’ accounts. Neither is the decision analyst divorced from the political process. Studies are commissioned by those that already possess power and are often seen as means to reinforce that power. Analysts enter the situation with power based on special skills and external status and acquire additional internal power through association with the dominant coalition. The conventionally presented role of the decision analyst allocates a considerable implied degree of power to him or her. It is not assumed there that the decision maker can understand the techniques by which the probability and outcome inputs to the model are translated into the model, nor is it assumed that the decision maker should understand what is going on in the utility assessment phase. If decision makers accept the full implications of t h s form of analysis, they are in fact abdicating control over it. In the long run this may well be perceived as a threat to their position. Indeed decision makers may well feel threatened by the notion that their ’real’ policies may be captured by models, let alone by simple models. There may be criteria or goals to which they are unwilling to admit, and unwilling to have revealed. Particularly in the case of middle managers, a subject might be placed in the position of covering up his or her true preferences and then being faced with a solution, based on declared preferences, that he or she finds unacceptable. It is remarkably difficult for a manager in a supposedly ’rational’ role to argue against a supposedly rationally derived solution to a decision problem. The willingness of subjects to go along without a reasonable understanding of the process is often dependent on the perceived status of the analyst. Analysts with lower perceived status may find it hard t o persuade subjects to participate in some of the tasks required. For example, Lock (1979, 1982) reports a case study where some of the difficulties in persuading subjects t o participate in certain elicitation exercises were due t o the analyst’s lack of perceived status in that case as a doctoral student.

152

A.R. Lock

Organisational Decision Processes Decisions in organisations emerge from the relative influence and interests of different individuals and coalitions. The specific individuals and coalitions involved in a particular decision will depend upon the groups of people affected and the level of decision w h c h is being considered. These levels range from the strategic, concerned with the overall direction of the organisation, down to immediate operational decisions at the most basic level. The political nature of the bargaining processes implicit in making these decisions is usually only seen when the distribution of interest, expertise and power is different from the explicit authority structure. In reading studies of organisational decision-making piocesses. one notices how little is written on the actual formal mechanisms and ntuals as opposed t o the line-ups in terms of coalitions and relative power. Little is said about the extent to which decisions are made by individual executive action, taken by groups or cabals in informal meetings, or taken in committee. Decisions are perceived to emerge rather than there being always a readily identified decision point. This reflects the extent to which 'decisions' in many instances are the cumulative consequence of a set of minor preceding actions (or inactions). Weiss (1980) has suggested the phrase 'decision accretion' for this pattern. Simdarly, the power of other groups may only be identifiable as internalised constraints on the part of the decision-maker . These form part of the 'frame of reference' or 'world view' of the decision maker and other organisational participants. Without an understanding of these viewpoints, as is noted by Boulding (1959), analysts will understand little of the process. Some studies of organisational decision processes suggest that goals are largely imputed by observers. Weick (1979) and March (1978) question the usefulness of notions of preferences at the organisational as well as the individual level, observing that they result from choice rather than determining it. In this framework choices are acquired, learnt, or they are the result of the influence of other participants. Where does all this leave practitioners of decision analysis'? It appears that in many organisations decisions are difficult t o identify and that there tends to be a very distinct desire to adhere t o and reinforce existing strategies (Staw, 1976). One possibility consists of changing the goal of an analysis as making a contribution to the organisational process rather than specifically recommending a course of action and getting it adopted. Fischhoff (1977) suggested that thls contribution is frequently the main benefit of decision analysis and saw this as a fundamental objection to Watson and Brown's (1978) proposals on the valuation of decision analysis. The rejection of any formal recommendations does not imply

DECISION ANALYSIS 1N ORGANISATIONAL CONTEXT

153

that no useful result has been achieved; the process of structuring the problem and the information relating actions to outcomes may have been a significant influence on the organisational decision process. Rejection of formal proposals does, however, suggest that somewhere along the line there was a lack of understanding of organisational culture, decision processes, and political structure.

Organisational Goal Structures

“Organisationsdo not have objectives, only people have objectives’’ (Cyert and March, 1963). Despite this view, it is rare that the goals of an organisation can be clearly identified as those of one individual participant and equally rare t o find an individual all of whose goals are satisfied by one organisation. It is clear that individuals satisfy their goals by multiple means inside and outside the organisation and that they adjust them in the light of experience. What is less clear is the processes by which organisational goal structures emerge from individual ones. Goal structures serve functions of coardination and direction ot organisations. Ideally, at each level in the organisation the overall goal structure should be broken down into goals appropriate to problems that ate encountered at that level while remaining consistent with the goal structure at higher levels. Decision problems are usually perceived in terms of a difference between some actual or forecast state and a desired state. Whilst the desired state or goal may be re-evaluated and adjusted in the light of experience, an absence of goals or aspiration levels removes the possibility of developing any coherent framework for action. The extent to which members of organisations have similar individual goal structures is affected by the way in which the organisation selects and admits recruits, the self-selection of potential recruits and the extent to wiuch newcomers are socialised into the organisational culture. Participants join internal coalitions to further personal goals, though the goals pursued by the coalition are an amalgam of members’ goals where the relative importance of individuals’ goals depend on their power. Some useful examples may be found in MacMillan (1978). The nature of the dominant coalition in the organisation depends on the patterns and diffusion of power in the organisation. Richards (1978) provides the following categorisation of dominant coalitions: - an executive coalition, based on the chief executive and those to whom he or she delegates power,

154

A . R . Lock

a bureaucratic coalition, where essentially the chief executive has lost power to the component departments, - a political coalition, where no one group has sufficient power of itself, and political slulls become dominant, - an expert coalition, where power stems from individual skills and expertise; Handy (1981) describes this as a 'person culture'. These different patterns of dominant coalitions result in markedly different goal structures. In the first case, there tend to be clearly stated corporate goals of a familiar conventional nature; in the second, overall goals are very much secondary to sub-unit interests; in the highly politicised organisation. it may be very difficult to identify any general goals at all; finally, in the expert dominated organisation, the overt pursuit of individual goals predominates. Unless the balance of power between coalitions is evenly balanced or coalitions are potentially unstable, decision analysts are introduced into the organisation by members of the dominant coalition. It may be seen that the strategy of the analyst will vary according to the nature of that coalition. 'The easiest one to deal with is the executive coalition, where problems and preferences are likely to be sufficiently articulated to produce an analysis acceptable to the decision maker and with the top-down commitment which is seen as the most successful route to change by specialists in Organisational Development (Porter el d., 1975). In the bureaucratic coalition, analysis a t the departmental level is likely to be similarly successful, but attempts to develop strategies at a higher level seem likely to fail. Within politicked and 'person culture' organisational contexts, there will be no single individuals with sufficient cornmitment and power to implement a set of recommendations even if it were possible to sufficiently discern an overall set of goals to actually make formal recommendations, though it may be possible to contribute to organisat.iona1 decision processes here by structuring available options and focussing attention on areas of agreement and disagreement.

-

Pulling Aside the Shroud of Ambiguity Whlle it is frequently claimed as an advantage of decision analysis that it throws light into areas that have previously been vague and shadowy, there remains what March (1978) has called the 'optimal clarity problem'. One aspect of this problem concerns the extent to which preferences need to be clarified. Explicit discussion of trade-offs and relative preferences has a tendency to accentuate differences in opinion and viewpoint. Sensitivity analysis on the final solution of an analysis frequently shows

DECISION ANALYSIS IN ORGANISATIONAL CONTEXT

155

that it is remarkably stable with respect to varying opinions and preference structures. Thc analyst may then concentrate on those differences of view to which the decision is sensitive, rather than creating unnecessary conflict at an earlier stage. The requirement that goals be clearly articulated at the outset conflicts with views (eg., Weick. 1979; Humphreys and McFadden, 1980) that goals are adjusted or changed as a consequence of investigating or experiencing the outcome of behaviour. Having 'fuzzy' goals may serve similar functions to other heuristics in a dynamic environment (Einhorn and Hogarth, 198 I). The insistence on a clear definition of goals at a particular point of time may lead t o an impoverished set, w h c h will not produce a decision that is 'better' in whatever subjective terms clients may choose to apply. One also has to consider the impact of attempts to structure decision problems on intermediate groups. The problem is particularly acute when one is dealing with middle managers, who are frequently unwilling t o expose either their forecast of actual outcomes or their own preferences to senior management for fear of ending up in a position they feel would be untenable. They would be responsible for the implementation of a chosen strategy and would carry the can for its success or failure. Conventionally middle managers survive by the exploitation of ambiguity in terms of goals and strategy outcomes. The difficulties faced by them are discussed by Uyterhoeven (1971). The maintenance of some level of ambiguity enables subordinates to protect their interests to some extent vis h vis the dominant coalition in the organisation. The pressure from senior management to expedite decision processes has a tendency to become an extension of their control over the organisation and a reduction of the autonomy of middle management . Thus there are advantages in not placing too much emphasis on precision. The need to maintain access and not to provoke political resistance unnecessarily seem to dictate the need for a certain level of flexibility in the definition of goals and for certain guarantees to intermediate participants, in order to prevent the collapse of an analysis due to political pressure or the production of information later aimed at discrediting it.

Problem Areas Encountered in Practice Who made the approach? Difficulties seem to occur when the initiative for a decision analysis did not come from the actual decision maker(s). Particularly in government organisations, the initial approach may come

156

A.R. Lock

from individuals higher up the organisation. In some cases, analysts have made the first approach. The initial ’contract’. As with many services provided by consultants, there is a danger of promising too much t o a client in order to secure a contract. It is also at this stage that the ’problem’ to be studied is defined, when the analyst may not have a full understanding of the client organisation. The result is that the full or ’real’ problem is not always defined or analysed and that it is difficult to reopen the issue of problem definition in order to explore it more deeply. The inhibitions about discussing real issues have been vividly highlighted by Rex Brown’s anecdote about an exercise with a governmental agency, when the study only really took off after the definition of a significant attribute named ’administrative morale.’ The process The degree of complexity of the required tasks is a recurrent issue. Both tasks that are seen as tedious and demeaning (Lock, 1979, 1982) and tasks whose complexity conceals their relevance or purpose have been advanced as causes of difficulty. Complex requisite models do not necessarily entail complex elicitation tasks. The nature of the innovation. When the goal of a particular analysis is the development of a technology for dealing with repetitive decision problems, radical departures from previous practice are less likely to gain acceptance. Tlus is dependent also on the type of culture of the organisation (Handy, 1981). Radical changes are most likely t o be accepted in the task culture, for reasons previously discussed, and in the power culture if the dominant individuals favour them. The role culture (or mechanistic organisation) is singularly ill-adapted to radical changes and is only likely to accept them if imposed by an external agency under an extreme threat. Finally, the person culture, for example, the medical profession, proves particularly resistant as innovations like decision analysis come from outside narrow professional areas and erode their mystique. The acceptability of the analyst. In organisations or situations with significant professionalised groups, it is important that the analyst is either of a similar professional background or has a status or social characteristics (e,g., in the U.K., social class or even belonging to the right club) that render him or her acceptable. Status may be derived from the organisation withm w h c h the analyst is based or from previous work of an academic or professional nature. The target group and the level of control With sufficient support from the top, business organisations find it relatively easy to impose innovations on individuals lower down the system. The more diffuse the target group (again medical decision makers are a prime example), the more difficult it is for proposals and innovations to be adopted.

DECISION ANALYSIS IN ORGANISATIONAL CONTEXT

157

Implementation. It is easy to assume that a client knows how to implement a particular option chosen by a decision analysis. Choices that may involve a greater level of conflict than the individuals in the organisation have previously faced will not be implemented unless those individuals can be shown how to confront it.

Developing Clinical Skills and Strategies Freud (1953) considered that resistance on the part of the subject towards the psychoanalyst’s interpretations was a healthy sign. Its absence would indicate complete vulnerability to outside influences. We have not all the same view when it comes to rejection of the recommendations of a structured and apparently scientific approach to problem-solving. A frequent response is to grumble about irrationality or to make veiled comments about Luddites. One has to recognise, however, that techniques or technologies that tend to exclude decision makers from much of the process, or take over from them, will be perceived as threats to their autonomy and their conclusions will be resisted. Indeed ’irrational’ resistance may be the only available strategy. There is also the more general resistance to change in particular organisational structures that has been discussed earlier. The problem is to develop skills that will reduce subject resistance. Fischhoff (1977) considers the problems that may arise in decision analysis as a result of low awareness of clinical issues, and defines the skills relevant to psychotherapists as follows: . . they must instil confidence in clients, choose the appropriate questioning procedures t o elicit sensitive information, handle crises, understand what is not being said, avoid imposing their own values and perceptions, and co-operating solutions”. The implication is that the skills required by decision analysts are similar. The problems involved may be broken down into seven areas: the initial ’contract’; the role adopted by the analyst; diagnosing, exploring and structuring the problem; the analysis process, presenting solutions; implementation; and dealing with conflict. The initial ’contract’.At the outset, entry to an organisation is based on a ’contract’, by no means always written and certainly more complex than a formal written one. It is clear that in a number of decision analyses the expectations and goals of the analyst and the client were markedly different. Some understanding of expectations and goals and a clarification of the likely process, procedures, and level of involvement are important for the overall success of the exercise. On the other hand, starting off with a rigid definition of the problem to be studied can prove to be a source of difficulty. The client’s initial definition of the perceived problem reff.

15 8

A.R. Lock

presents a set of symptoms or a ’felt need’. The nature of the underlying problem to be studied is likely only to emerge later. It seems a little naive to assume that the analyst can keep the terms on which he or she enters the organisation entirely flexible. Clients have distinct expectations of what should be considered and want to have some idea of the end result. Analysts sometimes also want to gain entry as an end in itself (for pecuniary or intellectual reasons) and as a consequence tend to comnu t themselves over-readily to some broad expected outcome. Some time often has to be spent by the analyst in the initial phase on explaining the requirements of the process in terms of inputs and the degree of access to others in the organisation. Even quite sophisticated managerial decision makers conceive decision analysis as simply drawing decision trees. The need for extensive problem re-structuring and the elicitation of preferences and utilities should be established at the outset. It should also be made clear that analyst may wish t o interview other members of the organisation to gain additional inputs. In highly politicised organisations, sponsoring coalitions or individuals may attempt to control access t o the analyst or the analyst’s access to other participants. This should be considered in planning the initial structuring of the problem and deciding which groups’ views have to be taken into account in devising acceptable strategies and representing preferences. In a typical decision analysis interested groups might include owners (the state, shareholders, community, etc.), employees, consumers, managers and society. It is important to realise that almost all the inputs come from a small subset of managers. Serious consideration has to be given to the extent to which one should include other groups and to the legitimacy of the goals of the sponsor (Kunreuther, this volume), but this can lead to a ’requisite’ decision model rather different from that anticipated, or welcomed, by the sponsor. The role adopted by the arzalyst. The literature on orgariisational change tends to take a philosophical stand on what should guide change agents. Bennis (1966) sees the goal as helping the system to become healther, in terms of a better ability to test and adapt to reality. The change agent is faced with a spectrum of possible roles ranging from the ’expert’, through the ’adviser’, to the ’trainer’. In the expert role the structuring and analysis is performed for the system but outside it. In contrast, the goal of the trainer is that the organisation members should carry out the structuring, analysis and strategy formation for themselves, and should be able to use the techniques in the absence of the analyst. Adviser roles involve a joint process between the analyst and organisation members, with the former retaining some special expertise. Obviously,

DECISION ANALYSlS IN ORGANISATIONAL CONTEXT

159

there are a wide range of options w i t h n this framework and there is no necessity for the same role to be adopted throughout the interaction. Involvement and the resulting commitment on the part of decision makers increases as one moves from the ’expert’ to the ’trainer’ roles. The tendency for conventional decision analysis to follow the first path partially explains the resultant low commitment to conclusions and recom mendations in a number of cases. It is clear that the adviser and trainer roles are more likely to be required when there is not a dominant executive coalition. Diagnosing, exploring and structuring the problem Client definitions of ’felt needs’ are based on discrepancies between some desired states and the aLTual current or future state. The question at this stage is to determine what problem or problems the analysis ought naturally to consider. One view of the decision analyst is that of a passive encoder of client-provided information, the ’man with the slide-rule’ as Rex Brown has put it. The problem with t h s is that it assumes that the decision maker has a fully developed representation of the problem. In fact, much of the value of decision analysis seems to come from the structuring phase (Jungermann, 1980; Humphreys, 1982) developing subjects’ representations of the situations and problems. This aids the retention, acquisition and use of information, which takes place within the structure of individual schemata (Abelson, 1976). We may point out several specific areas in which probing by the analyst is of considerable use. The first is the elicitation of the range of goals and decision criteria. These in turn assist the definition of the range of alternative actions (Jungermann et al., this volume). The second is exploring how actions are linked to outcomes. As well as specific questions of how different situation aspects are affected by particular actions, it is also necessary to identify who will be affected by a particular decision and their likely response. Crucial skills at this stage are the ability to listen and the ability to mirror and feed back the elicited information. The goal structuring phase starts with the elicitation of a list of criteria from the decision maker (it helps, particularly with business decision makers, to avoid terms like goals and objectives whose normative overtones hinder the process). These can be extended by focussing on the aspects on which decision alternatives differ, as in the MAUD programme (Humphreys and McFadden, 1980). Bauer and Wegener (1975) and Humpreys (1982) have suggested that analysts should try t o get subjects to imagme themselves in some future state or scenario and to come up with aspects they feel to be important. The analyst may also suggest criteria, while being careful not to impose these on the decision maker.

I60

A.R. Lock

At t h s point, the elicited aspects need some degree of organising. Firstly, they can be split into more global and more specific ones. The decision maker can then explore the links between the various attributes, arranging them in some form of goal hierarchy. Some new attributes may be found, others may be merged or renamed. Frequently, the elicitation of the goal hierarchy adds alternatives to the set defined at the outset by the decision maker. Further ones may be found by exploring the possibility of breaking a large, difficult choice into a number of reversible stages, the possibility of mixtures of options, and the way in which the various options relate to the goals. The latter process is extended in the process of exploring the full structure of the options ,and the particular areas about w h c h the decision maker feels uncertain (Jungermann, 1980). The aim of the structuring phase is to generate a ’requisite’ model, pruned as far as possible to leave a description that can be discussed with the decision maker to aid his or her understanding of the problem. As far as is possible, only diagnostic events and critical tradeoffs are left in. The remaining information gathered may be used in later sensitivity analyses. The analysis process. Having arrived at a ’requisite’ structure (con&tionally, at least) one can move to the collection of data on the probabilities and preferences identified within the representation of the structured decision problem. Given that a decision maker does not have pressing reasons why he or she does not want to provide accurate information, better estimates are obtained when it is understood what they are to be used for. Hence care should be taken to explain the nature of the specific tasks involved, and multiple simple tasks rather than single complex ones should be employed. Presenting solutions. Ultimately the analysis has to be useful to decision makers and they have t o feel confident about it. The presentation of a single best option does not always inspire this confidence. Strict optimisation is less attractive than the ability to explore the problem. A major role for computer-based models is the facilitation of manager-based sensitivity analyses and the ability to respond t o ’what-if questions. Decision makers acquire commitment t o the solution by feeling both that they have some control over the solution and that they have contributed to its develop ment. An alternative approach, where the exploration one is, for various reasons, unfeasible, is t o present the decision maker with a small number of alternatives with their predicted outcomes. Implementation. One of the major problems of proposed changes is that ranges of fears and expectations are raised which largely define indviduals’ attitudes to those changes. Many of these hopes and fears will

DECISION ANALYSIS IN ORCANISATIONAL CONTEXT

161

not be realised, and need to be modified if possible. In political or expert coalition dominated organisational structures, any discussion taking place about the choice of particular options will be vitiated by the concerns of participants about possible personal gains or losses within the overall outcomes. Prior agreement about how these could be redistributed after a decision can markedly reduce the conflict involved in the actual choice. It is important to realise that the tacit consent. at least. of most individuals in an organisation and the open commitment of key individuals are required for the successful implementation of a chosen option (Porter et al. . 197 5). It is worth invoking at this point a little of the literature on planned changes. Kurt Lewin defined three elementary stages t o the process-unfreezing the existing situation, changing it, and refreezing it once the desired changes have occurred (see, for example, Zaltman, 1973). The process of unfreezing occurs when levels of certain stimuli rise above a given threshold (Boulding, 1963). Thus change may occur as a response to an external threat providing a sufficiently strong stimulus. A possible internal strategy is to raise uncertainty about the present situation sufficiently for participants to start contemplating the possibility of changes. The presence of an outsider and the decision analysis process can form the source of this uncertainty that unfreezes the existing situation. One should note the implication of planned change models is that there is a high level of control and quality of feedback for the group seeking t o initiate change. A rather more plausible view is that in most organisations the impact of changes and the subsequent organisational equilibrium is not readily predictable (Weick, 1979). This is an issue that seems to have been played down in the organisational development literature. The main elements in devising an implementation strategy relate t o identifying the key groups and individuals, how they can be induced to contemplate change and how they will respond to any particular proposals. The person with the perceived responsibility for the decision should be in charge of the implementation strategy rather than the decision analyst. The reason for its inclusion here is that both the feasibility of implementation of a particular decision within a particular organisation and the way it is actually implemented are not separable from the analysis process. In the first case there is little point recommending policies that cannot be carried out for various reasons, and in the second, some responsibility has to be taken for the impact of the chosen policy and outside the organ isa t i on.

11

162

A . R . Lock

Dealing w i t h conflict. Real choice can be painful in organisational terms. In stagnant or declining organisations, change usually implies strong negative consequences. It is clesr that decision makers often dislike conflicts, particularly highly personal ones, and seek to avoid them. In many situations, changes and decisions are postponed until they are imposed by an external agency. By this time, the organisation’s survival may even be threatened. The problem of handling conflict may thus be a barrier to the adoption of otherwise ’optimal’ choices. For example, the author has observed decision makers in a U.K. higher education institution whose strategy is to try and maintain the present balance of subjects, courses and faculty. If choices are to be made in terms of what areas to cut and whch to expand. these will be made essentially by the intervention of outside bodies controlling the purse-strings. There is no implicit belief that the current balance of work is optimal, except in terms of the inertia of the status quo, an overriding desire to avoid compulsory redundancies, and concern about levels of conflict resulting from any other choice. It may be that in many cases where power is widely diffused and the organisation is highly politicised that it is the only feasible strategy. This unfortunate conclusion may mean that the organisation has little control over its own survival. The alternative is to consider to what extent it is possible t o improve the client’s or the client organisation’s ability to deal with conflict. Porter et al. (1975), Thomas (1976) and MacCrimmon and Taylor (1976) discuss a number of ways of resolving conflict. In cases where conflict is not directly resolvable one has to consider whether it is feasible to assist people to handle overt conflict and to confront political issues openly. This appears to be a development strategy over a longer time horizon than is usually available in a decision analysis study.

Conclusions Most of this paper has been concerned with developing the consciousness of analysts and would-be analysts about the organisational issues involved in practical applications. Analysts tend to enter decision problems with a large amount of normative and ideological baggage. Because of the tendency to be employed by the wealthier or more powerful segments of society, analysts have t o accept responsibility for the use to which their techniques and technologies are put, and to recognise the legitimacy of other interests than the original client group. On the behavioural side, sensitivity to the organisational climate and the organisational consequences of any decision are likely t o be crucial to

DECISION ANALYSIS IN ORGANISATlONAL CONTEXT

163

both the likelihood of implementation and the success of such an impiementation. The view of the role of the analyst as a change agent enables one to focus on the strategy that should be adopted in this role and the degree of client involvement that should be sought. This in its own way also tends to be an ideological issue reflecting the analyst’s goals. Considerable emphasis has been given of the need, in relation to problem structure and organisational goals, to avoid over-definition particularly in the initial stages. This is a recurrent theme in the suggestions of important clinical issues and strategies at the different phases of the decision analysis process, to avoid premature closure and as a means of maintaining sufficient room to manoeuvre for the analyst. In considering the overall organisational issues surrounding a major decision, it is appropriate to investigate what additional skills are required by an analyst and to what extent other specialist skillsshould be called upon. It would be probably inappropriate to define a specific professional training which should be undergone by all would-be decision analysts, but one should be aware of the very wide range of skdls that they seem expected to possess. It does not appear that decision analysis stands alone as a tool or skill, but that practitioners need to be aware of its and their own limitations and the point at which other approaches and other skills are more suitable to the problem and situation under consideration. It is possible to take perhaps a less pessimistic view of the relevance of decision analysis by emphasising its role in assisting organisational decision processes rather than viewing the latter as regrettable constraints to be circumvented wherever possible.

References Abelson, R. P., 1976. Script processing in attitude formation and decision making. In: 1. S. Carroll and 1. W. Payne (eds.), Cognition and Social Eehaviour. Hillsdale, N. J.: Lawrence Erlbaum. Ackoff, R. L., 1979a. The future of operational research is past. Journol of the Operational Research Society, 30, 93- 104. Ackoff, R. L., 1979b. Resurrecting the future of operational research. Journal o f t h e Operational Research Society, 30, 189- 199. Argyris, C , 1970. Intervention Theory and Method. Reading, Mass.: Addison-Wesley. Argyris C., 1976a. Explorations in consulting-client relationships. In: W. G. Bennis et al.. q. v. Argyris, C., 1976b. Theories of action that inhibit learning. American Psychologisf, 31, 638-654. Bauer, V. and M. Wegener, 1975. Simulation, evaluation and conflict analysis in urban planning. Proceedings of the 1E.E.E. 63,405-413. Bell, D., 1979. Thinking ahead. Harvard Business Review, 57, May-June, 20-42. Bennis, W. G.. 1966. Chaneine Oraanisations. New York: McGraw-Hill.

11*

164

A.R. Lock

Bennis, W. G.. K D. Benne. R. Chin, and K. E. Corey (eds.), 1976. The Planning of Change. New York: Holt, Rinehart and Winston, 3rd ed. Boulding, K. E., 1959. Natural images and international systems. Journal ofConj7ict Resolution, 3, 120. Boulding K. E., 1963. The place of the image in the dynamics of society. In: K Zollschan and W. Hlrsch (eds.), Explorarions in Social Change. London: Routledge, Kegan Paul. Burns T. and G. M. Stalker, 1961. The Management of Innovation. London: Tavlstock. Campbelh D.T., 1975. "Degrees of freedom" and the case study. Comparative Political Studies. 8, 178- 193. Cyert, R. M. and J. G. March, 1963. A Behavioural Theory o f t h e Firm. Englcwood Cliffs, N . J . : Prentice-Hall. Dale, A., 1974. Coercive persuasion and the role of the change agent. Interpersonal Development, 5. 102-111. Dunnette, M. D. (ed.), 1976. Handbook of Industrial and Organizational Psychology. Chicago: Rand McNally. Eden, C., S. Jones, and D. Sims, 1979. Thinking in Organisations. London: Macmillan Eilon, S., 1980. The role of management science. Journal of the Operational Research Society, 31. 17 - 28. Einhorn, A. and K. M. Hogarth, 1981. Behavioral decision theory: Processes of judgement and choice. Annual Review ofPsychology, 3 2 53-88. Elbing A. O., 1970. Behvioural Decisions in Organizations. Glenview, Ill.: Scott, Foresman Fischhoff, B . , 1977, Decision analysis: Clinical art or clinical science? Paper presented at the 6th Research Conference o n Subjective Probability, Utility and Decision Making, Warsaw, 1977. Freud, S., 1953. Inhibitions, symptoms and anxiety. In: The Standard Edition of the Complete Works of Sigmund Freud. London: Hogarth, Vol. 20, 77. Galbraith. J. K. 1973. Desiqning Complex Organizations. Reading, Mass.: AddisonWesley. Hammond, K. R. and L. Adelman, 1976. Science, values and human judgement. Science, IY4, 389- 396. Handy, C. B., 1981. Understanding Organizations. Harmondsworth, Middlesex: Penguin, 2nd ed. Humphreys, P. C., 1980. Decision aids: Aiding decisions. In: L. Sjoberg,T. Tyszka, and J. A. Wise (eds.), Decision Analysis and Decision Processes Lund: Doxa. Humphreys, P. C., 1982, Value structures underlying risk assessments. In: H.Kunreuther (ed.), Risk: A Seminar Series. ILaxenburg, Austria: International Institute for Advanced Systems Analysis. Humphreys, P. and W. McFadden, 1980. Experiences with MAUD: Aiding decision structuring versus bootstrapping the decision maker. Acta Psychologica, 45, 51-69. Hogarth, R. M., 1980. Judgement and Choice. Chichester, England: Wiley. Hogarth, R. M., 1981. Decision making in organizations and the organization of decision making. Working paper. University of Chicago, Center for Decision Research. Hogarth, R. M. and S. Makridakis, 1981. Forecasting and planning: An evaluation. Management Science, 2 7, 1 15- 138.

DECISION ANALYSIS IN ORGANISATIONAL CONTEXT

165

Jaques. E., 1976. A General Theory of’Bureaucracy. London: Heinemann. Jungermann, H., 1980. Speculations about decision-theoretic aids for personal decision making. Actu Psychologica, 4.5. 7 -34. Jungermann, H., I. von Ulardt, and L. Hausmann, 1983. The role of the goal for generating actions. In this volume, 223-236. Kunreuther, H., 1983. A multkattribute multi-party model of choice: Descripitive and prescriptive considerations. In this volume, 69 -89. Lansley, P., P. Sadler, and T. Webb, 1974. Organizatioii structure, management style and company performance. Omega, 2, 467-485. Lock, A. R., 1979. Multiple criterion strategic marketing problems. An analytical approach. Unpublished Ph. D. Thesis. University of London. Lock, A. R., 1982. A strategic business decision with multiple criteria: The Baliy men’s shoe problem. Journal of the Operational Research Society. MacCrimmon, K. R. and R. W. Taylor, 1976. Decision making and problem solnng. In: M. D. Dunnette (ed.), q. v. MacMillan, 1. C., 1978. Strategy Formulation: Political Concepts. St. Paul, Minnesota: West Publishing. March, J. G., 1978. Bounded rationality, ambiguity and the engineering of choice. Bell Journal of Economics, 9, 587-608. Mintzberg, H., 1978. Patterns in strategy formation. Management Science, 24, 934-948.

Neave. E. H. and E. H. Petersen, 1980. A comparison of optimal and adaptive decL sion mechanisms in an organizational setting. Management Science, 26, 8 10- 822. Neisser, U., 1976. Cognition and Reality: Principles and Implications of Cognitive Psychology. San Francisco: W. H. Freeman. Nisbett, R and L. Ross, 1980. Human Inference: Strategies and Shortcomings of Social Judgement. Englewood Cliffs, N.J.: PrenticeHall. Nord, W. R., 1974. The failure of current applied behavioural science: A Marxian perspective. Journal of Applied Behavioural Science, 10, 557-578. Pettigrew, G M., 1973. The Politics of Organizational Decision-Making. London: Tavistock. Pettigrew, A. M., 1974. The influence process between specialists and executives. Personnel Review. 3, 24-31. Pfeffer, J., 1981. Power in Organizations. London: Pitman. PhiIlips L. D., 1980. Organisational structure and decision technology. Actu Psychologica, 4.5, 247-264. Plott, C. R and M. E. Levine, 1978. A model of agenda influence on committee decisions. American Economic Review, 68, 146-160. Porter, L W., E. E. Lawler, and J. R. Hackman, 1975. Behaviour in Organizations. New York: McGraw-Hill. Pugh, D. S. and D. J. Hickson, 1976. Organizational Structure in Its Context: The Aston Programme I . Farnborough, England: Saxon House. Richards, M. D. 1978. Organizational Goal Structures St. Paul, Minnesota: West Publishing Simon, H. A, 1978. Rationality as process and as product of thought. American Economic Review, 68 ( 2 ) , 1-16. Simon, H. k , 1979. Rational dedsion making in business organizations. American Economic Review, 69,493-513.

166

A.R. Lock

Slovic, P., B. Flschhoff, and S. Lichtensteln, 1977. Behavioral dedsion theory. Annual Review of Psychology, 28, 1-39. Staw, B. M., 1976. Knee-deep in the big muddy: A study of escalating commitment to a chosen course of action, Organbational Behavior and Human Performance. 16, 27-44. Staw. B. M. and G. R. Salancik (eds.), 1977. New Directions in Organizational Behaviour. Chicago: St. Clair Press Thomas, K.,1976. Conflict and conflict management. In: M. D. Dunnette (ed.),q.v. Uyterhoeven, H. E. R., 1972. General managers In the middle. Haward Business Review, 50. March-April, 75 -85. Vroom, V. H. and P. W. Yetton, 1913. Leadership and Decision-Making. Pittsburgh, Pa.: University of Pittsburgh Press. Watson, S. R. and R. V. Brown, 1978. The valuation of decision analysis. Journal of the Royal Statistical Society, Series A, 141, 69- 18. Weick, K . E., 1979. The Social Psychology of Organizing. Reading, Mass,:AddisonWesley, 2nd ed.. Weiss, C. H., 1980. Knowledge creep and declsion accretion. Knowledge: Creation, Diffwsion, Utilization, 1, 381 -404. Zaltman, C. (ed.) 1973. Processes and Phenomena of Social Change. New York: Wiley.

PITFALLS OF DECISION ANALYSIS' Detlof von WINTERFELDT Social Science Research Institute, University of Southern California US.A

Introduction

Much of the craft of decision analysis consists of recognizing and avoiding pitfalls, which exist at each stage of an application between problem formulation and model implementation. Consider the nightmare of solving the wrong problem for a misconstructed client with an inappropriate model and gamed numerical inputs from experts. In this case one would almost hope that the model is never used or implemented, which, incidentally, is another common pitfall of decision analysis. Unfortunately, the published literature on applications of decision analysis provides little information about such pitfalls or the process by which analysts try to avoid them. The published "success stories" are usually cut and dry and, except for a few cases, they appear to be relatively straightforward (and often dull) applications of standard techniques. The failures are, of course, never published. Yet there is much to be learned from failures and mastered problems, especially if one wants to go beyond the technical surface of decision analysis and understand the craft of its application. One purpose of t h ~ s paper is t o systematically explore some typical traps and pitfalls of decision analysis, and thereby increase the practitioner's awareness of these problems. Another purpose is to present examples and lessons of how pitfalls -once recognized-can best be avoided. Further stimulatiori for this paper came from a series of articles and books (Majone, 1977a, b; Ravetz, 1973; Majone and Quade, 1980) which discuss, on a more general level, the pitfalls of analysis. Much of this discussion applies directly to decision analysis, e.g., the pitfall of addressing the wrong problem. Other pitfalls

'

With contributions by R.V. Brown, J.S. Dyer, W. Edwards, D.H. Gustafmn, P. Hurnphreys, L.D. Phillips, D.A. Seaver, A . V k i , and J. Vecsenyi.

168

D.von Winterfeldt

appear to be handled well by decision analysis, e.g., the neglect of intangibles. Sill others seem unique to decision analysis, eg., the "gaming" of utilities or probabilities. It is relatively easy to write about pitfalls in the abstract, but the most important lessons can be learned from problems and failures in real applications. To provide a rich set of pitfalls several practitioners of decision analysis contributed examples of their own applied experiences.' Thls paper discusses their contributions, following the stages of decision analysis : 1. Developing an analyst-client relationship; 2. Defining the problem; 3. Organizing the analysis; 4. Structuring and modeling; 5. Elicitation of utilities and probabilities; 6. Using and implementing the model. For each stage, we will describe the general nature of the possible pitfalls, give several examples, and briefly discuss the lessons.

Developing an Analyst-Client Relationship: Users and Hidden Agendas Decision analysis is still an emerging discipline and therefore largely supply driven. Often the analyst identifies a problem area for which decision analysis is useful and then approaches the potential client. While this picture is rapidly changing, it is still true that in a majority of cases the analyst "sells" the decision analytic approach to governmental agencies, businesses, or individual clients. As a partial consequence of this supply orientation and of the resulting eagerness of the analyst to practice his or her profession, a number of pitfalls can occur. These include misconstruction of the "red" client, misrepresentation of the "red" motives of the client, or simply confusion about who is to be served by the analysis and why. The problem is aggravated if the sponsors themselves are not sure about the ultimate user of the analysis. An example of multiple clients was decision analysis application for evaluating community anti-crime (CAC) programs (see Snapper and Seaver, 1980a). The sponsor of this evaluation was the Law Enforcement

Tht authors of these. contributions are Usted in the title of this paper. I would Uke to thank them all for thefr frankness and for lettlng me use.theh cases and materials. They are the real authors of this paper.

PITFALLS OF DECISION ANALYSIS

169

Administration Agency (LEAA), which also funded the programs that were to be evaluted. One of the purposes of the evaluation was therefore soon identified: to aid LEAA in its planning and funding decisions. However, two other clients of the proposed evaluation emerged soon: the CAC project managers who wanted to learn how their projects could be improved and Congress who had to set policy about programs and LEAA. A natural pitfall of decision analysis in this situation would be to focus the evaluation solely on the needs and concerns of LEAA, in essence becoming a watchdog of the subordinate project managers, and using the evaluation to ”sell” the program as a whole to Congress. The analysts avoided this pitfall with a quite elegant solution: they constructed a three level value tree whose upper level represented the values of Congress; one second level branch included LEAA’s overall program objectives; and one branch of LEAA’s objectives coincided with the local CAC program objectives. Tius approach may have idealized the harmony of the objeo tives of these major stakeholders somewhat, but it helped maintain an open mind about the use and users of the analysis. As it turned out, one of the more interesting implementations of the evaluation system was for project management purposes (see Snapper and Seaver, 1980b), rather than planning. Another pitfall in the client-analyst relationship are ludden agendas: a client wants a favorable evaluation of an activity; a decision, already made, needs justification; or the analyst’s technical skills and professional stature are to enhance the image of a project. Not all of these are necessarily detrimental to the analysis, but they are better handled when brought out in the open from the start. Hidden agendas sometimes occur in risk analysis applications of decision analysis if the client’s intention is to use the analysis to prove that the product or technology is safe (or unsafe), rather than solving a decision problem related to the safety of the product. In many such instances the analysis is in response to public or governmental concern about safety, and the client expects results that favor his or her position and therefore contribute to resolving the controversy in his or her favor. Ironically, risk analyses have frequently stirred up more controversy, rather than consolidating opinions (Mazur, 1980), partially because the public suspected such hidden agendas. A third pitfall is a direct consequence of the supply driven nature of decision analysis: analysts ldce to please their client and sometimes promise too much. An example is the American College of Radiation (ACR) study of the efficacy of X-rays (see Lusted et. al., 1978). The sponsors of the research and the analysts originally agreed that many X-rays were a waste

170

D.von Winterfeldt

of time, and the analyst in essence promised that this study would show exactly that, It was soon recognized that efficacy was a multi-level concept. To be efficacious, an X-ray has to have the potential for changing the medical doctors opinion, then to change his or her decision, and finally to change the state of the patient’s health. Recognizing that it is easiest to begin with the first link in this efficacy chain, the analysis began with a study of whether X-rays would change medical doctors’ opinions (this would later be called efficacy 1). The hypothesis was that it would not, and that hypothesis was apparently widely shared. But the large scale study showed that X-rays were indeed meeting efficacy I criteria-they did change the diagnostic likelihood ratio judgements of the physicians. While some of the researchers are still convinced that X-rays would not meet the efficacy I1 criterion (change physicians action), the efficacy I1 study was never funded. What is there t o be learned from these examples? First, understand your client, probe his or her motive, and carefully define the purpose of the analysis. This is usually simple when the client comes with a problem, so be more alert when you, as an analyst, approach a client. Second, actively search for organizations or individuals with a stake in the decision to identify the conflicting interests and motives. Third, establish an atmosphere of trust, encourage the client to reveal hidden agendas, stress that such agendas can become a legitimate part of the analysis. And finally: don’t promise too much or draw conclusions too early.

Defining the Problem: Real Problems, Apparent Problems, and Side Problems An important part of the early phases of decision analysis is devoted to the definition of the problem. A common pitfall is to take the client’s problem at face value. Another trap is to accept the client’s restrictions of the part of the problem for which he or she wants an answer. In contrast, analysts like to reformulate problems to make them accessible to decision analytic modeling, thereby fitting the problem to the model. Common to many of these pitfalls are the difficulties of defining the level of abstraction of alternatives (e.g., strategic vs. tactical) and objectives (e .g., overall organizational vs. cost-benefits) in response to an initially vague problem formulation. The first pitfall is illustrated by a study of chronic oil pollution from North Sea oil production platforms (von Winterfeldt, 1982; CUEP, 1976). The decision maker was the U.K.Department of Energy who had to set standards on oily water discharges from offshore oil production

PITFALLS OF DECISION ANALYSIS

171

platforms. These standards were to prevent negative environmental impacts, while at the same time avoiding costly restrictions on offshore oil development. This appears like a straightforward "choice among standards" problem and it was, indeed, handled that way informally by the CUEP, and formally, though experimentally, in von Winterfeldt (1 982). During the course of interacting with the U.K. Government agencies, the analyst began to suspect, however, that setting oil pollution standards was not really the problem of the U.K. government. Apparently, several countries of the European Community had been pushing very hard for a clean North Sea, enforced by uniform European standards. In the past, the U.K. government had avoided numerical standards, and instead stressed the flexible nature of pollution sources and of the local environmental carrying capacity, so setting numerical standards required a change in policy. Perhaps a better problcm formillation would have been how best t o counter the European Community's push for uniform standards. Obviously, the alternatives would have been much different from the maximum levels of oil concentration, that both the CUEP and v. Winterfeldt analyzed. Pitfalls in problem formulation frequently occur in analyses for personal decisions. In an experimental study of such decisions (see John, v. Winterfeldt, and Edwards, 1981), a woman initially formulated her problem as an apartment selection problem. The real problem turned out t o be much more complex and involved financial questions and family interactions. The problem could have been formulated, for example, as a problem of managing her relationship with her parents. Another problem formulation would have involved alternatives for managing her financial situation. This case was interesting, because out these deeper aspects of the "apartment selection" problem, the analysis incorporated them into objectives and maintained "apartments" as the alternatives. It was clear, however, that each apartment stood for a complex alternative for managing the subject's lifestyle. Another example of addressing only part of the "real" problem has been described by Viri and Vecsenyi (this volume). The decision maker, a Hungarian governmental agency, had to select among possible mixes of R & D projects in a typical resource allocation problem. The analystsintended to consider the complementarities and redundancies between projects in an explicit model of conditional utilities and probabilities. However, the decision makers preferred t o consider the utilities of the single projects separately and to use their own heuristics in combining the single project utilities to an overall evaluation. One reason for this preference was, presumably, that the decision makers wanted to engage some of their own values about

172

D.von Winterfeld t

projects and project interdependencies without making them an explicit part of the analysis. Brown and Ulvila (1981) provide an example concentrating on those parts of a decision problem which lend themselves easily to decision analytic tools (see also Ulvila and Brown, 1981). They developed a decision analytic model to aid the International Atomic Energy Agency (IAEA) in allocating inspection resources. On later reflection, it seemed that it might have been more useful t o develop and evaluate alternative strategies for motivating inspectors within a given resource allocation context. One conclusion of these examples is that a decision analyst should keep an open mind about problem formulations including those for which decision analysis does not seem especially suited. Another conclusion is that it is often necessary to examine different levels of problem formulations in the early stages of analysis. Problems usually come in hierarchies. In the oil pollution example the most general problem formulation would have involved alternative policies for dealing with international pressure on the U.K. government to clean up the North Sea. Once regulation strategies were accepted, the U.K. faced alternatives for regulating oil pollution: case by case regulation, emission taxes, standards, etc. Once standards were decided upon, the questions remained what levels these standards should be set at, how the monitoring and inspection procedures should be organized, etc. Decision makers sometimes like to “push down” the problem definition because it makes the analysis more technical, requires less sensitive political judgements, and it keeps the altenatives within their decision making capacity. This tendency t o suboptimize is not necessarily detrimental. A wider problem formulation does not always lead to a substantial improvement in the overall organizational objectives, and often leads to a more expensive analysis. Yet the analyst should be aware of higher order problem formulations and the tradeoffs involved.

Organizing the Analysis: Institutional Obstacles and Obstinate Individuals Setting up an analysis in an institutional environment often requires substantial managerial skill and political savvy. Decision makers have to be convinced that the analysis is useful and does not interfere with their activities or threaten their job. Experts have to contribute valuable time to perform what they may consider tedious and boring assessments. In politically controversial decisions, opposing stakeholder groups have t o be

PITFALLS OF DECISION ANALYSIS

173

involved in the analysis. Sometimes the institutional constraints prevent the analysis from heing carried out or being effectively implemented. Sometimes individuals or organizations simply refuse to cooperate. A particularly interesting case of institutional constraints was discovered in a descriptive case study of TVA’s power plant siting process (see b o p , 1980). One purpose of the descriptive study was t o examine how a decision analysis process could improve on the siting procedure then in use by TVA. That procedure was essentially a four stage screening and evaluation process in which, first, candidate areas were selected on the basis of power systems requirements. In the second stage numerous candidate sites were screened mainly on the basis of engineering, construction,and landuse features. More detailed comparative evaluations of cost, engineering, and environmental factors were done on the 3-5 sites that survived the screening. Finally, the preferred site was studied in great detail, mainly to meet federal and state regulations for environmental impact statements. An interesting feature of this process was the organizational arrangement. In the first two stages the process was coordinated and, in effect, controlled by the Division of Power Resource Planning, in the last two stages by the Division of Engineering Design and Construction. The Environmental Division, while technically reporting t o the Board of Directors, was in effect a subcontractor for performing surveys and research for the lead divisions. In addition, the Environmental Division became involved only in the later stages of the siting process. This institutional arrangement seemed t o contradict TVA’s policy which put environmental considerations on an equal footing with cost, engineering, and power systems considerations. The real process of decision making cLuld best be described as a sequential elimination process with shifting priorities in the order of power systems (first stage), engineering (second stage), cost (third stage), and environmental (fourth stage). Decision analysis by its very emphasis on tradeoffs could have been implemented only with organizational rearrangements. Interestingly, such reorganization was a focal point of discussion at TVA at the time of the case study: most divisions, headed by the Environmental Division pushed strongly for more lead time and an earlier involvement in the siting process-an institutional form of redressing the balance between TVA’s mu1ti ple objectives. A more straightforward pitfall of decision analysis is the refusal of important experts or stakeholders to qoperate. An example is Ward Edwards’ school board study (Edwards, 1979). In the late seventies and up to 1981 the board of the Los Angeles Unified School District (LAUSD) was presented several court orders to develop and implement a desegregation plan for the district. One order asked the school board to develop a

174

D. von Winterfeldt

scheme for evaluating alternative plans. Under the leadership of Edwards, a multiattribute utility analysis was developed that was meant to incorporate the concerns of the opposing stakeholders. Edwards developed a common value tree from inputs of all relevant stakeholder groups, school board experts performed the single attribute evaluations of the available plans, and school board members assigned the weights. While the general level of cooperation of stakeholders was remarkable, there were notable exceptions: a pro-bussing group and one member of the board refused t o provide inputs to the analysis. This did not cause a failure of the analysis, but it highlighted a potentially severe problem with decision analysis in politically controversial situations. There are many other examples of institutional constraints and lack of individual cooperation. Some can be very serious-as the rejection of some courts to allow probabilistic testimony. Others are more amusing, as the refusal of an expert to express his or her opinions probabilistically, because of a lack of "hard" knowledge. What can be done about it? Awareness of the institution, of the political arrangements and of individual psyches is an important step. Don't try to force an analysis that does not respect institutional barriers-it is likely to fail. Meshing analysis, institutions, and individuals, requires compromises from everybody-especially from the analyst.

Developing a Structure and Model: Bushy Messes and Analytic Myopia The initial steps of decision analysis are largely political in nature. Wldle the structuring, elicitation and implementation steps involve more of the craft and science of decision analysis, they nevertheless have their own traps. A frequent structuring pitfall is the acceptance of a traditional decision analytic paradigm that does not quite fit the problem. A problem is quickly labelled a "typical riskless multiattribute problem" or a "typical signal detection problem". Dynamic aspects and feedbacks in decision problems are often overlooked, because the traditional static structures of decision analysis d o not handle them well. A related pitfall is t o structure a problem too quickly in too much detail. Bushy messes in the form of overly detailed and often redundant decision, value, and inference trees are a frequent problem of beginning decision analysts. In the development of an appropriate inference, evaluation, and decision model, there are additional pitfalls: dependencies in inference models, redundancies in evaluation models, and dynamic features in decision models, to name only a few.

PITFALLS OF DECISION ANALYSIS

175

The previously mentioned school board study (Edwards, 1979) illustrates a structuring pitfall with value trees, Edwards’ value tree had 144 twigs, far too many according to his own rules (Edwards, 1977). Edwards began the value tree building process by constructing a relatively modest “strawman” tree. The tree was then shown t o members of various stakeholder groups, and their response was typically to add values. This addition of values not only increased the size of the tree but also created interdependencies and redundancies. The analyst accepted both the dependencies and the redundancies as a fact of life and the only reasonable way to deal with the political nature of the task he had set out to do. A fascinating case of misstructuring-mismodelingis the analysis of Ford’s decision whether to place the Pinto gas tank in front or behind the rear axle. The decision was based on a probabilistic cost-benefit analysis in which the chances of fires in rear end collisions were traded off against the costs of lives lost in these fires. As it turned out, the expected dollar value of lives saved by placing the tank in front of the rear axle was smaller than the cost of the tank relocation. Ford therefore chose not t o relocate. The problem with this model and structure is its myopia. It neglected the possible negative publicity that resulted from frequent occurrences of fires in rear end oollisions and it did not consider the possibility of punitive damages in liability suits. The analysis derived the value of lives from insurance premiums and past court awards. At the time of the analysis that value was estimated to be between #300,000and $400,000 (in 1970 dollars). From the company’s point of view, this number represented the average loss they would face if sued for liability. Punitive damages could, of course. be in the millions. The case had an ironic twist. Ford was sued in several instances of fatalities and injuries caused by fires in rear end collisions. In one suit the analysis was made public and that publication was partly responsible for punitive damages in the millions. The irony is that had Ford considered such punitive damages, it might very well have concluded that the expected costs of not replacing the tank would be larger than the costs of replacing it. Perhaps the best way to avoid the pitfalls of the above examples, and the many more that were mentioned in the beginning of this section is to develop several alternative structures and models early in the decision analysis, and run several sensitivity analyses t o see what matters and what does not. Further, the analyst should encourage creativity in structuring, and in inventing events and anticipating obscure secondary and tertiary impacts of the decisions, One way to improve this process is by involving groups of decision makers and experts covering a wide range of experiences, opinions, and values.

176

D.von Wintarfeldt

Elicitation: Gamed Numbere and Numbers Games Elicitation of utilities and probabilities is the technically best developed part of decision analysis. The main problems that an analyst encounters in this phase are inadvertent biases, misrepresentations, or even lies by respondents. A frequent bias is optimism or pessimism in assesments of probabilities of events on which the assessor has a stake. Misrepresentations can also occur in utility assessment t o bias the tradeoffs in order to make one alternative "come out better". And finally, there isthe possibility of outright gaming of the numerical inputs of the analysis. Perhaps the most intriguing example of this sort occurred in Dyer and Miles' (1976) evaluation of alternative trajectories for NASA's Jupiter/Saturn mission. In this study several science teams assigned utilities to alternative trajectories by considering the value of the research contribution for each trajectory. Since different science teams had different research tasks and interests, these utilities varied widely. Using the single team evaluations as inputs, several formal rules for collective decision making were applied t o find an acceptable trajectory. The science teams involved in the assessment knew, of course, that the purpose of the study was to develop a compromise evaluation and they were knowledgeable enough about collective decision rules to be able to game them if they wished to do so. In fact, two types of gaming occurred. The first was to simply give the trajectories which were considered good an extremely high evaluation, and to give all others an extremely low rating. This type of gaming gave the trajectories that a group preferred a better chance of surviving. Another type of gaming was coalition formation. Dyer and Miles report that some groups met during the evaluation process, and it is conceivable that they coordinated their assessments. The analysts were, of course, aware that such gaming could occur and they used several strategies to ensure that gamed numbers would not distort the analysis too much. Nevertheless, some groups, after the evaluation was completed, actually celebrated their "victory" in beating the evaluation system Humphreys and McFadden (1980) observed numbers games people play with the interactive computer program MAUD. In a group decision making task, a typical game was for individual members to argue for increased weights for those attributes on which their favored alternative scored most highly. If successfd, the group's evaluation would be biased in favor of the alternative that the "game player" originally preferred. It turned out that such gaming was much more difficult with MAUD than in interactions with group members, leading some group members to play "machine games".

PITFALLS OF DECISION ANALYSIS

177

While utilities are easier to game than probabilities, unintentional biases in probability judgments can occur quite easily. While much of the decision theoretic research literature focuses on cognitive biases in probability judgments, the more obvious biases are probably motivational. For example, researchers are notoriously optimistic in their estimates of the probability that a proposal will be funded c r In their point estimate of the amount of time it takes to finish a paper. To counteract gaming strategies, the analyst first of all has to know the motives of the decision makers and experts. This brings us back to Section 2, and involves questions of client-analyst relationships and trust. In utility assessment it furthermore helps to carefully anchor scales, and to ask for relative judgments rather than absolute ones. In a probability context unintentional optimism or pessimism can often be counteracted by phrasing the questions in terms of low stake lotteries. Other "debiassing" procedures that address specific cognitive illusions have been discussed in the literature (see, e.g., Kahnemann and Tversky, 1979). Implemntation: Being Better is Not Good Enough This section is about the unfortunate analyst who successfully avoided all the previous pitfalls only to fall into the last pit: the analysis is never implemented or used. There are, of course, many ways in which decision makers can ignore an analysis. One shot decision analyses are often compiled in voluminous reports and shelved away. A milder form of ignoring decision analysis consists of using parts of the analysis to justify a decision that is favored by the decision makers. Perhaps the most severe forms of the implementation pitfall occur when a decision analysis system that is built for repeated use, is never implemented. A particularly sad case of this nature is Gustafson e t d ' s (1977) system for predicting suicide attempts. They developed a computer based system which permitted a person complaining of suicidal thoughts to be interviewed by a computer prior to seeing the psychiatrist. The computer then prepared a narrative summary of that interview by the psychiatrist and also provided an estimate of the probability that a person would make a suicidal attempt. That probability calculation used a subjective Bayesian model where the person's characteristics are linked with the appropriate likelihood estimates and processed in a way that will give a probability of a serious suicidal attempt versus the probability that this person will not make a serious suicidal attempt. Both pilot studies and field test studies suggested that the Bayesian model was significantly superior to unaided judgment of psychiatrists and 12

178

D.von Winterfeldt

clinical psychologists. For instance, one analysis divided patients into two categories: Those that made a serious suicidal attempt and those that didn’t. The authors compared clinicians who were asked t o estimate whether the person would make an attempt on their life to the computer which was asked to make a similar prognostication. It turned out that the clinicians identified the serious attemptor about 37 per cent of the time and the non-attemptqr about 94 per cent of the time. The computer system identified the serious attemptor 74 per cent of the time and the non-attemptor about 94 per cent of the time. Thus, the ability of the computer t o detect serious suicidal attempts was much better than that of the expert clinician. But, even though this research has been published in several academic journals; even though the results have been described in a popular journal (TimeMagazine):even though the system has been set up for full scale implementation and attempts have been made t o implement on the full scale basis, the technique has never been adopted in practice. Nobody is at this point using it and while there have been expressions for interest in this system, those expressions have not led to implementation. There are a number of reasons for this failure. The system developers had paid little attention to the organizational and psychological aspects that can hinder the implementation of a formal decision analytic system. While the patients liked to interact with the computer, clinicians appeared to resist the implementation of the system. One reason was that the clinicians and other users in mental health counseling and guidance were not familiap with computers, and thus sceptical about their use; another reason was that the computer challenged the authority of the clinician, because it could be viewed as replacing clinical psychologists and psychiatrists in a critical task of their practice. Further, while the research scientists were very fond of the complexity of the Bayesian processing model, this model was hard to explain and somewhat obscure to the clinicians-another reason for a lack of trust. Edwards’ Probabilistic Information Processing (PIP) (Edwards et al. 1968) System is another instance of an inference system that never was adopted in significant real world applications. PIP was developed mainly for intelligence purposes to facilitate the task of processing intelligence information that bear on hypotheses of relevance to policy makers. PIP was built on a simple Bayesian inference structure using likelihood ratios for quantifying the diagnostic impact of information and prior odds for quantifying the apriori knowledge. PIP then computes the posterior probability or odds of the relevant hypothesis. In a simulation study (Edwards etuf., 1968) PIP was found to be superior to other on line systems that were evaluated. Yet PIP was never used. One reason was that inference structure in PIP was too simplistic.

PITFALLS OF DECISION ANALYSIS

179

Recent research by Schwn (1977, 1979) has generated several generic and useful inference structures which are quite a bit more realistic than the one stage Bayesian structure used in PIP. Another contributing factor was that PIP was an information processing system with no direct links intodecision making. There are also less dramatic examples of the implementation pitfall. The previously mentioned study of oil pollution standards was never used. While the analysis sharpened the decision maker's eye for the sensitive places in the process (e.g., monitoring procedures, sampling methods) it did not have precise numerical inputs nor were the environmental impacts modeled sufficiently. Consequently, the numerical analysis was not taken seriously by the decision makers. Edward's school board study had a unique fate. The study was successful in that it producsd an evaluation system and that the desegregation plans were actually evaluated. However, after the initial evaluation was completed, the school board developed its own plan, which was never evaluated by the multiattribute system. It is unclear why the school board did not use the system to evaluate the final proposal, in particular, since the proposal was developed with knowledge of the system and would have been a likely winner. What can be done to avoid the non-implementation pitfall? First, it helps to avoid all others that precede it. Second, during the process of the analysis keep in mind that the decision maker has to use the analysis, not the analyst. This should bias against overly complex modeling, too extensive use of incomprehensible or inaccessible machinery, and use of ideas or devices that may threaten the client. Third, the analysis or model should be presented as an aid, not a substitute for decision making, and the interaction between model and client should be user-friendly. A Brief Conclusion

Larry Phillips made a most pertinent comment about avoiding any of the pitfalls described above: "Over the past five years, I have completed about 25 decision analyses far a great variety of clients at all levels in organizations. None has been an outright failure, and on reflecting why this has been so, I can see only one common feature to all of these analyses: the client always came to me." Fortunately for decision analysis and decision analysts, more and more clients are doing just that. 12*

180

D. von Winterfeldt

References Brown, R. V . and J. W. Ulvila, 1982. The role of decision analysis in international nuclear safeguards. In this volume, 91 -103. Central Unit on Environmental Pollution, 1976. The separation of oil from water for North Sea oil Operations. Pollution Paper No, 6. Department of thvironmcnt, London: HMSO. Dyer, J. S. and D. Miles, 1976. An actual application of collective choice theory t o the selection of trajectorlcs for the mariner JupiterlSaturn 1977 Project. Operations Research, 24, 220--243. Kdwards, W., 1977. Use of multiattribute utility measurement for social decision making. I n : D. E . Bell, R . L. Keeney, and H. Raiffa (eds.). Conflicring Objectives in Decisions. New York: Wiley. Edwards, W., 1979. Multlattrlbute utllity measurement: Evaluatifig desegregation plans In a highly polltical context. In: R. Perloff (ed.), Evaluator Interventions: Pros and C o n s Baverly Hills: Sage. Edwards, W., 1980. Reflections o n and criticism of a highly politlcal multi-attribute irtihty analysis. In: L. Cobb and R. M. Thrall (eds.), Mathematical Frontiers of Brhavioral arid Policy Sciences. Colorado: Westvicw Press, 1 5 7 - 186. lidwards, W., L. D. Phillips, W. L. Hays, and B. C. Goodman, 1968. Probabillstlc information processing system: Design and cvaluation. IEEE Transactions of .Sy.vtems Science and Cybernetics, SSC-4, 3, 248-- 265. Gnstafson. D. H., J. H. Greist, F. F. Strauss, H. Erdman, and T. Laughren, 1977. A probabilistic system for identifying suicide attemptors. Compufers and Hiornedical Research, 10, 8 3 -- 89. Ilumphreys, P. and W. McFadden, 1980. Experiences with MAUD: Aiding decision structuring versus bootstrapping the decision maker. A c f a Psychologica, 45, 51 --69. Hurnphreys, P. and A . Wisudha, 1980. Multi-Attribute Utility Decomposition. Technical Report 79-212. Uxhridge, Middlesex, England: Brunel University, Decision Analysis Unit, John, R . S . , I). von Winterfeldt. and W . Edwards, 1981. The quality and user acceptance of decision analysis performed by coinputcr vs. analyst. Technical Report No, 81 - I . Social Science Research Institutc, University of Southern California. Kahncman, D. and A. 'I'versky, 1979. Intuitive prediction: Biases and corrcctivc procedures. Management Science, 12, 313-327. Knop, I{., 1979. The 'Tennessee Valley Authority: A field study. IIASA-RR-79-2. International Institute for Applied Systems Analysis, Laxenburg, Austria. Lusted, L . B ., H . V. Roberts, D. L. Wallace, M . Lahiff. W .Edwards, J . W . Loop, R . S. Bell, J. R. Thornbury, D. L. Seale, J. P. Steele, and D. C . Fryback. EffIcacy of diagnostic radiologic procedures. In : K , Snapper (ed.), Practical Evalrratio~i: Case Studies in Simplifying Complex Problems. Washington, D. C . : Information Resources Press (in press). Majone, G . and 1;. Quade (eds.), 1980. Pitfalls ofAnalysis. Chichcster: Wiley. Majone. C., 1977a. Pitfalls of analysis and the analysis of pitfalls. Research Memorandum IIASA- RM-77- 1. International Institute for Applicd Systems Analysis, Laxenburg, Austria.

PITFALLS OF DECISION ANALYSIS

181

Majonc, G., 1977b. l'echnology assessment in a dialectic key. Professional paper I I A S A , PP-77 1. International Institute for Applied Systems Analysis, Laxenburg, Austria. Mazur, A., 1980. Societal and scientific causes of the historical development of risk assessment. In: J . Conrad (ed.), Society, Teclinology, and Risk Assessment. New York: Academic Press, 1 5 1 -- 164. Ravetz, J., 1973. S('ieritifiic. Knowledge and Its Social Prohlenis. Harmondsworth, Middlesex, h g l a n d : Penguin. Sciium. D. A,, 1977. The behavioral richness of cumulative and corroborative tcstjnionial evidcnce. In: Castcllan, Pisoni, and Potts (eds.), Cognitive Theory, Vol. 2 . Hilisdalc, N . J . Laurence Erlenhaum Association. Schum, D. A,, 1979. On factors which influence the redundancy of' cumulative and corroborative testimonial evidence. Technical Report No. 79- 02. Houston, Texas: Rice University. Dept. of Psychology, Snapper, K, and D. A. Seaver. 1980a. The irrelevance of evaluation research for decision making: Case studies from community antkcrime program. Technical Report 80.-12. Decision Science Consortium, Inc. 1:alls Church, VA. Snapper, K. and D. A. Seaver, 19ROb. The use of evaluatior) models for decision making: Application to the community anti-crime program. Evaluafion and Program Planning, 197.- 208. Ulvila, J . and R. V. Brown, 198 1. Development of decision analysis for non-proliferation safeguards: Project summary and characterization of the IAEA decision structure, Technical Report No. 81 --7. Decision Science Consortium, Inc., Falls Church. VA. Viri, A. and J. Vecsenyi. Decision analysis of industrial K flr D problems. Pitfalls and lessons. In this volume, 183-195. von Winterfeldt, D. Setting standards for offshore oil discharges: A regulatory deck slon analysis. Operations Research (in press).

This Page Intentionally Left Blank

DECISION ANALYSIS OF INDUSTRIAL R & D PROBLEMS: PITFALLS A N D LESSONS Anna VARI and Janos VECSENYI Bureau for Systems Analysis State Oj]ke jor Technical Development Budapest. Hungary

Abstract This paper draws attention to a number of pitfalls of multi-attributive decision analysis, viz. : ( 1 ) Decision makers very often do not wish the whole decision problem to be analyzed. Only a limited part--more or less independent of the value system of the decision makers -is subjected to analysis, The multi-attributive comparison does not, as a result, refer to thc actual alternatives of the problem sulution but. e g . , to certain states, objects, etc. characterizing the decision situation. (2) There is a contradiction encountered when structuring the decision problem: while, by increasing the number of attributes the reliability of the assessments by attributes can be improved, the possibility for their simultaneous consideration thereby decreases. ( 3 ) During multi-attributive aggregation, the problem may arise that, if there are cause-effect, means-end relationships between the elements of the system of attributes those cannot be treated independently of the value system, of the decision makers. According to our experiences, the adequacy of utility models can be questioned on performing aggregations under these circumstances. (4) Another pitfall concerns the consideration of uncertainties. The models suggested by the decision analysts, like, e.g.. SEU, are often not accepted by the decision makers as an appropriate way of taking uncertainties (probabilities of success) into account when evaluating projects.

Introduction

Is decision analysis a successful enterprise? We have attempted t o answer tlus question by calling attention t o some pitfalls of multi-attribute decision analyses applied to research and development decisions in Hungary. In the first part we present some illustrative examples of experiences familiar to decision consultants who find themselves in analyzing not the real decision problem but only a limited part of it. We argue that the reasons for this usually stem from the efforts made by the decision makers aimed at simplifying and better understanding the problem, and at the same time hiding some of their goals and values, as well as some of the possible solutions.

184

A. Viri and J . Vecsenyi

In the second part we investigate some questions concerning the quantity and quality of the value relevant attributes. John et al. (1 98 1) point out that completeness and logical as well as value-wise independence of attributes are somewhat contradictory requirements. They also found that whle 'completeness' usually increases the impressiveness of the analysis, the level of acceptance of results based on fewer but more independent attributes is higher. In the third and fourth part we address some problems with aggregation rules. The usual methods for aggregating evaluations across multiple attributes are very poor in handling attribute-interdependencies, like cause-effect, means-end relationships, etc. Besides t h s , we often face the problem of means-end confusion which is a special case of incoherence of preference structure, pointed out by Humphreys and McFadden (1981) as goal confusion, and which cannot be resolved by applying an appropriate decision rule. The problem of finding appropriate rules is even more complicated in cases involving high uncertainties. We found that managers d o not accept SEU-based suggestions in the domain of low probabilities. We will illustrate these problems with our experience in decision analysis of the following three decisions: A. Selection among R & D projects at the branch level. B. Developing strategies for production mixes at the enterprise level. C. Selecting among development alternatives for a given product (choice between home development and buying a licence from abroad).

Case Descriptions

In Case A, a fixed budget had to be allocated across ten R & D projects (Vhri and Dhvid, 1982). Due to considerable long-term uncertainties, the individual projects were compared on the basis of their subjective expected utility (SEU), computed through considering the subjective probabilities of technical success of the research, the success in implementing the R & D results, and the success in applying the results. For each project, these probabilities were estimated using the certainty equivalence method. On the utility side an additive multi-attribute utility model with 14 attributes was used (David et al. ,, 1982). In Case B, a choice had to be made concerning which of the products of an enterprise should be developed, constrained by existing level or reduced budget levels (Vecsenyi, 1982). The products were evaluated by

DECISION ANALYSIS OF INDUSTRIAL R & D PROBLEMS

185

the decision makers on ten attributes and their uncertainties were expressed by using interval-estimations. The weights of the attributes were determined by pair-wise comparison and Guilford-transformation (Torgerson, 1967). The expected utilities were calculated using these estimations with the simulation model described by Kahne (1975)' In Case C, experts of an enterprise had to choose among three foreign licence offers and two home development strategies. A hierarchy of 28 means- and ends-type attributes was explored and transformed by canonical factor analysis into six independent factors. Alternatives were compared using the Kahne modal (Vari and Fustos, 1981). In all three analyses the comparison of alternatives (projects, products, development alternatives) was based on subjective expected utility and additive multi-attribute utilities. Evaluation attributes and input parameters of the decision model (weights of attributes, utilities, subjective probabilities) were elicited by applying various methods, depending on the particular problem. A common feature was that these operations were carried out in groups by experts from various fields and from different levels of the decision hierarchy. The group settings usually included feedback and discussions of the aggregation of individual evaluations, group characteristics (like indices of group agreement), etc. Pitfall 1 :The Domain of the Decision Problem vs. the Domain of Decision Analysis

In Case A, the decision makers faced an R & D resource allocation prob lem They had to choose the best alternative from the possible combinations of the competitive R & D projects. Because of several conditional, complementary and competitive relationships among the projects, we (the decision analysis consultants) suggested an evaluation method taking into consideration these interdependencies. This would have required estimating several conditional probabilities and utilities. Instead the decision makers first independently evaluated the individual R & D projects and several 'rational' combinations of projects. In this stage, peculiar heuristics were applied by the decision makers, which enabled them to engage values which so far had not been made explicit. The philosophy of these heuristics was explicated as the simultaneous consideration of the importance, cost and overlapping of the projects as well as the subjective expected The stages involved in the decision analyses in Cases A and B, and the contexts and participants, are discussed in Humphreys et ol., in press.

186

A. Viri and J . Vecsenyi

utilities computed for each project and project-combination. In fact, among the final choices they considered only importance and neglected all other parameters. (See the detailed description in Part 4.) A similar situation occurred in Case B. Here, a choice among improvement, maintenance and reduction of production of each project of the enterprise had to be made. The actual alternatives of the decision were, therefore, the possible development strategies of the enterprise. Analysis, however, was confined t o the comparison of the individual products. In order to make decisions about the actual development strategies, additional criteria were used, e g . , perspectives of the individual products related to governmental programs, costs required for development, capacity constraints, etc. Nevertheless, at this stage, decision makers did not wish t o apply formal analysis. Von Winterfeldt (1980) has pointed out that approaches based on multi-attribute utility theory (MAUT) are often not adequate in solving resource allocation problems, since MAUT enables only assessments of the individual alternatives and it is not suitable for facing the project-interdependencies and the continuous character of the budget. In our experiences, however, the decision makers still insisted on evaluating individual projects with MAUT. This could partly be due to the relative simplicity of MAUT and to the clarity of its results. But another similarly important reason for using MAUT for individual projects is the frequently observable phenomenon that decision makers prefer the analysis of particular elements of the decision problem to that of the decision problem as a whole. This provides them with additional information useful for surveying complex situations. At the same time, only a part of their values and preferences have to be made explicit and submitted to formal analysis, while implicit values can be taken into consideration intuitively by the decision makers during the actual decision. In h e r a r c h i d decision-making systems, tlus partial analysis helps the lower level decision makers to meet the expectations of higher-level decision makers. This would otherwise be fairly difficult to incorporate into formal decision analysis, as the experiences for other analyses have shown. In Case C, for example, the original intention of the decision makers of the corporate was to evaluate, as a first step, foreign licence offers only and then, in a later phase, to choose between the best foreign and the best domestic alternatives. They assumed that a comprehensive analysis and comparison of the foreign licence offers would justify for the higher level decision makers the importance of buying a licence from abroad instead of home development. Only after some deliberation could we convince decision makers to accept our proposal to compare all foreign and domestic alterriatives using the same criteria.

DECISION ANALYSIS OF INDUSTRIAL R & D PROBLEMS

187

Pitfall 2: Dependent and Overlapping Attributes According to our experiences, the development of attributes is one of the crucial phases of decision analysis. On the one hand, the attributes should allow reliable evaluations of alternatives, while on the other hand, they should facilitate the aggregation of values. These requirements have been more or less contradictory in our cases. The main contradiction as t o the structuring of the decision problem lies w i t h n the fact that while the reliability of evaluation in terms of the individual attributes can generally be improved by raising the number of attributes, the difficulties of their simultaneous consideration will increase. In Cases A and B, high-level categories of criteria were defined by elementary sub-criteria. In Case A, sub-criteria were obtained by brain storming. High-level categories of criteria were defined by clustering. Clustering was first made individually by the members of the group. Afterwards an automatic classification algorithm was used for creating the clusters best compatible with the group clustering. The results were interpreted and criterion categories were defined by the group. Figure la. shows one of the criterion categories and its sub-criteria used in evaluating R & D projects. In Case B, high-level criteria had already been available in the form of recommendations. These criteria were adapted by the participants of the analysis to the concrete situation and subcriteria were determined in the interpretation process. Figure l b shows one criterion category and its sub-criteria. The examples show that the relationship between hgh-level criterion categories and elementary sub-criteria is generally of a wholepart character which is not necessarily additive. In neither case did we seek to reveal these complicated relationshps. However, as a result, it was very difficult to interpret the criterion categories and to determine, at least qualitatively, the end-points of their scales. Since we used the sub-criteria for the purpose of definition only, and since we did not evaluate the alternatives in terms of these sub-criteria, we did not have the opportunity to use them for checking the evaluations in terms of the higher-level criteria. This resulted in substantial uncertainties on the evaluations reflected in differences between the various decision makers’ conceptions of the criteria, which we tried to reduce through group discussions. In Case C, we acted differently. We tried t o analyze the goals of the enterprise concerning the development strategies and the means of achieving them. Figure 2 shows a d e t d of the revealed goal hierarchy. We assigned criteria to each element of the goal hierarchy and asked the decision makers to evaluate the alternatives in terms of each criterion. Statistical analysis of the evaluations enabled us to check whether the

188

A. Vki and J. Vecsenyi

I

Llntelligible results

I

Up-to-date quality of the product

Technical parameters Novelty of the product Up-to-dateness of the materials used

I

~

~

~ service ~

~

Range of supply

Figure 1. Examples of Criterion Categories and Sub-criteria (a) Case A : A criterion used in evaluating R & D projects (b) Case B : A criterion used in evaluating products

m

e

r

DECISION ANALYSIS OF INDUSTRIAL R & D PROBLEMS

I I

Home market Wition

Quality of the product

Profitability

I

I

Exportability

Novelty of the prcduct

!

189

Technical development of the corporate

7

u Novelty01the technology

Figure 2. Goal Hierarchy for Development Strategies (Case C)

supposed means-end relationships did actually exist, and whether the evaluations did not contradict them. Preliminary estimations of the strength of interdependencies among criteria were made by the decision makers and these estimations were compared with the pair-wise productmoment correlation coefficients. By discussing the differences between assessments obtained through the two different approaches, we succeeded in reducing the contradictions between direct and indirect correlations. Making arguments more explicit also raised the consciousness and improved the reliability of the evaluations.

Pitfall 3: Means Don’t Aggregate to Ends As mentioned earlier, an additive model was applied in all three analyses

for aggregating single attribute evaluations. This did not cause us any trouble in Cases A and B, because the criterion categories were formulated at a high level of abstraction, and met the requirements of utility independence relatively well. In Case C, we originally intended t o aggregate the evaluations up to the highest level criteria. This approach turned out to be inappropriate, since the decision makers were not indifferent about the means by which the same goals might be achieved. We found it problematic to determine the importance of the means-criteria and to include such criteria in the summary model. Since the criteria located at lower levels of the goal

190

A . Vari and J . Vecsenyi

hierarchy are partly ends and partly means of fulfilling higher-level ends, their weights have a double character. In a multi-attribute utility model only the ends-component should be considered, but we have n o suitable method for separating this component from the one related t o the means. Further analysis of criteria revealed that not even the goals at the highest level of goal hierarchy have a pure ’ends’ character. They often can be each other’s means. For instance, a greater profit can establish a better economic base for technological development in the long run, while t e c h ological development in turn may influence profit. The above goals cannot be logically combined since they are environmentally related and can even contradict each other under certain conditions. Separation of the means-components of goal-criteria is as difficult as that of the goal-components of means-criteria. If, as in Case C, these components cannot be carefully separated, we face a contradiction: on the one hand, we gain reliability in single attribute assessments through the decomposition, on the other, the aggregation becomes less meaningful. To eliminate this contradiction, it seemed useful to identify quasi-independent means-end sub-hierarchies. In Case C, we tried t o obtain such sub-hierarchies through the use of canonical factor analysis, and the canonical factors obtained were used as inputs in the MAUT-model. The results are shown in Figure 3.

Organizational conditions

Novelty of the technology

Content and level of support

Range of the know- how

I

-M

Internal market position

Exportability

Quality of the product

Risk of the development

Possibility of further development

I

factor

, 2 nd main

.

3 rd main

Canonical factors

L

High level preferences

Profitability

External benefits

development

Organizational

Technical development

Goals -

I

A. Vari and J. Vecsenyi

192

Pitfall 4: Discounting for Uncertainties ys. Striving for Success In the assessment of R & D projects (Case A) overall uncertainties had t o be considered: the technical success of the research, the success in implementing the R & D results, and the success in applying them. The subjective probability of the above success categories were estimated for each project by the decision makers. The maximum feasible utility (U) calculated for each project was that which would be obtained in the case of total success in research, implementation, and application. Probabilities and utilities were combined to provide a SEU evaluation of each project. Prior to being informed about the SEU values and U values, the decision makers were asked to give a preliminary ranking for the projects. Comparing this ranking with rankings based on SEU as well as U values, we found that the preliminary ranking showed a strong positive correlation with the ranking based on U, but a small negative correlation with the ranking according to SEU. Our interpretation of this result is that the preliminary judgments of decision makers neglected the uncertainties and instead expressed a desire, a wish, or a necessity to obtain the maximum utility from the project. What happens if the decision makers get to learn the U and SEU values and the ranking of projects based on them? Figure 4 shows the location of the ten R & D projects in a space whose axes define SEU and maximum feasible utility. For some projects SEU

t t II

+I11

t v ---.

10

IV t*O

%

- -

t x

5

w 3 0

Figure 4. R & D Projects Located in t a m s of their SEU and Maximum 1:easible Utility

Maximum Feasible UtllltlJ (U)

DECISION ANALYSIS OF INDUSTRIAL R & D PROBLEMS

193

(11, X) both values are high, while for others the U and SEU values are

contradictory. Projects VIII and I, for example, have a high U but their success is less probable (low SEU), while projects 111, V, and VII have low U but their success is more probable (hgh SEW. The projects actually selected by decision makers aware of the results of the U/SEU analysis are encircled in Figure 4. Apparently, the decision makers favoured the high U projects rather than hgh SEU projects. The only deviation from a strict U rank ordering is that projects I and I1 are substituted by project VI. This can be explained by the fact that there are some overlappings in contents on these projects. The decision makers explained this final choice by stressing that these projects were very important and consequently their success should be important. The results of the analysis were utilized by alerting the organizations involved to the fact that in certain projects (VIand IX) "great attention should be paid to promoting the implementation of research projects and to the promotion of the use of the results in practice", as it was stated by a decision maker. The paradoxical situation arose that, although the decision makers had acknowledged the necessity of probabilistic thinking, and had made great efforts to estimate subjective probabilities, SEU was not accepted for actual decision making. This might have been due to the low values of thz probabilities of the overall success (plotted in Figure 5 against v). Instead of differentiating among the risks of failure of the projects, the decision makers put forth suggestions designed to alter the social world of the implementation of the projects in such a way that uncertainty about future events would be reduced and hopefully eliminated. Probability

of Ovpr(;ll Success (P)

1O 18b I/

+I1

t Ill

l2i

A

Maximum

Feasible 20

30

UtllltlJ (U)

Figure 5. R .& D Projects Located in terms of Probabilities of Overall Success and Maximum Feasible Utility

13

194

A. Vari and J . Vecsenyi

Conclusions 1. The first difficulty in our analyses was that being comprehensive (in terms of mapping all alternatives and their implications according to goals, sub-gods, means and their relationships) often created attribute structures which were incompatible with simple aggregation rules. In our opinion, there is no need, in every case, for aclueving both comprehensiveness and simple aggregations. In cases where, due to the character and time-scale of the problem, the uncertainty of estimations will not be reduced by a more detailed analysis, it is often sufficient to form more or less independent attributes at a relatively high level of abstraction, and to aggregate these high level goals. On the other hand, when the preferences of the decision makers cannot be reliably assessed (e.g., in the case of some toplevel decision makers) it is not always necessary to aggregate assessments across all attributes. Subaggregation across some attributes can be quite useful in a comprehensive analysis of alternatives, consequences, and means-end relationships. If both the detailed analysis and the multi-attribute aggregation seem to be promising, the gap between the two kinds of analyses should be adequately "bridged". This means that a multi-attribute evaluation should consider each attribute as well as the relationships among them. For this purpose, certain tools of mathematical statistics, such as the canonical factor analysis, employed here, may be useful. 2. The experiences of our case studies indicate that decision makers do not always accept the values aggregated from their "fragmentary" judgments and formed according to prescribed rules (e g.,SEU). If the results of decision analysis essentially contradict the preliminary ideas of decision makers, the decision makers are inclined to restructure the problems in order to reduce these contradictions (e.g., to eliminate certain attributes, to use new ones and to correct estimations and assessments). Humphreys and McFadden (1980) report similar findings. Their decision makers, using a computerized decision aid without help of a decision analyst, frequently restructured their problem in the course of the interactive sessions with the computer. In the cases described above, however, the elicitation of opinions and the use of computer programs were controlled by decision analysts. It is possibly due t o this and to the group interactions in the decision analysis that restructuring of the tasks and adaptation of the preliminary ideas were not attempted by the decision makers during the process of formal analysis of the projects. 3. Considering the observed reluctance of decision makers to allow an analysis of the whole problem, the question arises whether we can find methods which do not require the decision makers to "put their cards on

DECISION ANALYSIS O F INDUSTRIAL R & D PROBLEMS

195

the table", but still help in solving the real decision problems as a whole. Alternatively, we could resign ourselves to help the decision makers in structuring, learning, and estimating problem elements only. Usually, there is a compromise between the intention of the decision makers and the aspiration of the analysts as to w h c h methods to apply when analyzing a given problem. This decision is also multi-attributed, considering such criteria as, e.g., the connection between decision makers and analysts, the individual or group character of the decision making, the time factor, the efforts required, etc. We assume that considerable differences exist about the usage of decision aids, depending upon cultural factors (e.g., the expected role of the consultants, the function of group discussions, etc.) as well. We believe that further investigation of the above topics,e.g.,through crossnational comparative analysis, would be necessary for supporting the choice of the appropriate decision aiding tools under different circumstances.

References Humphreys, P. and W. McFadden, 1980. Experiences with M A W : Aiding decision structuring versus boot-strapping the decision maker. Acta Psychologica, 45, 51-70. Humphreys, P. C., 0. Larichev, A. Viri, and J . Vecsenyi. Comparative analysis o f use of decision support systems in R & D decisions. In: H. G Sol (ed.),Processes and Tools for Decision Support. Amsterdam: North-Holland (in press). John, R. S., D. von Winterfeldt, and W. Edwards. The quality and user acceptance of decision analysis performed by computer vs. analyst. In this volume, 30 1-31 9. Kahne, S., 1975. Procedure for optimizing development decisions. Automatica, 11. Torgerson, W. S., 1967. Theory and Method of Scaling New York: Wiley. V k i , A. and L. Fiistos. M6dszerek licenciavisklasi dontksek megalapozisha (Methods for decision making in buying licences) (in Hugarian and Russian). Presented at the joint meeting of MNIIPU and OMFB REI. Budapest: Szigma (in press). VLi, A. and L. Divid, 1982. R & D planning involving multi-criteria decision analytic methods at the branch level. Collaborative paper CP-82-73. Laxemburg. Austria: IIASA. Vecsenyi J., 1982. Product mix development strategy at the enterprise level. Collaborative paper CP-82 -74. Laxenburg, Austria: IIASA. von Winterfeldt, D., 1980. Structuring decision problems for decision analysis. Acta Psychologica, 45, 71-94.

13*

This Page Intentionally Left Blank

Section I11

AIDING THE STRUCTURING OF SMALL SCALE DECISION PROBLEMS

This Page Intentionally Left Blank

INTRODUCTION Anna

VARI

The widespread use of microcomputers has been associated with an increasing interest in computerized decision aids which can assist individuals in solving their problems. This interest is well reflected by the relatively rapid spread and use of programs discussed in this section, like that for "Multi-Attribute Utility Decomposition" (MAUD), developed by Humphreys and Wisudha, and that for "Goal-Directed Decision Structuring" (GODDESS), developed by Pearl, Leal, and Saleh. One of the aims of the authors in this section has been to provide extensive survey of computerized decision aids based on different principles and developed for supporting the stucturing of small-scale decision problems. At the same time, they have attempted t o answer general questions of principle, as for instance, what the division of labour should be like between the decision maker, the decision analyst and the computer. Which activities of the human decision-muking process should be automated and which should be augmented? To what extent can the help given by an unalyst in resolving a decision problem be translated into software? How can the human representation of decision problems be modelled and what are the requirements to be met by the procedures used in decision aids? In lus paper, Gordon Pitz has sought an answer to the general questionson the division of labour between the decision maker and the decision aid. in approaching the above questions, the author has suggested carrying out a kind of cost-benefit analysis where "the costs of automation are based on the difficulty of automating the process accurately; its benefit! depend on the difficulty faced by the human if left without mechanical assistance". In the case of certain types of abilities-like the integration of largc quantities of information - the superiority of computers over the human i!

200

A. Vari

quite obvious, and the advantages resulting from use for such tasks can be easily determined. Concerning other abilities-like information retrieval from long-term memory, evaluation of complex perceptual information and creative problem solving-the systems implemented at the present technical level d o not even approach human abilities. However, assistance lent by them may considerably increase (unaided) human abilities in these tasks. The degree of assistance depends, first of all, on the compatibility of the decision aid with human behaviour. Pitz’ paper offers an analysis of some extensively used decision aids from the point of view of Compatibility with human information processing procedures, comparing the applied techniques and procedures with recent results from psychological research on human information storage, retrieval and problem solving. One of the most debated questions is how the categories ( t e r m ) and procedures constructed during decision analysis are compatible with the human mechanisms and strategies used t o develop the structural representation of the decision problem “Is the knowledge that is retrieved in decision situations actually structured in terms of actions, events, and outcomes, as decision theory assumes? How are goals connected t o these elements? These are the questions posed by Helmut Jungermann, Ingrid von Ulardt and Lutz Hausmann. In their paper they attach special importance t o the goals which they consider the key elements of problem representation. They propose a network model in which goals and actions are connected and the actions can be activated by the related goals. In order to validate the proposed model, they have investigated the assumptions as follows. When goals are made more explicit and specific -which means an increase in goal activation-the activation of the connected actions will also be increased. As a consequence, more options will be generated. On the other hand, when the problem is considered from the point of view of one’s personal priorities-which means the limitation of goal activation compared with the less self-oriented approach -the number of actions generated will be limited as well. The results of the experiments support the hypotheses on the above two factors, i.e., goal explicitness and personal involvement, thus demonstrating the importance of goals in the process of generating options. In Dimiter Driankov and Ivan Stantchev’s paper a similarly great importance is attached t o the goals in the process of option generation and evaluation. Goals are considered as complex wholes which are described in terms of constituent parts and interrelations among these parts. The authors suggest a fuzzy structural modelling approach w h c h offers an efficient procedure for describing imprecise, ill-defined andlor ’I

Section Ill :INTRODUCTION

20 1

nonlinear interrelations among the constituent parts of the goals. The suggested procedure helps the decision maker in measuring the attainment of the goals, in assessing the strength of influence among the constituent parts of the goals, as well as in generating and testing options. The case study discussed by Patrick Humphreys is concerned with otherwise rather neglected methodological problems of option generation. The case studied involved the selection of a set of computers intended for meeting a relatively complex system of goals and requirements by which, from a largely extensive set of options, the specification of options worth of consideration can be made. The suggested procedure is based on the thorough analysis of the goals to be achieved. The first step is the hierarchical decomposition of the goals into computing facilities. This is followed by the mapping out of three requirements spaces in which the interrelatedness of the computing facilities can be described from the point of view of the hardware and software requirements to be met as well as from the point of view of the groups of potential users. Partitioning the requirements spaces in different ways is used for generating different configurations of systems meeting the whole set of requirements. First, options are generated under different philosophies (e. g., assuming the use of microcomputers, "multi-micro" approach, etc.), and afterwards actually purchasable computers are specified which meet the requirements prescribed for the parts of the composite options. The case study demonstrates the practical applicability of certain procedures of goal analysis, as e. g., the herarchical decomposition of goals, the use of multidimensional scaling for mapping the requirements spaces, for aiding the generation of options. It shows also how flexibility analyses can be performed to options under consideration. Mule the above papers discussed the issues of the relationships between goals and options, with special regard to option generation, other papers included in this chapter have focussed their attention on computerized procedures of aiding the evaluation of options All the programs investigated are based on multi-attribute utility decomposition (MAUD) comprising automated procedures for elicitation and weighting of attributes as well as for computing multi-attribute utilities. The experiments described by Fred Bronner and Robert de Hoog have double aims. In part, they attempt to elaborate evaluation techniques which can help developers of interactive computer-based decision aids in revealing and eliminating the deficiencies of software still in the process of development. On the other hand, by using these techniques, they evaluate the experiences of the use of a MAUD-based decision aid (developed by themselves) by people without a special training in decision analysis.

20 2

A. Vari

For the evaluation of the decision aid, two criteria were introduced: the overall ease of interaction with the program and the helpfulness of the aid The value of the first criterion depends on how independently, i. e., without any technical or conceptual assistance, can the subjects go through the decision analysis, namely, how much the program is compatible with the procedures of the individual problem structuring of the decision makers. In order to measure the degree of helpfulness, a so-called scale of consciousness raising was introduced. The scores on this scale are dependent on the subjects’ judgements as to the importance of the program from the points of view of problem structuring, awaicness raising and decision justification. The authors’ experiment for testing the proposed criteria and measurement techniques in evaluating their program has proved t o be successful. For example, through investigating subjects’ requests for (external) assistance in different phases of the analysis, the stages of the program requiring considerable improvements can be identified. In comparing different decision problems in terms of ”perceived applicability” scores provided by users of the program, guidelines for the application area of the software can be defined. Analysing the relationshps between scores on the proposed evaluation criteria and characteristics of the subjects (education, age-group, sex, technical experience) can lead to important decisions related to the education and training of the potential users. Similarly to Bronner and de Hoog, k c h a r d S. John, Detlof von Winterfeldt, and Ward Edwards also consider the attribute elicitation procedure to be the most crucial phase of multi-attribute utility analysis. They compare the effectiveness of decision analyses performed by a computer running the MAUD 3 program (written by Humphreys and Wisudha) and by the analyst from the point of view of the number and quality of the elicited attributes. According to the results of their experiments, the number of attributes elicited (“completeness” of the analysis) was greater for analyst sessions than for MAUD 3 sessions. However, the attribute set elicited by MAUD 3 was judged t o be more independent, both logically as well as valuewise. Judgements of overall quality of the attribute set hghly correlated with those of ”completeness”. Further findings from John, von Winterfeld and Edwards’ experiments point out the contradictory character of the evaluation criteria for decision analysis as well. The users’ satisfaction with the process was better in analyst sessions, while the acceptance of the resulting alternatives was higher in MAUD3 sessions. This points to the fact that the subjects’ feelings were influenced by factors different from the quality of the results, eg., by the supposed “completeness” of the elicited structure, the

Section 111: INTRODUCTION

20 3

need for personal interaction, the experience with or aptitude for computer-like tasks, etc. Consequently, instead of raising the dogmatic question of "computer or analyst" it is more to the point to create the most satisfactory division of labour between the computer and the analyst, considering the characteristics of each decision situation. The paper by Stuart Wooler and Alma Erlich deals with another critical phase of multi-attribute utility analysis: attribute weighting. They made their investigations in connection with a MAUD-based set of interactive computer programs whch had been developed for university students to structure and evaluate their own career options. The weighting assessments of two groups of students were investigated: Members of the first group were at the stage in their decision processes where they had fully determined the option set they wished to evaluate, whereas the second group was at the stage where they had structured a preliminary option set, which they wished to evaluate with a view to determining those options worthy of further consideration. The weighting strategies of the two groups differed signlficantly. The weights assessed by participants concerned with structuring correlated with the degree of satisfaction with the current option set on each attribute, namely, the degree of achievement of the subgoals represented by the attributes. On the other hand, the weights assessed by those currently concerned with evaluating options were based on weight judgements across "grand scale" attributes, without incorporating a measure of the range of the option set on each attribute. The fact that the weightingstrategy of both groups contradicts the prescriptions of decision theory hghlights the point that, in every case, the choice of attribute weighmg procedures should be made with due consideration. Special attention should be devoted to the anchors to be used in decision situations where-as here in the case of career choice-the decision makers usually face a lack of knowledge necessary for constraining the attribute scales by reality and feasibility.

This Page Intentionally Left Blank

HUMAN ENGINEERING OF DECISION AIDS' Gordon F. PITZ Southern Illinois University at Carbondale, US.A.

Broadly speaking, this paper is about helping people to make decisions when confronted with situations in which they are not sure what to do. When one reflects on the issues involved in this topic, it becomes clear that almost everythmg known about human behavior is potentially relevant. T o make the topic manageable, I have defined certain boundary conditions on the kinds of decision problems to be considered, and on possible approaches to solving these problems. Someone interested in a more thorough survey of decision aiding methods and principles can sample from a variety of recent books and articles written from different points of view (e.g., Janis and Mann, 1977; Hogarth, 1980; Jungermann, 1980a; Humphreys, 1981). The situation I had in mind in writing this paper was one in which a problem of personal choice is to be assessed from the point of view of a formal model of decision analysis. 1 have assumed that external expertise is not available either in the form of a human expert or as part of the decision analysis. Several computer programs have recently appeared that can implement such a decision analysis (e. g., Humphreys and Wisudha, 1979; Weiss, 1980; Pearl etal., 1980). What makes these programs interesting t o a psychologist is their generality; they make few assumptions about the nature of the person's problem, and include procedures for helping the person to describe any problem in terms that would be suitable for analysis. The programs attempt to formulate the problem according to some formal structure or representation; they vary in how they help a person to achieve this representation.

'

I would like to thank Michael Franzen, Natalie Sachs, and three anonymous reviewers of the manuscript for their helpful comments on earlier versions of the paper.

206

G.F. Pitz

Programs of this sort are likely to proliferate, and the ones I have nmtioned will no doubt be modified on the basis of experience with their use. The purpose of this paper is to discuss some psychological issues involved in the design of decision aids of this kind. "Human Engineering is the name given to that branch of applied psychology that uses general principles of human behavior to guide the design and construction of operational systems involving people (Alluisi and Morgan, 1976). I have used the term in the title of tlus paper because it captures quite precisely the issues involved in the design of decision aids. One approach used by human engineers t o issues of design is to regard the human as an integral part of a larger system, and t o investigate ways in which the contribution of each part of the system to its overall functioning can be made most efficient. One may divide t h s approach into two parts: First, find an appropriate division of labor between the human and the rest of the system; second, given that one has determined the responsibilities of the human component, design the total system in such a way that the person can meet these responsibilities most effectively. By focussing on the interaction between decision aid and decision maker I may give the impression that there are no other issues involved in the design of the decision aid. In typical applications of decision analysis there is usually a third party involved, namely, the decision analyst. Furthermore, many applications will incorporate other sources of information, especially when something is known about the substantive concerns of the decision maker. For example, an analysis of career decision making would certainly need to include relevant sources of vocational information. My failure to discuss these issues should not be taken as an indication that they are any less important than those that I have discussed.

Automation and the Division of Labor The systems approach to design regards the decision maker and the decision aid as parts of a single decision making system. The overall goal of the system is to arrive at a choice that maximizes some explicit criterion, given certain assumptions about the nature of the problem. The importance of the conditionality implicit in this statement has been emphasized by Einhorn and Hogarth (1981). There may be few assertions to be made that are independent of the assumptions made about the problem. Nevertheless, it should still be possible to offer general guidelines that apply to the design of specific systems. Achieving the objective of the decision system is clearly dependent on input of some sort from the human decision maker. The person must

HUMAN ENGINEERING OF DECISION AIDS

201

provide some information about the nature of the problem, about the uncertainties that are involved, about the values t o be taken into consideration, etc. On the other hand, a mechanical component of some sort is necessary to provide an integration of all this information. The integration task is normally carried out by calculating utility functions, expected utilities, or other quantities. The output of the system, therefore, can be regarded as a set of descriptive statements calculated by the mechanical component. Between the initial human input and the final mechanical output, however, it is not clear which components of the decision task should be automated and which should be left to the individual decision maker. The important question is, what should a decision aid try to do for the decision maker? Deciding which parts of the process t o automate requires a kind of cost-benefit analysis. The costs of automation are based on the difficulty of automating the process accurately; its benefits depend on the difficulty faced by the human if left without mechanical assistance. In some cases, the relative superiority of human and machine are clear cut. For example, the computational abilities of the machine are clearly far superior; once probabilities and utilities are known, no one would suggest leaving the calculation of expected utilities to the human. Machines are better able to follow formal algorithms without error, they can deal better with known information in a predetermined way, and they are able to insure that no information is ignored that is known to be relevant. Conversely, we know that limitations on human short-term memory make it difficult to integrate large quantities of information (Newell, 1973). It is presumably for this reason that people revert to simplifying heuristics in making judgments (Lopes and Ekberg, 1980). Identifying the unique strength of humans is more difficult, since advances in artificial intelligence may make today’s human strength tomorrow’s mechanical advantages. Nevertheless, there are a number of respects in which no machine can yet approach the effectiveness of a person. One such area is the encoding, storage, and retrieval from longterm memory of large quantities of information organized in complex ways. In contrast to the limits on short-term memory, the capacity of, and the speed of information retrieval from long-term memory are quite remarkable. As yet there is no mechanical substitute for human memory in this respect, even with quite narrowly defined content areas. Weiss (1980) has proposed a decision aid (GENTREE) that would have access to a very large data base for decision structuring. While such an effort is admirable, one must be pessimistic with respect to its success. The kind of data base with which computers deal efficiently is one in which a large number of items are organized in fairly simple ways (long lists of people, places, etc.). The

G.F. Pitz

208

kind of data base that seems most useful for representing a decision problem is one involving multiple, complex interconnections among items of information. In recent years, some understanding of the organization of human memory has developed, and artificial models of memory organization have appeared (e.g., Anderson, 1976; Bobrow and Norman, 1975; Minsky, 1975). These models give some insight into the nature of human memory, but can hardly be said to replace it, One of the responsibilities, then, for the human component of the decision system will be to provide relevant information about the problem from knowledge stored in l o n g term memory. A second, uniquely human ability that may be important in designing decision aids is the rapid evaluation and categorization of complex perceptual information. While integrating complex information within short-term memory is awkward and error prone, the automatic integration of perceptual information is very fast and accurate (see Schneider and Shiffrin, 1977). This perceptual skill is related t o the human ability to deal rapidly with large amounts of information stored in memory. For example, expert chess players derive much of their expertise from the rapid identification and evaluation of the dynamic features of a chess position. This is made possible by having in memory a very elaborate storage of relevant information (Chase and Simon, 1973). Computer models of this process are not very impressive. For example, recent advances in computerized chess programs rely on the rapid search and the calculation abilities of a computer, rather than on attempts to model human perceptual and mnemonic abilities.l Thus, insofar as decisions depend on perceptual evaluations, these appear to be best left to the human. Other areas in which human abilities seem t o be unique include certain aspects of problem solving, inference and creativity. Pitz et al. (1980) have suggested that identifying new, creative options is an important aspect of decision aiding. While this creativity can be assisted, it is unlikely that it can be automated. As in the case of memory, computer models of human problem solving with a well-defined task can often be quite impressive. Nevertheless, I expect that i t will be many years before generalized mechanical problem solvers can outperform humans if the problem is not well defined. This limitation will be particularly important if we are concerned with decision aiding systems that are intended t o have general applicability.

See Scientific American, April 1981. 244 (4).

83-85.

HUMAN ENGINEERING OF DECISION AIDS

209

The design of a decision aid requires that much of the relevant information be made available to the decision system from the memory of the human decision maker. The remainder of the decision analysis process will probably not be fully automated. For any given substantive problem, it may be possible to provide useful technical information, as well as an overall structure for objectives and attributes. However, there will always be advantages to having human perceptual skills and creative problem solving abilities used to evaluate the information. Nevertheless, the use of human memory, perception, and problem solving within the decision system is not without need of mechanical assistance. There are characteristic limitations to human memory and problem solving. An important part of any decision aid, therefore, might consist of devices designed to overcome these limitations. A good decision aiding mechanism would help an individual retrieve from long-term memory (and, perhaps, from other sources) information that might be relevant for the analysis of the problem. In addition, it might help a person to recognize new connections between previously unassociated items in ways that lead to creative problem solving. In order to develop procedures for assisting humans in performing these tasks, it will be necessary to know something about the nature of the information retrieval and problem solving processes. For example, if we know how retrieval from memory occurs, it may be possible to design methods for assisting the retrieval process. If we know something about how inferences are made, it may be possible to construct automatic devices that help a person to make new and appropriate inferences. Thus, we need to address the issues of designing an interface between the human and the machme. Communication Between Decision Maker and Decision Aid Problems in designing a mechanical decision aid range from the traditional concern of human engineers with the design of displays and controls, to the more subtle problem of making sure that information stored in human memory is adequately represented to the system. It is helpful in discussing these problems to divide the decision making process into four separate stages. Stage one consists of selecting an appropriate abstract model that can be used for analyzing the problem. Stage two consists of formulating the original problem in terms of the structure that is required by the analytic model. Stage three is the quantification process, in which the decision maker’s values and beliefs are represented in some form suitable for computation. Stage four consists of the feedback of results to the 14

210

G.F. Pitz

decision maker, and possible reevaluation of the judgements made at the three earlier stages. In any complex analysis these stages will not be clearly distinct, and the decision malung process will not follow such an orderly sequence. Nevertheless, the stages can be helpful in identifying some of the psychological issues involved. Design issues can be addressed separately for each of these four stages.

Model Selection There is not a lot to be said about the model selection process. A characteristic of decision aids currently available is that they offer little flexibility in the kind of analytic model used for decision analysis. Such a limitation is perhaps inevitable. The demands made on the system at later stages of the analysis are frequently unique t o one particular kind of model. To design a system capable of working with several different models would probably not be worth the effort involved. Some of the issues involved in model selection have been discussed by von Winterfeldt (1980). An effeo tive choice of model depends upon having a useful taxonomy of problems that might be used for diagnosis purposes. Von Winterfeldt has discussed existent taxonomies, and he outlines an approach based on the construction of prototypical problem types that might describe broad categories of substantive problems. Subsequent sections make no particular assumptions about the type of decision model to be used, except that it falls into the category of "orthodox decision analysis". This consists of a combination of expected utility analysis of the sort described by Raiffa (1968), and its extension to the multiattribute utility models described by Keeney and Raiffa (1976). Issues of problem structuring and quantification that arise within the context of these models will be discussed.

PToblem Structwing

Humphreys and McFadden (1980) suggest that one of the most important things a decision aid can do for a person is to provide some better understanding of the interrelationships among elements of the problem. In any event, structuring a problem appropriately, i. e., formulating it in such a way that it is amenable to analysis according to the analytic model, is far more important than the small amount of research devoted to the topic might imply (Jungermann, 1980b). There are many issues involved in this stage of the decision analysis that are not yet well understood. From a

HUMAN ENGINEERING OF DECISION AIDS

211

human engineering point of view, it may be the most critical stage in the decision analysis. In discussing problem structuring, I shall assume that we are interested in eliciting from the decision maker two kinds of information. On the one hand, we need t o know the range of options that might be available, and the possible outcomes that might result from each choice. On the other hand, we need information about the decision maker’s goals or objectives, from which we can derive an attribute structure that can be employed to evaluate options and outcomes. Assume for simplicity’s sake that the only source of relevant information is the individual decision maker. In some cases we might want t o consider the possibility that the decision aid could direct the decision maker to external sources of information (see, for example, the career guidance system designed by Katz, 1973). However, the present discussion will not address these possibilities. The simplest approach that a decision aid can take is to ask the decision maker to provide a list of options and a set of attributes. The program QVAL (Weiss, 1980) does little more than use this direct approach Humphreys and Wisudha’s program M U D possesses a more elaborate set of procedures for eliciting attributes. The elicitation approach taken by MAUD is based upon Kelly’s personal construct theory (see Humphreys and McFadden, 1980). MAUD includes a number of techniques that help the decision maker to develop a multiattribute representation of the problem. An equally elaborate approach, but based on a very different set of principles, is described by Pearl et a1 (1980). The program GODDESS is based on a goal-directed approach to decision structuring. It begins with an attempt to specify the decision maker’s primary objective, and then attempts to describe the means-end relationshlps among elements of the problem. A decision structuring mechanism needs to address three separate issues, each of which has important psychological implications. First, it is necessary to retrieve from the decision maker’s long-term memory information that might be relevant for generating options and for describing objectives and attributes. Second, it must help the decision maker to go beyond the available information. For example, it may be necessary to provide generalized, abstract characterizations of options, outcomes, and objectives, In addition, it may be useful t o help the person exercise some creativity in generating options or outcomes that have not previously been considered. Third, the mechanism must help the decision maker to d o all this in a way that is compatible with the analytic model.

14”

212

G.1:. Pitz

Information Retrieval Retrieval of information from memory is a process that is well understood by cognitive psychologists. Its role in structuring decision problems has been discussed by Gettys and Fisher (1979). It is possible to identify a number of reasons why important information might not be retrieved at the appropriate time. For example, retrieval is known to be context dependent; information stored under one set of conditions may not be retrievable in a different context (Light and Carter-Sobell, 1970). The best general solution to this problem is t o devise procedures for encouraging a person to view problems from a range of different perspectives. Another way in which the decision aid might help in the retrieval of information is by providing cues that have a high probability of being associated with relevant information. In this way, the aid can direct the decision maker’s search for information in long-term memory. Since we are assuming that initially the decision aid has no information about the specific problem faced by the decision maker, retrieval cues must obviously be derived from the decision maker. Any procedure for assisting retrieval must therefore include a boot-strapping process, in w h c h preliminary information obtained from the decision maker is used to generate retrieval cues that might be helpful in generating new information. This approach to structuring has been discussed by F’itz, Sachs, and I-leerboth (1980) and by Pitz, Sachs, and Brown (1981). We described two different techniques that seem to be effective in helping a person to generate options and objectives. The technique that we found to be most helpful is based on the theoretical concept of “schemata“ (see below), and on the kind of means-end analysis that is represented by the GODDESS program (Pearl et al., 1980). It involves using information about options t o suggest relevant objectives, and using information about objectives to elicit possible options. Since in many cases one cannot differentiate the retrieval of existent information from the generation of new information, a discussion of this technique must include a consideration of the inference process. Retrieval and inference appear to be closely interrelated mechanisms.

Generating New Ideas It is almost impossible t o rely on retrieval alone to provide an adequate structure for a decision analysis. By specifying options, outcomes, and attributes, the decision maker will inevitably begin to explore new ideas. It is clear that memory processes cannot in general be separated from infer-

HUMAN ENGINEERING OF DECISION AIDS

213

ence processes (Bransford et al.. 1972). Current theories of memory make extensive use of this interdependence. Several recent theories suggest that memory is organized around "schemata" or "scripts", which are abstract scenarios that describe typical sequences of events and actions in one's daily life. It is by using such schemata that a person is able t o understand and remember information. Establishing the connections between generalized schemata and specific event sequences involves the use of inference processes (see, e.g., Schank and Abelson, 1977). By reference to the concept of schemata, we can understand how an individual might have difficulty in structuring a problem completely, according to the set of scripts that the person thinks are relevant for the problem. T h s kind of fixation can be seen in numerous studies of problem solving. The phenomenon of "set" (Luchins, 1942) refers to the continued use of strategies that at one time were useful, but are no longer relevant. The similar phenomenon of "functional fixity)) (Duncker, 1945) is the identification of some component of the problem in terms of its most obvious function, and the consequent failure to recognize alternative functions that might be served by that component. In both cases, a person is limited by sequences of ideas that follow well established lines. Creative solutions t o problems are typically found by breaking away from such structured thinking. One way of doing this is to stimulate alternative scenarios. Pitz et nl. (1980) described how this might be done. They found that the number of options that a person could generate for a typical decision problem was increased by providing the person with a list of the relevant objectives, one at a time, and asking the person to think only of choices that might be helpful for acheving that objective. In a further study of the method, Pitz et al. (1981) found that the additional options generated in that way were not identifiably poorer choices. They also found that the same procedure could be used effectively in reverse. By presenting a list of options, one at a time, and asking the decision maker to think about possible outcomes of each choice, it was possible to generate a more complete attribute structure for analyzing the problem. The procedure used for generating objectives and attributes by Pitz et nl. (1981) differs in some interesting ways from the procedure used by MAUD (Humphreys and Wisudha, 1979). The essence of MAUD's procedure is to focus on dimensions that define differences between options. The Pitz et al. procedure focuses on outcomes of individual options. In a recent experiment, Michael Brown and I compared the effectiveness of these two procedures. We were unable to find any difference between them in terms of the completeness of the attribute structure that results, or its ability to differentiate among the options. Nevertheless, there appear

214

G.F. Pitz

to be some important psychological differences between the two procedures that would be worth exploring further. There may be other ways in w h c h the retrieval and generation of relevant information can be assisted by a mechanized decision aid. As mentioned earlier, one of the reasons why human problem solving sometimes fails is the limited ability of short-term memory to deal simultaneously with several items of information. Many failures to recognize new solutions may occur because the individual never had the necessary pieces of information together in short-term memory. It should be possible t o design display systems that present elements of the problem structure in groups in such a way that the decision maker can use them creatively. One must be cautious, however, in using such a procedure. The ability of a machine to present the decision maker with large quantities of information can actually interfere with the efficiency of the decision making process. For example, management information systems now exist that enable a decision maker to retrieve and display large quantities of information. However, one effect of increased information may be to increase the decision maker’s confidence in his or her eventual decision without any necessary increase in accuracy (Oskamp, 1965). Einhorn and Hogarth (1978) describe how it is possible under some conditions to learn to be very confident about poor judgements, and never to discover this error.

Problem R eprcsen tation Generating a formal problem structure requires that a decision maker’s knowledge, beliefs, and preferences be translated into a format appropriate for the analysis. The decision maker’s description of the problem, even if complete, will not necessarily be suitable as a formal structure. For example, the typical multiattribute utility model considers outcomes and options to be points in a continuous multidimensional space. There may be times when the decision maker’s description of the information can be characterized in this way-for example, when the important concerns for the decision maker can be described by variables such as cost, time, and distance. However, in most cases a spatial representation of the information may be very difficult to obtain. Recent theories of the organization of knowledge, and theories of preference, tell us something about how relevant information is organized for the decision maker. While there are some theories of knowledge that are spatial in character (e. g., Rips et al., 1973), most recent theories suggest that knowledge consists of a network of associations among discrete nodes. Information may consist of features, relations, hierarchical

HUMAN ENGINEERING OF DECISION AIDS

21 5

structures, or production rules, but the way in which these are related to continuous quantities is not at all clear (see Pitz, 1980). Similarly, there are non-spatial theories of preference (e.g., Tversky and Sattath, 1979) that may help to explain how problems are represented cognitively. An important concern in the design of decision aids is to find procedures for translating non-spatial information about preferences and knowledge into the format required by a normative model. As an example of the difficulties involved in creating an appropriate problem representation, consider an application of multiattribute utility theory to the problem of selecting a contraceptive (Sachs and Pitz, 1981). Both cognitive theory and actual experience suggest that the most suitable representation of the problem is one that characterizes the options in terms of discrete features that are unique t o each contraceptive. The pill and the IUD pose certain health hazards (different in each case), rhythm methods require a period of abstinence, and so on. There is no reason in principle why a multiattribute utdity model could not use such features as attributes. In practice, however, the problem of assigning weights to the features is then unmanageable, due largely to the difficulty of establishing tradeoffs among attributes of this sort. The alternative approach is t o use more abstract attributes that are continuously variable (we used such attributes as convenience, personal feelings, etc.). To establish such attributes, however, requires a great deal of effort; it took six of us several months t o establish the attribute set, and we still ran into unexpected difficulties later. To ask a decision maker to perform this abstraction process for each separate problem is to violate the original purpose of decision analysis. The assumption behnd the analysis is that one should minimize the amount of integration that the decision maker has to perform by asking the individual to evaluate only the basic elements of the problem There are other recent findings concerning the representation of information in long-term memory that may have implications for decision structuring. Wilks (1977) has suggested that knowledge can be divided into two broad categories, causally-oriented and goal-oriented. The former deals with cause and effect sequences among events, the latter with actions that have some intentional or goal-directed property. When representing a problem for purposes of decision analysis, goal-oriented knowledge may be most relevant when describing the objectives that are to be achieved; causally-oriented knowledge may be most important’ in describing tree structures that connect possible options with subsequent outcomes. A recent study by Graesser, Robertson,and Anderson (1981) suggests that these two kinds of information are organized in different ways. If this is the case, problem representation procedures that deal with objectives may

216

G.F. Pitz

have to be different from those that deal with options and outcomes. Note that the program MAUD, which is primarily concerned with the attribute structure that relates to the objectives, has a set of procedures quite different from those used by GODDESS, which is concerned with the means-end relationships between options, outcomes, and goals. One important property of goal-oriented knowledge is that it is hierarchical. Achieving a single goal may depend on first achieving a number of sub-goals, while each sub-goal requires establishing intermediate goals, and so on. For this reason, goals at each of these levels may be regarded as options; one may establish as an option the pursuit of a given objective. For example, suppose that a college student is trying to select a career. One option is t o pursue a career in medicine, which requires that the student first pass a number of science courses, then make application to, and be accepted by, a medical school, find the necessary financial support, and so on. In this situation the distinction between options and objectives becomes confused. Any option may require that certain conditions be met before it can be selected, which leads to the specification of new objectives. On the other hand, one may choose whether or not t o include a certain objective in the analysis. Descriptive models of human problem solving suggest that people do think in terms of such means-end relationships (e.g., Atwood and Polson, 1976). Yet the usual form of decision analysls has difficulty in capturing this representation (see Pearl et al., 1980). Problem Translation

Because a person’s cognitive representation of a problem may be so different in character from that required by a decision analysis, it would be helpful if decision aids could somehow translate a direct description of the problem into a format compatible with the analyis. To some extent this is what the programs MAUD and GODDESS try to do. MAUD, however, still requires the decision maker to identify and name the relevant dimensions, GODDESS is more compatible with a means-end representation of problems, but it is not clear that it can capture the feature-based structure of many personal decisions. Problem translation would be a two-stage process. First it is necessary to assess the structure of the decision maker’s representation of the problem; then that representation must be translated into an analytic structure. There are several procedures that have been described recently that might be adaptable for the problem representation component. For example, Reitman and Rueter (1980) describe a technique for assessing

HUMAN ENGINEERING OF DECISION AIDS

217

the structure of information in long-term memory that is based on an analysis of free recall; it uses item sequences and pauses between items to infer the underlying structure. Craesser et al. (1981) describe a procedure that uses a series of "how", "why", "when", and "where" questions to generate a representation of a person's knowledge structure. Both of these techniques lead to representations in terms of discrete nodes and associations among nodes, rather than as points in multidimensional space. To be useable for decision analysis purposes, one must develop algorithms that allow mapping a representation of t h i s sort onto a multiattribute space. In order to develop such a mapping procedure one must know something about the kind of problems that people confront, and how these problems are represented cognitively. One might employ something like von Winterfeldt's concept of problem prototypes, but using a taxonomy that is defined in terms of cognitive representations rather than situational characteristics. One way to initiate such a taxonomy is to develop a syntax for problem representation. It may be possible to identify a set of rules that describe the form of a problem, independent of its content. Some preliminary research of mine has been concerned with the way in which people describe their problems. A person is asked to characterize the problem in a single sentence, beginning with the words "I don not know ..." One can construct a grammar for parsing the sentences that are produced this way. That is, each sentence can be described as a sequence of transitions betwesn nodes of a syntactic net. What is of interest is the possibility of inferring important properties of the problem from a purely syntactic analysis of this sort. For example, a distinction that may be important for decision structuring is between problems in which the options are well defined, and problems in which the decision maker needs to expend some effort to discover suitable options. It appears that sentences beginning "where", "when", or "which" are more often associated with problems with well defined options. "Whether" typically identifies twwhoice problems. "How", on the other hand, is usually associated with problems in which a search for suitable options would be an important part of the decision analysis. The purpose of a syntactic representation is to provide a formal mechanism for identifying problem characteristics that are important from a decision analytic point of view. The goal of such an approach is to describe a problem without having to use concepts or frameworks that are incompatible with the cognitive representation of the problem. Only if one can establish translation rules that map cognitive descriptions onto formal representations will it be possible to automate the most difficult part of the problem structuring process.

218

G.F. Pitz

Quantification Problems involved in quantifying a decision maker's values, preferences, and beliefs are widely known and have already been discussed by others (Tversky and Kahneman, 1981; Fischhoff et al., 1980). It isclear that one can obtain mutually inconsistent judgements from the decision maker by asking equivalent questions in different ways, or in the context of different statements of the problem. Apart from inconsistencies that stem from information processing limitations, there are other biases that might be due to the way in which an individual is asked to translate subjective feelings into numbers. Some of these issues have been discussed by Weiss (1980); I shall not pursue them here.

Evaluating the Results The purpose of a decision aid is to provide an analysis of a problem that is acceptable to the decision maker, and that makes recommendations for future action. The final recommendation must be consistent with all other information provided by the decision maker. Presumably the decision aid contains procedures for detecting inconsistencies, whether they occur among the initial judgements, or between the final recommendations and the person's global evaluation of the options. However, when inconsistencies do exist it is not clear how one should proceed; they may arise for any number of reasons. Inconsistencies among quantifications of beliefs and values, for example, may result from an inappropriate representation of the problem rather than from biased judgements. The source of concern at this stage may be deeper than the identification and resolution of inconsistencies. The concept of consistency, the foundation stone of decision analysis, may itself be a chimera. Demonstrating consistency for any complete formal system is known to be impossible; there must always be gaps in the demonstration (Godel's theorem). In a fascinating discussion of this issue, Hofstadter (1979) suggests that the gaps specified by Godel's theorem arise under conditions of "self-reference", that is, when a formal system somehow makes reference to itself. The formal system of decision analysis concerns the specification of beliefs and values. Since decision theory itself provides the rationale for evaluating any behavior, self-reference occurs whenever one attempts to evaluate the statements of value. An implication of Hofstadter's thesis is that such self-reference is bound to lead to paradox and confusion. One resolution of the paradox was provided by the Persians

HUMAN ENGINEERING OF DECISION AIDS

219

over two thousand years ago. According t o Herodotus, in Book One of 7he Histories,

’

“If an important decision is to be made, they discuss the questit n when they are drunk, and the following day the master of the house where the discussion was held submits their decision for reconsideration when they are sober. If they still approve it, it is adopted; if not, it is abandoned. Conversely, any decision they make when they are sober is reconsidered afterwards when they are drunk.”

In other words, invariance of recommendation under changes in context may be the safest guide to action. Humphreys and McFadden (1980) have suggested that the value of a decision analysis lies in the insight that it provides into the decision maker’s value system, rather than in specific recommendations for action. Their conclusion is consistent with observations of our own; following a decision analysis, individuals frequently make statements t o the effect that they have a better understanding of their problem, and that they see more clearly what to do. However, in a world characterized by uncertainty and conflict, it is difficult to argue that such an outcome is inevitably desirable. Have we provided the person with insight, or have we converted realistic confusion into dogmatic determination? If the latter, is such dogmatism necessarily undesirable? In other words, can we ever know if, or how, the decision aid is aiding anything? A formal analysis of a decision problem may be an internally coherent representation of the problem, but translating the analysis into action requires a leap of faith that cannot be justified within the system.

References Alluisi, E. A. and B. B. Morgan, Jr., 1976. Engineering psychology and human performance. Annual Review of Psychology, 27, 305-330. Anderson, J. R., 1976. Language, Memory, and Thought. Hillsdale, N.J.: Erlbaum. Atwood, M. E. and P. G. Polson, 1976. A process model for waterjug problems. Cognitive Psychology, 8, 191-216. 3obrow, D. G. and D. A. Norman, 1975. Some principles of memory schemata. In: D. G. Bobrow and A. M. Collins (eds.), Representation and Understanding: Studies in Cognitive Science. New York: Academic Press.

1 would like to thank Roger Garberg for bringing to my attention this early form of decision aiding

2 20

G.F. Pitz

Hransford, J. I),, J. R. Barclay, and J. S. Franks, 1972. Sentence memory: A constructive versus intcrpretivc approach. Cognitive Psychology, 3, 193-209. Chase, W. G. and H. A. Simon, 1973. The mind’s eye in chess. In: W. G. Chase (ed.), Visual Information Processing New York: Academic Press, Duncker, K., 1945. On problem solving. Translated by L. S. Lees from the 1935 original. PsychologicalMonographs, 58, (270). Einhorn, H. J. and R. M. Hogarth, 1978. Confidence in judgment: Persistence of the illusion of validity. Psychological Review, 85, 395-416. Einhorn, H. J. and R. M. Hogarth, 1981. Behavioral decision theory: Processes of judgment and choice. Annual Review of Psychology, 32, 53-88. Fischhoff, B., P. Slovlc, and S. Lichtenstein, 1980. Knowing what you want: Measuring labile values. In: T. S. Wallsten (ed.), Cognitive Processes in Choice and Decision Behavior. Hillsdale, N. J.: Erlbaum. Gettys, C. F. and S. D. Fisher, 1974. Hypothesis plausabllity and hypothesis generation Organizational Behavior and Human Performance, 24, 93-1 10. Graesser, A. C., S. P. Robertson, and G. A. Anderson, 1981. Incorporating inferences in narrative representations: A study of how and why. Cognitive Psychology, 13, 1-26. Hofstadter, D. R., 1979. Godel, Escher, Bach: An Eternal Golden Braid. New York: Basic Rooks. Hogarth, R. M., 1980. Judgement and Choice: The Psychology o f Decision. Chiches ter, England: Wiley & Sons. Humphreys, P. C. 1981. Decision aids: Aiding decisions. In: L. Sjoberg, T. Tyszka, and J. A. Wise (eds.), Decision Analysis and Decision Processes Lund: Doxa. Humphreys, P. C. and W. McFadden, 1980. Experiences with MAUD: Aiding decision structuring versus bootstrapping the decision maker. Acta Psychologica, 45, 5 1-69. Humphreys, P. C. and A. Wisudha, 1979. MAUD: An interactive computer program For the structuring, decomposition, and recomposition of preferences between multiattributed alternatives. Technical Report 79- 2. Uxbridge, Middlesex: Declsion Analysis Unit, Brunel University. Janis, I. L. and L. Mann, 1977. Decision Making, New York: Free Press. Jungermann, H., 1980a Speculations about decision-theoretic aids for personal decision making. Acta Psychologica, 45, 7- 34. Jungermann, H., 1980b. Structural modeling of decision problems. Paper presented at American Psychological Association, Montreal. Katz, M., 1973. Career decision making: A computer-based system of interactive guidance and information (SIGI). Proceedings of 1973 conference on testing problems: Measurement for Self-understanding and Personal Development. Princeton, N.J.: Educational Testing Service. Keeney, R. L. and H. Raiffa, 1976. Decisions with Multiple Objectives: Preferences and Value Tradeoffs New York: Wiley. Light, L. L. and L. Carter-Sobell, 1970. Effects of changes in semantic context on recognition memory. Journal of Verbal Learning and Verbal Behavior, 9, 1-11. Lopes, L. L. and P. H. Ekberg, 1980. Test of an ordering hypothesis in risky decision making. Acta Psychologica, 45, 161-167. Luchins, A. S., 1942. Mechanization in problem solving. Psychological Monographs, 54 (248).

HUMAN ENGINEERING OF DECISION AIDS

221

Minsky, M. A., 1975. A framework for representing knowledge. In: P. H. Winston (ed.), The Psychology of Computer Vision New York: McGraw-Hill. Newell, A,, 1973. Production systems: Models of control structures. In: W. G. Chase (ed.), Visuallnformation Processing. New York: Academic Press. Oskamp, S., 1965. Overconfidence in case-study judgements. Journal of Consulting Psychology, 29, 261 - 265. Pearl, J., A . Leal, and J . Saleh, 1980. GODDESS: Agoal-directed decision structuring system. UCLA-ENG-CSI 8034. School of Engineering and Applied Sciences, Ilniversity of Calitoi Ilia, Los Angeles. E’itz, G. F., 1980. The very guide of life: The use of probabilistic information for making decisions. In: T. S. Wallsten (ed.), Cognitive Processes in Choice and Decision Behavior. Hillsdale, N. J. : Erlbaum. Pitz, G. F., N. J. Sachs, and M. T. Brown, 1981. Eliciting a formal problem structure for individual decision analysis. Unpublished manuscript. Southern Illinois University. Carbondale. Pitz, G. F., N. J. Sachs, and J. Heerboth, 1980. Procedures for eliciting choices in the analysis of individual decisions. Organizational Behavior and Human Performance, 26, 396-408. Raiffa, H. 1968. Decision Analysis. Reading, Mass.: Addison-Wesley. Reitman, J. S. and M. R. Rueter, 1980. Organization revealed by recall orders and confirmed by pauses. Cognitive Psychology, 12, 554-581. Rips, L. J., E. J. Shoben, and E. E. Smith, 1973. Semantic distance and verification of semantic relations. Journal of Verbal Learning and Verbal Behavior, 12, 1- 20. Sachs, N. J., and C . F. Pitz, 1981. Choosing the best method of contraception: Application of decision analysis to contraceptive counseling and selection. Unpublished manuscript. Southern Illinois University, Carbondale. Schank, R. C. and R. P. Abelson, 1977. Scripts, Plan4 Goals, and Understanding. Hillsdale, N.J.: Erlbaurn. Schneider, W. and R. M. Shiffrin, 1977. Controlled and automatic human information processing: l. Detection, search and attention. Psychological Review, 84, 1- 66. Tversky, A. and D. Kahneman, 1981. The framing of decision and the psychology of choice. Science, 211, 453-458. Tversky, A. and S. Sattath, 1979. Preference trees. Psychological Review, 86, 542-573. von Winterfeldt, D., 1980. Structuring decision problems for decision analysis. Acta Psychologica, 45, 7 1 - 9 3. Weiss, J. J., 1980. CVAL and GENTREE: Two approaches t o problem structuring in decision aids. Technical Report 80- 3-97. Decisions and Designs Incorporated, McLean, VA. Wiks. Y.. 1977. What sort of taxonomy of causation do we need for language understanding? Cognitive Science, 1, 235- 264.

This Page Intentionally Left Blank

THE ROLE OF THE GOAL FOR GENERATING ACTIONS* Helmut JUNGERMdNN, Ingrid von ULARDT, and Lutz HAUSMANN Technical University, Berlin

Abstract In thls paper the initial phase of decision processes is conceptualized as the develop ment of a structural representation of relevant knowledge. Goals are viewed as playing an important role in representing decision problems when they have some specific content and are not purely formal (e. g,maximize SEU). A network model is proposed f a the representation of goals and actions, and several assumptions are made regarding the spread of activation through the network. In an experiment, hypotheses about the effects of two factors were investigated: Goal explidtness (E) was varied by presenting to Ss goal hierarchies of different speciflcity (one to three levels), and goal importance (R)was varied by letting Ss either rank-order goals with respect to their personal priorities, or not. The results show that the number of actions generated increases with the degree of goal explicitness, thus supporting the Ss creative search process, whereas the number of actions is lower for Ss who focus on their own values compared to Ss who do not, thus pointing to ego involvement as a factor restricting creativity. On the other hand, the actions generated by the personally involved group were rated higher on goal achievement scales than the actions generated by the other group. The results axe in accordance with the model which, however, needs elaboration

The Structural Representation of Decision Problems Any decision problem can be assumed to be structurally represented somehow. This representation may be more or less explicit, more or less precise, more or less aware to the subject, of course, but in any case it reflects the subject's perspective of the situation from which the decision

* We would like to thank Susanne Dlbbelt for her cooperation in the preparation of the experiment, and we are indebted to Robin Hogarth and Lennart Sjoberg for comments on a previous version of this paper. Requests for reprints should be sent to Helmut Jungermann, Institut f i i Psychologie, Technische Universitat Berlin, Doverstr. 1-5, D-1000 Berlin 10, West Germany.

224

H . Jungermann, I. von Ulardt,and L. Hausmann

process starts. A structure is a set of components of a complex whole and their interrelations; developing a structure, then, implies the generating of the components of the problem, and relating these components to each other. Both processes are closely intertwined and can be distinguished only analytically: The generating process is mostly guided by some implicit assumptions about the relations among the elements (e.g., their similarity or their mutual influences), and the structuring process often leads to a redefinition of the element set (e.g., adding or eliminating elements). In decision problems, the components may be possible actions, relevant events or states, potential outcomes, or goals and objectives; relations may be of a categorical or means-end sort (e.g., in goal hierarchies) or of a causal sort (e.g, in decision trees). A person’s representation of a problem, may it be ’real’ or ’experimental’, draws on two sources: (a) Knowledge already stored in memory, that is activated and retrieved in a particular situation; (b) Information that is searched for in the environment, and subsequently stored and integrated in permanent memory, Knowledge may be stored as relatively unconnected pieces of information that are structured only in a specific situation, or it might be stored in some already existing structure, e.g., as a schema or script. The mechanisms and strategies people use to generate and structure the components of a decision problem are not yet well understood. This is important, though, since it is this first initial phase of a decision process in which an ill-defined problem becomes well-defined, and the definition of a problem, of course, predetermines strongly the subsequent process. For instance, the selection of actions taken into consideration, or not, is certainly an important decision made before the usual decision process starts (i.e., the selection of one of the actions). This is particularly important when actions are not somehow given (e.g., sites for energy facilities), but must be designed or created (e.g., in urban planning), i.e., when imagination, phantasy, creativity are required. Whereas many authors have pointed out the importance of studying the process of representation and the factors that affect it (e.g., Vlek and Wagenaar, 1979; Jungermann, 1980; von Winterfeldt, 1980; Einhorn and Hogarth, 1981; Pitz, t h s volume), little empirical research exists on the issue. Exceptions are Gettys and Fisher (1979), who studied how people generate hypotheses about possible states of the world, or Pitz, Sachs, and Heerboth (1980), who investigated the generation of options and goals.

ROLE OF THE GOAL FOR GENERATING ACTIONS

225

The Goal Concept in Decision Theory Of particular importance for the problem formulation is the goal concept, but its function for representing decision problems has received little theoretical attention in decision theory. 'Goals' have been used in different meanings and functions: - Originally, the 'goal' of a decision maker was understood as nothing else than to 'maximize' sometiung. A goal in this sense does not have any particular content, it is a purely 'formal' goal, it is conrext-free. Its function is to serve as a criterion for selecting a course of action. - With the extension of the classic approach towards a 'multiple-objectives' approach, however, the goal concept got another meaning: goals stand for specific outcomes the decision maker tries to achieve (Keeney and Raiffa, 1976). A goal has a particular 'content'; it is contextbound- which has consequences for rhe understanding of 'rationality' and 'optimality' (cf. Einhorn and Hogarth, 1981; Pitz, this volume). Mostly so far, the function of this goal concept has been to generate and structure outcomes, i.e., to define attributes or dimensions on which potential outcomes may be evaluated. T h s is one of the goal(s) in representing decision problems. Another role might be considered, however, namely, to generate and structure actions, i.e., prior to all further steps (evaluation and selection) to design possible alternative courses of action. It is surprising that tlus interpretation has not been explored more systematically, since we can think of no other way of defining the set of actions than to consider the goal or goals to be achieved. The reason for this neglect might be that mostly decision-theoretic approaches, particularly the prescriptive ones, assume a well-defined problem in the sense that the options are given; the focus of these 'option-driven' approaches is on evaluation and selection. Only recently, in what may be called 'goal-driven' approaches, ill-defined problems have come under closer study, in the sense that the options are not given but must be created (e.g., Toda, 1978; Vlek and Wagenaar, 1979; Hogarth, 1980; Pearl, Leal,andSaleh, 1980; Pitz, Sachs, and Heerboth, 1980). Pearl, Leal and Saleh (1980) were apparently the first who, within a decision-theoretic framework, used the subject's goal(s) for generating actions, GODDESS is a computerized goal-directed decision structuring system for representing decision problems. The system allows the user to state relations among aspects, effects, conditions, and goals, in addition to actions and states which are the basic components of the traditional decision-theoretic approach. The system begins with assessing a structure of goals and subgoals and then elicits possible actions that would help produce improvements in each of the subgoals. 1s

226

H. Jungermann,l. von Ulardt,and L. Hausrnann

In an experimental study, Pitz, Sachs, and Heerboth (1980) investigated the effect of techniques on the generation of options and objectives, based on means-end analysis as used in the GODDESS system, and on script theory. Groups were given a decision problem and a list of objectives related to this problem, and were then asked to generate as many reasonable options as possible. Three groups, for example, were given the list of objectives and asked to generate options that satisfied the objectives one at a time, two at a time, or all simultaneously. Although the differences were small, there was a tendency for the goup focusing on one objective at a time only to produce more options than the group who tried to meet all objectives simultaneously. Our present interest is to study this role of goals for creating actions, i.e., in situations in which these are not given a priori. We want to know whether and how the explicit consideration of the goal, and various ways of thmking about the goal, affect the representation of the problem in terms of alternative actions. For instance, is the representation of a problem dependent on the framing of the goal (e.g, in terms of seeking positive or of avoiding negative consequences, of Tversky and Kahneman, 1981)? Under which conditions does it help and under which does it hinder people to create actions when they think about their goals(s) beforehand (Pitz, this volume)? What effect has the degree of detail with whch people think about their goals for the representation of a problem? Are there situations in which thinking about the goal results in the creation of actions which do not need to be evaluated any more, because a solution has ’emerged’ meanwhile (Beach and Wise 1980)?

A Cognitive Approach to Study the Role of the Goal

Our approach to study the role of the goal is based on conceptions of the representation of knowledge in human memory. Specifically, we made use of the idea that human memory can be modeled in terms of an associative network of concepts and schemata. The basic unit of thought is a proposition. The basic process is activation of the network’s nodes, i.e., concepts; activation presumably spreads from one to another by associative linkages (Anderson and Bower, 1973; Collins and Loftus, 1975). We assume generally that goals, actions, events, and outcomes are components of the memory structure; in the present study, however, we focus on goals and actions only. We do not distinguish here between goal-directed and causally-directed knowledge (Wilks, 1977), but rather treat goals and actions as elements of the same knowledge structure.

ROLE OF THE GOAL FOR GENERATING ACTIONS

221

A god is assumed to be represented as a node that is connected to many subordinate and superordinate goals by associative pointers. The relation between a subordinate and a superordinate goal can be interpreted as 'is meant by' or 'is implied by'. This assumption entails the idea of a cognitive goal hierarchy. Connected to goals are actions that, by reasoning or by experience, are related to that goal. The relation between an action and a goal can be read as 'helps achieve'. These actions may differ in their efficiency with respect to the goal. For simplicity we assume further that actions are directly connected to the more specific (lower level) goals and only indirectly connected to the more abstract (higher level) goals (see Figure 1). \ \ \

\

--

7

\

0 , , ,

, ,

Figure 1. Schematic Representation of a Cognitive Network Structure with Hierarchically Ordered Coals G and Actions A as components. (The numbers attached to action components indicate hypothetical efficiencies, and shaded actions indicate their belonging to a personal set.)

15*

228

H. Junprmann, 1. von Ulardt, and L. Hausmann

Activation of a goal node spreads throughout the nodes in the network to which it is connected, creating excitation at those nodes. Activation of a node might be interpreted as the attention the element is getting from the subject. The degree of excitation of an action node is dependent on the distance, i.e., the number of elements between the action node and the activated goal node; the greater the distance, the weaker the excitation. We postulate that, whether and how goal nodes are activated, influences which actions or sets of actions will be excited, retrieved, and generated. Specifically, the following assumptions are made with respect to the generation of actions: (1) When the activation of a goal concept is increased (e.g., by activating more of the goal nodes constituting the concept), then the excitation of action nodes connected to the goal concept will also be increased. If the chance of an action to be generated is a function of the excitation level of the node, which appears reasonable, the number of actions that can be generated increases with the degree of the general goal activation. For each individual, a goal concept has an impersonal (seman(2) tic) but also a personal (episodic) meaning which forms a part of the semantic meaning. When a subject considers a goal from his or her own perspective, i.e., what it means to him or her personally, only the elements of the personal goal concept, including the associated action nodes, will be activated. Consequently, the number of actions that can be generated is lower when goals are personally interpreted than when they are not. A further assumption refers to the quality of actions generated for the achievement of personal goals: Relating goals to personal preferences might also induce an unequal distribution of activation among the goals, i.a, higher-valued goals might be stronger activated than lower-valued goals. This implies, due to the spread-of-activation effect, that the action nodes connected with the higher-valued goals are also more excited than the action nodes connected with lower-valued goals. Since discrimination and search among a set of more excited actions should be easier than among a set of less excited actions, the quality of actions generated with respect to some goal is assumed to be a function of the value that goal has for the subject.

ROLE OF THE GOAL FOR GENERATING ACTIONS

229

The Experiment Material and Variables We chose an issue that was presumably familiar to all subjects: The goal was "I want to have a nice vacation", and the actions to be generated and evaluated were the possible means to achieve this goal. Two independent variables were manipulated in a 3 x 2 design: The first factor (E) was the goal explicitness, i.e., the degree of elaboration of the goal; there were three levels of E: In E l , Ss were only given the general goal, i.e., "I want to have a nice vacation". In Ez,Ss were presented a two-level hlerarchy in which the elements on the second level represented aspects of the top-level goal. In &, Ss were presented a three-level hierarchy with elements on the third level representing aspects of the elemensts on the second level. The goal hierarchy had been developed beforehand with the help of 20 student Ss, using a sorting technique. A part of the three-level hierarchy is shown in Figure 2. The second factor (R) was the way in which Ss had to think about the goals; there were two levels: In condition R,,Ss had to rank-order the goals that were presented to them, according to their personal priorities. In Rz,Ss were requested t o think about the goals in a general way, i.e., as goals people commonly have for their vacation. Procedure In the first step, Ss were presented the goal, or goal hierarchy, and requested to think about it for a while. To ensure that they actually considered each element, they had to give a specific example for each. Depending on the experimental condition, they then had to rank-order the goals, or not. In the second step, Ss were asked to generate actions that would help to achieve the goal. They had to put together packages consisting of 8 elementary actions, one from each of 8 categories (location, transport, accommodation, company activity, transport at destination, food, all other). Ss were given a booklet, and used for each package a new sheet. The first 'vacation package' was to represent their personal, ideal combination; they had then to form as many other packages as possible that might be attrao tive to other people. The number of different elementary actions generated was the first dependent variable. In the third step, all Ss were shown the complete hierarchy, had to rank-order the goals (if they had not done this before) and then evaluated their own ideal package on a 0 to 100 scale as to which degree it met each second-level goal. The rating on the respeo tive scales was the second dependent variable.

h, w

0

I want to have a nice vacation i. e.,i want

Recovering physically

activity

H . Jungermann, I. von Ulardt, and L.Hausmann

Improving

some sports

Figure 2. Fragment of the Whole Three-Level Goal Hierarchy

hobbies

education

sightseeing

H . Jungermann, I. von Ulardt, and L.Hausmann

Q

To engage in

To recover

ROLE OF THE GOAL FOR GENERATING ACTIONS

231

Hypotheses The hypotheses can be stated as follows: H 1 (General Activation Hypothesis): With increasing goal explicitness, as operationalized through factor E, an increasing number of actions will be generated. H 2 (Set Activation Hypothesis): Ranking of goals with respect to personal importance, as operationalized through factor R, will reduce the number of actions generated. H 3 (Evaluation Hypothesis): The personal package of actions will have higher values on the goal achievement scales for the group who rank-ordered the goals with respect to personal importance before generating the actions than for the group who did not.

Subjects Ss were 130 students, nurses, secretaries, and post office workers. The number of Ss was not the same in all groups: In group El /Rl , N=21; in group E2/RI, N=19; in group E3/R1, N=17; in group EI/Rz, N=34; in group Ez/R2,N= 19; in group E3/Rz, N=20.

Results For each S, three scores were determined: The P-Score represents the number of packages the S has generated; all packages that were either incomplete (i.e., one or more categories left out) or impossible under the given contingencies (e.g., three weeks vacation) were not counted. The A-Score represents the number of generated actions, the variable of main interest in our study; only semantically or functionally different elements were counted (e.g., a VW and a Ford were not counted as two different means of transportation). The E-scores represent the evaluations of the personal package with respect to the goal achievement scales. Our main two hypotheses concerned the number of actions generated, depending on goal explicitness (factor E) and goal importance (factor R). The A-scores, that are of primary interest here, are given in Table 1. Goal explicitness (factor E). The General Activation Hypothesis (H 1) said that Ss who think more explicitly about the goal will produce more possible actions. The data support this assumption: The number of elementary actions (A-score) was significantly different among the three

H. Jungermann, I. von Ulardt ,and L. Hausmann

232

Table 1. Mean Number of Generated Actions (A-score)

I Importance of Goal

EI

E2

I

E3

ii

Rl

29.90

2758

32.18

29.8 1

R2

28.21

38.1 1

42.55

34.71

x

28.85

32.84

37.78

~

groups El, E2, and E3 (Kruskal-Wallis test: H = 16.15; df = 2; p < 0.001). In particular, group E3 generated more actions (A = 37.78) than group Ez (A = 32.84), and group E2 more than group El (A = 28.85). The main effect of E was also found when P-scores were analysed: The mean number of packages ('P) produced by E l , E2, and E3 were 5.36, 7.42, and 8.08, respectively. These differences are highly significant (Krwkal-Wallis H = 21.43; df = 2; p < 0.0001). Goal importance (factor R). In our Set Activation Hypothesis (H 2) it was assumed that Ss who rank the goals with respect to personal importance will generate fewer actions than Ss who do not. The data provide evidence for this hypothesis also: Group R1 produced significantly less = 29.81) than group R2 (i = 34.71) (Kruskalelementary actions (i Wallis H = 6.16; df = 1; p < 0.02). The analysis of the P-sere shows the same effect. For group R I , 5 = 6.37, and for group Rz,P = 7.03 (Kruskal-Wallis H = 7.75; df = 2; p < 0.01). Some further evidence on E and R effects. Since an analysis of variance could not be applied to explore possible interactions, separate analyses with the Kruskal-Wallistest were performed of the effects of E on the two levels of R, and of the effect of R on the three levels of E. The analysis of E effects shows that only under condition R2 are the differences between E l , Ez, and E3 significant (p < O.OOOl), but not under condition R 1 in which Ss ranked the goals with respect to personal importance. The analysis of R effects shows that only the differences on levels Ez and E3 are significant (for both, p < 0.01), but not the difference on level El . Evalwtion. The Evaluation Hypothesis (H 3) postulated that Ss who relate the goals to themselves will produce better personal packages in t e r n of potential goal achievement than Ss who do not. Table 2 shows the mean evaluations (E-scores) of groups RI and R2 with respect to each of the 8 second-level goals. As predicted, the values for group R1 are

ROLE OF THE GOAL FOR GENERATING ACTIONS

233

1

2

3

4

5

6

7

8

R1

88.8

86.9

85.8

19.3

81.1

70.1

75.3

66.6 N = 5 1

RZ

89.6

84.1

18.8

18.0

73.6

76.1

66.0

58.6 N = 19

significantly higher than the values for group Rz (Kolmogorov-Smirnov test: Chi' = 11.91; p < O.Ol).However,ifonesplitsthe datain evaluations for the four most important goals and the four least important goals, the groups differ significantly only on the latter scales.

Discussion The results support the hypotheses and are basically in accordance with our theoretical approach based on a network model of the representation of knowledge. The main effects of goal explicitness (factor E) and goal importance (factor R) were evident in our data. The finding that factor E is not effective when Ss rank the goals may be seen as an indicator for the strong effect of R irrespective of the other factor: Relating goals to personal values and thus interpreting them from a personal perspective seems to level the effect of goal explicitness. The finding that factor R is not effective on level El of the goal is relatively easy to understand: Under condition El, only one very general goal is presented to Ss, namely, the top element of the goal hierarchy. An instruction to relate to the self, as given in R1,might actually be expected not to have any particular effect, although, of course, some Ss might implicitly rank aspects of this goal with respect to their importance. However, the lack of any difference between groups El/Rl and El/Rz seems more plausible. Finally, we have a very tentative explanation for the finding that evaluations differ only on the less important goal achievement scales. Ss in group R2,when asked to generate their personal ideal package, had only a very brief time to reflect about their own goals, and they might have directed their attention therefore primarily t o the most important goals while neglecting the less important ones. Consequently, they could select relatively efficient actions with respect to the more important goals, thus

234

H . Jungerrnann, I . von Ulardt,and L. Hausrnann

not differing from Ss working under condition R1,but produce only less efficient actions with respect to the other goals. Our results demonstrate the importance of goals for the process of generating options. On the one hand, the consideration of goals can support the creative activity of subjects, namely, when goals are made explicit and specific. On the other hand, goals may limit subjects’ search for options when the perspective is strongly self-oriented. In both cases, however, goals direct the attention and effort subjects spend on generating actions for goal achievement; the effect differs only depending on the way goals are introduced and framed. Thus result may be linked to other psychological research. For instance, in problem-solving psychology it has always been said that one needs to define the goal status in order to be able to generate the operators that help transform the status quo into the desired status, and various theoretical and empirical approaches have studied problemsolving processes using this conception (e.g., Newel1 and Simon, 1972; Dorner, 1976). Effects of set and direction on creative problem-solving has clearly been demonstrated in many studies (e.g., Hyman, 1961), and characteristics of people’s handling of goals in complex decision situations (like running a town) have recently been investigated by Dorner (in press). Another area is the research on goal setting and task performance. A recent review (Locke, Shaw, Saari, and Latham, 1981) concluded that the “beneficial effect of goal setting on task performance is one of the most robust and replicable findings” (p. 145). One particular finding is that specific goals lead to higher output than vague goals such as ”do your best”. This, of course, sounds similar to the contrast between concrete, substantial goals and abstract, formal goals like ”maximize your SEU”, as discussed above in Section 11. Of course, the model proposed in this paper is very simple. However, the use of models of long-term memory for research on decision processes is only beginning. More specific models of the acquisition, storage and retrieval of decision-relevant knowledge must be developcd and tested. Questions worth to be studied would include, for instance: Is the knowledge that is retrieved in decision situations actually structured in terms of actions, events, and outcomes, as decision theory assumes? How are goals connected to these elements? Which factors influence the retrieval process in which ways? Can the effects of heuristics suchas availability or representativeness (Tversky and Kahneman, 1975) be explained with the help of models of the representation of knowledge? One problem for such research is that it seems difficult to design experiments and to develop measurement procedures for eliciting subjects’ r e p resentation of decision problems or for studying factors that affect this

ROLE OF THE GOAL FOR GENERATING ACTIONS

235

representation. New experimental paradigms are needed that parallel those in cognitive psychology, e.g., recognition and recall. Finally, what are the implications for the application of decision theory? In decision analyses, clients are required to retrieve, and then externalize knowledge in terms of the prescriptive conception (i.e., actions, events, and outcomes), and this seems usually to work, at least it does not seem to be counterintuitive. However, this approach might not be sufficient when the representation of the problem is itself an important step of the decision process, as many decision analysts claim it is most of the time. But it is exactly this step where attention and memory play an important role. Our conclusion is that, if we want to improve our understanding of people’s decision behavior, and also if we want to improve our ability to aid people making decisions, we will have to study how this process of representing problems is performed and what techniques we need to develop for adequately eliciting people’s knowledge about the problem. If the representation of a problem depends on which knowledge is activated and retrieved, the procedures used for eliciting this knowledge and their effects on cuing attention and activation are as important as the techniques for eliciting utilities and subjective probabilities.

References Anderson, J.R. and C.H. Bower, 1973. Human Associative Memory. Washington, D.C.: V.H. Winston & Sons. Beach, L.R and J.A. Wise, 1980. Decision emergence: A Lewinian perspective. Acta Psychologica, 45, 343-356. Colhs, A.M. and E.F. Loftus, 1975. A spreading-activation theory of semantic memory. Psychological Review, 82, 407-428. Dorner, D., 1976. Problemlosen als Informationsverarbeitung. Stuttgart: W. Kohlhammer. Dorner, D. Heuristics and cognition in complex systems. In: R. Groner, M. Groner, and W.F. Blschof (eds.), Methods of Heuristics Hillsdale, N.J.: Lawrence Erlbaum (in press). Einhorn. H.J. and R.M. Hogarth, 1981. Behavioral decfsion theory: Rocesses of judgement and choice. Annual Review of Psychology, 32, 53-88. Cettys, C.F. and S.D. Fisher, 1979. Hypothesis plausabillty and hypothesis generation. Organizational Behavior and Human Performance, 24, 9 3 - 100. Hogarth, R.M, 1980. Judgment and Choice: The Psychology of Dedsion. Chichester, England: Wiley & Sons. Hyman, R, 1969. On prior Information and creativity. Psychological Reports, 9, 151- 161.

Jungermann, H., 1980. Structural modeling of decision problems. Paper presented at American Psychological Association, Montreal.

236

H. Jungermann, I. von Uiardt, and L. Hausmann

Keeney, R.L and H. Raiffa, 1976. Decisions with Multiple Objectives: Preferences and Value lladeoffs. New York: Wlley. Lmke, E.A., K.N. Shaw, L.M. S d , and G.P. Latham, 1981: Goal setting and task performance: 1969- 1980. Psychological Bulletin, 90, 125- 152. Newell, A. and H.A. Simon, 1972. Human Problem Solving Englewood Cliffs, N.J.: Rentice-Hall. Pearl, J., A. Leal, and J. Saieh, 1980. GODDESS: Agoal-directed decision structuring system UCLA-ENG-CSL-8034. School of Engineering and Applied Sciences, University of California, Los Angeies. Pitz, G.F. Human englneering of decision alds. In this volume, 205 -221. Pit& G.F., N.J. Sachs, and J. Heerboth, 1980. Procedures for eliciting choices In the analysts of Individual decislons. Organizational Behavior and Human Performance, 26, 396-408. Toda, M., 1976. The dedslon process: A perspective. International Journal of General Systems, 3, 79-88. Tversky, A. and D. Kahneman, 1976. Judgment under uncertainty: Heuristics and blases. In: D. Wendt and C. Vlek (eds.), Utility Probability, and Human DecC sion Making DordrechtlBoston: D. Reidel. Tversky, A. and D. Kahneman, 1981. The framing of dedslons and the psychology of choice. Science, 2 1 I , 4 5 3- 458. Vlek, Ck and W.A. Wagenaar, 1979. Judgment and declsion under uncertalnty. In: J.A. Michon, E.G. Eijkman, and L.F.W. DeKlerk (eds.), Handbook ofPsychonomics, 11. Amsterdam: North-Holland, 253-345. von Winterfeidt, D., 1980. Structuring decision problems for dedsion analysis. Acta Psychologica, 45, 11-93. WiIks, Y., 1977. What sort of taxonomy of causation do we need for language understandlng? Cognitive Science, 1, 235 -264.

FUZZY STRUCTURAL MODELLING-AN AID FOR DECISION MAKING Dimiter S. DRIANKOV and Ivan STANTCHEV Higher Institute of Economics "K. Marx" Sofia, Bulgaria*

Introduction An issue which has received little attention in decision theory is the role of

the goal for representing decision problems. Our hypothesis is that in situations in which the alternatives and their consequences are not a p r w n given, goals play a major role in the process of,generation andevaluation of the alternatives and their consequences. In the present paper goals are considered as complex wholes which are described in terms of constituent parts and interrelations among these parts. In the field of complex humanistic systems practical experience shows that it is quite impossible to assess in precise terms all the quantitative and/or qualitative data necessary for elaborating a relevant representation of the goals. That is why we employ here a linguistic approach ior describing the Constituent part of the goals, the values they take, and the interrelations among them. On the other hand, the structural modelling approach offers a relatively simple procedure appropriate to the imprecise and/or ill-defined interrelations among the constituent parts of the goal. It is to be stressed here that the use of fuzzy conditional propositions helps us t o retain the nonlinear character of- these interrelations, whde the classical structural modelling approach simply linearizes them. It should also be stressed here that whde in the classical structural modelling approach the strengths of the influences among the constituent parts are assessed subjectively, this is not so in our approach: the assessment of the strength of influence among the constituent parts of the goal is made on the basis of objectively existing relationslups among the constituent parts.

* Authors' address: Department of Systems Analysis, Higher Institute of Economics "K. Marx", Ekxrh losif 14 St.. Sofia. Bulgaria.

238

D.Driankov and I . Stantchev

Statement of the Problem In the process of decision making, once the Goal System (GS) has been constructed, an acute problem arises. This concerns the possible relationships of influence among the so-called Goal Indicators (GI), the strength of these relationships and the mechanism and the type of interaction among the goal indicators. According to Griiber and Strobe1 (1977), the goal indicators are placed at the lowest level of a tree-like structure which describes the goal system. They represent measurable system parameters which describe the behaviour of the system within which the decision maker takes and executes his decisions. Thus, the goal indicators and certain values taken by them, which we shall call ”aspiration values,” express in measurable terms the meaning of the GS in the context of a certain real life system. The “aspiration values” of the goal indicators give the decision maker the conditions under which he measures the attainment of his goals. In general, there is more than one GI characterizing a certain goal. On the other hand, these goal indicators may describe some other goals. Thus, if there exist relationships of influence among these goal indicators, and some of them have been influenced by a particular decision, this eventually may change the degree of attainment not only of that particular goal but of the whole GS. Thus there is a necessity for a descrip tion of the GI’s structure in terms of: - relationshps of influence among the goal indicators; - the strength and the type of these relationships; - the mechanism of interaction. The necessity for this description arises in order to help the decision maker to determine the full range of consequences of a particular decision with respect t o the attainment of the whole GS.

Constructing the Structure of the Goal Indicators After the goal indicators expressing the meaning of the GS in measurable terms have been assessed, the construction of their structure follows. In our approach, this is done through a man-machine procedure consisting of the following stages:

Stage 1: Identification of the relationships of influence between each pair of goal indicators

239

FUZZY STRUCTURAL MODELLING

Definition I : The GI, Xi is said to be fuzzy with respect to the decision maker if he describes the values taken by this GI in terms of fuzzy sets, i.e.,

T(X')

=

{LK(Xi)

= xik ]

k

="t

where

X' is the name of the i-th GI. X' is a linguistic variable (Zadeh, 1975) with a universe of discourse U' = {u'} and (Xi)is one of its linguistic valries assigned to it by the decision maker in order to behaviour of Xi in measurablc terms. T(X') determ:nes the so-called term-set of Xi; xk is a fuzzy set expressing the meaning of the linguistic value Lk(Xi). I t is defined on the universe of discourse U' and is characterized by a membership function pcC&(u') such that pi( (u'): u' + [0,1]. Notion 1: Informally, a linguistic variable, Xi, is a variable whose values are words or sentences in a natural or artificial language. For example, if "salary" is interpreted as a linguistic variable with termset T (salary), the set of linguistic values Li(, k = 1,2 ..... might be:

Ti (salary) = {$ (salary)] k = 1, 2 , . = {high, low, above low, approximately high, between low and medium, very high, . . .] where each of the terms in T (salary) is a label of a fuzzy set of an universe of discourse, say U' ={$7,000 $40,000). More generally, U may be an arbitrary collection of objects or constructs of objects. Then a finite fuzzy set, x, of U = u is expressed as: ,

-

x = pi/ u i + ~ 2 1 + ~ 2 . + Pnlun

(2)

where the pi, i = 1,2, . . . n , represent grades of membershp (compatibility) of the ui in x. More generally, a finite fuzzy set x of U = { u ) is expressed as: x

=

PX(U)/U

(3 )

where, px(u): U + [0,1] is the membership function (compatibility function) of x. Thus, the linguistic value, say Lk(Xi) = high salary may be expresse d by a fuzzy set as:

D. Driankov and I. Stantchev

240

.I,= 1/$40,000 + 0.8/$32,000 + 0.7/$30,000 + t 0.5/$21,000 + 0.2/$12,000 Definition 2: A fuzzy relationship of influence of Xi on XJ exists for the decision maker if he is able to construct a fuzzy conditional proposition expressing his cognition that certain values of Xi induce certain values of xJ,i.e., IF Ln (X') THEN L, (Xj)

(4)

Formally,(4) is expressed by a fuzzy relation R, where R is a fuzzy set defined on the Cartesian product U' x Uj and is characterized by a membership function defined in the following way:

Example 1: Let

+

+ +

U i = Uj = 1 2 + 3 f 4 5 6 + 7 L, (Xi) = small integer = 1/1 + 0.8/2 + 0.613 + 0.4/4 + 0.2/5 L,,, (XJ)= medium integer = 0.4/2 + 0.813 + 114 + 0.8/5 + 0.2/6 Then a fuzzy conditional proposition, say IF Xi is small THEN Xj is medium translates into fuzzy relation R, characterized by a membershp function p~ obtained according to ( 5 ) :

1 PR

=

2 3

2

3

0.4 0.4

0.8 1

4 1 1

5 6 0.8 0.2 1 0.2

4 5 Using the apparatus described above, the decision maker can construct all the fuzzy conditional propositions describing fuzzy relationships of influence between Xiand Xi. To do this he proceeds as follows: Step I: The decision maker defines linguistically all the expressions of type (4).

FUZZY STRUCTURAL MODELLING

24 1

Step 2: The decision maker, using a man-machine procedure, constructs the fuzzy sets representing the meaning of the linguistic values taken by Xi and XI. Step 3: Using (5), for each expression of type (4) its R is built. Thus, at the end of stage 1 , the decision maker has obtained both the linguistic and the formal representation of all the relationships of influence between each pair of goal indicators.

Stage 2. Determining the degree of influence between each pair of goal indicators

The degree of influence of X' on Xj is determined according to the follow ing procedure : Step I: Given p~ and ph(u') it is possible t o obtain gm(u-) by the use of a fuzzy inference rule and the so-called "composition" operation. T h s is always possible if p~ has been constructed according t o (5). This is done as follows:

where "0" denotes the "composition" operation given by the expression

in the right hand side of (6); "V"denotes "max"; "A" denotes "mii?.

Notion 2: We often make an inference such that from the relationship between some objects Xi and XJ and the information about Xi, we deduce some information about X' as a consequence. For this, Zadeh (1975) suggested an inference rule named the "compositional rule of inference". In our case we consider an inference rule of the form:

Antecedent 1: IF Xi is L ( X 1 )THEN Xj is Lm(XJ) Antecedent 2: Xi is L ( X i ) Consequence: XJ is ~ ( x J ) If we translate the antecedent of the inference which represents the relationship between the objects Xi and (Antecedent l), into a suitable fuzzy relation R, given by p R , and the antecedent, which represents information about Xi (Antecedent 2), into a suitable fuzzy set x i , charac16

D. Driankov and 1. Stantchev

242

terized by pf (ui), then the consequence can be obtained by the "composition" of p i (u') and p K , as given in (6). This may be expressed as follows:

Xi = L(XL) and XJ = L ( X J ) are R = p~ xi = h ( X i ) = ph(u') Xj = pL(ui) 0 p~ =

&m(d)

which is L,,,(XJ)

Example 2: Let

U i = U l = 1 + 2 + 3 + 4 + 5 + 6 -I- 7 Ln(Xi) = very small integer 1.011 + 0.6/2 + 0.4/3 + 0.114 h ( X J ) = big integer = 0.2/4 + 0.4/5 + 0.8/6 + 1.017 Applying ( 5 ) , we translate the antecedent of the inference, say:

If Xi is very small THEN X' is big into a fuzzy relation R: 1 2

4

4 0.2

5 0.4

0.2

0.4

6 0.8 1

7 1

1

The antecedent which represents information about Xi = h ( X i ) = very small, is translated into a fuzzy set as: x i = 1.011

Xi, i.e.,

+ 0.612 t 0.4/3 + 0.1/4

which is characterized by a membership function pl(ui) taking values: (1,0.6,0.4,0.1). Then the consequence Xj = b ( X j ) is obtained by the composition of g',(ui) and C(R :

r0.2 (1,0.6,0.4,0.1) 0

0.4

0.8

;1

FUZZY STRUCTURAL MODELLING

24 3

(max (min(l,0.2), min(0.6,0.2), min(0.4,0.2), min(O.l,l)), max (min(l,O.4), min(0.6,0.4), min(0.4,1), min(O.l,l)), max (min( 1,0.8), min (0.6,1), min(OA,l), min(0.1, l)), max (min( 1, l), min(0.6, l), min(OA,l), min(0.1,l))) = (max (0.2,0.2,0.2,0.1 j, max (0.4,0.4,0.4,0.1), max (0.8,0.6,0.4,0.1), max (1,0.6,0.4,0.1)) = (0.2,0.4,0.8,1) Thus we have obtained the values of the membership function drn(.') which expresses the meaning of h ( X J ) big. It is easy to see that if we consider the values taken by ph(u') as unknown, then (6) defines a system of fuzzy equations, as defined by Driankov and Petrov (1977). This can help the decision maker to obtain all values of Xi which give one and the same value of Xj, expressed by bm(d).Th'is means that he/she can find out those values of Xi which do not change a certain value taken by XJ. If p~ is constructed according to (5) then (6) has a solution; i.e., there exist some values of Xi represented by the set of memberslup functions, [p:(u*)} which also includes p:(ui) and such that:

It has been proved that lower and upper bounds o f the set of the solutions of (6) exist. These are denoted by p i m i n (u') and p\max(ui) respectively . Applying the so-called "linguistic approximation procedure"(Zadeh, 1975) to the elements of the set of the solutions, we can obtain the linguistic value corresponding to each solution.

Example 3: Following the previous step, we deal with the inverse problem of (6): Given a fuzzy relation R , defined on the Cartesian product U' x and characterized by a membership function p R , o b t a i n e d according to ( 5 ) find all p:(ui), such that

u,

p:(ui) @ C(R = drn(d), vt

Let

I 1

0.4 0.0 0.9 0.6 0.8 0.7 0.8 0.3 1.0 0.5 IrR = 0.6 0.4 0.3 0.4 0.9 0.2 1.0 0.5 0.8 0.4

pJrn(d) = (0.6,0.5,0.9,0.6,0.8) 16*

D. Driankov and I. Stantchev

244

We look for p:(u*) such that p:(u') @ PR = pfn(uj)~v t and these are: (0.9, 0.5,0.6, O.O),Lm, ( x J ) > L ~( X , J)>L,

( x j ) > L m 3 (xJ)

we say that Xi exerts a negative influence on Xj for these particular values taken by Xi.

If

'

(xi) 2 Lk6 (xi)2

L,asp. h

6

Lk5 ( X i ) a n d ~ s p . ( X J ) 2 L r n(XJ) ,

(xJ)

we say that Xi exerts positive influence on XJ for these particular values of

xi.

Step 1: Determ'ning the type of influence for each two nodes of the structure. Using a simple search procedure for finding increasing or decreasing sequences in a list of fuzzy conditional propositions, determined by a particular degree of influence, we mark the links in the structure thus determining the type of influence between each pair of nodes of the structure. This will help the decision maker to,see the effect which Xi has on Xj with respect to the aspiration value of XJ for particular values taken by X'. Step 2: Determining the type of the circuits. The following types of cyclic relations may exist among the elements of a given circuit: ( I ) "Snowball" type: all links in a certain circuit are marked with +1 sign. The effect which this type of circuit causes is as follows: Suppose we have a circuit consisting of three nodes, like that shown in Figure 2.

D. Driankov and 1. Stantchev

250

XJ

Figure 2. The "Snowball"Type of a Circuit

If the value of Xi is increased this leads (according to Definition 4) to an increase of the value ofXJ,and due to that, the value of Xk will also increase. This leads again to an increase of the value of Xi, etc. So, we can see that if we increase the value of an arbitrary element of the circuit this leads to a "snowball" like increase of the values of all elements of that circuit. It is easy to show that if we decrease the value of a n arbitrary element of the circuit this will cause a "snowball" like decrease of the values of all elements of the circuit. q

(2)

"Colhpse" type: This type of circuit exists if fl 6k > 0, where k=l 6, e {- 1 , -I-1) . An example is given in Figure 3. The effect which this type of circuit causes is as follows:

x'

X' Figure 3. The "Collapse" Type of a Circult

FUZZY STRUCTURAL MODELLING

25 1

& increase of the value of X' leads to an increase of the value of XJ. According to Definition 4, the value of Xk vill decrease, which

leads to an increase of the value of X' . Since the influence of X' on Xi is positive, the value of Xi increases again and thus the value of XJ also increases, thus decreasing the value of Xk for the second time. So, we can see that the values of Xi, XJ and X' increase permanently while the value of Xk decreases permanently. If we start with a decrease of the value of Xi it is easy t o see that the effect is a permanent increase of the value of Xk and at the same time a permanent decrease of the values of Xi: Xj and X I . ( 3 ) "Resistance-to-change" type: This type of a circuit exists if 9

n 6 k < 0,where G k E {-I,

+I} . An example is given in Figure 4.

k=l

The effect of this type of circuit is as follows: XJ

+I Hgure 4. "Resistance-toChange" Type of a Circuit

An increase of the value of Xi leads to an increase of the value of Xj. As a result, the value of Xk decreases. Since there is a positive influence of Xk on Xi the value of Xi will decrease. Then the value of XJ,also decreases. But due to the negative type of influence of XJ on Xk this leads to an increase of the value of Xk,etc. Thus, we can see that in this case the values of the goal indicators pulsate around certain initial values. The effect will be the same if we start with a decrease of the value of Xi. It is easy to see that if the structure contains circuits of the types described above, this may cause problems with respect to the attainment of the aspiration values of the goal indicators. One way out of this is to rebuild the structure in such a way that in the new structure there will be no circuits. Two alternative possibilities exist in this direction. The first

D. Driankov and I. Stantchev

25 2

one is to change Xi in such a way that the direct influence it exerts on XJis to increase, ie., to increase the value of a,]. The second one is to determine some significant value for the degree of influence such that for each k holds: if a: Q y then

a,] > y or the inverse.

Conclusion The structure of GIs thus built up is used by the decision maker in the following manner: - the decision maker measures the attainment of the goals in terms of certain goal indicators taking into consideration their aspiration levels and their actual values resulting from a particular decision; - he or she inserts these actual values into the GI’s structure and relates them to each other, thus causing changes to some other goal indicators. Thus, situations and problems are perceived and identified, alternatives are sought and tested. Finally, the experiences following the decision are stored and inserted in the GI’s structure. References Dimitrov. V. and D. Driankov, 1977. Pointwise prediction based on fuzzy sets. Pro. reedings o f the International Conference on Information Sciences and Systems, Patras. Greece, 19 76. Hemisphere Publishing Corporation. Driankov, I). and A. Petrov, 1977. Fuzzy equations with appllcations. Proceedings of the 4th Congress of the Balkan Mathematical Union, Varna, Bulgarh, 1975. Bulgarian Academy of Sciences. Zadeh, L.A., 1975. The concept of a llnguistic variable and its appllcatlon to approximate reasoning, Parts 1, 2, 3. Information Sciences, 8, 9.

USE OF PROBLEM STRUCTURING TECHNIQUES FOR 0PTI ON GENERATION : A COMPUTER CHOICE CASE STUDY Patrick HUMPHREYS London School of Economics and Political Science, England

Abstract Structuring techniques developed so far for use in decision analyses have concentrated on structuring uncertainty or the decompostion of worth of alternatives while the specification of the options to be considered within the analysis has usually been considered as something which happens a priori to the decision analysis, which does not comprise activities specifically addressed to this problem. This case study of a decision analysis dealing with the choice of a set of computers to meet the overall requirements of a university psychology department demonstrates structuring techniques useful in providing a representation of a decision problem at the stage prior to the development of specific options. It shows how this representation can be used to generate alternative options, each of which could provide adequate solutions to the problem. The evaluation of individuiu elements within each option is discussed, together with subsequent rvaluation of complete sets of options, where properties of the configuration of elements within each option may be taken into consideration. In each case a computer-based interactive decision aid, MAUD4, was used in forming preferences on the basis of these evaluations. The advantages and limitations of the option generating methodology are discussed, together with a discussion of how one can use these techniques to assess the flexihilify of a potential option set in terms of the provision of a method for judging its responsiveness both (i) to future changes in requirements under the assumption that there is in the present total uncertainty about what such changes might be PO thcy cannot be structured and (ii) to future changes in states of the world about which there is currently too much uncertainty to permit explicit act-event modelling.

25 4

P. Humphreys

Introduction: Problems of Option Selection in Purchasing Decisions

This paper is based on a case study of a deckion analysis concerning the optimal set of computers to buy to completely re-equip a university psychology department.' Its principal aim, however, is to examine a method capable of aiding the decision making process at a n earlier stage than that typically addressed by decision theoretic analyses: structuring the options which will subsequently be evaluated and chosen between. In text-book analyses any need for structuring rarely appears at this stage. Options somehow have already arrived on the scene; indeed it is often the awareness of a set of options being present which triggers the perceived need for a decision analysis. Emphasis is laid on structuring the uncertainty surrounding the links between the immediate acts involved in selecting an option and the ensuing consequences (e.g., Howard and Matheson, 1980), or in structuring one's objectives in a way that allows one to form preferences between options, either by clarifying goal relationships (Pearl, Leal, and Saleh, 1980) or by identifying relationships between objectives and preferred levels on attributes characterizing the various options (Keeney and Raiffa, 1976, Section 2). However, there are many decisions relating t o the purchasing of goods within a commercial environment where a large part of the problem stems from the fact that at the outset there is considerable uncertainty what should comprise the set of options to be evaluated. Methods traditionally recommended for clarifying the specification of options t o be considered through exploring goals or initiating schemata (Pitz, Sachs, and Heerboth, 1981) d o not always help, since they lead to the specification of yet more options, whereas the principal problem here is that the potential range of options is too wide for the decision maker to be able to give proper consideration to the tradeoffs involved in making choices between them. Svenson (1 983) and Montgomery (1 983) discuss how, when faced with decisions of this type, people will try to find screening (non-compensatory) methods to remove less favourable options from consideration To preserve the realism of the material presented here, no attempt has been made to disguise the identity of the decision maker's institution, or the computers considered. It should be remembered that all the characterizatbns, evaluations and pre ferences between computers identlfied here are subjective and made relative to the specific requirements of particular users. Hence information presented here should not be taken as generally applicable in assessing the various computers. For this task, it is recommended that a new decision analysis be carried out using requirements and criteria speclflc to the new application.

USE OF PROBLEM STRUCTURING TECHNIQUES

25 5

before engaging in the more difficult task of making tradeoffs between the remaining smaller set of options which have emerged as serious contenders for choice under the decision maker’s various (and possibly conflicting) objectives. In the decision situation discussed in this paper the option-selection problem is particularly acute: the decision maker was the computer engineer for the Department of Psychology at Brunel University which has about 100 undergraduate students, 12 teaching staff and some 20 active research staff and students. The computer engineer (henceforth called the decision maker to reflect her role in this analysis) was charged with t w o objectives: (i) to replace an obsolete computer serving the department with computing facilities which would meet the various (and as yet ill-defined) requirements of users within the department, (ii) to d o so on a budget of €27,000. The second objective serves as a constraint on the development of options: all options need to comprise configurations of computers costing no more than C27,OOO. Beyond this, it does not require further elaboration. The first objective is more problematic. Figure 1 shows the decision maker’s ’first-cut’ attempt a t achieving this objective through breaking it down into a hierarchy of activities requiring computer facilities, with the pattern of distinctions within the hierarchy reflecting the formal organization of activities within the department. Figure 1 highlights a special problem in the generation of potential options in this study. It is not just a question of selecting a number of computers which might be purchased, each constituting a separate option, but a question of how many separate computer configurations should comprise each particular option. One could consider candidates for a single multipurpose time sharing machine, capable of meeting all requirements. Alternatively, a set of three computers could be purchased, each supporting one of the three activities shown at the first level of decomposition (Research, Secretarial, Teaching); or one could purchase a set of 18 small dedicated machines, each supporting one of the 18 facilities shown at the right hand side of Figure 1. In general, an option can comprise a set of n computers, each meeting the requirements of a cluster of activities formed by grouping the 18 facilities into n groups, and there are a vast number of ways of forming such groupings. When one couples this realization with the fact that there were computers available from over 40 different manufacturers which, on grounds of price and capabilities, could be candidates for an element in the set of computers comprising each option, one can see that the number of different options that could be generated is very large indeed, and far too

P. Humphreys

25 6

Overall objective: provide

Decomposed objectives

computer facilities for

facilities 1 Psychophysiological Laboratory research

~

Laboratory

\ 2 Human experimental

experiments

laboratory research 3 Cognitive simulation research

Si mulat i on

I//

4 Interactive decision aiding research

Research Conversational

1 5 Interactive educationall interview research

6 Statistical package / development research Statistics S(PU) =S(FU), where S is surprisingness. However, if temporal context does influence judgments, then the surprisingness of future stories will be greater than past stories; thus S(F1) > S(P1) and S(FU) > S(PU). Experiment 1

Method Design The design of the experiment was similar to that of Falk (1975). Four different coincidence stories were created, each having different content. Four versions of each of the four stories were created: past-intersection (PI), past-union (PU), future-intersection (FI), and future-union (FU).

COINCIDENCES

493

Altogether 16 stimulus stories comprised the design. Each subject was given four coincidences to read and rate, one (different) version from each of the four stories, presented in random order. All 24 permutations were used, most of them repeating four times (except for 5 permutations that appeared 3 times).

Stimulus material The stories were essentially the same as in Falk (1975), adapted to an American setting. The coincidences described an unexpected meeting with an old friend, a fortunate hitchhike, a convergence of birthdays, and a peculiar numerical combination; versions of each were constructed as follows: The pustiitersection (PI) version enumerated all the specific details that converged in the event that had occurred. For example: Recently, I met a student who told me how fortunate he was: Precisely on his 19th birthday, h e tried to hitchhike home t o John Day for his birthday party and started waiting at the outskirts of Eugene. Quite soon, a driver stopped to offer him a ride, and it turned out that he was going exactly to John Day, a small town in Eastern Oregon. The past-union (PU) version related the same story that had occurred, however, it also gave the reader several hints about the universe of possibilities of which this story represented but one: Recently, I met a student who told me how fortunate he was. His home town is John Day, a small town in Eastern Oregon situated on a main highway. One day earlier, precisely on his birthday, he tried to hitchhike and started waiting at the outskirts of Eugene. Quite soon, a driver stopped to offer him a ride, and it turned out that he was going directly to John Day. He wasn’t that lucky up t o that day through a year of frequent hitchhiking. In the future-union (FU) version, the same scope of possibilities was described but the question of surprisingness referred t o a hypothetical event: Recently, I met a student who told me that his hometown was John Day, a small town in Eastern Oregon situated on a main

R .Falk and D. MacCregor

4 94

highway. He is a student in Eugene. He prepares himself for frequent hitchhiking and a lot of waiting on the roads. He often asks himself whether throughout the year he will be fortunate enough to get a ride directly from Eugene to John Day. The future-intersection (FI) version described a hypothetical event in which all the above mentioned components converged: Recently 1 met a student on his way to the outskirts of Eugene in an attempt to hitchhike. He told me that his home town is John Day, a small town in Eastern Oregon. He added that it was exactly his 19th birthday and he was going home for his birthday party. He wondered whether he would be fortunate that day and that the first driver to stop and offer him a ride would go directly t o John Day. Parallel versions were constructeg for the three remaining stories.

Procedure The task was introduced to subjects as follows: In this task we would like you to read four stories, each of which describes coincidences which occur in people’s lives. Read each one carefully. At the end of each story, you will be given an event from the story and asked t o rate it for how surprising it is. Each story was presented on a separate page. Subjects were asked to judge the surprisingness of each story on a 20-point scale where 1 = ”not at all surprising” and 20 = “very su~prising”.~ In addition, subjects were asked t o write and describe a coincidence that had happened to them, and rate it for swprisingness.

An alternative task would have bym to ask subjects to judge surprlsingness via For example, how many more tlmes surprising did you find magnitude this event? than some specified event. Such judgments would form a log-ratio scale, in wNch case their subsequent analysis would have involved division rather than subtraction Though the affective character of ”surprisingness”may seem to call for a logarithmic response, the judgmental scale used was chosen for simplicity and to parallel Falk (1975).

tima mat ion.

495

COINCIDENCES

Subjects A total of 91 subjects recruited through an advertisement in the University

of Oregon student newspaper took part in the experiment. Each was paid $5/for participating in a 1-112 hour session in which they completed this and a number of other unrelated tasks dealing with judgment and decision making.

Results

Since each subject rated four stories, six paired comparisons can be made for each subject. An ordinal analysis was performed noting only the dkection of the difference (whether more, less or equally surprising) between the ratings of each pair of stories. Hence, six signs (each being >, < or =) were obtained for each subject. The results of this analysis are presented in Table 1. The paired comparisons are arranged in the Table such that S(Y) >S(X), where S( Y) is the rated surprisingness of the version noted on the Y axis and S(X) the corresponding rating of the version on the X axis. The ordering of the versions along each axis is such that the preponderance of instances correspond to S(Y) > S(X). The z statistics presented in the cells of Table 1 are significance tests on the sign of the comparison between the row and column entries. The lower line (in parenthesis) presents analogous results for the 79 subjects in Falk (1975). Table 1 . Analysis of Version Effect Within-In Subjects: Unadjusted cores^ @J = 91)

Analysis of the Hypothesis S ( Y ) > S ( X ) Y PU (z = 0.97)

PI

FI

z = 5.16

z = 4.10

(Z = 3.35) (Z = 2.02)

FU a

PU

z = 2.49 (z = 4.36)

X

PI

Values in parentheses are for the 79 Israeli subjects from Falk (1975).

R. Falk and D. Madregor

4 96

Since the same subjects contributed their responses t o all six cells, the z ratios should be interpreted only as descriptive measures. The most apparent result in Table 1 is that the future-intersection versions were judged considerably more surprising than any of the others. Furthermore, a PI coincidence was less surprising than a comparable FI event, although both mentioned the same list details. The same result can be seen from consideration of the mean ratings given to the four versions by all subjects. These means, written in decreasing order of surprisingness, are: S ( F I ) = 14.l;S(PI)= 11.5;S(PU)= 10.9;S(FU)=9.20 However, the four stories differed in mean surprisingness (across versions) about as much as the four versions. Therefore, each raw rating of a specific story-version combination was adjusted by subtracting from it the mean rating of that story (across subjects and versions). If S(X) is the raw surprisingness rating of version-story A', then S'Q denotes the adjusted rating of X,that is, the difference between S(X) and the mean rating of the same story. Hence, S'(X) measures the net version effect. The differences between pairs of adjusted ratings were organized in a format as previously. Table 2 presents the results of this analysis. The ordering of the versions along the axes matches that of Table 1, confirming the general Table 2. Analysis of Version Effect WithinSubjects: Mjusted Scores

(N = 91) d = S' (Y)- S' (X)

6=1.71

PU

-I

u = 7.976

t = 231*

2 = 2.43

PI

u

Fl

* **

***

p < 0.05

p < 0.01 p 80

E

k

F if u($30),80

Figure I . Possible Representation of an Allais-Type Decision task.

Thus, the risk dimension would receive a smaller weight than in problem 5. People for whom the value of w is above 0.5 in problem 5 and below it in problem 7, a very plausible set of assignments, will prefer A to B and F to E. Note that this pattern of preferences is not a reversal, is not incoherent, and is not a violation of expected utility theory, if the problem structures in Figure 2 are assumed. In other words, the Allais paradox illustrated by the supposed preference reversal from problem 5 t o 7 is only a paradox if you accept Allais' formulation of the problem. It is perfectly possible that all those who have investigated the Allais paradox are correct in saying that many subjects make inconsistent shifts of preference. Certainly I am not questioning the data that many people do change their preference order. But to call this a reversal, to say it is 34

5 30

L.D. Phillips

poblrm 5

Problem 7 Criteria

mqilts: w, risk

A

>B

Criteria

4-IN,

WL

I-w2

payoff

risk

payoff

if w, > 0.5

E > F if wz > 0.5 However, w,

.'.

it

IS

possible for A

> w2

> 6 and F > E

Figure 2. An Alternative Representation of the Allais Problem

incoherent or that it violates expected utility theory is too strong when we do not know how subjects structured the problem. Thus, the importance of the psychophysical paradigm. Lacking inforination about how their subjects structure the problems presented, investigators have to assume that one particular problem structure is veridical or objective or else the data cannot be interpreted. But to prefer the structures of Figure 1 to those of Figure 2 is a value judgement; decision theory is largely silent about how tasks should be structured. There is, then, a substantial valueladen component (or bias) in most of the work on judgemental heuristics and biases when investigators conclude, without knowledge of their subjects' subjective problem structures, that people exhibit systematic biases in making judgements of uncertainty. Where decision research is concerned, the simple psychophysical model is untenable. There are no 'objective' problems, no 'veridical' structures. A group can reach concensus about the statement of a problem but disagree about specific structural representations of it. In decision conferences, as the decision analyst facilitates the group in working toward a requisite structure (a socially shared and agreed structural representation of the problem), the very nature of the problem often changes as the explicit modelling allows new intuitions about the problem to emerge and then to be captured in a revised structural representation. This same process of iterating towards a more satisfactory representation of a decision problem is evident also in individual decision making. Wooler and Erlich

PERSPECTIVE ON HEURISTICS AND BIASES

531

(1 983) observe that students using multi-attribute utility theory typically change and revise their problem representation over the course of a one- to two-hour session, adding and deleting attributes as they gain new intuitions about their job choices. As for laboratory research, or ’demonstrations’, as Tversky and Kahneman (1981) now call their research, investigators usually assume that their representation of a problem is veridical and can be considered objective. Decision researchers have ignored the theoretical stance of other psychologists who have long since recognised that people do not construct identical representations of their phenomenal world (Evans, 1972). To reject the psychophysical model deals a severe blow to most of the work over the past twenty years on judgement research conducted within the expected utility framework. That work has almost exclusively been concerned with comparisons of people’s behaviour to that predicted by the expected utility rule, the probability laws (including Bayes’ theorem), and multi-attribute models. Issues of structure have been ignored, yet it should now be clear that conclusions about how people process infornlation cannot be made without knowing how subjects structure problems. Since this lack is characteristic of most research t o date, the conclusion of ’bias’ is possibly wrong, or at best, premature.

The Test Thcoty Model Some decision research, particularly work on scoring rules, assumes a paradigm that sounds very much like the classical psychometric model: test score = true score + error assessed probability = true probability + error judged value = true value + error The ’true probability’ or ’true value’ is presumed to reside inside the assessor as an indication of the individual’s true state of uncertainty or true utility, but becomes distorted by motivational, cognitive, and other biases while being translated into a response (de Zeeuw and Wagenaar, 1974). Proper scoring rules (Stael von Holstein, 1970) were thought t o keep the assessor ’honest’, to deter the influence of biasing processes. However, more than a decade of research using scoring rules has shown that their main function is to serve as rather uninformative outcome feedback, of some use in the early stages of teaching people how t o assess probabilities. It appears that scoring rules lend some meaning to the scale of probability, and so help people to formulate numerical assessments. There never was a ’true probability’, only a rather diffuse feeling of uncer-

5 32

L.D. Phillips

tainty. Not surprising, for probabilities are numbers, while uncertainty is registered as a feeling. The concept never was very tenable, and it is not made more so by claiming that people behave ’as if‘ they experience probabilities. A more serious objection to the test theory paradigm is that it propagates wllat may well turn out t o be a fundamental misconception about the nature of probabilistic thinking, that the ability t o assess probabilities varies in degree from one person t o the next. This is clearly the assumption behind all research o n calibration where the majority of studies have shown that most people are too sure of themselves (Lichtenstein, Fischhoff,and Phillips, 1982). Just as some people are more intelligent than others, so some are better calibrated than others; both are thought to be matters of degree. But an alternative forniulation is beginning to find ernpirical support. Under this view, probabilistic thinking is one aspect of an individual’s capacity to d o work, but there are qualitative differences, discontinuities, in capacity, not quantitative degrees of capacity as with IQ. These differences are described by Jaques (1978) as consituting seven levels of abstraction, with each higher level implying increased ability by iridividuals to deal with problems in an abstract way. Capacity develops as a person matures, so it is possible for an individual to move up this hierarchy of levels, though few will ever achieve the highest levels. According to Jaques’ theory, people differ in their capacity to think probabilistically. There is no hint that people at lower levels are in any way deficient; they are qualitatively different, not suffering from some cognitive deficit. Capacity theory also reveals two different ways in which uncertainty may arise (Humphreys and Berkeley, in press). At levels I arid 2 , Uncertainty is about how a defined task is to be done, whereas at higher levels uncertainty concerns the possible consequences of tasks and the characteristics of the ’small worlds’ within which these tasks are located. This clearly suggests that task characteristics will influence the handling of uncertainty. Since an individual’s ability to deal with uncertainty depends on characteristics of the task as well as on the person’s level of capacity for abstraction, three criticisms can be made of most work o n heuristics and biases. First, research has typically used subject populations that are biased; they are deficient in people with higher levels of capacity, the very people who could be expected to be better at probabilistic thinking. Indeed, Schoemaker (1980) found that subjects whose mean age was about 40 make hypothetical insurance decisions more consistently and more in accordance with expected utility theory than did student subjects with mean age about 20. The second criticism concerns the selection of tasks. Capacity theory

PERSPECTIVE ON HEURISTICS AND BIASES

5 33

predicts that tasks involving uncertainty will be dealt with differently depending on whether the problem is stated in a concrete or abstract context. Schoemaker found that 13 percent of his studelit subjects preferred a zero deductible insurance policy when they were presented with an abstract representation of the two choices facing them, but that 45 percent of these same subjects then preferred this policy when it was presented in an insurance context. The heuristics literature abounds with abstract problems, hypothetical gambles, imaginary games with abstract probabilities of winning or losing amounts that materialise out of thin air, imagined balls and urns with samples generated by unseen forces, life and death decisions, choices between large amounts of money and stated probabilities that differ by only 0.01 and 0.05, unfamiliar contexts, mostly tasks for which good performance could be expected only from people whose capacity is above level 3.’ Virtually no intercorrelations were found between 12 different measures of probabilistic thinking derived from the two tasks given to 143 adult volunteers in a study by Wright and Phillips (1979b). Nor did Evans and Pollard (1982) find any correlation between errors on two tasks involving the perception of non-randomness. Both these studies highhght the importance of considering task characteristics before generalisations can be made to other tasks. Taking these two criticisms together, only when the task characteristics are appropriate for the level of abstraction of which the individual is capable, should we expect good performance. If a group of subjects of varying capacity is given a collection of concrete and abstract tasks, a subjects-by-tasks interaction should appear. Usually, though, group perform n c e is examined for individual tasks, and between-task comparisons are rarely made. This preference for analysing only group data is my third criticism. Elsewhere (Phillips, 1980) I have reported that at least one person in 20 in the West and more in the East shows no inclination to t h n k probabilistically. As we have seen, capacity theory predicts qualitative differences among people in probabilistic thinking. By analysing group data we lose the sense of some people doing well on some of the tasks. It is a revealing observation that researchers in this area prefer to focus on the deficiencies, t o develop explanations and models to account for these deficiencies, rather than t o look for the characteristics of tasks that would enable people with different capacities t o do well. Since level 3 is the typical capacity of middle managers, this theory explains the reluctance observed by Harrison (1977) of many middle managers to use decision analytic structures that incorporate uncertainty, such as decision trees.

5 34

L.D. Phillips

In short, because tasks have not been matched t o people’s capacities, it is quite wrong to assume that current research justifies the generalisation that people are inconsistent and biased in their handling of real-life tasks. One should not give bank clerks judgemental tasks appropriate only for the manager of the bank, and then believe that any useful psychological theory has resulted from the categorising and labelling of the observed errors of judgement. The psychophysical paradigm is inappropriate because although perceptual processes do not differ substantially from one person to the next, problem solving capacity does. Further, even if the ’true probability’ concept were tenable, the test theory paradigm would still be inapplicable because a given task would call up the ’true probability’ in some people but not in others. By tacitly accepting either paradigm, researchers construct experiments or ’demonstrations’ whose generalisability to other tasks and other people is negligible. The Information Processing Paradigm When researchers focus on how subjects use the information that is presented to them in arriving at an assessment of probability or in making a choice or preference judgement based on probabilities and values, they are assuming an information processing paradigm. Our subjects are assumed to be processors of information, so any failure to come u p with the normatively prescribed answer indicates a failure in processing. When we use the representativeness heuristic, we do not incorporate base rate information in making our assessments. When the availability heuristic influences our judgements, we process only information that comes readily to mind and d o not consider the information contained in less available cases. The anchoring or adjustment heuristic causes us to over-value the information implied by the starting value, while we are not sufficiently influenced by the information that should pull us away from the anchor. Each of these is interpreted as a failure to process relevant information. This may be so, but another explanation is possible, too. Consider this simple problem. A fair coin is to be tossed twice. What is the chance of obtaining at least one head? Most people would reason that there are four equally likely outcomes, HH, HT, TH and TT, three of which are favourable cases so the probability is 3/4. We would assume that any other answer is a failure to combine the probabilities at each toss according t o the laws of probability, and if enough people made the same mistake, we would infer a cognitive limitation on the ability to combine information. But D’Alembert, as Todhunter (1865) reports, claimed that the probability was 213, reasoning that if the head appeared on the first toss, there

PERSPECTIVE ON HEURISTICS AND BIASES

5 35

was no need for a second toss, leaving only three cases, H, TH and TT. Thus, the probability is 213. Today, we would argue that D’Alembert erred in assigning equal probabilities in those cases. Suppose, however, that a drawing-pin (thumbtack) is being tossed. It can fall with its point up, U,or down, D. What is the probability that in two tosses it will fall point up at least once? If you have no idea of the direction ar extent of the drawing pin’s bias, 0, which might be interpreted as the long-run ratio of Us to total tosses, you might choose p(0) = 1/2, assume that the tosses are independent, thus making the four possible outcomes equally likely, giving the answer 314, as with the coin. A Bayesian, on the other hand, might assign a uniform prior distribution over 0 before the first toss. T h s has its mean at 0 = 112, so for the first toss p(U) = p(D) = 112. If the drawing-pin comes up U on the first toss, that datum revises the prior so it now becomes a triangle with its mode at 8 = 1.0, and with its mean at 8 = 213. Thus, if the first toss results in U, then for the second toss p(U/U) = 2/3 and p(D/U) = 113. By a similar line of reasoning, if the first toss is a D, then for the second toss p(U/D) = 113 and p(D/D) = 2/3. A little multiplication gives the probabilities for the joint events: p(U,U) = p(D.D) = 112 x 213 = 113 and p(U,D) = 1/2 x 1/3 = 1/6. Thus, the probability of at least one U is 113 + 116 + 1/6 = 213, in agreement with D’Alembert’sresult if not his reasoning. The poirit of this simple example is to highlight once again the important role of structure. Information has been processed correctly in the two aproaches to the drawing-pin problem, but the structures are different. It is not possible to tell whether information has been processed correctly lacking knowledge of how the person has structured the problem internally. Thus, the ghost o f the psychophysical model lurks behmd the information processing paradigm. Experimenters must assume, usually implicitly, that some structure is ’objectively’ correct, otherwise nothing could be inferred about people’s information processing capabilities at all. This is true of very nearly all experiments conducted to date, for no attempt is made to discover how subjects structure the task presented. There is an even more serious objection to the information processing paradigm that is assumed by most research. Most investigators seem to assume that information in the task is encoded in some way by the subject, who may selectively filter or preprocess the information, that additional information thought to be relevant may be retrieved from the subject’s memory, that these inputs are then combined or ’processed’ with the individual’s needs, goals, and motives possibly influencing the processing, all being affected by group norms and cultural values, with the resulting brew somehow governing the subject’s response. It is an open system; information in, response out.

5 36

L.D. Pliillips

Problem solving is conducted in interaction with other people and with the environment (Hogarth, 1981 ). Because of this, the production rule version of the information pocessirlg paradigm (Pitz, 1977; T u g l e and Barron, 1980) is inadequate since production rules reside in memory unchanged by attempts t o solve a particular problem. Yet experience in using decision analysis as a framework for helping decision makers t o generate ’requisite’ representations of problems, suggests that this opensystem information processing paradigm fails to capture the interactivc, iterative nature of good problem solving. In the research literature, s u b jects are almost never given feedback about the logical implications of their judgements, never shown their inconsistencies and invited to resolve then4 rarely asked for redundant judgements so that inconsistency can he utilised as part of the assessment process, and almost never asked to make judgements in a group setting. Yet many of these characteristics can be seen in recent studies that demonstrated good calibration of probability assessments (Balthasar, Boschi,and Menke, 1978; Kabus, 1976). It is perfectly possible that many people, given the right tasks in the right circumstances, could make precise, reliable, accurate assessments of probability, as I have argued elsewhere (Phillips, 1982b), but as long as research is governed by the open-system version of the information processing paradigm it is unlikely that those circumstances will ever be identified.

Summary

So far I have tried t o show the restrictive effect our implicit acceptance of three paradigms has had on decision research into heuristics and biases, with the consequence that we believe our subjects to be cognitively deficient, intellectual cripples. This generalisation was shown not to be justified by current research because it depends on untenable assumptions embedded in the paradigm. From the discussion of these paradigms and their limitations, four problems with current research can be identified. 1. 7he problem of objectivity. Most investigators assume the existence of an objective problem, which then justifies interpretations of incoherence, violation of expected utility theory, etc. We ignore the fact that our subjects generate their own structural representations of a stated problem that assessed probabilities are conditional on the assessor’s structure, not the experimenter’s, and that judged values are also conditional on structure which may be multi-attributed even though a single attribute is given in the stated problem. Even where structure is not at issue, it may be misleading to use statistical or actuarial data as an objective standard to

PEP.SPECT1VE ON HEURISTICS AND BIASES

5 37

which subjects' judgements can be compared, as in some risk studies (Slovic,Fischhoff,and Lichtenstein, 1381). The age a t which I will die is an event that is conditional on other events and information: my current age, my family history, general health, smoking and drinking habits, etc. Some of these conditioning events are explicitly identified in the actuarial standards, but many are not. Thus, without knowing the conditioning events I consider in making my judgement, it is not possible to ascribe bias t o me when my assessment differs from the actuarial standard. In any event, actuarial standards are notoriously difficult to develop. Insurance companies rely substantially on the subjective judgements of their underwriters in developing differential rates for all classes of business other than life insurance. Private motor insurance is a good example: companies differ in their rating schemes, offering different premiums for the same risk. If insurance companies cannot develop objective rating schemes, the standards used in risk assessments studies must surely be suspect. 2. The problem of generalisability. Heuristics research has ignored the possibility that subjects are characterised by qualitative differences in their capacity to deal with the research tasks. Instead, biases are likened t o error scores, different from one person to the next in degree, not kind. Furthermore, no consideration has been given to the representativeness of tasks used in the research. In particular, no distinction has been made between concrete and abstract tasks, nor has any research attempted to match the task to the level of abstraction of which the subject is capable. Also, by relying almost exclusively on group data, experimenters have ignored the people who d o well on these tasks. No attempt has been made to discover the task characteristics and subject characteristics that lead to a good performance. A good example is the almost exclusive reliance o n the fractile method of assessing probability distributions described in narrative form in Raiffa (1968). It is well-documented that this procedure results in distributions that are too tight, which has led to the generalisation that people are too sure of themselves when assessing continuous distributions. However, the encoding method recommended by Stael von Holstein and Matheson (1979, pp. 45-53), which requires the subject t o make cumulative probability judgements for various possible values of the uncertain quantity, so reliably gives more spread-out distributions than the fractile method that the two methods can be demonstrated in a class-room setting with one assessor using both approaches. For the most part, it is decision analysts, not decision recearchers, who have sought and found the conditions under which good assessments can be obtained. Until decision researchers take a greater interest in task taxonomy and individual differences, they are likely t o continue making unwarranted generalisations from unrepresentative experiments.

L.D. Phillips

538

3. The problem of cognitive de.ficiency. To their credit, Tversky and Kahneman have been careful not t o suggest the cognitive deficit interpretation of observed biases in decision research, but most other researchers have not been so discrete, as Edwards (1982) observes. i n anthropological research ’cognitive deficit’ interpretations have been attacked since the turn of the century. Boas (1911) was an early critic. Summarising his argument, Cole, Gay, Click, and Sharp (1971) write:

”...the thrust of Boas’s argument seems to be that previous observers failed to understand the people they were describing and then mistook their own lack of understanding as evidence of their informants’ stupidity.” Lacking any understanding of the internal problem representations generated by their subjects, decision researchers may actually be describing themselves when they ascribe ’bias’ to their subjects. 4. The problem of measurement, Much decision research is predicated on the notion that the subjects’ assessments, preferences and choices are stable, that an enduring internal state is being measured. An alternative view is that assessments, preferences and choices are generated in the process of solving a problem, that they develop as the individual explores alternative problem structures, attempts to resolve inconsistencies and incorporates new intuitions and information that arises in the course of this iterative process. Eventually stability is achieved and a decision can be made with confidence. Nisbett and Ross (1980) observe: ”It would be interesting to know how any of the inferential errors reported in this book would survive intact after an open discussion among groups of ten or twelve subjects. Our guess is that most of the errors would at least be substantially reduced.” (p. 267) We need far more research which allows subjects to explore problems that are meaningful to them. At the moment, research results characterise only our subjects’ first reactions, which in real-life is only the starting point for serious problem solving. Research on heuristics and biases has become a psychology of first impressions.

The Generation Paradigm One line of the argument so far has been that unless we know the subject’s internal structural representation of the task, we cannot adequately interpret his or her data. That applies to us as decision researchers, too. Unless

PERSPECTIVE ON HEURISTICS AND BIASES

5 39

we are clear about the paradigm or model that we bring to our research design, our interpretations of the data are likely to be confused, muddled, or wrong. I have already sketched some of the elements of a new paradigm that seems more realistic and less restrictive than the three discussed so far (Phillips, 1983a). Here I will revise that view somewhat and extend the model, but it is still incomplcte. The key idea in the generation paradigm is that process is embedded in structure. We cannot talk about information processing without reference to the internal representation of the task. Structure works in different ways and at different levels. Here, we distinguish two types, general structure and problem structure. The research on cultural and individual differences in probabilistic thinking (Wright and Phillips, 1 979a; Wright and Phillips, 1979b) suggests that people impose a general structural framework on a problem. Causal and fatalistic structures have been mentioned, but others are possible: deductive logic structures, inductive logic structures, similarity structures, diagnostic or conditional probability structures, and more. For example, Hamniond, McClelland, and Mumpower (1 980) have identified six general structures as characterising research on decision and choice processes. You interpret the same data differently if you use a lens model rather than expected utility theory. Problem structure represents the task at hand. Even if you accept the general structure of expected utility theory, different problem structures are possible (Yates and Carlson, 198l), as Figures 1 and 2 show. hoblem structuring may include aspects of the editing phase of prospect theory, but other operations, such as comparing parts of the internal model against features of the presented problem, or testing consequences of the internal representation against the environment, or scanning the internal model for coherence, are also possible. It is because prospect theory omits these additional operations and fails to consider the impact of general structure, that I claimed earlier it was incomplete. Composition rules are applied witlun the problem structure. These may take the form of production rules, compensation rules such as expected utility, non-compensatory rules such as elimination-by-aspects, or any other rule judged by the subject as appropriate for the task in hand. Application of these rules allows information to be processed. The capacity of the individual, indicated by the level of abstraction, will affect the kinds of general structures used by the individual, the types of problem structures developed, and the information processing rules used. In this way, the generation paradigm allows for individual differences in judgements and choice. A second feature of the generatim paradigm is that general structur-

5 40

L.D. Phillips

ing, problem structuring, and information processing are carried out in any order, and they are done iteratively. (Note the contrast with prospect theory, in which editing comes before evaluation; see Kahnenian and Tversky, 1979.) The failure to find an adequate problem structure niay lead the individual t o shift to a different general structure. Processing some information may lead to inconsistent results, motivating a change in problem structure. Problem solving activities continue within the practical constraints of time and resources. In many cases, the goal of the subject may be t o develop a requisite solution, requisite in the sense that the internal representation includes all the features required to solve the problem, and judgements are no more considered than is required t o make a decision or to express a preference. Note that this is not a 'cost of thinking' or 'satisficing' criterion; it is a pragmatic rule that exploits the insensitivity of decisions to particular aspects of the problem (for an example, see Phillips, I982c). A third feature of this paradigm is that task characteristics can be influential at any level. In studies of calibration, questions about future events nught be more likely t o induce probabilistic general structures than general knowledge questions. Pitz showed that a simple rewording of one of the problems appearing in Tversky and Kahneman (1971) caused nearly every respondent t o answer the question correctly, whereas most people answered the original problem incorrectly. Pitz argued that the rewording blocked the higher-order production usually given to the original problem and allowed a lower-order production, which gives the right answer, to emerge. Schoemaker (1980) speculated that the context effect of his insurance study is an evohng process, the insurance context calling u p influences like societal norms that were not evoked by the hypothetical wagers. This latter observation highlights a fourth feature of the generation paradigm: it acknowledges the multi-dimensional nature of even the simplest tasks. In building their own internal representation of the problem, subjects may abstract features of the presented problem, but they may also incorporate dimensions drawn from experience, influenced by group norms and cultural values. There is no assumption that an 'objective' problem exists; the reality is the subject's current internal representation. The experimenter may have a different one, most likely corresponding closely to the stated problem. But there are no correct representations, only different ones. Thus, supposed preference reversals are not necessarily the result of different frames imposed o n the 'real' problem, they are different preferences resulting from different internal representations. A minor change in interpretation? Not so, for the 'reversal' interpretation

PERSPECTIVF ON HEURISTICS AND BIASES

541

implies inconsistency, whereas applying the generation paradigm to the results may well indicate no incoherence. In summary, the generation paradigm sees decision making and the forming of judgements as dynamic, social processes, involving interaction between the individual and his or her environment and with other people. Problem statements are seen as the starting point for an iterative develop ment of a requisite structural representation within which information can be processed. Task characteristics and individual differences are seen as interacting and influencing the problem solving process at all levels. In short, judgements of value, assessments of probability, representations of structures are all generated by the individual. Investigators working from this new paradigm will not be content with asking subjects for first impressions to static problems. Instead, it will be necessary to allow subjects time t o work toward a requisite representation, and investigators will have to discover how subjects internally represent stated problems if inferences are to be made about consistency. Rather than investigate ’de-biasing’ procedures, experimenters will examine the effects of aids, from paper-and-pencilto interactive, conversational computer programs. The effects of gioLips on decision representation will be investigated (strangely, this was never done in any of the numerous studies on the risky-shift phenomenon). Effort will be devoted to finding the circumstances in which people are ’intellectual athletes’ not ’intellectual cripples’. As a final observation, it is worth noting that the generation paradigm has important implications for the status of expected utility theory. There seems little point from thn perspective in asking whether people violate fundamental axioms of preference in either experimental or applied settings. Instead, the generation paradigm suggests that it would be more appropriate to ask whether people would accept the axioms and resulting theory as a useful framework within which a requisite structure can be developed, and which will guide individuals in generating a coherent set of assessments which are themselves requisite. Instead of worrying about grand world violations of consistency, we might be more modest in aslung of expected utility theory that it serve as a guide for decision makers who wish to develop a small-world model of their problem, not an optimal model but a requisite one that goes as far as is deemed useful toward achieving internal consistency. In this sense, expected utility theory is neither normative nor descriptive. Rather, it is a guide to thinking that polices consistency. Thus, attacks on its normative or descriptive adequacy are irrelevant, and contribute only t o a destructive devaluation of the power of expected utility theory as an organising principle and demonstrably useful guide to choosing and deciding.

542

L.D. Phillips

References Boas, F., 1911. The Mind of Primitive Man New York: Macmillan, 1938, rev. ed. Balthasar, H.U., K.A.A. Boschi, and M.M. Menke, 1978. Calling the shots in R & D. Harvard Rusincss Review, May-June, 15 1 - 160. Cohen, J., 1960. Cliance, Skill and Luck, Harmondsworth, Middlesex: Penguin. Cole, M., J. Gay, J.A. Click, and D.W. Sharp, 1971. The Cultural Context of Learning and Thinking U,S.A.: Basic Books; London: Methuen and Tavistock. de Zeeuw, G. and W.A. Wagenaar, 1974. h e subjective probabilities probabilities? In: C-A.S. Stael von Holstein (ed.), The Concept of Probability in Psych@ logical Experiments Dordrecht: D. Keidel. Edwards, W., 1983. Human cognitive capabilities, representativeness and ground rules for research. In this volume, 509 --5 15. Evans, J.St.B.T., 1972. On the problems of interpreting reasoning data: Logical and psychological approaches. Cognition, 1, 373-384. Evans, J.St.B.T. and A.E. Dusoir, 1977. Proportionality and sample size as factors in intuitive statistical judgement. Acta Psychologica, 41. 129%138. Evans, J3.B.T. and P. Pollard. Statistical Judgement:A further test of the representativeness construct. Acta Psychologica (in press). Hammond, KR., G.H. McClelland, and J. Mumpower, 1980. tfuinan Judgment and Decision Making New York: Praeger. Harrison, F.L., 1977. Decision making in conditions of extreme uncertainty. The Journal of Management Studies, 169- 178. Hogarth, RM., 1981. Beyond discrete biases: Functional and dysfunctional aspects of judgmental heuristics. Psychological Bulletin, YO, 197-217. Humphreys, P.C., 1977. Application of multi-attribute theory. In: H. Jungermann and G. de Zeeuw (eds.), Decision Making and Change in Human Affairairs Dordrecht: D. Reidel. Humphreys, P.C. and D. Berkeley. Problem structuring calculi and levels of knowledge representation in decision making. In: R . Scholz (ed.), Decision Making Under Uncertainty. Amsterdam: North Holland (in press). Humphreys, P.C., S. Wooler, and L.D. Phillips, 1980. Structuring decisions: The role of structuring heuristics. Technical Report 80- 1. Uxbridge, Middlesex: Decision Analysis Unit, Brunel University. Jaques, E., 1978. Level of abstraction in mental activity. In: E. Jaques, R.O. Gibson, and D.J. Isaac (eds.), Levels of Abstraction in Logic and Human Action London: Heinemann. Kabus, I., 1978. You can bank on uncertainty. Harvard Business Review, May-June, 95-105. Kahneman, D. and A. Tversky, 1979. Prospect Theory: An analysis of decision under risk Econometrica, 47, 263-291. Kuhn, T.S., 1970. The Structure of' Scientific Revolutions Chicago: The University of Chicago Press, 2nd ed. Lichtenstein, S., B. Fischhoff, and L.D. Phillips, 1981. Calibration of probabilities: The state of the art to 1980. In: D. Kahneman, P. Slovie, and A. Tversky (eds.), Judgement Under Uncertainty: Heuristics and Biases Cambridge University Press. Llndley, D.V., A. Tversky and R.V. Brown, 1979. On the reconciliation of probability assessments. Journal of the Royal Statistical Society, 142, 146-162.

PERSPECTlVE ON HEURISTICS AND BIASES

543

Nisbett, R. and L. Ross, 1980. Human Inference: Strategies and Shortcomings of Social Judgeitlent. Englewood Cliffs, New Jersey: Prentice Hall. Phillips, L.D., 1980. Organisational structure and decision technology. Acta Psyche logica, 45, 247 -- 264. Phillips, L.D., 1982a. Generation theory. In: L. McAlister (ed.), Research in Marketing, Supplement I : Choice Models for Buyer Behaviour. Greenwich, Conn.: JAI Press. Phillips, L.D., 1982b. The evaluation of risks estimates: Limitations to human judgement? In: G. Volta (ed.), Risk and Safety Assessments in Industrial Activi. ties. Phillips, L.D., 1 9 8 2 ~ Requisite . decision modelling: A case study. Journal of Opere tiotial Research Society. Pitz, G., 1977. Decision making and cognition. In: H. Jungermann and G. de Zeeuw (eds.), Decision Making and Change in Human Affairs. Dordrecht: ReideL Raiffa, H., 1968. Decision Analysis: Introductory Lectures on Choice Under Uncertainty. New York: Addison-Wesley. Schoemaker, P.J., 1980. Experiments on Decision Under Risk: The Expected Utility Hypothesis Boston: Martinus Nijhoff. Slovic, P., B. Fischhoff, and S. Lichtenstein, 1981. Perceived risk: Psychological factors and social implications. In: The Assessment and Perception of Risk. London: The Royal Society. Stael von Holstein, C-A.S., 1970. Assessment and evaluation of subjective probability distributions. Stockholm: Economic Research Institute. Stael yon Holstein C-A.S. and J. Matheson, 1979. A manual for encoding probability distributtons. Menlo Park: SRI International. Todhunter, I., 1865. A History of the Mathematical Theory o f Probability. Cambridge and London: Mamillan. Tuggle, F.D. and F.H. Barron, 1980. On the validation of descriptive models of decision making. Acta Psychologica, 45, 197-210. Tversky, A, 1974. Assessing uncertainty. The Journal of the Royal Statisticd Society, Series B, 36, 148- 159. Tversky, A. and D. Kahneman, 1971. Belief in the "law of small numbers". Psychclogical Bulletin, 76, 105-1 10. Tversky, A. and D. Kahneman, 1974. Judgement under uncertainty. Science, 185, 1124- 1131. Tversky, A. and D. Kahneman, 198 1. The framing of decisions and the psychology of choice. Science, 211, 453-458. Wooler, S. and A. Erlich, 1983. Interdependence between problem structuring and attribute weighting in transitional decision problems, In this volume, 321 -334. Wright, G.N. and L.D. Phillips, 1979a. Cross-cultural differences in the assessment and communication of uncertainty. Current Anthropology. 20, 845-846. Wright, G N . and L.D. Phillips, 1979b. Personality and probabilistic thinking: An exploratory study. British Journal of Psychology, 70, 295-303. Yates, F. and B.W. Carlson, 1981. A synopsis of representation t b o r y . Paper p r e sented at the Eighth Research Conference on Subjectfve Probabllity, Utility and Decision Making, Budapest.

This Page Intentionally Left Blank

AUTHOR INDEX

Abelson, R.P. 148, 159, 163, 213, 221, 358, 367,444,453,455 Ackerman, B. 6 5 , 6 7 Ackerman, S. 67 Ackoff, R.L. 146, 163 Acton, J.P. 24, 37 Adelbratt, T. 3 4 6 , 3 5 0 , 3 6 7 , 3 6 8 Adelman, L. 164 Ahern, W. 5 7 , 6 7 , 81,88 Ajzen, J. 387,400 Alluisi, E.A. 206. 219 Altshuler, G.S. 127, 141 Alvarez, L.W. 492,504 Anderson, G.A. 220 Anderson, J.R. 208,215, 219,226,235 Argyris, C. 163 Arrow, K.J. 133, 141 Aschenbrenner, K. 134, 142 Aschenbrenner, K.M. 290,298.326,333 Aschenbrenner, M. 138,141 Asher, S.I. 493, 504 Atwood, M.E. 216, 219 Atz, H. 5 9 , 6 7

Balthasar, H.U. 538,544 Barbour, F. 354, 369 Barclay, J.R. 220 Barron, F.H. 538,545 Bauer, V. 159, 1 6 3 , 2 8 4 , 2 9 1 , 2 9 8 , 3 2 8 Beach, L.R. 226, 235, 282, 284, 298, 400,458,468, Becker, G.M. 419,429

35

Becker, S.W. 458,468 Bell, D. 147,163 Bell, R.S. 180,514 Benne, K.D. 164 Bennis, W.G. 158, 163,164 Berkeley, D. 5 1 8 , 5 2 5 , 5 3 4 , 5 4 4 Berlyne, D. 384,400 Bernard, H.R. 493,504 Bettman, J.R. 134, 141, 367, 371, 372, 373,381,444,455 Beyth, R. 137, 142 Biel, A. 386, 387,400 Bieri, J . 338, 386,400 Boas, F. 540,544 Bobrow, D.G. 208,219,455,456 Boichenko, V.S. 134, 142,143 Bonczek, R.H. 268, 277 Boring. E.G. 5 1 9 , 5 2 5 Boschi, R.A.A. 538,544 Boulding. K.E. 152, 161, 164 Bowel, G.H. 226, 235 Brach, E.W. 103 Bransford, J.D. 21 3, 220 Braunstein, M.L. 371,382,430,442 Braybrooke, D. 75,88 Bronner, A.E. 293, 297,298 Bronner, F. 201,202,281 Brown,M.T. 212,213,221,304, 319 Brown, R.V. 1 8 , 1 9 , 2 0 , 2 1 , 9 1 , 9 3 , 1 0 3 , 152, 156, 159, 166, 172, 180, 181, 277,458,468,529,544 Brownson, F.O. 458, 468 Brunswik, E. 511,514

546 Budge, 1 , 2 8 8 , 2 9 8 Buedc, D.M. 274,277,318 Burns, T. 148, 164

Campbell, D.1'. 164 Campbell, F.L. 298 Carlson, B.W. 541,545 Carnap, R. 138,142 Carroll, J. S. 371,382,430,442,455 Carter-Sobell, L. 212,220 Casper, S. 105 Chase, W.G. 208, 220 Chin, R. 164 Christen, F.G. 303, 318 Chuev, Ju.V. 144 Churchman, C.W. 135,142 Clayton, li. 85,88 Cohen, J. 529,544 Cole, M.J. 540,544 Collins, A.M. 226, 235,366,367 Coombs, C.H. 261,278, 328,333 Corbin, R.M. 3.54, 367 Corey, K.E. 164 Crcwe, 1. 289 Crozicr, R. 339,417 Cyert, R.M. 153, 164

Dale, A. 164 Da'vid, L. 184, 195 Davis, L.N. 8 3 , 8 8 Dawes, R.M. 2 6 1 , 2 7 8 , 3 2 8 , 3 3 3 Dean, B.V. 129, 142 de Bono, E. 268,278 Decker, L. 473,490 de Finetti, €3,458,468 de Groot, M.H. 429 de Hoog, R. 201, 202, 281, 293, 297, 298 Derby, S.67,525 Detmer, 1).119 de Zeeuw, G . 533,544 Dimitrov, V. 246,252 Dolbear, F.T. 138,142 Dosher, B.A. 138,144,363,368 Driankov, D. 200,237,243,246,252 Duncker, K. 213, 220,267,278 Dunnette. M.D. 164

Dusoir, A.E. 544 Dyer, J.C. 134, 142 Dyer, J.S. 176, 180 Eden, C . 164 Edwards, W. 8 4 , 8 8 , 1 0 5 , 1 1 9 , 136, 137, 142, 171, 173, 175, 178, 179, 180, 195, 202, 270, 278, 298, 301. 310, 318, 322, 323, 326, 333, 334, 344, 367, 507, 509, 510, 514, 515, 528, 540,544 Egan, J.P. 475,490 Eilon, S. 146, 164 Einhorn, H. J . 133, 138, 142, 155, 164, 206, 214, 220, 224, 225, 235, 322, 325, 333, 348, 350, 368, 411, 415 Ekberg, P.H. 207,220,373, 381 Elbing, A.O. 164 Elliot, P.B. 4 7 6 , 4 7 7 , 4 9 0 Ellsberg, D. 458,468 Emelyanov, S.V. 1 3 9 , 1 4 0 , 1 4 2 Englinder,T. 453,454,455 Enthoven,A. 1 2 6 , 1 2 7 , 1 2 9 , 1 4 2 Erdman, H. 180 Ericsson, K.A. 371, 381, 4 1 7 , 4 2 9 , 4 3 1 , 432,440,442 Erlich, A. 203,321. 532,545 Evans, J.St.B.T. 533, 544

Falk, R. 340, 491, 4 9 2 , 4 9 3 , 4 9 4 , 4 9 5 , 496,497,499,504 Farkas, E. 453,455 Farlie, D. 298 Feldman, Ph. 282,299 Fellner, W. 458,468 Ferreira, J . 32, 37 Ferrell, W.R. 339, 340, 471, 472, 476, 478,479,480,490 liestinger, L. 350,368, 385 Filippov, V.A. 144 Fidler, E.J. 337,431,434,442 Fishburn, P.C. 134,142 Fischer,G.W. 137,142, 304,318 Fischhoff, B. 37,57,67,74,88,131,136, 137, 142, 143, 144, 146, 147, 152, 157, 164, 166, 218, 220, 303, 319, 339, 346, 348, 368, 371, 382,417, 428, 429, 430, 471, 472, 4 7 9 , 4 8 3 ,

547 490, 493, 503, 504, 5 0 7 , 5 1 7 , 520, 521,523,525,534,539,544,545 Fishbein, M. 387,400 Fisher, S.D. 2 1 2 , 2 2 0 , 2 2 4 , 2 3 5 Fiss, C. 119 Fleiss, J.L. 453, 455 Ford, D. 119 Fox, J. 524,525 Frankfurt, H. 461,468 Franks, J.S. 220 . Freud,S. 157,164 Fryback, D. 1 1 9 , 1 8 0 , 5 1 4 Fryback, J. 119 Fustos, L. 185,195 Galbraith, J.K. I64 Gay, J . 540,544 Girdenfors, P. 458, 4 5 9 , 4 6 0 , 461,468, 469 Gerds, U. 134, 142 Gershuny J.1. 130, 142 Gettys,C.F. 2 1 2 , 2 2 0 , 2 2 4 , 2 3 5 Glass, A.L. 268, 278 Click, J.A. 540,544 Gointein, B. 368 Goldsmith, R.W 339, 340, 457, 460, 469 Good, I.J. 4 5 8 , 4 6 0 , 4 6 9 Goodman, B.C. 1180 Graesser, A.C. 215,217,220 Granovskaya, K.M.136,142 Green, D.M. 4 7 6 , 4 7 7 , 4 9 0 Green, P.E. 288,298 Greenberg, G.Z. 475,490 Greenwald, A.C. 385,400 Greist, J.H. 180 Grey, D.R. 4 7 6 , 4 9 0 Gruemm, H. 93, 103 Gustafson, D.H. 1 8 , 20, 105, 106, 107, 119,177, 180 Gustafson, D.J. 333 Guttentag, M. 119 Hackman, J .R. 165 Hagman, D. 8 7 , 8 8 Hall, J . 107, 119 Hallden, S. 461,469 Hammond,K.R. 1 6 4 , 5 4 1 , 5 4 4 Handy,C.B. 149, 154,156,164

Hardy, A. 492,504 Harrison, F.L. 535,544 Harvie, R. 492,504 Hausmann, L. 1 6 5 , 2 0 0 , 2 2 3 Havens, J. 4 7 , 6 8 Hays, W.L. l 8 0 , 5 1 0 , 5 15 Hederstierna, A . 461,469 Heerboth, J. 212, 221, 224, 225, 226, 236,304,319 Heerboth, M.T 254,259,278 Helson, A . 356, 368 Henderson, D. 67 Heuston, M.C. 127, 142 Hickson, D.J. 148, 165 Hiles, M. 119 Hintzman, D.L. 493,504 Hoessel, W. 333 Hofstadter, D.R. 218,220 Hogarth, R.M. 133, 138, 142, 155,164, 205, 206, 214, 220, 224, 225, 235, 413,415,489,490,538,544 Holling, C.B. 74,88 Holsapple, C.W. 268,277 Holyoak, K.J. 268,278 Hoos, I. 133, 142 Hoos, J.R. 130,133,136, 142 Hosseini, J . 476, 490 Howard,R.A. 2 4 , 3 7 , 2 5 4 , 2 7 8 , 5 1 4 Huber, G. 119 Huber, G.P. 323, 333 Huber, 0. 339,353, 363, 368,428,429, 443,454,455 Huff, M. 282,299 Humphreys, P.C. 15, 66, 68, 84, 88, 155, 159, 164, 176, 180, 184, 185, !94, 195, 199, 201, 202, 205, 210, 211, 213, 219, 220, 253, 259, 260, 269, 270, 277, 278, 282, 284, 291, 292, 298, 301, 302, 304, 308, 309, 319, 322, 324, 325, 333, 334, 337, 350, 368, 518, 525, 528, 530, 534, 544 Hyman, R . 234,235 leromin, S. 142 Jackson, J . 72,88. Jacoby, J . 372,381 Janis, I.L. 205, 220, 350, 357, 362, 368,384,385,400

548 Jaques, E. 148,165,534,544 Kulkalni, R. 68 Jaus, D. 298,333 Kunreuther, H. 17,21,42,68,69,70, Jeffrey, R.C. 461,469 72,74,75,85,88,158,165 John, R.S. 171, 180, 184, 195, 202,

269,278,284,298,301 Johnson, D.M. 323,333 Johnson, E.J. 353,368 Johnson, E.M. 323,333 Johnson, R.M. 295,299 Johnson-Lenz, P. 291,299 Johnson-Lenz, T. 299 Jones, S. 164 Jones-Lee, M. 24,37 Jotwani, P. 119 Juedes,G. 119 Jungermann, H. 134, 143, 159, 160,

Lacan, J. 268,278 Lahiff, M. 180,514 Lakatos, 1.518,525 Lansley, P. 165 Larichev, 0.1. 123,125, 136,137,138,

139,140,142,144,185,195 Larson, J.R. 459,463,467,469 Lathani, G.P. 234,236 Lathrop, J.W.16,39,42,55,68,70,74,

75,80,82,84,88

Laughren, T. 180 165, 200,205,210, 220,223, 224, Lave, L.V. 138,142 Lawler, E.E. 165 235,291,299,522,525 Lawless, J. 74,88 Leal, A . 199,221,225,236,254,278, Kabus, 1,538,544 299,302,319 Lee,W.461,469 Kahne.S. 185,195 Kahneman, D. 73, 88, 137, 144,177, Levi, 1.460.469 180,218,221,226,234,236,330, Levine, M.E. 73,88,150,165 334, 346,353,368,428,429,430, Lewin,K. 161,384,389,390,400 492,493,504,510,514,515,517, Lichtenstein, S . 37, 67, 74, 88, 137,

525,529,530,533, 540,542,544, 139,143, 144, 166,220,339,371, 382,423,427,429,430,439,442, 545 Karmarkar, U S . 463,467,469 471,472,479,483,490,525,534, Katz, M. 21 1,220 539,544,545 Lichtenstein, W.350,368 Keating, G.W. 298 Keeney, R.L. 15,46,23,24,30,31,32, Light, L.L. 212,220 36,37,66,67,68,70,88,134,136, Lindley, D.V. 458,468,529,544 143,210, 220,225,236,254,269, Lindman, H. 367 278,309,319,322,323,333,525 Linncrooth, J. 16, 24, 37,39, 42,68, 70,75,84,88 Killworth, P.D. 493,504 Lock, A . R . 124, 145, 147, 151, 156, King, D.C. 107,119 165 Klayman, J. 338, 401, 409,413,415, Locke, E . A . 234,236 428 Loftus, E.F. 226,235 Klee, A.J. 137,143 Loop, J.W. 180,514 Kleinmuntz, B. 368 Lopes, L.L. 207,220,373,381 Kleinmuntz, D . N . 368 Luchins, A.S. 213,220 Kneppreth, N.P. 322,333 Lusted, L.B. 169,180,513,514 Knop, H.173,180 Kochen, M. 298,299 Koestler, A. 492,504 MacBride, J.N. 295,299 Kopetsky, E. 105 Macco, A. 105 Kozhukharov, A.N. 140,143 MacCrimmon, K.R. 162,165,323,333 Kroh-Puschl, E. 142 Magill, S.M.59,68 Kuhn, T. S. 528,544

54 9 MacCregor, D. 340.491 MacMillan, 1.C. 153. 165 MacPhillamy, D. 138, 144, 353, 369, 445,451,456 Maheshwari, A . 298 Majone, G. 45, 60. 6 2 , 68, 133, 135, 143,167,180,181 Majone, N. 7 0 , 8 4 , 8 8 Makridakis, S. 164 Mandl, C. 5 5 , 6 8 , 8 L , 84,88 Mann, L. 205, 220. 350,356, 362,368, 384, 385,400 March, J.G. 73, 74. 89, 152, 153, 154, 164, 165 Marlatt, G.A. 385,400 Marschak, J. 137, 143,429 Martin, E. 519,525 Martino, J.P. 493,504 Matheson, J. 254,278,539,545 Mazur, A . 169,180 McClelland, G.H. 541,544 McCoach, W. 322,325, 333 McCnmmon, K.R. 137,143 McFadden, W. 84, 88, 155, 159, 176, 180, 184, 194, 195, 210, 219, 220, 269, 278, 282, 292, 298, 304, 308, 319,322, 333, 350, 368 McGoey, P.J. 471, 472, 478, 479,480, 490 McGuire, W. 387,400 McKelvey, R.D. 439,442 McNamara, D.E. 107,119 Meadows, D.H.132,143 Meadows, D.L. 132,143 Mellor, D.H. 461,469 Menke, M.M. 538,544 Miles, D. 176. 180 Miles, R.F. 134, 142 Mills,C.W. 519,525 Minin, V.A. 142 Minsky, M.A. 208,221 Mintzberg, H. 150, 165 Misczynski, D. 87, 88 Mitchell, R.C. 72,88 Mokken, R.J. 292,299 Montgomery, H. 138, 139, 143, 254, 278, 328, 333, 337, 338, 343, 344, 346, 347, 350, 367, 368, 371, 372, 373, 378, 380, 381, 384, 385, 387,

399, 4 0 0 , 4 0 2 , 4 1 1 , 4 1 5 , 4 2 6 , 429, 4 30 Moore, L. 85, 88 Morgan, B.B. 206, 219 Morgan, B.J.T. 476,490 Morgcnstern, 0. 25, 37 Moshkovich, H.M. 142,143 Mulkay, M . 6 1 , 6 8 Mumpowcr, J. 541,544 Murphy, A.H. 513,515 Nagara, H. 268,278 Nagy, G. 444,456 Nair, K. 68,322, 333 Nappelbaum, E.L. 140,142 Neave, E.H. 165 Neisscr, U. 165 Neniiroff, P.M. 107,119 Newell, A . 207, 221, 234, 236, 381, 442 Nisbett, K. 165, 499, 503, 504, 530, 540,545 Nisbett, R.E. 371, 380, 381, 4 1 7 , 4 3 0 , 441,442 Nord, W.R. 165 Norman, D.A.208,219,444,455,456 Novick, D. 129,143 0gawa.G. 127,142 O'Hare, M. 72, 87 Okrent, D. 37 Olsen, J. 7 4 , 8 8 Olshavsky, R.W. 371,372,381 Olson, M. 72,88 Olsson, G. 385,400 Optner, S.L. 135, 143 Ortendahl, M. 361,368 Osgood, C.E. 376, 382 Oskamp, S . 214,221 Otway, H. 6 7 , 6 8 Parducci, A. 356, 368 Passmore, W. 119 Parkes, C.M. 325,333 Payne, J.W. 350, 353, 354, 368, 371, 372, 382, 402, 403, 408, 4 0 9 , 4 1 0 , 413, 415, 417, 427, 430, 4 3 1 , 4 4 1 , 442,455

550 Pearl, J . 199. 205, 211, 212, 216, 221, 225, 236, 254, 278, 282, 299, 302, 319 Pctersen, E J I . 165 Peterson, C.R. 274, 277 Peterson, K. 105 Petrov, A. 243, 252 Pettigrew,A.M. 149, 151, 165 Pfeffer, J . 150, 165 Phillips, L.D. 143, 146, 147, 148, 165. 179, 180, 259, 278, 301, 333, 339, 367, 471, 472.490. 508, 510, 515, 5 2 1 , 528, 534, 535, 538, 542. 544, 545 Pitz, G.1:. 199, 205, 2 0 8 , 2 1 2 , 2 1 3 , 2 1 5 , 221, 224, 225, 226, 236, 254, 259, 267, 278, 302, 304, 319, 323, 333, 538,541,542,545 Plott, C.R. 7 3 , 8 8 , 1 5 0 , 1 6 5 Pollack, I. 4 7 3 , 4 9 0 Pollard, P. 535,544 Polson, P.G. 216, 219 Polyakov,O.A. 137. 143 Porter, L.W. 1 5 4 , 1 6 1 , 1 6 2 , 1 6 5 Poulton, E.C.4 8 9 , 4 9 0 ~ug11,D.S. 1 4 8 , 1 6 5 Quade, E.S. 70, 88, 127, 128,129,132, 1 3 3 , 1 3 5 , 1 4 3 , 1 6 7 , 180 Quillian, M.R. 366, 367

Raiffa, H. 1 3 4 , 1 3 6 , 1 3 8 , 1 4 3 , 2 1 0 , 2 2 0 , 221, 225, 236, 254, 269, 278, 309, 319, 322, 323, 333, 514, 515, 539, 545 Ranyard, R. 339, 417, 418, 421, 426, 427,430 K ~ O ,V.R. 298 Ravetz, H. 6 2 , 6 8 Ravetz, J . 1 6 7 , 1 8 1 Reitman, J.S. 216, 221 Richards, M.D. 153,165 Rios, M. 5 5 , 6 8 Rips, L.J. 214, 221 Rittel, H. 130, 133, 143 Rivett, P. 137, 143 Roberts, H.V. 1 8 0 , 5 1 4 Robertson, S.P. 215, 220 Ronis, D.L. 385.400

Rose, J. 119 Rosen,L.D. 136, 137,144.372, 382 Rosennian, M . I . 1 4 0 , 1 4 3 Ross, L. 165. 499, 503, 504, 530, 540, 545 Rossineissl, J . 105 Roy, B . 137, 144 Rueter, M . R . 216, 221 Rugg, T. 282,299 Rubso, J.E. 136, 137, 138, 144, 353, 363,368,372,382

Saari, L.M. 234,236 Sdchs, N.J. 212, 215, 221, 224, 225, 226, 236, 2 5 4 , 2 5 9 , 2 7 8 , 3 0 4 , 3 1 9 Sadler, P. 165 Sagaria, S.D. 492,504 Sahlin, N . - E . 339, 340,457, 458, 459, 460,461,468,469 Salancik, G.R. 166 Saleh, N.J. 199, 221. 225, 236, 254, 278, 299,302. 319 Samet, M.G. 303, 318 Sattath, S. 215.221 Savage, L.J. 25, 37, 260, 278, 323, 333, 457,469 Sawyer, J., Jr. 67 Sayeki, Y. 3 2 2 , 3 3 3 Schank, R.C. 213, 221 Scher, J.M. 299 Schlesinger, J . R . 131, 132, 144 Schneider, W. 208, 221 Schoemaker, P.J. 534,535, 5 4 2 , 5 4 5 Schulman, A.F. 475,490 Schum,D.A. 1 7 9 , 1 8 1 Scott Morton, M . 282,299 Seale, D.L. 1 8 0 , 5 1 4 ,Sewer, D.A. 105, 120, 168, 169, 181 Seery, J . 260,262,278 Shanteau, J . 4 4 4 , 4 5 6 Shapka, Z. 3 3 1 , 3 3 4 , 3 6 8 Sharp, D.W. 5 4 0 , 5 4 4 Shaw, K.N. 234,236 Shaw, M.L.G. 297,299 Shea, T.E. 9 5 , 9 6 , 1 0 2 , 103 Sheptalova, L.P. 1 4 3 Shiffrin, R.M. 208, 221 Shinners, S . M . 127, 144 Shneiderman, B. 282,299

551 Shoben, E.J. 221 Todhunter. I. 536, 545 Shuford, E.H. 260,278 Torgerson, W.S. 185, 195 Shuman, J . 136,144 Townes, B.D. 298 Simon, H.A. 147, 165, 208, 220, 234, Tuggle, F.D. 538,545 236, 354, 369, 371, 381, 4 1 7 , 4 2 9 , Turner, C. 519,525 431,432,440,442 Turoff, M. 282,291,299 Sims, D. 164 Tversky,A. 73, 88, 136, 137, 138,144, Sin, J.K. 137, 143 177, 180, 215, 218, 221, 226, 234, Sjoberg, L. 63, 68, 259, 278, 331, 334, 236, 261, 278, 328, 330, 333, 334, 338, 350, 357, 361, 362, 368, 369, 343, 346, 350, 353, 361, 363, 368, 383, 385, 386, 387, 388, 392, 399, 369, 403, 415, 418, 428, 429, 430, 400,523,525 442, 458, 469, 492, 493, 504,510, Skyrms, 8 . 4 5 8 . 4 6 9 514, 515, 517, 525, 529, 5 3 0 , 5 3 3 , Slesin, L. 32, 37 540,542,544,545 Slovic, P. 32, 33, 37, 67, 74, 88, 133, Tyszka, T . 3 5 4 , 3 6 9 , 3 7 2 , 3 8 2 , 4 5 3 , 4 5 5 137, 138, 144, 166, 220, 343, 350, 353, 355, 368, 369, 371, 382,417, 423, 427, 429, 430, 439, 4 4 2 , 4 4 5 , Ulvila, J.W. 18, 19, 20, 2 1 , 9 1 , 93, 9 5 , 451, 456, 458, 469, 517, 5 2 5 , 5 3 9 , 96,103,172,180,481 545 Uyterhoeven, H.E.R. 155,166 Smelser, P. 119 Smith, E.E. 221, 268,279 Smith, M. 339, 340.471 van Houten, H.J. 293, 298 Snapper, K. 105, 119, 120, 168, 169, van Koningsveld, R. 105 VLi, A. 124, 171, 181, 183, 184, 185, 181 Stael von Holstein, C-AS. 533, 539, 195,199 Vaupel, J.W. 6 2 , 6 8 545 Sta1ker.G.M. 148,164 Vecsenyi, J . 123, 124, 171, 181, 183, Starr, C. 33. 38 184,185,195 Stantchev, I. 200,237 Vickers, J.M. 461,469 Staw, B.M. 1 5 2 , 1 6 6 Villani, C. 298,333 Steele, J.P. 180, 514 Vlek, Ch. 224,225, 236 Stern, L.D. 493,504 von Ulardt, 1.165,200,223 Strauss, F.F. 180 von Neumann, J . 25, 37 Suci, G.J. 376, 382 von Winterfeldt, D. 5 5 , 6 7 , 6 8 , 105.120, Svenson. 0. 254, 279, 337, 338, 343, 123, 124, 137, 144, 167, 170, 171, 344, 347, 350, 353, 354, 359, 361, 180, 181, 186, 195, 202, 210, 217, 368, 369, 371, 372, 373, 374, 375, 221, 224, 236, 270, 278, 297, 298, 380, 381, 382, 402, 411, 4 1 3 , 4 1 5 , 299, 301, 309, 319, 322, 323, 325, 418, 428, 430, 431, 442, 4 4 5 , 4 5 3 , 334, 454,456 Vroom, V.H. 166 Swets, J.A. 476,477.490

Tannenbaum, P.H. 376,382 Taylor, R.W. 162,165 Thomas, K. 166 Thornbury, J.R. 180,514 Thurow, L.C. 140,144 Toda, M. 225, 236, 260, 279, 362,369

Wagenaar, W.A. 224, 225, 236, 492, 504,507,533,544 Wagner, H.M. 126, 144 Walker, J. 74,88 Wallace, D.L. 1 8 0 , 5 1 4 Watson, S.R. 152,166 Watson, W. 107, 119

552 Webb, T. 165 Webber, M. 1 3 0 , 1 3 3 , 1 4 3 Wcgcner, M. 159, 163, 284, 291, 298, 328 Wehrung, D.A. 323, 333 Weick,K.E. 152,155,161,166 Weiss, C.H. 152, 166 Weiss, J.J. 205, 207, 211, 218, 221, 322,334 Wickelgren, W.A. 366,369 Wildavsky, A. 87.88 Wilks, Y . 215, 221,226,236 Wilson, J.Q. 7 2 , 8 8 Wilson, T.D. 371, 380, 381, 417, 430, 441,442 Winkler, R.L. 513,515 Winston, A.B. 268, 277 Wise, J.A. 226, 235, 384, 400, 458, 468 Wisudha, A. 199, 202, 205, 211, 213, 220, 269, 218, 284, 298, 302, 308, 309, 319 Wooler,S. 203,259,270,218,282,284, 298, 321, 325, 326, 333, 334, 528, 532,544,545

Wright, G.N. 535,541,545 Wright. P. 354, 369 Wynne, B. 6 1 , 6 8 Yates, P. 541,545 Yates, J.F. 458,469 Yetton, P.W. 166 Young, F.260.262,278 Young, S . 127,144

Zadeh, L.A. 2 3 9 , 2 4 1 , 2 4 3 , 2 5 2 Zajonc, R.B. 357, 358,369 Zaltman, G. 161.166 Zaus, M. 142 Zavoina, W. 439,442 Zeckhauser, R. 2 4 , 2 6 , 3 7 Zeleny, M. 324,334 Zink, D. 350,368 Zins, M.A. 367 Zuev, Ju.A. 139, 144 Zukowsky, L.G. 458,469

SUBJECT INDEX

compromises in 195 Absolute judgment see Judgment Action development of 168,177 justification of 259 Analytic bias, false approach t o 61 Anomaly in nuclear safeguards analysis quality of 228 definition of 97 representation of 224 observation of 92 Actions, packages of 229 Aspiration value 238. 243 Activities, hierarchy of 255 Attributes Actuarial data, mistaken use of 538 attractiveness scale for 372 Actuarial standards, difficult t o develop collapsing of 363,373 539 de-emphasising importance of 360 Additive rule see Decision rules defining of 225 Affective reaction determining preferdesirahle vs. undesirable 268 ences 357 development of within decision Agenda setting 73 analysis 187 Allais paradox reformulated 531 elicitation of, procedurc 21 1 , 284, Alternatives see also Options 296,308 comparison o f , in risk assessments 66 generation of 21 3 discarding of 354 hierarchy of 185 finding a promising one 355,372 independence of 202 holistic ratings of 305 independence, test of 284 limitations of techniques for cornparnumber of, and reliability 187 ison of 1 3 1 reference images for 333 rejection of 399 relative importance of 85, 269, 310, s c r c c n i g of 354 322,353 Alternatives. multi-attributed see also salience of 265 Multi-Attribute Utility Analysis weighting of 185, 203, 308,439 acceptance of 202 Attribute sets, judgments of completedecision rules for choosing betwcen ness, independence and quality 306, 34 3 311, 315 Ambiguity in organizational decision Availability of information see Informaking 1 5 5 mation Analyst-clicnt relationship Availability heuristic see Heuristics

36

554

Background knowledge see Knowledge Base rate effects 480 Basic Reference Lottery Ticket (BRLT) see Lottery methods Bayesian model for suicide attempts 117 Bayesian inference structure 178 Belief-vdlue correlation 384 Biases in probabilistic thinking 507 focus on 509 likened to error smres 539 part and parcel of science 6 1 premature conclusions 533 Bidding and choice, comparison between 427 Bidding task procedure 339.419 Binary choice procedure 339,420 Bolstering support for an alternative 361,373 Bootstrapping proccsy 21 2 Calibration of subjectivc probabilities 471,534,538 curve for 477,484 hard and easy questions 483 model 472 Cancellation of differences on attributes 363, 373 Capacity theory 534 Care of patients, long term 108 Care, quality of see also Quality Assurance Process index of 108 monitoringof 19,106,113 Multi-Attribute Utility Model 108, 115 Catastrophes avoidance of 16, 32 calculation of probabilities of 4 8 deaths in 81 insurance as protection against 86 probabilities plotted against fa talities 51 risk measures sensitive to 65 Causal path models, use of 398 Causally directed knowledge 226 Change agents, effective 113 Change model, planned 160 CUent-analyst relationship see Analystclient relationship

Clinical judgment compared with Bayesian model 178 Clinical skills in decision analysis 157 Coalitions in collective decision making 176 in organizations 154 Cognitive deficiency, problem of 540 Cognitive dissonance. theory of 385. 397 Cognitive goal hierarchy 227 Cognitive operations, availability of 45 5 Cognitive reprcsentation of decision problems 6ee Decision problem representation Cognitive script 21 3, 358, 444 Coincidences experienced by self and others 341, 496,499 surprisingness of 340,491 Collapse circuit 250 (:ollective decision making see Social decision making Compensation payments 8 7 Compensatory decision rules see Decision rules Compensatory thinking 363 Computerized decision aids see Decision aids Computers, choice between 254 Conflict antecedents of 395 and differentiation between and within options 386 motivation t o avoid 384 in organizational decision making 162 Conflicting evidence, forgetting of 366 Consciousness raising 189, 202,291 Conservatism in probabilistic inference 510 Consistency, difficulties in demonstrating 218 Context effects see Decision problem I epresent at ion Contingent regulation of quality of care 107,119 Convergence among intuitive and prescribed preferences 291, 304, 312, 329

555

increase over time 31 3 measures of 3 12 Convergent validity 433 salience ineasures 441 Correlation between information dimcn. sions 449 Cost-benefit analysis 127, 199 focus on outcomes, rather than process 84 Cost-effectiveness technique I27 ill-suited to strategic choice problems 131 uselessness of constructing objective models 130 Court awards, use of, to evaluate mortality risks 27 Criteria, elicitation of 159 Cultural setting, influence on attributes considered 85 Culture, organizational 149

Database for decision structuring 207 structured 268 Decision aiding ability, improvement of 235 Decision aids, computerized see EVAL, CENTREE, GODDESS, MAUD, OPINT, and QVAL ease of interaction 288 evaluation of 202,284 helping to find a dominance structure 368 human engineering of 206 improvement of 283 interactive 199, 282, 302 perceived applicability 290 user satisfaction with 202, 312, 314 Decision analysis art of 302 comparison with psychoanalysis 146 compromise between decision maker and analyst 195 development of attributes within 186 developing clinical skills 157 formal models of 205 goals of 18,257 and ideology 147 impact of judgment research on 524

36*

implementation of 157, 160, 167 importance of awareness of organizational context 146 institutional constraints on 173 personalist 91 pitfalls of see Pitfalls of analysis potential as an evaluation methodology 105 reasons for rejection 303 task taxonomy for 514 use of prescriptions to justify decisions 276 valuation of 152, 219 Decision analytic model for inspection of activities 96 selection of 210 Decision behavior, improvement of 235 Decision conferences 532 Decision criterion, location of 477 Decision latency 437 Decision marking compared with systems analysis 133 defective avoidance in 362 distinction from judgment blurred 5 24 levels of abstraction in see Levels of abstract ion multi-criteria see Multi-criteria decision making personal 303 psychological and sociological factors in 141 Decision making process automation of 207 control of 150 directionality of 357 missing information 434,445 model for 341.380 rounds in see Sequential decision making process phases in 352 Decision making research paradigms 528 Decision making strategies development of 184,409 with missing information 445 as points in multidimensional space 41 1 repertoire of 402 selection of 425,454 Decision making systems, hierarchical 186

556 Decision objectives see Objectives Decision option see Options Decision problem representation 200, 224 cognitive 216, 372,444 context effects 542 differences between subjects 530 editing phase in 353 methods for eliciting 234 preferences for incompleteness 186 prototypical structures for 210 role of the goal in 2 2 3 , 237 syntax for 21 7 testing against the environment 541 veridicality of 530 Decision problem structure see Structuring decision problems Decision problem taxonomy see Taxonomy Decision process, sequential see Sequential decision making process Decision rules abstract vs. concrete 349 additive difference 402 additive MAUT model 189,309 compensatory, application of 373 compensatory vs. noncornpensatory 297,346,365,402 complex 402 conjunctive 345,373,402,410 disjunctive 345 dominance 344 elimination by aspects 345,403,410 expected utility 344,533 formulated by subjects 422 integrating qualitative and quantitative factors I39 lexicographic 345,407,410 limited applicability of 347 non-compensatory 346, 380 problem of finding appropriate ones 184 subjective nature of 130 validity of 350 Decision structure see Structuring decision problems and Requisite decision structure Decision support system 268,282 Decision strategy see Decision making strategies

Decision theoretic approach lo program evaluation 105 Decision thcory 15 application of 235 goal concept in 225 Decision variable partitioning of 473 relation to propositions 474 Defensive avoidance of stress see Stress Deregulation, e r ; ~ of 1 I8 Detection, probability of, in diversion path analysis 20, 94 Detection of problems 116 Directed graphs 246 Discriminability parameter (d’) 474 Discrimination proccss, probabilistic 473 Diversion paths in nuclear safcguards analysis 19, 93, 99 Dominance structure hierarchical 365 search for 360,372,387 unrelated to thought concerns 392 Dominance testing 352 Drawing pin probleni 536

Ecological validity of research o n probabilistic thinking 507 backing in process models 508 Editing operations 353 Effective change agents 1 1 3 ELECTRE technique 137 Elicitation o f attributes see Attributes of options see Options Elimination by aspects see Decision rules Ellsberg Paradox 458 Emergence of decisions 384 Empirical findings, unrepresentative of tasks and subjects 528 Environmental effects, analysis of 105 Environmental impact report 44 Epistemic risk model 462 Equitable distribution of risk 31 Error, focus on, in research 510,524 Errors reduced by group discussion 540 EVAL 302 Evaluation methodology, use o f 105

557 Evaluation of dccision aids see Decision aids of human judgment 522 of mortality risks 28, 35 Event, exogenous, role in triggering social interest 73, 82 low probability 73 Evidence, rule\ of, in risk assessments 17,64 Expected Utility see Subjective expected utility Expected utility analysis 210,531 Expected utility theory influence on decision making 135 interpretation of violations of 538 neither normative nor prescriptive 543 Experts advocacy role of 17 involvement in decision analysis 175 Exploring decision problems 159 Facility siting, Liquefied Natural Gas see Liquefied Natural Gas powcr plant 173 sequential decision making in 4 3 Facts vs. judgment in risk assessments 61 Fatalities see also Risk analysis confused measures of 56 factors to be taken into consideration 2 3 , 6 5 "good" and "bad" kinds 34 indices of 49 use in risk indices 15 Flexibility analysis 200, 276 Forgetting of conflicting evidence 366 F+aming of decisions 428 of goals 226 Functional fixity 213, 267, 276 Fuzzy set 239 Fuzzy structural modeling 2 0 0 , 2 3 7 Games played by decision makers 176 General system\ approach see Systems analysis Generalizations from laboratory tasks 511,520,535 problems of 539

C;eneralizations froni research on heuristics and biases 528 Generation of options see Option generation Generation ofoutcomcs 225 Generation paradigm 540 Generic problem structure 318 see also Structuring decision problems GENTREE 201 GERT see Graphic Evaluation and Revicw Technique Goal achievement, potcntial 232 Goal concept, activation of 200, 228 Goal confusion 184 Goal directcd behavior contrastcd with policy making process 45 Coal directed decision structuring system sce GODDESS Goal driven approach to decision making 225 Goal hierarchy 160, 1 8 7 , 2 0 0 , 2 2 9 , 2 3 3 means and ends mixed I90 Goal indicators 238 Goal-oriented knowledge 215, 226 Goal specification, problems of 130 Goal structure, organizational 153 Goal structuring 159, 302 Goal system 238 Goals analysis of, in R & D decisions 187 associated with phases in the decision process 350 assumed to be clearly definable 147 cognitive hierarchy of see Goal hierarchy conflicting in social decision making 63 context-bound 225 differences between analyst and client 15 7 elicitation of 159, 211 explicitness (General activation hypothesis) 231 formal, context free 225 furthered through coalitions 153 importance of 233 meaning of 224 and objectives of parties with different interests 83 related to personal preferences 228, 233

558 of sponsor 158 value of 228 GODDESS 199,225 procedures contrasted with MAUD 216,302 Good configuration 264 Graphic Evaluation and Review Technique (GERT) 18,85 Group discussion, role in reducing errors 540 Group process, integrative 107 Guttman scaling 291

Information availability in memory 455 defensive restructuring of 386 epistemic reliability of 460 introduced from memory 444,452

missing,roleof434,445,453

neglect of, in noncompensatory rules 347 about preferences (non-spatial) 215 presented, interpretation of 454 separated from judgment 61 storage in memory 208 substitution hypothesis 447 utilization of 444 Heuristics, availability and representa- Information boards 402 tion of knowledge 73,234 Information gathering patterns 402 classification vs. computation 423 Information integration, minimization of employed by decision makers 185 215 evaluation of research on 518 Information processing approach to generation of 224 decision research 527,536 identification of 524 Information processing capacity in probabilistic thinking 507 increase of 136 Hierarchical decomposition of goals see. limitations of 135 Goal hicrarchy types of limitations 455 Human engineering tradition in psy- Information processing paradigm chology 5 12 production rule version 538 Information retrieval from mcmory see Memory IAEA see International Atomic Energy Information search patterns 338, 392, Agency 403 Ideal point, positioning of 284,308 and decision rules 35 1,403 Identifiable fatalities vs. statistical fataltask dependent charactcristics 407, ities 29 44 1 Ideology and decision analysis 147 Inspection activities 18 Imagination, role of in creating action allocation of time in 100,113, 116 224 effectiveness of 91 Importance weights see also Attributes timeliness of 97 and Tradeoffs Inspection histories 97 as ascaling parameters 31 1 Inspection resources, allocation of 95, Improvement of decision aids see Deci172 sion aids Institutional decision problems 22 Incoherence Insurance as a protection against catasinterpretation of 538 trophes see Catastrophes of preference structures 184 use of, for evaluating mortality risks Inconsistencies, detection of 218,436 27 Individual differences, importance of Integrative group process 107 539 Intellectual tasks, taxonomy of 51 1 Inference structure, Bayesian 178 Influence, degree of between goal and Interactive decision aids see Decision aids indicators 241

559 Interest groups characteristics of 1 6 , 4 0 , 7 1 , 1 5 8 conflict between standpoints 130 differences in value structures 5 5 problem of different viewpoints 140 refusal to cooperate 173 International A,tomic Energy Agency 18,81,172 Interpretation of situations by subjects 518 Involvement of decision makers, and commitment 159 Judgment absolute vs. relative in tasks 423,425 need for in analysis 62 ordinal 439 Judgmental abilities, results of underestimation of 523 Judgmental heuristics see Heuristics Judgmental phenomena, robustness of 518

Judgment-free measurement 92 Justification of decisions 293, 337,343, 417,427 Knowledge background, effect of 452,455 causally oriented vs. goal oriented 215 goaldirected 226 in long-term memory see Memory non-spatial 215 theories of organization of 214 Knowledge base in memory 208 interfacing with 297 KYST 260 LNG see Liquefied Natural Gas Laboratory experiments on decision making, controversy over 507 Laboratory tasks, generalizations from 511 Language used in measurement techniques 138 Lateral thinking 268 Legitimacy of group membership 108

Levels of abstraction of alternatives 170 in dealing with decision problems 5 34 Limitations of information processing capacity see Information processing Linear judgment model 438 explanatory power of 439 Linguistic approach in describing goals 237 Linguistic approximation procedure 243 Linguistic variable 239 Liquefied Natural Gas, facility siting case studies 4 0 , 7 5 economic aspects of 80 environmental aspects of 80 risks of see Risk assessments worst case scenarios see Scenarios Long-term memory see Memory Lottery methods for attribute weighting 138,284, 308 difficulties with 31 7 Lotteries over mortality risk vectors 25

MAMP see Multi-Attribute Multi-Party model Management information systems 214 Mathematical programming techniques I27 MAUA see Multi-Attribute Utility Analysis MAUD 84, 176, 194, 199, 201, 211, 269,284,302 Procedures contrasted with GODDESS 216,302 MAUD3 compared with decision analyst 304 involvement of subjects with 316 peculiarities of 317 Maximum feasible utility of R & D projects 192 Meaning episodic 228 of goals 225 Means-end confusion 184 Means-end relationship between op tions, outcomes and goals 216 in goal hierarchy 189 in goal structuring 21 1

560 Measurement, judgment-free see Judgment Measurement problems in systems a n d ysis and decision making 137. 540 Measurement techniques, use of amenable language in 138 Medical services, evaluation of cffectiveness of 105 Membership function 242 Memory, limitations of 5 10 Memory, long-term, retrieval of information from 2 0 0 , 2 0 9 , 4 5 2 , 4 9 3 representation of knowledge in 208, 215, 226 use of models of 234 Memory, organization of 208 Memory, rote, study of 509 Memory, short-term limitations of 136, 207, 214 limits of, and compensatory rules 349 Modeling, prescriptive 9 2 Modeling, qualitative 107 Modeling, structural see Structuring decision problems Mortality risks 24 evaluation of 28, 35 utility function for 30. 35 Motivation of the client 170 sec also Goals Multi-Attribute MultiLParty model 17, 21, 7 0 , 7 5 focus o n process 84 interpretation of 80 lessons from 83 Multi-attribute risk aversion 309 Multiattribute Utility Analysis 210 characterizing discrete features in 215 comparison of alternatives in 185, 322 compared with other techniques 136 decision rules 373 focus o n outcomes rather than process 84 incorporating concerns of stakeholders 174 performed by computer vs. analyst 304 pitfalls of 183

role in social decision making 18, 84 self-reports of usefulness of 305 use in resource allocation problems 186 use by students 533 weighting of attributes see Attributes Multi-Attribute Utility model for quality of care 108 Multi- criteria decision making 128 see also Multiattribute Utility Analysis clustering of sub-criteria 186 problems of attribute weighting 137 Multi-criteria linear programming, manmachine techniques in 137

Negative evaluations, dominance of 380 Network analysis 127 Network model connecting goals and actions 200 of the representation of knowledge 233 Network theory, use of, in GERT 8 5 Non-conipensatory rules see Decision rules No n-proli fera t ion treaty 9 2 Normative models, lack of psychological validity 508 Nuclear material, diversion of 91 Nursing homes, quality of care in 106

Objective models built o n inadequate information 131 replacement by subjective models in systems analysis 135 Objective surrogate for subjective assessments 19, 94 Objectives conflicting 254 focussing o n 226 generation of 2 1 3 multiple 225 Objectivity, problem of 538, 5 4 2 Operations research techniques, application of 126 Opinions, elicitation from knowledgeable people 114

56 1 Opinion leaders, use of 108 OPlNT 302 Optimal behavior, in weather forecasters 513 Optimality, understanding of 225 Optimization of procedures rather than outcomes 2 2 Option-anchored scales in multiattribute utility analysis 324 Option comparison, differences between systems and O.R. approaches 127 Option-driven approach to decision making 225 Option generation 200, 208, 224, 257, 30 2 role of the goal in 234 Option structuring technique 254,272 Options differentiation within and between 385 elicitation of 2 1 1 kept open 277 screening of 303,35 3 tradeoffs between see Tradeoffs Organizational culture 156 Organizational decision-making process 146,152 dealing with conflicts 162 procedural rules in 148 reconciling contradictions in goal hierarchies 189 Organizational structure and opportunities for decision analysis 147 Paradigms used in decision-making research 528 Personal construct theory 21 1 Personal decision making see Decision making Personalist decision analysis see Decision analysis PIP see Probabilistic information processing system Pitfalls of analysis 33, 61, 146, 167, 183 Planned change model 16 1 Policy-making process 5 8 , 6 0 characteristics of 45 introduction of competing expertise 65

Positivist view of science, challenged 61 Posterior probability of an event 474 Power in organizations 149 PPB system 129 Pre-editing operations 352, 354 see also Editing operations Preference independence 30 Preference reversals 53 1 Prescriptive modeling 92 Pressure groups see Interest groups Probabilistic discrimination see Discrimination process Probabilistic information processing system, failure to adopt 178 Probabilities second order 339,457 "true" 533 Probability assessments elicitation of 176 problems in research on 528 resistance in their use 5 1 1 Probability distributions, methods of encoding 5 39 Probability revision experiments 5 10 Problem, definition of, in organizational decision making 156 Problem detection in monitoring quality of care 116 Problem representation see Decision problem representation Problem solving creative 200, 208 descriptive models of 216 reasons for failure 2 14 Problem statements, incomplete 530 Problem translation 216 Process tracing analysis 337, 368, 371, 380,402,417 Process tracing tools 41 3 Production rules in information processing paradigm 5 38 relation to continuous quantities not clear 215 Proper scoring rules 533 Prospect theory contrasted with generation paradigm 542 Prototypical decision problem structures see Structuring decision problems

56 2 Psychophysical paradigm in decision making research 527,529 Psychosocial transitions 325 Public interest groups see Interest groups Qualitative factors measurement of 138 problcm of how to integrate them 139 Qualitative model see Modeling Quality Assurance Process (QAP) 20, 109,113 Quality control, statistical 106 Quality of care see Care QVAL 211

R & D see Research and devclopment Rare events, attribution of causality to 504 Rational procedures for problem analysis 140 Rationality, understanding of 225 Rationalization (postdecision) 60 Reasons for choice. formation of 339, 428 see also Justification of decisions Recidivism rate, for nursing homes 115 Regret attempts to decrease 399 in giving up alternatives 350, 366 Regulated industries, implications of deregulation 116 Regulation, contingent 20 Regulatory agencies, role of in decision making 4 2 , 7 2 Regulatory system 18,105 Relative judgment see Judgment Relevancy, requirement of 310 Reliability of evaluations, improvement of 186,189 of information 139 of screening process 11 2 Repertory grid procedure 2 I 1 , 284 problems with 286,296 Representativeness of intellectual tasks and performers 5 12 Requirements, structuring of 257 Requirements space 259 efficient partitioning of 264

Requisite decision structure 158, 160, 258,532,542 Research and dcvelopment problems, decision analysis of 129 Rcscarch and development projects, selection of 184 Resistance to change circuit 25 1 Rctrospectivc verbal reports see Verbal reports Risk acceptability of 57 equitable distribution of 16, 31 health 22 to individuals 25 institution perspective 25 lack of definition of, in risk assessments 55 mortality see Mortality risks multidimensional concept 64, 67 multiple causes of 33 public 23 of re-oprning an issue, counteraction of 400 safety 23 Risk analysis distinguished from decision analysis 41 generation of false expectations rcgarding 4 0 hidden agendas in 169 increase in polarization of arguments 63 modeling fatalities in 47 prescriptive 25 results viewed as evidence 40 use in LNG facility siting 40 use of utility analysis in 24 Risk assessments advocacy of 62 comparison of 52 conflicting 84 effectiveness of 64 as evidence 17 improvement throuph rules of evidence 64 omission of "unquantiliable" risks 51

parameters used in 54 purpose of 1 7 , 5 9 , 6 4 role in sequential decision making

563 process 47 scope of 5 4 subjectivity of 16 suspect standards 5 39 timing of 5 8 , 6 4 use of 1 6 , 5 3 , 6 0 , 8 4 value judgments in 17, 24, 3 5 , 8 3 viewed as facts rather than evidence 58,61 Risk aversion, multiattribute 309 Risk equity preference for 16 vs. catastrophe avoidance, preference for 32 Rounds in decision making see Sequential decision making process Rules, decision see Decision rules of evidence in risk assessments 1 7 , 6 4 , 70 procedural in organizational decision making 148 production see Production rules

Safeguards nuclear 92 aggregate measure of effectiveness 98 objective of 96 Safeguard system, vulnerabilities of 99 Satisficing, rules used for 4 1 1 Scenarios impact of changes in context on 86 stimulation of 21 3 use of 159 viability of 304 worst case 1 6 . 4 2 , 81 Schemata 2 1 2 , 2 5 4 Screening of decision optionsseeOptions Screening instrument for assehsing quality of care 1 1 1 Screening process 1 9 , 1 1 7 reliability of 1 12 Script, cognitive see Cognitive script Search for information see Information search Self-reference in systems 21 8 Self-report, use of in inspections 114 Semantic memory see Memory

Sensitivity analysis 154, 175, 277, 302 vs. strict optimization 160 Sequential decision-making process 6 3 , 73,82 roundsin 1 6 , 4 1 , 7 5 , 1 7 3 SEU see Subjective Expected Utility Severity measure of illness 105 Short-term memory see Memory Signal detection theory 473 Simple Multi-Attribute Rating Technique (SMART) 310, 317,326 Small worlds 258,277 SMART see Simple Multi-Attribute Rating Technique Snowball circuit 249 Social decision making 15, 18, 139 problem of applying systems analysis 130 use of multiattribute utility analysis 174 Stakeholders see Interest groups States of the world, generation of hypotheses about 224 Statistical quality control 106 Status quo, transformation of 234 Strategic choice problems 131 Stress defensive avoidance as response to 384 and directionality of decision making 351 Structural modeling, fuzzy 2 0 0 , 2 3 7 Structure of goals and subgoals 225 important role of 537 search for 344 Structuring decision problems 24, 159, 224,254,282,302,372,532 consequences of failure 542 database for 207 goal directed approach 2 1 1 , 3 18 pitfalls of 174 prototypical structures 210 restructuring to reduce contradictions 194 small scale problems 199 Subjective Expected Utility see also Expected utility theory applicability of model 344 compared with maximum feasible

564 utility 1 9 3 neglect o f , by decision makers 186 Subjective-objective probability functions 5 29 Subjective probabilities, calibration of see Calibration Substitution hypothesis see Information Surprisingness of coincidences see Coincidences Swrogate measures see Objective surrogate for subjective assessments Systems analysis application t o ill-structured problems 129 art of application 130 cost and benefit models in 128 current trends in 131 and decision making 126 general systems approach 127, 131 pitfalls of 133 psychological and sociological factors 141 qualitative and quantitative aspects of 129 Systems engineering 127 Task characteristics choice of decision rule dependent o n 402 complex nature of 542 effects of 4 4 1 , 5 3 4 Task performance, effects of goal setting o n 234 Task requiremcnts and reasons for choice 419 Task treated as an end in itself 522 Taxonomy of decision problems 210 of intellectual tasks 5 1 1 , s 39 Test theory paradigm in decision research 527,533 Think-aloud procedure 337, 350, 368, 371,436,455 Think-aloud protocol see Verbal protocols Tools for aiding decision making see also Decision aids absence o f , in laboratory experiments 522 use of 512

‘I’rddeoff ratios assumed fixed in Multi-Attribute Utility Analysis 269 Tradeofls see also Attributes, weighting of between attributes, difficulties with discrete features 215 between efficiency and equity in social decision making 8 6 between problcm formulations 1 7 2 between risks and costs 25, 27 between types of decision-making procedures 85 in modeling uncertainties in risk assessments 66 problems in making cxplicit 46 Tradeoff techniques 136 Transformation, topological 262 ’Truc” probability 5 33

Uncertainties involved in R & D project assessmcnt 192 modeled in risk assessments 6 5 Uncertainty ability to cope with 148 as an attribute describing conscquences 5 30 structuring of 254 Utilities, see also Expected utility theory and Subjective Expected Utility Maximum feasible 192 problems in the assessment of 176, 323 Utility analysis in risk analysis see Risk analysis Utility functions for mortality risks 30 for equitable risk vs. catastrophe avoidance 32 Utility independence 3 0 , 1 8 9 , 3 0 8

Value functions, assessment of 308 Value of a life, lack of consensus o n 24,30 Value judgments classification of 421 coniplexity of 347 in risk assessments see Risk assessments

565 Value structures, differences between interest groups see Interest groups Value systems insights provided by decision aids 219 only part made explicit 186 Value tree 169 Verbal protocols, analysis of 337, 371, 374,450,467 Verbal reports consistency of 3 3 8 , 4 3 2 , 4 3 7 dominance of negative evaluations in 380

reliability of 431 retrospective 4 3 1 , 4 3 6 validity of 431 Verbalization, effect of 4 3 2

Weighting of attributes see Attributes Wishful thinking 361 Worst case scenario see Scenario ZAPROS technique 139

This Page Intentionally Left Blank