A Guide for Uncertainty Analysis in Dose and Risk Assessments Related to Environmental Contamination (N C R P Report)

NCRP COMMENTARY No. 14

A GUIDE FOR UNCERTAINTY ANALYSIS IN DOSE AND RISK ASSESSMENTS RELATED TO ENVIRONMENTAL CONTAMINATION

Issued M a y 10, 1996

National Council on Radiation Protection and Measurements 791 0 Woodmont Avenue / Bethesda, MD 2081 4-3095

LEGAL NOTICE This Commentary was prepared by the National Council on Radiation Protection and Measurements (NCRP). The Council strives to provide accurate, complete and useful information in its commentaries. However, neither the NCRP, the members of NCRP, other persons contributing to or assisting in the preparation of this Commentary, nor any person acting on the behalf of any of these parties: (a) makes any warranty or representation, express or implied, with respect to the accuracy, completeness or usefulness of the information contained in this Commentary, or that the use of any information, method or process disclosed in this Commentary may not infringe on privately owned rights; or (b) assumes any liability with respect to the use of, or for damages resulting from the use of any information, method or process disclosed in this Commentary, under the Civil Rights Act of 1964, Section 701 et seq. as amended 42 U.S.C.Section 2000e et seq. (Title VZZ) or any other statutory or common law theory governing liability.

Library of Congress Cataloging-in-PublicationData

A guide for uncertainty analysis in dose and risk assessments related to environmental contamination. p. cm.-(NCRP commentary ; no. 14) "Issued May, 1996." Includes bibliographical references. ISBN 0-929600-52-5 1. Environmental risk assessment. 2. Radiation dosimetry. 3. Uncertainty. I. National Council on Radiation Protection and Measurements. 11. Series. RA566.27.G85 1996 96-14605 615.9'02'02874~20 CIP

Copyright O National Council on Radiation Protection and Measurements 1996 All rights reserved. This publication is protected by copyright. No part of this publication may be reproduced in any form or by any means, including photocopying, or utilized by any info11nation storage and retrieval system without written permission from the copyright owner, except for brief quotation in critical articles or reviews.

Preface The question of uncertainty in environmental dose calculations and the associated risks is essential in the decision-making process as it relates to radiation exposures that might result from radionuclides in the environment. Mathematical models are often used to extrapolate environmental information beyond the realm of direct observation. The reliability of mathematical models is relevant given the widespread use of these models in the performance of dose and risk assessments to evaluate environmental contamination. The National Council on Radiation Protection and Measurements (NCRP) addressed some of these issues in Commentary No. 8, Uncertainty in NCRP Screening Models Relating to Atmospheric Transport, Deposition and Uptake by Humans, and has a study underway focused on uncertainty in human metabolic models. The purpose of this Commentary is to answer three questions: (1)when should one perform an uncertainty analysis, (2) how should an uncertainty analysis be performed, and (3) how does one use the formal elicitation of expert knowledge in the absence of site-specific data? Example problems are given for both chemical and radioactive contamination. This material is being released as a Commentary to make it quickly available, but it is anticipated that it will subsequently be published as an NCRP report, followingthe review by the Council members that is necessary for publication in that form. This will also provide an opportunity for the Council's Collaborating Organizations to proffer comments on the document. The Council appreciates the financial support received from the U.S. Environmental Protection Agency and the Centers for Disease Control and Prevention in the preparation of this Commentary. Serving on NCRP Scientific Committee 64-17 for the preparation of this Commentary were: F. Owen Hoffman, Chairman SENES Oak Ridge, Inc. Oak Ridge, Tennessee

Members

David E. Burmaster

Eduard Hofer

Alceon Corporation Cambridge, Massachusetts

Gesellschaft fiir Reaktorsicherheit GmbH Garching, Germany

William J. Conover Texas Technological University Lubbock, Texas

Stephen C. Hora University of Hawaii Hilo, Hawaii

Richard 0 . Gilbert

Ronald G. Whitfield

Pacific Northwest National Laboratories Richland, Washington

Argonne National Laboratory Argonne, Illinois

iv

1

PREFACE

Consultants

Timothy M. B a r r y U. S. Environmental Protection Agency Washington, D.C. Michael A. Callahan U.S. Environmental Protection Agency Washington, D.C. Jana S. H a m m o n d s SENES Oak Ridge, Inc. Oak Ridge, Tennessee

Jon C. Helton Arizona State University Tempe, Arizona

M. G r a n g e r Morgan Carnegie Mellon University Pittsburgh, Pennsylvania William R. Rish Woodward-Clyde Consultants Cleveland, Ohio Kathleen M. Thiessen SENES Oak Ridge, Inc. Oak Ridge, Tennessee David Waite EBASCO Bellevue, Washington

NCRP Secretariat E. I v a n White, Senior StaffScientist Cindy L. O'Brien, Editorial Assistant The Council wishes to express its appreciation to the Committee members for the time and effort devoted to the preparation of this Commentary.

Charles B. Meinhold President

Contents Preface

.................................................................................................................................

.

1 Introduction .....................................................................................................................

. .............. 3. Methods for Uncertainty Analysis .............................................................................

2 Guidance on When to Perform a Quantitative Uncertainty Analysis

3.1 Limiting the Scope through Screening ..................................................................... 3.2 General Approach t o Uncertainty Analysis ..............................................................

3.3 3.4 3.5 3.6 3.7 3.8

3.2.1 Parameter Uncertainty Analysis ..................................................................... 3.2.2 Model Uncertainty ............................................................................................ Analytical Methods for Propagation of Uncertainty ................................................ Numerical Methods for Uncertainty Propagation ................................................... Alternative Methods for Sensitivity Analysis .......................................................... Guidance on the Selection of (Subjective) Probability Destributions for Uncertain Model Parameters .................................................................................... Interpreting the Result of an Uncertainty Analysis ................................................ Uncertainty Analysis for an Assessment Endpoint that is a Stochastic Variable .......................................................................................................................

.

4 Elicitation of Expert Judgment .................................................................................. 4.1 Meaning and Use of Expert Judgment ..................................................................... 4.2 Choosing Between Formal and Informal Elicitation Methods ................................ 4.3 Formal Elicitation Process ......................................................................................... 4.3.1 Selection of Issues and Associated Parameters ............................................ 4.3.2 Selection of Experts ....................................................................................... 4.3.3 Preparation of Background Information ....................................................... 4.3.4 Training of the Selected Experts in the Elicitation Method ....................... 4.3.5 Presentation of Issues, Questions and Background Information to

Experts ............................................................................................................. 4.3.6 Analysis of Issues, Questions and Background Information by Experts ....... 4.3.7 Discussion of Analyses of Issues and Background Information by Experts ........................................................................................................ 4.3.8 Elicitation of Judgment from the Experts ................................................... 4.3.9 Consolidation (and Possibly Aggregation) of the Elicitation Results ......... 4.3.10 Review and Communication of the Results of the Elicitations ...................

.

5 Summary and Conclusions

..........................................................................................

Appendix A. Equations Used to Calculate the Mean and Standard Deviation of Various Distributions Used in Analytical Error Propagation .......

vi

CONTENTS

References

.............................................................................................................................

48

The NCRP ..............................................................................................................................

53

NCRP Commentaries ........................................................................................................

54

1. Introduction Dose and risk assessments related to environmental contamination rely heavily on mathematical models to extrapolate information beyond the realm of direct observation. Projections into the future are made to support decisions about siting of potentially hazardous facilities, licensing of planned releases, and safeguarding against accidents. Projections into the past are made using incomplete records and data about historic operations and conditions leading to human exposure. Projections are also made in space and time for present day situations in which direct measurements are impractical or the levels in the environment are below the limits of detection. In most cases, these evaluations must be made when complete data for the site in question are unavailable. Whenever mathematical models are used, questions inevitably arise about the reliability of the results. To address these questions, the true but unknown values of dose and risk may be bounded through screening calculations. However, when the objective of the assessment is a "best estimate" value, uncertainty about this value should be disclosed. This uncertainty should be expressed as a subjective confidence interval within which there is a high probability of encompassing the true but unknown exposure, dose or health risk. The confidence interval is qualified as "subjective" because extensive use of judgment is required to quantify uncertainty in both the parameters and the structure of the model, particularly in the absence of relevant sitespecific data. Complete site-specific data sets are seldom available for environmental radiological assessments, and the use of judgment to quantify uncertainty is unavoidable. Incorporating uncertainty analysis into a dose or risk assessment provides an essential ingredient for decision making. An uncertainty analysis provides the assessor with a subjective probability distribution to represent the uncertainty in the assessment endpoint so that decisions can be made. The analytical and numerical methods of error propagation suggested in this Commentary also allow sensitivity analyses1to be performed to identify the parameters in model components that contribute most to the total uncertainty in the model result. By combining a sensitivity analysis with the quantitative uncertainty analysis, the relative contributions of each parameter, pathway and contaminant to the overall uncertainty of the assessment endpoint can be evaluated. Uncertainty analysis combined with sensitivity analysis allows the assessor to determine where efforts should be invested to improve the state of knowledge and thereby efficiently reduce the amount of uncertainty in the assessment endpoint. The purpose of this Commentary is to address key issues involved in performing and reviewing quantitative uncertainty analyses. This Commentary emphasizes the importance of rigorously defining the assessment endpoint, the need for expert judgment in the absence of relevant data sets, and the use of uncertainty analysis in an iterative mode to define model components warranting more detailed investigation. Guidance is given on when and how to perform an uncertainty analysis, and how to use formal elicitation of expert judgment in the absence of site-specific data to obtain the most defensible quantification of the current state of knowledge about critical parameters and model components.

'In this Commentary,the term "sensitivity analysis"refers to the identification of the relative contribution of the uncertainty in a given parameter or model component to the total uncertainty in the model result. In other sources, this process may also be referred to as an analysis of uncertainty importance (Morgan and Henrion, 1990).

2. Guidance on When to Perform a Quantitative Uncertainty Analysis Before beginning a project, most risk and dose assessment analysts ask: "Under what conditions is it necessary to perform an uncertainty analysi~?"~ In some circumstances, a formal assessment of uncertainty may not be necessary. These circumstances should be identified a t the outset of planning for an assessment of exposure, dose or risk. They include the following: 1. If conservatively biased screening calculations indicate that the risk from possible exposure is clearly below regulatory or risk levels of concern, a quantitative uncertainty analysis may not be necessary. Conservative screening calculations are designed to provide a risk estimate that is highly unlikely to underestimate the true dose or risk. Therefore, a more detailed analysis will likely demonstrate that the true risk is even less. 2. If the cost of an action required to reduce exposure is low, a quantitative uncertainty analysis on the dose or risk assessment might not be warranted. For example, for small contaminated sites with inexpensive options for remediation, it may make more sense to clean up the property than to undertake a detailed analysis of potential health risk and its attendant uncertainty. 3. If data for characterizing the nature and extent of contamination a t a site are inadequate t o permit even a bounding estimate (an upper and lower estimate of the expected value), a meaningful quantitative uncertainty analysis cannot be performed. Under these conditions, it may not be feasible to perform a n exposure or risk estimate, unless the assessment is restricted to a nonconservative screen that is designed to not overstate the true value. A nonconservative screen is obtained by eliminating all assumptions that are known to be biased towards an upper bound estimate. In this case, if the nonconservative screen suggests that potential exposures and risks will likely be above regulatory criteria, plans for remediation can begin even before the full extent of contamination is adequately characterized. Once the extent of contamination is more h l l y characterized, a quantitative uncertainty analysis would be useful in guiding costeffective measures for remediation. In contrast to the above situations, a quantitative uncertainty analysis of a dose or risk assessment will be justified in the following situations: 1. A quantitative uncertainty analysis should be performed when an erroneous result in

the dose or risk assessment may lead to large or unacceptable consequences. Such situations are likely to occur when the cost of regulatory or remedial action is high and the potential health risk associated with exposure is marginal. At some Superfund sites and associated military or weapons facilities, for example, the anticipated costs of cleanup may exceed many millions of dollars per facility. In the face of such large costs, it is useful to distinguish between those estimated risks which are truly high and deserve 2"Uncertaintyanalysis is the computation of the total uncertainty induced in the output by quantified uncertainty in the inputs and models, and the attributes of the relative importance ofthe input uncertainties in terms of their contributions" (Morgan and Henrion, 1990).

2. GUIDANCE ON WHEN TO PERFORM A QUANTITATlW UNCERTAINTY ANALYSIS

1

3

strong intervention from those which have been exaggerated due to the application of sets of compounded assumptions with a conservative bias. In cases where the estimate of exposure, dose or risk could lead unnecessarily to expenditures of large quantities of financial and human resources, the uncertainty in the estimate should be disclosed. If the uncertainty is unacceptably large, consideration should be given to an investment in gathering critical data to reduce uncertainty prior to making decisions about contaminant remediation. 2. A quantitative uncertainty analysis is needed whenever a realistic rather than a conservative estimate is needed. This is especially the case for epidemiological investigations that attempt to evaluate the presence of a dose response. Failure to include uncertainties in the dose estimate could lead to misclassification of the exposed individuals and thus decrease the power of the analysis. Established dose-response relationships that are not corrected for uncertainty in the exposure or dose estimates will often be biased towards a lesser effect than is actually present (Prentice and Thomas, 1993). 3. A quantitative uncertainty analysis should also be used to set priorities for the assessment components for which additional information will likely lead to improved confidence in the estimate of dose and risk. Without an uncertainty analysis, inconsistencies in the application of conservative assumptions may obscure those assessment components (i.e., contaminants, exposure variables, pathway models, etc.) that dominate the uncertainty in the estimation of dose and risk. With an uncertainty analysis, the variables that contribute most to the overall uncertainty in the result are readily identified. These are the variables warranting the highest priority for further investigation and are those for which an increase in the base of knowledge will effectively reduce the uncertainty in the calculated result.

Methods for Uncertainty Analysis Methods used to quantify uncertainty in dose and risk assessment involve both analytical equations for simple models that can be implemented with hand calculations and numerical approaches requiring the use of a computer. Whenever a quantitative uncertainty analysis is performed, a stepwise procedure should be used. The first step is the use of simple bounding (or screening) calculations to determine whether further investigation is warranted before making a decision. One set of calculations is intentionally biased to produce high confidence that the true but unknown value is not underestimated by the calculated value ( a conservative screen). For the second set of calculations, known conservative bias is removed and the intent is to be highly confident that the true value is not below the calculated value (a nonconservative screen). If further investigation is warranted, the second step is an exploratory uncertainty analysis; appropriately wide probability distributions are specified that represent conservative estimates of the analysts' state of knowledge for each uncertain variable and discrete functional term in the model. Uncertainty analysis combined with a sensitivity analysis will indicate where further investigations should be focused t o improve the state of knowledge. As new information is obtained, the analysis is repeated; each time remaining information needs are identified until sufficient reduction in uncertainty is obtained or until time, resource limitations or other considerations force a decision. At that point, the uncertainty will, for practical purposes, be either acceptable for decision making or irreducible. Within this iterative approach, the cost of dose or risk reduction must be considered a t each stage, along with the alternative risks of taking no action.

3.1 Limiting the Scope through Screening When the assessment is directed at a large number of contaminants and exposure pathways, it is usually prudent to narrow the scope of the problem to those aspects that particularly warrant a more detailed investigation. The contaminants and exposure pathways that require a more detailed analysis can usually be identified rapidly through a set of preliminary screening calculations. Screening can be considered a first step in an uncertainty analysis because a rough estimate is provided of upper and lower bounds of the dose and risk to exposed individuals. For screening calculations to be used successfully, decision criteria must be established that define a level of dose or risk below which the need for further analysis is not necessary. Such a decision criterion might be the NCRP negligible dose level of 10 pSv per y (NCRP, 1993), but other criteria could be justified as well on a case-by-case basis. For example, for dose reconstruction, the National Academy of Sciences has suggested that an individual lifetime dose of 0.07 Sv be used as a decision criterion for establishing the need for more detailed investigation (NASNRC, 1995). All concerned parties, including the public, should be involved in the establishment of specific decision criteria.

3.2 GENERAL APPROACH TO UNCERTAINTY ANALYSIS

/

5

Without a decision criterion for screening, the process becomes more problematic. Decisions must then be based on the relative importance of specific contaminants and pathways to the total estimated dose and risk. Such relative ranking procedures, in the absence of a quantitative uncertainty analysis, are sensitive to hidden inconsistencies in the amount of conservatism employed in the assumptions of the screening calculations. If a series of conservatively biased assumptions is used for estimating the dose or risk to potentially maximally exposed individuals, high confidence can be achieved that the true but unknown dose or risk will not be underestimated. If these conservatively biased screening estimates of dose or risk fall below an established decision criterion, a more detailed analysis of these contaminants can defensibly be given a low priority (Hoffian and Gardner, 1983; NCRP, 1989; 1996). Conversely, the results of a nonconservative screen, produced by removing the known sources of conservative bias from the calculation, can be compared to a decision criterion to identify contaminants that should have the highest priority for further investigation (Hoffman et al., 1991; 1993). Once screening estimates have been made and compared against established decision criteria, the scope of the problem can be refined to a smaller list of contaminants and exposure pathways. For each contaminant on this smaller list, a more detailed assessment can be made that should include a quantitative analysis of uncertainty. The components of the assessment that dominate the overall uncertainty should then be identified, more information obtained for these dominant components, and the analysis repeated. This process should be continued in an iterative manner until the uncertainty is either acceptable or irreducible.

3.2 General Approach to Uncertainty Analysis

In most situations, uncertainty in the estimate of dose or risk can be calculated from an estimate of uncertainty in each of the parameters used in the model equations. This approach is referred to as a "parameter uncertainty analysis" (IAEA, 1989). The technique of parameter uncertainty analysis provides a quantitative method for estimating the uncertainty in the model result, assuming that the structure of the model is an adequate representation of the real world and that correct values for model parameters will produce correct results. 3.2.1

Parameter Uncertainty Analysis

A parameter is uncertain if there is lack of knowledge about its true value. Parameter uncertainty analysis should be composed of the following steps (IAEA, 1989): 1. Define the assessment endpoint. This may be an annual dose to a member of a "critical group," the collective dose to a defined population, or the lifetime health risk to an average individual, a maximally exposed individual, or an unspecified individual of a population. 2. List all potentially important uncertain parameters. Include additional parameters, if necessary, to represent uncertainty in the model structure. 3. Specify the maximum conceivable range of possibly applicable values for uncertain parameters with respect to the endpoint of the assessment. 4. For this range, specify a (subjective)probability distribution that quantitatively expresses the state of knowledge about alternative values for the parameter (i.e., defines the probability that the true value of the parameter is located in various subintervals of the indicated range). 5. Determine and account for dependencies that are suspected to exist among parameters.

6

1

3. METHODS FOR UNCERTAINTY ANALYSIS

6. Using either analytical or numerical procedures, propagate the uncertainty in the model parameters to produce a (subjective) probability distribution of model predictions [i.e.,a probability density function (PDF) representing the resulting state of knowledge for the assessment endpoint]. 7. Derive quantitative statements of uncertainty in terms of a (subjective) probability or confidence interval about the assessment endpoint. 8. Identify the parameters according to their relative contribution to the overall uncertainty in the model prediction. 9. Present and interpret the results of the analysis.

The choice of the assessment endpoint will determine the center and spread of the probability distributions used to represent uncertainty in the parameters of the mathematical model used to perform the assessment. For example, if the assessment is targeted a t a maximally exposed individual, the distributions obtained for the uncertain model parameters should be representative of the analysts' state of knowledge about individuals who are likely to be those with the highest exposure. These individuals may eat more fish, may drink more water, may Live longer in the region, etc. Distributions that are centered on the "average" or "typical" person may not be adequate representations of the state of knowledge for uncertain parameters intended to be representative of maximally exposed persons. In other words, if there is interest in values occurring in the extremes of a distribution, the conditions that bring about these extremes should be modeled explicitly, and the uncertainty analysis should be centered on these conditions. Procedures that can be used to propagate parameter uncertainty through dose and risk assessment models are described in Sections 3.3 and 3.4. Known (or suspected) dependencies among the uncertain parameters can be incorporated either by specifying a correlation coefficient or by changing the model structure to incorporate functional relationships that define those interdependencies. A brief discussion of how to incorporate correlations among parameters is included in Section 3.4. In the absence of directly relevant data sets, Steps 3, 4 and 5 are usually carried out by using professional judgment based on a n extensive review of available literature, evaluation of related data, and interviews with experts. Formal procedures for elicitation of expert judgment are discussed in Section 4.

3.2.2 Model Uncertainty The term "model uncertainty" is used to represent lack of confidence that the mathematical model is a valid formulation of the assessment problem. The term "model" is used in this Commentary to represent a functional relationship. Model uncertainty exists if an incorrect result can be obtained even if exact values are available for all of the model parameters. In attempting to quantify the state of knowledge for uncertain models, it will be necessary to ask questions such as the following: Has the model been tested against independent data sets that are relevant to the domain of model application? Are uncertainties involved with respect to the manner in which discrete processes are represented? Do alternative models exist? Among competing models, are there any significant differences in model structure, discretization strategies, data sets used for model development, etc.?

3.2 GENERAL APPROACH TO UNCERTAINTY ANALYSIS

1

7

The best method for assessing model uncertainties is through "model validation" (Hoffman et al., 1984). Model validation is the term often used for the process in which model predictions are compared to data sets that are independent of the data used to develop the model. However, the process of model validation is often incomplete due to limited experimental and financial resources or to the fact that the test data are not entirely relevant to the conditions for which the model will be applied. In many instances, the endpoint of the assessment cannot be confirmed through direct measurement, as is the case for potential health effects from low doses of contaminants. Uncertainty due to lack of confidence that the model structure is an adequate representation of actual processes can be approached in several ways. One is to place additional parameters into the model that represent residual uncertainty when all other inputs are known exactly (IAEA, 1989).Another approach is the explicit consideration of alternative model formulations. Alternative model formulations may differ in a t least three ways. First, they may differ by the physical, chemical and biological processes represented in their structure (Weber et al., 1992). In this situation, model uncertainty is due to questions about the relevance of the respective processes. Second, alternative models may differ in their assumptions about the unknown spatial and temporal pattern of some model parameter (Helton et al., 1992; LaVenue and Rao, 1992). Examples are alternative representations of parameters such as the spatial distribution of a radionuclide, population density, agricultural land use, or the rate of evacuation and alternative representations of future weather sequences to be considered for an assessment. Third, alternative models may be derived from different sets of possibly relevant experimental data or from the same data set using different sets of assumptions about the underlying processes to obtain comparable fits (Glaeser et al., 1994). Rarely, if ever, can any alternative model formulations be considered as "true." However, if there is a preferred alternative model, multiplicative and/or additive uncertain correction terms can be assigned to the output from the model or to specific subjects of the model that are believed to be uncertain. In this manner, model uncertainty is again reduced to parameter uncertainty. Quantification of the state of knowledge of the correction term may be a function of the model output or input. The set of models actually implemented in the assessment will usually consist of a relatively small number of alternatives. In general, the set of alternative models is only an approximate expression of the state of knowledge, since this set is merely an approximation of the set of all possible alternative models. If the uncertain "difference" from the true relationship could be mathematically expressed as an uncertain correction factor, and if the state of knowledge about the necessary degree of correction can be quantified, then it would suffice to consider only one model formulation-or, the preferred one (Draper et al., 1987; Evans, 1990; Glaeser et al., 1994; Weber et al., 1992). Sometimes uncertainties about model structure are combined with the uncertainties about the model parameters. In these cases, two different approaches can be taken. First, the subjective probability distributions that have been specified for the uncertain parameters of the model can be modified by expert judgment and made sufficiently wide to include the effect of model uncertainty. Second, subjective probability distributions that quantify uncertainty in the model results can be obtained from expert judgment or experiments, and these distributions can then be used to adjust the distributions specified for the parameters of the model through mathematical fitting procedures (Cooke and Vogt, 1990). The latter approach will be limited to relatively simple models. In practice, a model structure may be adequate in certain parts of the input domain, where it is sufficient to quantify parameter uncertainty only, while in other parts of the input domain,

8

/

3.

METHODS FOR UNCERTAINTY ANALYSIS

the uncertainty in the model formulation may range from "clearly inadequate" to merely a suspicion of the existence of severe model deficiencies. When model uncertainty becomes dominant it will be difficult, if not impossible, to contain this source within the uncertainty assigned to the parameters of the model. When a set of alternative model structures is explicitly considered, each alternative is assigned a number and a subjective probability of being an adequate representation of the actual system (i.e.,uncertainty due to an inadequate model structure is negligible with respect to uncertainty from other sources). For example, let us assume we have three alternative models for ground water flow. These three models differ in their spatial discretization. The first alternative, which estimates flows and concentrations in one dimension, is given the Number 1.The second, which estimates flows and concentrations in two dimensions, is assigned the Number 2. The third, a three-dimensional model, is assigned the Number 3. Each of these numbers is then assigned a subjective probability that the designated alternative model is an adequate representation. If each alternative model were considered equally plausible, a subjective probability of 0.33 would be assigned to each of the identifying numbers. If one alternative were considered more plausible than the others, it would be assigned a higher probability. In this manner, the selection of alternative models is treated as an uncertain parameter within a parameter uncertainty analysis.

3.3 Analytical Methods for Propagation of Uncertainty

For relatively simple equations, uncertainty analysis can be performed using analytical methods. The analytical approach most frequently used for uncertainty analysis of simple equations is variance propagation (IAEA, 1989; Martz and Waller, 1982; Morgan and Henrion, 1990). Variance propagation is best demonstrated by a simple model that is a summation of terms:

where R p

the quantity computed using the model the number of uncertain parameters k, and hi = coefficients xi = the ithuncertain parameter = =

For an additive model like this, the mean value of the result, R, is equal to the sum of the mean values of the model parameters. The variance of the result, assuming statistical independence among the parameters, is equal to the sum of the variances of the parameters. Thus,

and

3.3 ANALYTICAL METHODS FOR PROPAGATION OF UNCERTAINTY

1

9

where ZR = the mean value of the result

p

=

the number of parameters in the model

Ti = the mean value of the parameter i si = sz =

k, and hi

=

the variance of the model result the variance of parameter i coefficients

When many uncertain parameters are summed, the result will tend to conform to a normal distribution even if the shapes of the distributions assumed for the model parameters are not normal. Multiplicative models can be reduced to additive form by logarithmically transforming the variables (provided the model result and the parameters can assume only positive values). For example:

Therefore, for a sufficiently large number of parameters, the distribution of R will tend to be approximately log-normal even when the parameters are assigned distribution shapes other than log-normal (Gardner, 1988; O'Neill e t al., 1981). For multiplicative models, the median value (50thpercentile or geometric mean) of the log-normal distribution of R is simply the exponential of the sum of the mean values of the logarithms for the model parameters:

where &,R

=

CL, =

p~ =

the geometric mean of the result, R the mean of logarithms for parameter i the sum of the means of logarithms of the model parameters

The geometric standard deviation of the result is found (assuming independence among the parameters) by computing the square root of the sum of the variances of the log-transformed parameters and exponentiating

where

Sg,R = the geometric standard deviation of the result, R u; = the variance of the logarithms for parameter i u i = the sum of the variances of the logarithms for the model parameters The formula used to obtain the mean and variance oflogarithms for each uncertain parameter depends on the type of subjective probability distribution chosen to represent the state of knowledge of the uncertain parameter (see Section 3.6). Equations describing the mean and variance of logarithms of log-normal, log-uniform and log-triangular distributions are provided in Appendix A.

10

/

3. METHODS FOR UNCERTAINTY ANALYSIS

In the case outlined above, if a log-normal distribution is selected for R, a 90 percent subjective confidence interval for R is obtained by:

and

where

XI5 = the subjective 9Eitbpercentile of the result, R XF = the subjective Eith percentile of t h e result, R A 95 percent subjective confidence interval is obtained by replacing 1.65 by 2 in Equations 3.8 a n d 3.9. Example 3.1 demonstrates the use of variance propagation. Example 3.1 This Example is entirely hypothetical and is for demonstration purposes only. The problem illustrates common applications of the general Equations 3.1, 3.4 and 3.5. Situation. Let us assume that l3ICs is continuously released to a nearby lake. Using the technique of variance propagation, obtain a 90 percent subjective confidence interval on the excess lifetime cancer risk to a maximally exposed individual resulting from eating fish from the lake. After review of the literature and available data, and after consultation with other experts, the (subjective) probability distributions shown in Table 3.1 are selected for this problem. The contaminant concentration in fish (C) includes uptake of the contaminant from water by the fish; the contribution of drinking water to the individual's ingestion dose is assumed to be negligible in comparison to the dose from consumption of fish. Solution. The form of the equation used for this problem is as follows:

TABLE3.1-Information for Exumple 3.1. Central Parameter Distribution Minimum Maximum Value 8.OE - 01 Concentration in fish Log-uniform 3.9E - 02 (C), Bq kg-' Intake of fish by Log-uniform 5.6E- 03 9.2E - 02 humans (I),kg d-I Exposure frequency Constant 3.5E +02 (EF), d y-' Exposure duration Constant 3.OE + 01 (ED),y 2.8E - 08 1.4E - 08" Dose conversion Log-triangular 7.OE - 09 factor (DCFj', Sv Bq-I 2.4E - 01 6.0E - 02" Risk conversion Log-triangular 1.5E - 02 factor (RCFI. Sv-' T h e central value as given for the DCF and RCF refers to the exponential of the most likely value (mode) of the logarithm of the triangular distribution. The distributions describe the uncertainty associated with estimating an unknown value for each model parameter.

3.3 ANALYTICAL METHODS FOR PROPAGATION OF UNCERTAINTY

/

11

where LRRAD = excess lifetime cancer risk from radionuclides (unitless) C = concentration in the contaminated medium (Bq kg-') I = estimated intake of the contaminated medium (kg d-') ED = exposure duration (y) EF = exposure frequency (d y-') DCF = dose conversion factor for the radionuclide of interest (Sv Bq-') RCF = risk conversion factor (Sv-l)

By log-transformation, Equation 3.10 becomes:

The logarithmic mean and variance of the excess lifetime cancer risk, assuming independence among the model parameters, are found by using variance propagation. The equations given in Appendix A must be used to find the mean and variance of logarithms for each of the model parameters. Equations A.3 and A.4 produce a mean of the logarithms for the fish concentration of - 1.73 and a variance of the logarithms for the fish concentration of 0.76. Equations A.3 and A.4, for the log-uniform distribution, will also be used to find p (the mean of logarithms) and u2 (the variance of logarithms) for the intake:

Equations A.5 and A.6, given for the log-triangular distribution, are used for the dose and risk conversion factors. To demonstrate, the IJ, and u2 for the dose conversion factor are calculated as follows:

This same process is performed for the RCF to produce a mean value of - 2.81 and a variance of 0.32. The mean of logarithms and, thus, the geometric mean of the excess lifetime cancer risk, is calculated as follows:

The variance of the excess lifetime cancer risk and, consequently, the geometric standard deviation of the excess lifetime cancer risk can then be calculated:

12

/

3. METHODS FOR UNCERTAINTY ANALYSIS

Based on the assumption that the lack of knowledge about the excess lifetime cancer risk is log-normally distributed, the approximate upper and lower limits for a 90 percent subjective confidence interval are calculated as:

X,P = &,%.S:;L = (3.6E-O8)(3.84)'= 3.33-07

(3.21)

g ~ %, ~ % 0

(3.6E-08) = 3.93-09 (3.22) s:;E5% (3.84)'.'j5 Therefore, there is a subjective confidence of approximately 90 percent that the excess lifetime cancer risk lies between 3.93-09 and 3.33-07. -

-

Variance propagation is a straightforward process for simple additive and logarithmically transformed multiplicative models where the parameters are statistically independent. For more complex calculations, variance propagation techniques are more difficult to derive analytically, and in many, if not most, cases their derivation may not be practical or possible.

3.4 Numerical Methods for Uncertainty Propagation To overcome problems encountered with variance propagation equations for complex models, various numerical methods can be used to perform an uncertainty analysis with the aid of a computer. Perhaps the most commonly applied numerical technique, and the one emphasized in this Commentary, is Monte Carlo simulation (Rubinstein, 1981). Other approaches include the following: (1)differential uncertainty analysis (Cacuci, 1981; Scavia et al., 1981; Worley, 19871, in which the partial derivatives of the model response with respect to the parameters are used to estimate uncertainty; (2) Monte Carlo analysis of simplifications of complex models (Downing et al., 1985; Mead and Pike, 1975; Morton, 1983; Myers, 1971); and (3) nonprobabilistic methods [for example: fuzzy sets, fuzzy arithmetic, and possibility theory (Ferson and Kuhn, 1992)l. Monte Carlo analysis is usually performed using one of two random sampling processes: Simple Random Sampling (SRS) or Latin Hypercube Sampling (LHS) (LAEA, 1989; Iman and Conover, 1980; 1982; Iman and Shortencarier, 1984; McKay et al., 1979). In SRS, a random value is taken from the distribution specified for each uncertain model parameter, and a single estimate of the desired endpoint is calculated. This process is repeated for a specific number of samples or iterations. The result is an empirical approximation to the probability distribution of the model output or assessment endpoint. SRS, however, is less efficient in many cases than LHS, especially when the sample size is less than a few hundred. To generate a n LHS of size n from the variables xl, x2, . . ., x,, the range of each variable is divided into n intervals of equal probability (i.e., n-I). A single variable value is randomly selected from each interval. The n values for xl are randomly paired without replacement with the n values for x2 to produce n pairs of variable values. These pairs are randomly combined without replacement with the n values for x3 t o produce n triples of variable values. This process is then continued until all n variables have been incorporated into the sample. The resultant LHS is a sequence (xlj,xzj,. . ., xmj),j = 1, 2, . . ., n, of sample elements, where xijis the value of variable xi in sample element j. The preceding process has the desirable property of producing a sample of relatively small size that covers the full range of each variable. Replicated LHS designs, wherein several smaller LHS designs are run, may also be used, and approximate quantitative statements may be obtained for the impact that sampling error may have on estimates of the mean value and specific percentiles of the subjective probability distribution of the assessment endpoint. For example, instead of one LHS or SRS set of 100

3.4 NUMERICAL METHODS FOR UNCERTAINTY PROPAGATION

1

13

runs, five LHS sets of 20 runs each may be used. The efficiency of a replicated LHS in approximating the probability distribution of the model endpoint is somewhere between that of SRS and a single LHS (Iman and Conover, 1980). Figure 3.1 shows a PDF resulting from a Monte Carlo simulation of several uncertain input parameters. Other methods of representing the output of a Monte Carlo simulation include cumulative distribution functions, complementary cumulative distribution functions, and probability plots. These alternative ways of representing the results of a Monte Carlo analysis are shown in Figure 3.2. Monte Carlo analysis may be implemented by writing a sampling code or by utilizing one of several currently available software packages. Several available Monte Carlo simulation tools are listed in Table 3.2 with their references. Example 3.2 provides a more detailed description of a Monte Carlo simulation. Example 3.2 This Example is entirely hypothetical and is for demonstration purposes only.

Situation. Assume the scenario presented in Example 3.1. As in Example 3.1, we wish to obtain an interval that contains the true risk to the maximally exposed individual with 90 percent subjective confidence,this time by employing Monte Carlo simulation. This Example does not address dependencies among parameters; the effect of correlations among parameters will be demonstrated in Example 3.4. Solution. To begin a quantitative uncertainty analysis, the uncertainty or state of knowledge about each parameter must be described with a probability distribution. Each distribution will be subjective

Parameters 1.2,

...,n --, Model

-

Fig. 3.1. PDF resulting from a Monte Carlo simulation.

Model Result

14

/

3. METHODS FOR UNCERTAINTY ANALYSIS

Cumulative Distribution Function (CDF) *

dose *

the probability that any value in the distribution is equal to or less than a given value of dose

Complementary Cumulative Distribution Function (CCDF) *'

dose **

the probability that any value in the distribution is greater than a given value of dose

Probability Plot '**

1 0.1

*** Fig. 3.2.

5

50

cumulative probability

95

99.9

a probability plot permits close examination of values in the tails of the distribution

Alternative ways of representing the results of a Monte Carlo analysis.

to some degree because it will be obtained through judgment after extensive review of all relevant data. The information presented in Table 3.1 is used as input for a Monte Carlo simulation for this problem. During a Monte Carlo simulation, a value is selected a t random from the (subjective) probability distribution for each uncertain parameter, and the assessment model is run to produce a value for the prediction or assessment endpoint. This procedure is repeated for a specified number of iterations to

3.4 NUMERICAL METHODS FOR UNCERTAINTY PROPAGATION

/

15

TABLE3.2-Monte Carlo simulation tools. Program Reference MOUSE Klee (1986) TAM3 Gardner (1988); Kanyar and Nielsen (1989) PRISM Gardner and Trabalka (1985); Gardner et 61. (1983) Crystal Ball Decisioneering, Inc. (1994) @RISK Palisade Corporation (1991) ORMONTE Williams and Hudson (1989) DEMOS Henrion and Wishbow (1987) GENIVSUNS Leigh et al. (1992) RiskQ Bogen (1992); Murray and Burmaster (1993) SUSA-PC Krzykracz et al. (1994)" "Unpublished material, Krzykracz, B., Hofer, 3. and Kloos, M. (1994). "A software system for uncertainty and sensitivity analysis of results from computer models," Processings of PSAM-I1 Conference, Vol. 2, Session 063. provide a distribution of predicted values. A set of 500 predicted values of the excess lifetime cancer risk was obtained by first selecting 500 sets of parameter values obtained through LHS and running the model for each set of parameter values (Table 3.3). This process yields an empirical (subjective) probability distribution for the excess lifetime cancer risk. Figure 3.3 displays the 500 values of the lifetime cancer risk. An approximate 90 percent subjective coddence interval of 3.63-09 to 2.63-07 is obtained by ordering the 500 values according to their cumulative probability and identifying the values that occur at the 5~ and 95'" percentiles (this process is often automated in commercial software packages). The 90 percent subjective confidence interval for this Example is indicated by tick marks on the graph provided in Figure 3.3. Thus after taking into account the uncertainties on the parameters, one is confident (at a subjective level of approximately 90 percent) that the true excess lifetime cancer risk lies between 3.63-09 and 2.63-07. Once the user becomes familiar with a software package for Monte Carlo simulation, this technique is very efficient. Even if the assessment model for the risk analysis becomes more complicated, the Monte Carlo technique does not. One reason that Monte Carlo calculations are more useful than analytical approaches to uncertainty analysis is that analytical solutions based on variance propagation techniques provide only approximate probability or confidence intervals and can become very complicated and time-consuming for more involved risk analyses. Their usefulness is limited to linear models of simple structure. The inputs required for Monte Carlo simulations are the probability distributions for each parameter. These distributions are obtained by applying professional judgment about the

TABLE3.3-A Sample Number 1 2 3

set o f 500 predicted values of the excess lifetime cancer risk size 500 for Example 3.2. Concentration Intake ED DCF in Fish EF of Fish 0.) (SvBq-l) (Bq kg-') (kg d-') (d y-') 1.30E - 01 2.20E - 02 3.50E + 02 3.00E + 01 1.363 - 08 8.16E - 02 1.26E - 02 3.50E + 02 3.00E + 01 1.15E - 08 9.353 - 02 7.02E - 03 3.50E + 02 3.00E + 01 1.17E - 08

obtained using LHS o f

RCF (Sv-l) 1.14E - 01 7.283 -02 5.723 - 02

Lifetime Risk 9.853 - 10 1.113 - 09 1.33E -09

16

/

METHODS FOR UNCERTAINTY ANALYSIS

3.

Excess Lifetime Cancer Risk

Fig. 3.3. Subjective PDF of the excess lifetime cancer risk for Example 3.2. The 90 percent subjective confidence interval is from 3.63-9 to 2.63-7.

current state of knowledge about each uncertain parameter, after extensive review of the available literature and d a t a a n d evaluation of t h e relevancy of this information with respect to t h e assessment endpoint (see Section 3.6). Because professional judgment is used, the probability distributions are subjective. With t h e various i n p u t (subjective) distributions, the Monte Carlo simulation program then provides a subjective probability distribution of risk (or dose) from which subjective confidence intervals c a n be easily obtained. A demonstration of this technique for a more complicated risk analysis situation is presented i n Example 3.3. Example 3.3

This Example is entirely hypothetical and is for demonstration purposes only. Situation. Let us assume that as the result ofwaste management practices, a mixture of contaminants is released inadvertently to the environment. Through various pathways, this contamination is transported to aquatic systems such as rivers and lakes where fish and other biota are exposed. After further investigation, it is discovered that the contaminants released were I3lCs,Aroclor-1254@,Aroclor-1260@, chlordane and methylmercury. Suppose that contaminated fish are caught and eaten by humans. As with Example 3.1, the contaminant concentration in fish includes uptake by the fish from the water, and the contribution of drinking water to human exposure is assumed to be negligible. What are the Hazard Index (HI), the sum of the total lifetime cancer risks from carcinogenic chemicals, and the excess lifetime cancer risk from 13'Cs for the maximally exposed individual f?om consumption of contaminated fish? The hazard quotients for chlordane and methylmercury, treated as noncarcinogenic contaminants, are calculated from the following equation:

where

HQ C

= =

hazard quotient (unitless) concentration of the contaminant in fish (mg kg-')

3.4 NUMERICAL METHODS FOR UNCERTAINTY PROPAGATION

I ED EF BM AT RP

1

17

= estimated intake rate of fish by humans (kg d-l) = exposure duration (y)

exposure frequency (d y-l) body mass (kg) averaging time (d) = reference dose for the chemical of interest (mg kg-' d-')

= = =

The HQs for various chemicals are summed for each exposure pathway to obtain an HI for an individual exposed from ingestion of multiple contaminants (EPA, 1989). The excess lifetime cancer risk for Aroclor-1254@,Aroclor-1260Band chlordane are determined using the following equation:

where

LR*,, C I ED EF SF BM AT

= excess lifetime cancer risk from chemicals (unitless) = concentration of the contaminant in fish (mg kg-') = estimated intake of fish by humans (kg d-')

exposure duration (y) exposure frequency (d y-') slope factor (or cancer potency factor) for the chemical of interest [(mg kg-' d-')-'I = body mass (kg) = averaging time (d) = = =

The excess lifetime cancer risk for 137Csis determined using Equation 3.10. The uncertainty or state of knowledge associated with each of the parameters introduced in these equations is expressed by a subjective probability distribution that has been derived from very limited sets of data and other relevant information in the published literature, using a considerable amount of judgment. Once these distributions have been specified using judgment, Monte Carlo techniques can be utilized to obtain the resulting subjective probability distributions of the HI and the total lifetime cancer risks. From these distributions, subjective confidence intervals (90 percent) are produced for use in decision making. Table 3.4 contains quantitative estimates of the state of knowledge for each of the parameters that might be used in an environmental risk assessment for Aroclor-1254@,Aroclor-1260@,chlordane, methylmercury and l3ICs in the fish potentialIy harvested fiom a contaminated freshwater system. For this Example, the exposure duration, exposure frequency, and averaging time are assumed to be known, fixed values:

ED EF AT

= =

=

30 y for carcinogens and 1y for noncarcinogens 350 d y-' 25,550 d (70 y) for carcinogens and 365 d for noncarcinogens

Solution: The values given in Table 3.4 were used to find the median, the 5"'percentile, and the 95"' percentile of the probability distribution for the noncarcinogen HI for the probable maximally exposed individual for chlordane and methylmercury, for the total cancer risk from the given concentrations of Aroclor-1254@,Aroclor-l26O@and chlordane in fish, and for the excess lifetime cancer risk from a chronic exposure to 13Tsthrough the ingestion of contaminated fish. The results (presented in Table 3.5) were obtained by using a sample of size 500 with the LHS Monte Carlo technique. As shown in Table 3.5, the primary contaminant contributing to the total cancer risk is Aroclor1260@,and the chemical contributing the majority of the total HI is chlordane. The parameter that has the most effect on the total uncertainty in the total cancer risk or the total HI (i-e., on the probability distribution of the total cancer risk or total HI) can be determined by performing a sensitivity analysis (see Section 3.5). Many different approaches can be used for the sensitivity analysis (IAEA, 1989; Morgan and Henrion, 1990). In this Example, a regression was performed on the rank order of the model results against the rank order of the uncertain model

18

/

3. METHODS FOR UNCERTAINTY ANALYSIS

TABLE3.4-Subjective probability distributions specified for the Monte Carlo analysis i n Example 3.3. Subjective Probability Distribution Log-uniform Log-uniform Log-triangle Triangle

Minimum 4.00E - 03 5.60E - 03 4.50E + 01 O.OOE +00

Fish conc. Intake Body mass SF

Log-uniform Log-uniform Log-triangle Triangle

3.19E - 01 2.293 + 00 5.60E - 03 9.20E - 02 4.50E + 01 1.20E + 02 O.OOE + 00 1.00E + 01

9.753 - 01

Fish conc. Intake Body mass SF

Log-uniform Log-uniform Log-triangle Triangle

3.963 - 02 5.60E - 03 4.50E + 01 O.OOE + 00

3.06E - 01 9.20E - 02 1.20E + 02 5.00E + 00

1.27E - 01

Fish conc. Intake Body mass RfD Fish conc. Intake Body mass

Log-uniform Log-uniform Log-triangle Log-triangle Log-uniform Log-uniform Log-triangle Log-triangle Log-uniform Log-uniform Log-triangle Log-triangle

3.963- 02 l.lOE - 02 4.50E + 01 3.00E - 05

3.06E - 01 1.80E - 01 1.20E+02 1.90E - 03

2.553 - 02 1.10E- 02 4.50E + 01 1.50E - 04

1.57E - 01 1.80E- 01 1.20E+ 02 3.00E - 03

7.10E - 02

3.90E - 02 5.60E - 03 7.00E - 09 1.50E - 02

8.00E - 01 9.20E - 02 2.80E - 08 2.40E - 01

1.80E - 01

Contaminant Parameter Aroclor-1254@ Fish conc. Intake Body mass SF Aroclor-1260B

Chlordane (carcinogen)

Chlordane (non-carc)

Methylmercury

RP Cs-137

Fish conc. Intake Dose factor Risk factor

Maximum 3.793 + 00 9.20E - 02 1.20E+ 02 1.00E+ 01

Central Value (mean or mode) 5.343- 01

7.00E + 01 7.70E+ 00

7.00E + 01 7.70E + 00

7.00E + 01 1.30E + 00 1.27E - 01 7.00E + 01 6.00E - 05

7.00E + 01 3.00E - 04

1.40E - 08 6.00E - 02

Units mg kg-' kg d-' kg mg-' kg d mg kg-' kg d-' kg mg-' kg d mg kg-' kg d-' kg mg-' kg d mg kg-' kg d - ' kg mg kg-' d-' mg kg-' kg d-' kg mg kg-' d-' Bq kg~.' kg d-' Sv Bq-.' Sv--'

parameters. The index of sensitivity for the uncertain model parameters is the square of Spearman's Rank Correlation Coefficient (with the sum of squares of the ranks of each of the uncertain percentiles adjusted to the total sum of squares of the ranks of the model result). This sensitivity index is employed in the software package Crystal Ball (Decisioneering, Inc., 1994) and provides an approximation of the fractional contribution of the uncertainty in a given parameter to the total uncertainty in the model

TABLE3.5-Results Contaminant Aroclor- 12540 Aroclor-12600 Chlordane Cs-137

obtained from Monte Carlo simulation using the values i n Table 3.4. ij"' Median 95th Percentile (50fi percentile) Type of Result Percentile 9.6E - 05 3.OE - 03 Cancer risk 2.OE - 06 5.9E - 04 4.1E - 03 7.9E - 05 Cancer risk 1.9E- 04 4.OE - 06 2.4E - 05 Cancer risk 3.1E-08 2.8E - 07 3.1E - 09 Cancer risk

1.5E - 04

Total cancer risk" Chlordane Methylmercury

Noncarcinogen HQ Noncarcinogen HQ

9.6E - 04

6.1E- 03

0.05 0.01

Total HIa 0.08 0.55 'Risks may not be directly additive due to the random sampling used in the analysis.

4.5

3.4 NUMERICAL METHODS FOR UNCERTAINTY PROPAGATION

1

19

result. The parameters having the highest index of sensitivity are the parameters for which additional data should be obtained to reduce the amount of overall uncertainty in the results. Other methods of performing sensitivity analyses are introduced in Section 3.5. In this Example, the uncertainty in fish intake by humans was identified as contributing approximately 59 percent of the overall uncertainty for the total cancer risk from chemical exposure. The next most important parameter is the concentration of Aroclor-1260" in the fish, the uncertainty in which contributes approximately 13 percent of the overall uncertainty. For the noncarcinogens, the sensitivity analysis showed that the two parameters that are the most significant contributors to the total uncertainty are the uncertainty in the fish intake (contributing approximately 41 percent of the overall uncertainty) and the uncertainty in the R/D for chlordane (contributing approximately 36 percent of the overall uncertainty). One of the key steps in setting u p a quantitative uncertainty analysis is determining whether or not any of the parameters a r e correlated with each other. For known or suspected dependencies among model parameters t h a t can be quantified through correlation coefficients, simulation techniques are available t o address these dependencies (Devroye, 1986;Iman and Conover, 1982). Dependencies which are suspected but a r e hard to quantify can be included using subjective judgment. Sometimes the effect of assumed minimum versus assumed maximum levels of correlation is investigated to evaluate the importance of including a n explicit estimate of dependency among the parameters. In some cases, explicit modeling of the dependency between model parameters is possible, based on knowledge about the explicit mechanistic reasons for the dependencies (McKone a n d Bogen, 1992; Morgan and Henrion, 1990). In general, it becomes more important to consider the effect of dependency when correlations are strong among the model's most sensitive variables; weak correlations among sensitive parameters and strong correlations among insensitive parameters will have very little impact on the overall result.

Example 3.4 This Example is entirely hypothetical and is for demonstration purposes only. The purpose of this Example is to study the effect of correlation coefficients on the model result.

Situation. Two scenarios are investigated. 1. The effect of the correlation between body mass and intake on the total cancer risk from carcinogenic chemicals and the total HI for the situation given in Example 3.3 is analyzed. First, assume that a rank correlation of 0.3 has been determined to exist between body mass and intake, and second, compare the results with those obtained with a rank correlation of 0.5. 2. The effect of a correlation between the fish concentration and the intake on the total cancer risk from carcinogenic chemicals, the excess cancer risk from 137Cs,and the total HI for the situation described in Example 3.3 is analyzed. This correlation would exist for those fishermen who eat only a certain species and size of fish when the concentration in the fish is partially determined by species and size. A correlation of - 0.9 has been assumed for this Example.

Solution. 1. In this case, rank correlations (Iman and Conover, 1982)are used to account for interdependencies between body mass and intake. As can be seen from Table 3.6, for results produced with a sample of size 500 using LHS, the correlation coefficients do not have a dramatic effect on the total risk or total HI. This, of course, is to be expected, as rank correlations of 0.3 and 0.5 represent weak dependencies. In both cases (total cancer risk and total HI), the values obtained with a rank correlation of 0.3 are virtually the same as those obtained with a rank correlation of 0.5. Another reason that the correlation does not have an obvious effect on the results is that uncertainty in

20

1

3. METHODS FOR UNCERTAINTY ANALYSIS

TABLE3.6-Selected percentiles of the probability distribution of total cancer risk from carcinogenic chemicals and of the total HI when correlations between body mass and intake are present. Rank Correlation Coefficient Total cancer risk 5'" percentile 50'h percentile (median) 9Ph percentile Total HI 5&percentile 50&percentile (median) 9Fhpercentile

9.OE - 2 5.3E - 1 4.2E + 0

9.OE - 2 5.3E - 1 4.OE + 0

the body mass is not an important contributor to the overall uncertainty for either the total cancer risk or the total HI. 2. Rank correlations were also used to account for interdependencies between fish concentration and intake. Table 3.7 provides a comparison of the results obtained in Example 3.3 (no correlation) and the results obtained for the total lifetime cancer risk, the total HI, and the excess cancer risk from '37Cswhen a correlation coefficient of -0.9 between the fish concentration and intake is assumed. As can be seen from Table 3.7, the assumption of a strong negative dependence between the concentration in fish and human consumption has a noticeable effect on the lower and upper confidence limits of the total lifetime cancer risk and the total HI. The effect of the assumed strong negative dependency is evident because of the importance of the two correlated parameters to the overall uncertainty. Example 3.4 shows that correlations among parameters may have a n effect when the correlation is between important parameters a n d may be important when the risk assessor is interested i n values occurring i n the extremes (tails) of the distribution.

3.5 Alternative Methods for Sensitivity Analysis The use of Spearman's Rank Correlation Coefficient for sensitivity analysis was described in Example 3.3 (Section 3.4). Another method for identification of important parameters is TABLE3.7-Selected percentiles of the ~ r o b a b i l i distributions t~ for total cancer risk, total HI, and excess cancer risk from '37Cswhen a hypothetical correlation o f -0.9 between fish concentration and intake is assumed, compared to the distributions obtained when no correlation is assumed. No Correlation - 0.9 Correlation (Example 3.3) Total cancer risk 51hpercentile 1.5E - 04 3.OE - 04 9.6E - 04 8.8E - 04 50thpercentile (median) 1.8E - 03 95h percentile 6.1E - 03 Total HI !jLh percentile 50'h percentile (median) 95thpercentile Excess Risk (lS7Cs) 5'h percentile 50thpercentile (median) 2.8E - 07 1.2E - 07 95'h percentile

3.5 ALTERNATIVE METHODS FOR SENSITrVITY ANALYSIS

/

21

the use of scatter plots of the Monte Carlo samples of the uncertain parameters against the Monte Carlo simulations of the model result (Iman and Helton, 1988). For example, if the risk is determined by the addition of two independent parameters, which parameter is more important? This can be determined by plotting the 500 LHS values of Parameter 1 against the 500 simulations of the model result and comparing this graph against a similar graph for Parameter 2, as demonstrated in Figure 3.4. In this Example, a more distinct trend exists for Parameter 2 than for Parameter 1. Therefore, we can conclude that the more important parameter for the overall uncertainty in the model result is Parameter 2. Other methods of performing sensitivity analyses include (1)simple regression [on untransformed and transformed data (Brenkert et al., 1988)l; (2) multiple and stepwise multiple regression [on transformed and untransformed data (Downing et al., 1985)l; (3) correlation coefficients and partial correlation coefficients (Bartell et al., 1986; Gardner et al., 1981); (4) stepwise regression and correlation ratios [onuntransformed and transformed data (Glaeser et al., 1994)l; and (5) differential sensitivity analysis (Griewank and Corliss, 1991; Worley, 1987). Other references discussing the use of regressions of the model prediction on the uncertain parameters to determine a ranking of parameters as the main contributors to the overall uncertainty in the model result include Hamby (1994), Helton (1993a), IAEA (19891, Iman and Helton (1991), Iman et al. (1981a; 1981b), Morgan and Henrion (1990) and Saltelli and Mariovet (1990).

I

I

1

10

100

result

result

Fig. 3.4. Scatter plots of LHS values for Parameter 1(top) and Parameter 2 (bottom)against the simulations of the model result.

22

/

3. METHODS FOR UNCERTAINTY ANALYSIS

3.6 Guidance on the Selection of (Subjective) Probability Distributions for

Uncertain Model Parameters For an uncertainty analysis to be performed using Monte Carlo procedures (or some analytical approaches), probability distributions must be specified that quantitatively express the state of knowledge about each parameter. The distributions characterize the degree of belief that the true but unknown value of a parameter lies within a specified range of values for that parameter. Unless the distributions are obtained from directly relevant data sets, they can be considered subjective. Distributions commonly used in uncertainty analyses include uniform, log-uniform,triangular and log-triangular. Where data are Limited but uncertainty is relatively low (less than a factor of lo), a range may be used to specify a uniform distribution. If the current state of knowledge suggests a most likely value, in addition to a range, a triangular distribution may be assigned. When the range of uncertainty exceeds a factor of 10, it is often prudent to assume either a log-uniform or log-triangular distribution instead of a uniform or triangular distribution, respectively. The assumption of normal, log-normal or empirical distributions is usually dependent on the availability of relevant data. Identification of the best data available to represent the state of knowledge of an uncertain parameter is obviously a prerequisite to selection of an appropriate probability distribution for the parameter. However, for a given environmental assessment objective, the best data may not be obvious. Often the available data are not direct measurements of the parameter of interest. For example, risk assessors are often faced with data obtained on spatial or temporal scales that are significantly different from the scale of the problem under consideration; examples include the use of national fish consumption data to represent consumption patterns in a particular locality and the use of short-term data to represent long-term behaviors (e.g., lifetime home occupancy estimates based on short-term surveys). Risk assessors are frequently faced with the question of whether or not to use unrepresentative or surrogate data to represent the state of knowledge for a particular parameter. For some parameters, physical plausibility arguments may be used to establish ranges for the parameters, and uniform or log-uniform distributions could be assumed within these ranges. Assumption of bounding distributions such as uniform, log-uniform, triangular and log-triangular can be helpful in establishing which uncertain parameters have the greatest potential influence on the assessment endpoint. The risk assessor should be careful to avoid assigning overly restrictive ranges that suggest an unwarranted precision in the state of knowledge. Specification of ranges that are too narrow or distributions that are insufficient by wide margins constitutes a statement of overconfidence. The specification of unreasonably large ranges that do not account for what is known about a parameter should also be avoided. Piecewise uniform or piecewise log-uniform distributions are used when judgment is elicited about ranges and quantiles from an expert or a group of experts (see Section 4; IAEA, 1989). Many other distribution types may be suitable representations of the present state of knowledge for uncertain parameters; these include the gamma, beta, Poisson, Weibull and truncated normal distributions, plus a variety of discrete distributions (Decisioneering, Inc., 1994; Palisade Corporation, 1991). The shape of the distribution can be of any form necessary to describe the present state of knowledge about the parameter. In situations that include mixed distributions, truncated distributions andlor parameter dependencies, analytical methods for uncertainty propagation become intractable and Monte Carlo simulation is the only practical choice. Analytical methods for error propagation are options, however, when one has independent parameters and untruncated distributions for

3.8 UNCERTAINTY ANALYSIS FOR A STOCHASTIC ENDPOINT

1

23

all of the parameters and when the risk'assessment is composed of fairly simple sets of arithmetic operations.

3.7 Interpreting the Result of an Uncertainty Analysis The information obtained from a quantitative uncertainty analysis can be used to guide decisions. For example, if the estimated 5thpercentile of the subjective probability distribution for an endpoint is above a regulatory standard of concern, then it is very likely that the standard will be exceeded. On the other hand, if the 951h percentile is below the standard, it is very likely that the standard will not be exceeded. If the 95thpercentile is above the standard but the 50thpercentile (the median) is below the standard, further study should be recommended for those parameters that dominate the overall uncertainty, as determined by a sensitivity analysis. However, if the 50'" percentile is above the standard, further study may still be recommended, but under some circumstances one may choose to proceed with regulatory action, depending on the cost-effectiveness of measures for risk reduction. Under many circumstances, it is prudent to gather additional data to reduce uncertainty in the overall result before making final decisions. In these cases, data should be gathered for the components of the model that dominate the uncertainty in the estimate of dose or risk. The gathering of additional data should continue until the uncertainty in the dose or risk estimate is either acceptable for decision making or, because of limitations of time and financial resources, irreducible. Under these conditions, decisions must be made in the face of uncertainty that, for practical purposes, cannot be reduced.

3.8 Uncertainty Analysis for an Assessment Endpoint that is a Stochastic Variable The general subject of this Commentary up to this point has been the evaluation of uncertainty about a fixed but unknown value [referred to in IAEA Safety Series No. 100 ( M A , 1989) as "Type B" uncertainty]. However, some risk assessments may have an endpoint defined as a stochastic variable. An example is the true but unknown distribution of doses among individuals in a population in which the individuals can be thought of as selected from the population at random. An uncertainty analysis dealing only with stochastic variability (i.e., simulation of true distributions) is referred to in IAEA Safety Series No. 100 (IAEA, 1989) as a "Type A" uncertainty analysis. Both stochastic variability and lack of knowledge uncertainty occur when the assessment objective is to simulate the distribution of individual doses or risks within an exposed population group where the true shape and spread of this distribution are unknown. For performance of an uncertainty analysis for an assessment in which the endpoint is a stochastic variable, correct interpretation of the results requires that the two sources of uncertainty be analyzed in a distinctly different manner. This is accomplished by producing numerous alternative realizations of distributions that represent the assessment endpoint (analogous to the numerous alternative values obtained for the result in an uncertainty analysis where the endpoint is a fixed but unknown single value). To distinguish between stochastic variability (Type A) and lack of knowledge uncertainty about fixed values (Type B), Monte Carlo simulation should be applied in two stages (see Example 3.5).The first stage involves identification of uncertain quantities that are fixed with

24

/

3. METHODS FOR UNCERTAINTY ANALYSIS

respect to t h e assessment endpoint. These a r e quantities t h a t have only one true value (even though the assessment endpoint is a distribution of values). An example of a quantity t h a t is fixed but unknown is the total amount of a contaminant released or deposited i n a region over a period of time. Another example is the mean, variance a n d shape of the distribution for those parameters t h a t a r e themselves stochastic variables and t h a t determine stochastic variability in t h e assessment endpoint. For each of these fixed quantities, subjective probability distributions are specified t h a t reflect t h e state of knowledge about the t r u e but unknown values. From these distributions, a Monte Carlo sample of alternative realizations of true b u t unknown values is drawn (see Example 3.5). These alternative realizations represent Type B uncertainty (IAEA, 1989). Example 3.5

Example of a two-staged approach to distinguish between lack of knowledge about fixed values and uncertainty due to the presence of stochastic variability using RESULT

= (x)(y)

(3.25)

where x

=

a fixed but uncertain parameter (Type B uncertainty)

y = a stochastic variable that is a distribution of values (Type A uncertainty) with fixed but unknown mean p, (Type B uncertainty) and a fixed but unknown standard deviation a, (Type B uncertainty)

RESULT

=

the simulation of a true distribution of values (Type A uncertainty)

Procedure:

Step 1: Specify subjective probability distributions that represent the state of knowledge for x, and a,.

p,,

Step 2: Use Monte Carlo sampling to produce a set of alternative realizations for the values of x, py and a,.

The set of alternative realizations of CL?, and a, defines alternative realizations of the distribution of the stochastic variable y.

3.8 UNCERTAINTY ANALYSIS FOR A STOCHASTIC ENDPOINT

1

25

Y'

Each realization of the distribution ofy represents Type A uncertainty. The set of alternative realizations represents Type B uncertainty. Step 3: For each alternative realization of x and the distribution of y, multiplication of the value of x by samples from the distribution ofy will produce an alternative realization ofthe RESULT.

Y'

result'

Step 4: A set of alternative realizations of the RESULT is produced from the set of alternative realizations for parameters x and y.

result'

Y'

result'

This set of alternative realizations represents Type B uncertainty on the RESULT, which is itself a stochastic variable representing Type A uncertainty.

26

1

3. METHODS FOR UNCERTAINTY ANALYSIS

Step 5: A sufficiently large number of alternative realizations of the result (i.e.,the distribution for the stochastic variable) is produced to obtain subjective confidence intervals at any desired percentile. (Each alternative realization of the result is represented by a single line.)

Ninety percent subjective confidence intervals at the 2.5, 50, and 97.5 percentiles.

Percentiles of the distribution Step 6: Subjective confidence intervals can also be produced for the percentile occurring at a given value of the result (e.g.,RESULT = 10).

Giy.?? yalye of the result

............

Percentiles of the distribution The second stage involves use of Monte Carlo procedures to produce an alternative realization of the true but unknown distribution (the assessment endpoint representing stochasticvariability) for each set of alternative realizations of fixed but unknown quantities. Each alternative realization of the true but unknown distribution is a representation of Type A uncertainty. The number of alternative realizations of the assessment endpoint is made sufficiently large

3.8 UNCERTAINTY ANALYSIS FOR A STOCHASTIC ENDPOINT

1

27

to enable a subjective confidence interval t o be obtained at each percentile of t h e assessment endpoint. T h e set of alternative realizations of t h e t r u e but unknown distribution t h a t is used to obtain t h e subjective confidence intervals represents Type B uncertainty (IAEA, 1989). Subjective confidence statements can be made either for the value of dose o r risk occurring at a given percentile or for the percentile associated with a given value of dose or risk. T h e ranking of the model components that contribute most to the subjective confidence interval for risk at a given percentile of t h e population will depend on the percentile of interest (IAEA, 1989). A detailed illustration of t h e technique described in this Section is provided in Example 3.6. Additional readings on this issue can be obtained from a number of authors (Bogen, 1990; Frey, 1993; Helton, 1993b; 1994; Helton and Breeding, 1993; Helton et al., 1992; 1993; Hofer, 1990; Kaplan a n d Garrick, 1981). Example 3.6 This Example is entirely hypothetical and is for demonstration purposes only. Situation: Let us assume that there has been an accidental spill of mercury in a small lake. The problem is to determine the value of the HQ that is not likely to be exceeded by more than five percent of the population of potentially exposed individuals. The primary route of exposure is from the ingestion of contaminated fish. Solution: For this problem, we use a simplified form of an equation from EPA Superfund guidelines for human health risk assessment (EPA, 1989):

where HQ Cf I BM

= =

RfD

=

=

=

hazard quotient (unitless) concentration of contaminant in fish (mg kg-'), estimated intake rate of fish by humans for 1y, averaged over 1y (kg d-') body mass (kg) reference dose for the chemical of interest (mg kg-' d-')

The assessment endpoint is the true but unknown distribution of the HQ for individuals in the exposed population. The stochastic variability of the contaminant concentration in fish, of body mass, and of daily intake rate contribute to the interindividual variability of the HQ and must be considered as Type A uncertainty. Type B uncertainty (lack of knowledge about values that are fixed with respect to the assessment endpoint) is also present because the true values of the means and standard deviations of these three stochastically varying parameters are unknown. In this Example, only Type B uncertainty is considered for the RfD for mercury, i.e., the RfD is assumed in this Example to have a fixed but unknown value that does not vary from individual to individual. The first step is to quantify the state of knowledge associated with all fixed but unknown values (Type B uncertainty). By use of judgment after a thorough review of existing data, a log-uniform distribution is assigned to represent the uncertainty (due to lack of knowledge) about the fixed but unknown value of the RfD. The fixed but unknown means and standard deviations of the distributions used to describe the stochastically varying parameters Cf,B M and I are assigned subjective probability distributions representing lack of knowledge about the true values. From these subjective probability distributions, 59 alternative realizations of the RfD and of the means and standard deviations of Cf, B M and I are obtained using SRS in a Monte Carlo calculation. The magnitude and spread of values for Cf, I and B M are determined by the selected values of the means and standard deviations defining the Type A distributions of these parameters (Table 3.8). From each of these Type A distributions, 100 values are obtained using LHS. These values are combined with one of the 59 values selected for the RfD to produce a resulting distribution composed of 100

28

1

3. METHODS FOR UNCERTAINTY ANALYSIS

TABLE3.8-Subjective Fixed quantity

probability distributions representing the state of knowledge about fixed but uncertain quantities. RfD C,(mean) Cf(s.d.) I(mean) Z(s.d.) BM(mean) BM(s.d.)

(units) 1 (kg) Type 'B' Distribution Tb Uc U LU LUa min 9.503 - 05 2.003 - 01 5.003 - 02 3.00E - 03 0.4*I(mean) --mode 5.003- 01 ------max 3.003 - 03 1.00E + 00 5.003 - 01 2.00E - 02 2.0*I(mean) aLog-uniform distribution. bTriangular distribution. "niform distribution.

T 60 70 90

T 7 10 14

values for the HQ. This process is repeated 59 times to produce 59 Type A distributions (Figure 3.5 and Tables 3.9 and 3.10). For this Example, the HQ value exceeded by five percent of the population lies between 0.6 and 1.3 at a subjective confidence level of a t least 95 percent (Figure 3.5, top). It is also possible to determine a 95 percent subjective confidence interval of the percentile for a given value of the HQ. For instance, for a n HQ of one, the range of percentile values from 85 to 97 is a subjective confidence interval of a t least 95 percent, indicating that the true proportion of the population having values of HQ less than or equal to one is expected to be greater than 85 percent but less than 97 percent (Figure 3.5, bottom). Ideally, the Monte Carlo simulation would have included many more than 59 realizations of the distribution of individual values of the HQ. However, for the sake of calculational convenience, the number of alternative realizations in this Example was limited to 59 (and the sampling error in each realization is assumed to be negligible). Using nonparametric tolerance limits (Conover, 1980), the classical statistical confidence is at least 80 percent that the value of HQ associated with the percentile of interest (or the percentile associated with the HQ of interest) lies between the minimum and maximum values a t a subjective confidence level of a t least 95 percent.

TABLE3.9-Alternative realizations produced using Monte Carlo methods to sample from the subjective probability distributions described i n Table 3.8. Rfo Alternative Realizations 1 2 3

(mg kg-' d-') 4.31E - 04 3.473 - 04 7.323 - 04

Cf(mean) (mg kg-') 8.41E - 01 3.643 -01 8.283 - 01

I(s.d.) BM(mean) BM(s.d.1 I(mean) Cf(s.d.) (mg kg-') (kg d-') (kg) (kg) (kg d-') 75.85 10.26 9.413 - 02 1.093 - 02 5.053 - 03 81.27 11.54 3.723 - 01 1.70E - 02 2.873 -02 76.62 9.29 2.163 - 01 7.653 - 03 1.363 - 02

3.8 UNCERTAINTY ANALYSIS FOR A STOCHASTIC ENDPOINT

0.1

5

50

95

1

29

99.9

Percent

Fig. 3.5. Fifty-nine alternative realizations of Type "A" distributions of the HQ plotted on a probability plot. Top: Subjective confidence internal for the value of HQ associated with the 95thpercentile of the true distribution. Bottom: Subjective confidence interval for the percentile associated with a n HQ value of one. Due to the small number of alternative realizations, the subjective confidence intervals have been drawn from statistical tolerance limits.

30

1

3. METHODS FOR UNCERTAINTY ANALYSIS

T ~ L 3.10-Fifty-nine E alternative realizations of Type "A"distributions defining stochastic interindividual variability of HQ in a population. Alternative Cf I BM RP HQ Realization (mg kg-') (kg d-l) (kg) (mgkg-' d-I ) (unitless)

h

59

x

/cI +

u

u

P.,&

Llrl

U., L b 2

LI*,

+

6.21EM

=

!

t

0

0175

PJJ

OdU

I.#

4. Elicitation of Expert Judgment The previous sections have described the mechanics of error propagation through a mathematical model. This Section describes the most defensible approach for quantifyrng the state of knowledge about uncertain parameters and submodels in the absence of data sets that are directly relevant to the assessment question. When relevant data are available, they can either be used directly or summarized using classical statistical methods to obtain distributions for use in an uncertainty analysis (see also Section 3.6). When data are unavailable or only partially relevant, experience with similar situations, reviews of the literature, and judgment must be used to quantify uncertainty. Probability distributions obtained in this manner are subjective. This Section focuses on methods for eliciting subjective estimates of the current state of knowledge from individuals acknowledged for their specialized expertise. The material described in this Section is primarily from Cooke (1991), Hora (19931, Hora and Iman (1989), Meyer and Booker (1991), Morgan and Henrion (1990), and Ortiz et al. (1991). The advantage offered by a formal approach to the elicitation of expert judgment, as described in Hora and Iman (1989) and Ortiz et al. (1991), is transparency in the production of a subjective PDF for an uncertain parameter.

4.1 Meaning and Use of Expert Judgment

In order to discuss the meaning of expert judgment, the term "expert" must be defined. An expert is someone who (1)has training and experience in the subject area resultingin extensive knowledge of the field, (2) has access to relevant information, (3) has an ability to process and effectively use the information, and (4) is recognized by his or her peers or those conducting the study as qualified to provide judgments about assumptions, models and model parameters a t the level of detail required (Bonano et al., 1990; Meyer and Booker, 1991). Expert judgment is a quaIitative or quantitative inference or evaluation based on an assessment of data, assumptions, criteria, models and parameters in response to questions posed in the expert's area of expertise (Bonano et al., 1990). It pervades all scientific inquiry, but is often unacknowledged, poorly understood or neglected (Hora, 1993). Expert judgments are expressions of opinion that represent the expert's state of knowledge at the time of response to the technical question (Ortiz et al., 1991), but expert judgment is not opinion substituted for objective scientific research. The expert's role "is not creating knowledge, but synthesizing disparate and often conflicting sources of information to produce an integrated picture" (Hora, 1993). Expert judgment is a complement to, rather than a substitute for, other sources of scientific and technical information; expert judgments should not be considered equivalent to technical evaluations based on scientific laws or data (Bonano et al., 1990). As new data, calculations or scientific understanding become available, expert judgments will change. Expert judgment is needed at all stages of the risk assessment process, from the preliminary screening efforts that determine the scope of a risk assessment effort, to the estimation of risks using the most defensible information and data that can be determined to meet the objectives of the assessment. Expert judgment as used in the risk assessment process has

32

1

4. ELICITATION OF EXPERT JUDGMENT

been discussed in detail by Bonano et al. (1990), Meyer and Booker (1991), and Rechard et al. (1992) for the following activities: specification of parameter values and their uncertainties for conducting exploratory risk assessments to identifjr parameters or models for which additional information is required quantification of key parameters and their uncertainties as subjective PDFs determination of gaps in knowledge and areas where research is needed to reduce uncertainty priority setting for data collection determination of the level of effort needed to reduce uncertainties to a desired level formulation of approaches for validating conceptual and computational models and for verifying computer codes reconstruction of individual exposures from the historical release of contaminants to the environment

4.2 Choosing Between Formal and Informal Elicitation Methods

Expert judgments are of two types: informal (implicit) judgment and formal (explicit)judgment. Informal judgment methods may be used in the selection of data sources and models; when conducting preliminary screening risk assessments; or when specifying problems, questions, required decisions, exposure scenarios and boundary conditions. Informal methods include self-assessment, brainstorming, casual elicitation from an expert (without structured efforts to control biases), and taped group discussions of the issues by project staff or readily available experts. Such informal methods are integral to all of science and are a necessary part of the risk assessment process. An example of a common situation in which informal judgment is utilized is in the specification of the PDF for a parameter which will be used in a screening study to determine the sensitivity of risk estimates to changes in parameter values and PDFs. A knowledgeable project staff person may be asked to use self-assessment to provide the initial PDF based on that person's knowledge of the site and information in scientific reports and papers. The PDF provided using this informal judgment may be adequate for the screening objective, but completely inadequate for a final risk assessment that must pass rigid peer review and public scrutiny. Formal elicitation methods have a predetermined structure for selecting and training experts and for eliciting, processing and documenting expert judgments and their rationales. Some advantages of using a formal approach include the following:

Improved quality of expert judgments that occurs due to careful selection of experts, use of multiple experts, dealing with psychological biases, definition of problems, improved communication, systematic analysis of issues, and the careful documentation of rationales and results. Reduced likelihood of critical mistakes, such as inclusion of experts with motivational biases, not checking for consistencies, and permitting individuals to dominate group interactions. Such mistakes can make information suspect or biased and reduce the credibility of risk estimates. Improved scrutability or accountability of the process, i.e., the procedures and assessments are documented so that reviewers and users of the results can understand what was done, why it was done, and what the results mean.

4.3 FORMAL ELICITATION PROCESS

/

33

Improved consistency ofprocedures as compared to procedures that change with the whims and desires of participants. Enhanced communication among the experts as well as between experts and project staff. Reduced chance of unexpected delay because critical steps were skipped. Use of a formal approach limits the opportunity for major mistakes to occur. Formal methods of eliciting expert judgments are more costly and time-consuming than informal methods. Hence, it is important to determine when formal methods are necessary. In general, formal methods are needed for problems where widely accepted models or sufficient data do not exist, and when results are needed that are both scrutable and defensible (Hora, 1993). For example, a formal elicitation method may be used to obtain a scrutable PDF for a parameter of the risk assessment model(s) whose uncertainty, as specified by the PDF, is expected or known to have a significant impact on the magnitude and uncertainty of the risk estimate. In this situation, a PDF obtained via informal self-assessment or brainstorming may not withstand the scrutiny of peer reviewers. In the following situations, formal methods of expert elicitation may be needed (Bonano et al., 1990; Hora, 1993): when the relevance of existing data is questioned and when acquisition of new data is impractical when information to be gathered will have a major impact on the results of a study of high visibility when costs of redoing an informal study that failed to withstand criticism or that used faulty methods are expected to exceed the costs of doing a formal study when the findings are apt to undergo close scrutiny, especially where legal action is possible when a detailed level of documentation is required when several experts will be employed individually or as a team when expert opinions will be extensively used by other studies Clearly, formal and informal elicitation methods should be balanced so that project expenditures are optimized to achieve the project objectives. The decision whether to use formal elicitation methods is the responsibility of the project staff. However, the needs and requirements of all stakeholders (e.g., project staff, the funding agency, regulatory agencies and the public) should be considered in making this decision. The decision and its rationale should be thoroughly documented (Bonano et al., 1990). Because subjective judgment is required for any activity requiring the use of models to extrapolate information beyond the realm of observation, it is important to understand that different results can be expected from different assessment groups. Therefore, in the absence of formal expert elicitation, important assessments should be assigned to a t least two if not more independent groups of assessors to identify potential discrepancies (BIOMOVS, 1993; IAEA, 1995). A final step in which these discrepancies are resolved should serve to improve the overall credibility of the assessment.

4.3 Formal Elicitation Process

The remainder of this Section focuses on the essential components of the formal elicitation process for developing or encoding probability distributions for key parameters. This discussion is based in large part on the ten-step methodology used to obtain expert judgment on issues expected to be main contributors to the uncertainty in estimates of offsite risk for five nuclear

34

/

4. ELICITATION OF EXPERT JUDGMENT

plants (NRC, 1990);this process was developed further for a later study (NRC, 1995). Whether this detailed formal procedure, or some variant of it, is needed for a given parameter of a risk assessment model is determined in each individual case on the basis of the objectives and goals of the study, as well as on funding and time constraints and stakeholder concerns. The ten-step formal process in NRC (1990), as discussed in Hora and Iman (1989) and Ortiz et al. (1991), is as follows: 1. Selection of issues and associated parameters 2. Selection of experts 3. Preparation of background information 4. Training of the selected experts in the elicitation method 5. Presentation of issues, questions and background information to experts 6. Analysis of issues, questions and background information by experts 7. Discussion of analyses of issues and background information by experts 8. Elicitations of judgment from the experts 9. Consolidation (and possibly aggregation) of the elicitation results 10. Review and communication of the elicitations

Each individual step is briefly discussed below, with discussions based on Hora (19931, Hora and Iman (1989), and Ortiz et al. (1991). 4.3.1 Selection of Issues and Associated Parameters

The selection and clear statement of issues and associated model parameters are critically important parts of the formal elicitation process. Selected issues and parameters should be those (1)that have a major impact on estimates of risk andlor risk uncertainty, and (2) for which alternative sources of information (e.g., data, observations and validated computer models) are not available. Important issues may be identified initially by project staff or policy makers. Issue selection can be aided by conducting sensitivity and uncertainty analyses, keeping in mind that such analyses are model-dependent. The cost-effectiveness of using formal expert elicitation methods as opposed t o alternative sources of information (if they exist) should be considered in selecting issues and parameters. Issues that will attract public attention and outside peer review are candidates for formal elicitation methods. Both selected issues and excluded issues and the rationale for selection and exclusion should be presented to the experts (see Section 4.3.2) for their evaluation. 4.3.2

Selection of Experts

Selected experts should have recognized experience and expertise in the subject area addressed by the issue as well as expertise in answering questions in the required response mode or form (probabilities, continuous scales, ratings, etc.). It is advisable to use many experts who have different views of the problem and who may solve the problem in different ways. Selection of well-known experts who have credibility among their peers and the public can provide greater credibility to the elicitation process as well as encourage other experts to participate. If controversy is a possibility, a formal nomination process for selecting experts may be used. The number of experts selected will depend on the number of experts who are identified, available and willing to participate, and on the selected elicitation method (faceto-face, telephone or mail). Meyer and Booker (1991) discuss in detail how to select issues, questions and experts.

4.3 FORMAL ELICITATION PROCESS

1

35

The ideal number of experts in a study depends upon many factors. The breadth and difficulty of the issues, the cost and difficulty of obtaining experts, the method of organizing the experts, the importance and uses of the judgments, and the statistical dependence among the judgments all affect the decision. When teams of experts are used, the number of experts is apt to be large because each team provides an overall judgment rather than each individual. Practical evidence on the best number of experts to use is hard to come by. Winkler (1971) found little benefit from using more than five or six judgments when predicting football scores. Practice in risk analysis indicates that from 4 to 12 experts should be used in a panel format, with 6 to 8 being a good middle ground. Because experts usually share knowledge to some extent, judgments are often correlated, and thus the combined judgment of many experts carries less information than would be the case if the experts were independent. In an interesting theoretical work, Clemen and Winkler (1985) show that if the correlation among experts' judgments is 0.8, no matter how large the number of experts is, their information cannot be greater than that obtained from two independent experts of equal capability. 4.3.3

Preparation of Background Information

Background information for the experts includes both a description of the issue(s) and specific elicitation questions about the issue(s). Hora (1993) emphasizes the critical importance of achieving an accurate, logically complete (without unstated assumptions) and understandable description of an issue to be addressed by an expert. This is a difficult task. Formulating correct and clearly worded questions regarding each selected issue or parameter requires a precise definition of the issue. If possible, complex issues and questions should be broken down into simpler parts for the elicitation process. A team consistingof project personnel and representatives of the various stakeholders should develop the elicitation questions. The experts should be provided with clearly synthesized and presented background information on the issue or parameter. The questions should be welldefined and not open to differing interpretations; they should elicit the required information in a reasonable amount of time without extreme effort. If time permits, the description of the issue and specific questions regarding the issue should be tested in a dry run with project staff, as discussed by Meyer and Booker (1991). Questions should pertain to physically realizable values (Winkler, 1968). For example, some models have parameters that are simply fitted parameters and do not have a physical meaning. A good example is the Gaussian dispersion model which describes the horizontal spread of a plume in terms of a standard deviation that grows according to a power law. The exponent in the power law should not be an elicitation variable. Instead, questions should be asked about the concentrations in the plume and, perhaps, the spread of the plume a t various points. The exponent in the power law can be obtained from the experts' results. 4.3.4

Training of the Selected Experts in the Elicitation Method

The primary purpose of training is to help prepare experts for expressing (encoding) their beliefs about issues and parameters as probability distributions (Hora, 1993).Training includes informing the experts about the general elicitation process and schedule to be followed, as well as the issues and questions that will be asked, the documentation process, and the common biases that can occur during the encoding process. Common biases include (1)a tendency to underestimate uncertainty, e.g., encoding distributions whose tails are too short, (2) a tendency

36

1

4. ELICITATION OF EXPERT JUDGMENT

to anchor beliefs on a single experiment or model, which results in not considering the full range of available information, and (3) use of only the most readily available information. Training and the use of experienced project staff can help overcome these biases. Training should also include practicing the elicitation process. Meyer and Booker (1991) discuss training aspects in detail. 4.3.5

Presentation of Issues, Questions and Background Information to Experts

Either before or after training, the issues are presented to the expert (or group of experts) in a meeting (Hora and Iman, 1989). The issues may be presented in the form of issue papers. Each paper should clearly and concisely state the issue and associated model parameters; present background information on models, parameters, data and computer analyses related to the issue; describe the relationship of the issue to the overall risk assessment; and provide an example of the breakdown of the issue into simpler parts, if warranted. This meeting is held to make sure that all experts understand the issues and will be responding to the same issues, and that the experts will have the information needed to begin studying the issues in subsequent weeks. The meeting provides experts with the opportunity to begin discussing whether issues should be excluded or modified and to be presented with alternative data sources and models. Project staff may propose division of the problem into simpler pieces that can be answered more easily by the experts. The experts are encouraged to challenge the proposed division and to begin considering modifications. 4.3.6 Analysis of Issues, Questions and Background Information by Experts

The experts are allowed several weeks to study the issues and questions, to exchange information, and to propose and discuss alternative consolidations. The purpose of these analyses is the accumulation of information and preparation by experts of summaries (or full reports) of their approaches, which are then circulated among all experts. 4.3.7

Discussion of Analyses of Issues and Background Information by Experts

A meeting of project staff and experts is held to review previous activities, to provide an opportunity for the experts to present the results of their analyses and research, and to assure that all experts are exposed to the same information. The technical rationale of favored models and experiments should be explained by the experts. However, to avoid biasing each other, the experts do not yet present their probabilistic characterization of the uncertainty in the issues under consideration. The meeting also provides for project staff, stakeholders andlor individuals from universities and consulting companies to present other relevant information. 4.3.8

Elicitation of Judgment from the Experts

The purpose of the elicitation is to encode the judgment of the expert or group of experts in a PDF that expresses what is known and what is unknown about the parameter. Methods for conducting elicitations are discussed by Meyer and Booker (1991), Morgan and Henrion (19901, and von Winterfeldt and Edwards (1986). Formal elicitations should be conducted with the aid of one to three support personnel. The nuclear power reactor risk assessment project (Hora and Iman, 1989) used two support

4.3 FORMAL ELICITATION PROCESS

1

37

personnel: a normative analyst and a substantive analyst. The normative analyst was an expert in probability assessment and decision analysis; the substantive analyst was knowledgeable in the subject area being addressed. A third person may be needed to assure proper documentation of the elicitation process. There are several formats for eliciting the PDF: an expert, in private, develops the PDF and provides it to the support personnel a group of interacting experts develops the PDF with the aid of support personnel the support personnel meet with each expert privately to elicit the PDF When elicitations are made individually from many experts, differences in response can be expected because of differences in their experience and methods of solving the problem. However, differences in response due to differences in knowledge, information, assumptions and biases should be minimized by the expert training discussed in Section 4.3.4. Sometimes group elicitations may generate more ideas and information and also produce more accurate data than elicitations from each expert in private. Several methods for group interaction are available, including open forums, Delphi panels, group probability encoding, and formal group evaluation (Roberds, 1990). The Delphi panel method was developed to reduce bias arising from group dynamics. This approach requires each expert to develop his information in private. Then the information from each expert is collected by the support personnel, made anonymous, and distributed to all experts, who are then allowed to change their original input. This procedure may continue until responses no longer change or consensus is achieved (Meyer and Booker, 1991). For the case where support personnel meet with each expert privately, Hora (1993) indicates that the elicitation session is usually under the control of the normative analyst (probability specialist). The analyst has three goals: to extract rationales for judgments, to quantify judgments, and to assist the expert in making the elicited PDFs accurate reflections of the expert's knowledge. Achieving these goals may require extensive interactions between the expert and the analyst. Several elicitation methods may be used in a single assessment session to reveal and resolve inconsistencies (Bonano et al., 1990; Hora, 1993; Hora et al., 1992). Physical aids such as the probability wheel have been developed to help experts quantify probabilities (Morgan and Henrion, 1990).As the elicitation session draws to a close, the expert is presented with a summary of his estimates. This summary can help the expert detect any important omissions, inconsistencies or misinterpretations.

4.3.9

Consolidation (and Possibly Aggregation) of the Elicitation Results

The expert may have divided the parameter (or issue) into smaller parts and provided a PDF for each part. If so, this information must be consolidated to produce a single PDF for the original parameter. This consolidation may be done using decision analysis methods implemented by computer programs (Hora and Iman, 1989; Ortiz et al., 1991). The individual consolidated PDFs from the experts can then be aggregated to obtain the required PDF. This aggregation may be done by averaging the probabilities of the various experts a t fixed values of the parameter. Alternatively, Monte Carlo methods may be used wherein equal numbers ofvalues from each expert's consolidated PDF are randomly generated and placed in ascending order to obtain the aggregated PDF. These methods are discussed and illustrated by Hora and Iman (1989) and Ortiz et al. (1991). Chapter 16 in Meyer and Booker (1991) provides examples of aggregation methods.

38 4.3.10

4. ELICITATION OF EXPERT JUDGMENT

Review and Communication of the Results of the Elicitations

The entire elicitation process (Steps 1through lo), including results, must be documented. Each expert should be sent a written record of his elicitation and results for review and comment. Modifications to the assessments should be made if requested by the expert. Hora (1993) emphasizes that "the detailed rationales for the assessments, t h e methods, and results of any post-elicitation processing of the judgments should be provided." Also, documentation of the rationales for the subjective PDFs will allow for updating (rather t h a n discarding) the PDFs a s new information becomes available. The interpretation of t h e results is particularly important. For example, when the endpoint is a true but unknown value, the elicited PDF should not reflect variability of reported data, but rather, uncertainty about the fixed but unknown value. The elicited P D F is constructed after consideration of the assessment endpoint and all other possible sources of information, including published d a t a i n the literature. Finally, the results must be effectively communicated to decision makers and other stakeholders or the elicitation effort will have been wasted. Graphical methods for communicating uncertainty and elicited PDFs are discussed and illustrated in Chapter 9 of Morgan and Henrion (1990). The utility of elicitation of expert opinion for a n uncertainty analysis is demonstrated in Example 4.1. Example 4.1

An example of expert elicitation i n a n uncertainty analysis: The ozone chronic risk assessment During the 1988 to 1989 review of the ozone National Ambient Air Quality Standard, the issue was raised as to whether repeated or long-term exposure to ozone at levels observed in areas such as southern California and the Northeast leads to chronic effects in the form of irreversible lung injury or lung disease. The results of the animal toxicology studies discussed in Chapter 10 of "Air Quality Criteria for Ozone and Other Photochemical Oxidants" (EPA, 1986, hereafter referred to as the criteria document or CD) suggest that ozone exposure at sufficiently high concentrations over a prolonged period results in permanent structural changes to the lung. Among the possible effects of prolonged exposure are narrowing of the distal airways (i.e., thickened respiratory bronchiole walls), remodeling of the centriacinar airways, and thickening of the interalveolar septa resulting in part from increased collagen and epithelial thickening. Because few direct data addressed this issue, EPA's Office of Air Quality Planning and Standards (OAQPS) chose to obtain the judgments of experts to better understand possible effects of prolonged ozone exposure. This effort was called the ozone chronic risk assessment. The ozone chronic risk assessment (McKee, 1994; Whitfield et al., 1991; Winkler et al., 1995) is discussed in the context of the ten-step formal process presented in Section 4.3; the assessment followed the 10 steps very closely, but not exactly. Two encoding sessions were conducted with each expert. The first session was conducted at each expert's office. The second session was a workshop that all experts attended and where first-round judgments were presented and discussed. The workshop was followed by a second round of encoding sessions-one encoding session for each expert. Two or three members of the risk assessment team3were present to assist the expert and record the proceedings ofeach session. Select issues a n d parameters For the assessment conducted in 1989 (McKee, 1994; Whitfield et al., 1991), EPA chose mild and moderate lesions in the centriacinar region of the lung as the health endpoint for which judgments 3Members of the risk assessment team were Stan R. Hayes and Arlene S. Rosenbaum (Systems Applications, Inc.),Thomas S. Wallsten (University of North Carolina at Chapel Hill), Robert L. Winkler (Duke University), and Ronald G. Whitfield (Argonne National Laboratory).

4.3 FORMAL ELICITATION PROCESS

/

39

about probabilistic exposure-response relationships were to be elicited. This choice was made during the process of developing, in consultation with cognizant EPA staff and outside experts, an encoding protocol. Mild centriacinar lesions were defined as "those detectable only by highly sensitive techniques and quantitative analysis'' (e.g., morphometric techniques, special stains, and electron microscopy). Moderate centriacinar lesions were defined as "those detectable by qualitative assessment using light microscopy but not necessarily gross techniques." The centriacinar region of the lung was pinpointed because it has been shown to be the location of maximum ozone dose and maximum lung injury in a number of species (e.g., rats, guinea pigs and monkeys) exposed to elevated ozone concentrations (EPA, 1986). The health scientists consulted in the early phases of this assessment agreed that centriacinar lesions are a good indicator of potential chronic lung injury caused by ozone exposure. For this assessment, EPA selected children aged 6 through 18 and all adult outdoor workers in two urban areas-New York City and Los Angeles-as the populations of interest. These two groups were thought to be a t greater risk for chronic effects because their members tend to be outdoors more than most other individuals when ozone concentrations are a t their highest. Select experts In consultation with the EPA Environmental Criteria and Assessment Office (ECAO) staff (who are responsible for writing CDs), OAQPS chose six experts and invited them to participate in the chronic risk assessment. All agreed to participate. The experts were chosen on the basis of their knowledge and familiarity with the health effects of ozone exposure as well as their various viewpoints and approaches. The experts were not, and should not have been, chosen randomly. Rather, they were selected for their expertise and because, as a group, they represent the range of respected scientific opinion on the questions of interest. The individuals who participated were: Dr. Daniel Costa, U.S. Environmental Protection Agency Dr. James Crapo, Duke University Dr. Charles Plopper, University of California, Davis Dr. Russell L. Sherwin, University of Southern California Dr. Walter Tyler, University of California, Davis Dr. Elaine Wright, General Motors Corporation Prepare background information A carefully developed encoding protocol was crucial to the success of the project. This protocol ensured that the questions for the experts were always phrased identically, that specific assumptions and definitions were always the same, and that the encoding process proceeded similarly for each of the participants. Consequently, the differences in probability judgments could be attributed to true differences of opinion and not to differences in assumptions, understandings or procedures. In developing the encoding protocol, background material was assembled, and the assessment team consulted extensively with individuals (named in the acknowledgments in Whitfield et al., 1991)having expertise in different aspects of the problem. Ofparticular concern were the appropriate health endpoints to consider and the best way to represent exposure conditions. However, many other points also required careful attention. Dealing with chronic rather than acute effects greatly increased the number of factors that had to be considered. On the basis of the literature and discussions with consultants, a draft of the encoding protocol was prepared. At various points, the draft was circulated to some or all of the participating consultants. Written comments were received in some cases; meetings were held in other cases. In this manner, several drafts of the protocol were generated and revised. After several revisions, the protocol was pilot-tested on an EPA staff member and an EPA contractor. Some weaknesses not previously recognized became apparent during the pilot testing, and the protocol was revised further to correct them. Consultations and discussions among members of the risk assessment team led to some final tuning to create the protocol that was actually used in the encoding process. The following is a brief annotated outline of the contents of the encoding protocol:

40

/

4. ELICITATION OF EXPERT JUL)GMENT

Introduction: This section of the protocol presents the motivation for and objectives of the encoding process. Centriminar lesions: In this part of the protocol, the health endpoints of interest are defined as mild and moderate centriacinar lesions. The results of animal studies suggest that long-term ozone exposure can lead to numerous structural changes in the lung, such a s narrowing of distal airways, remodeling of the centriacinar airways, and thickening of the interalveolar septa. Each of these health effects is important, and all are correlated. However, it is only when one of them is precisely defined that the concept of a dose-response function can be applied to it. Even then, it is not always clear what level of the endpoint should be considered adverse. It was therefore necessary for the encoding process to select an endpoint for which mild and moderate levels could be precisely defined. The mild level was a t the lower end of what might be considered adverse, and the moderate level was somewhat farther along on the continuum. Exposure conditions: To deal with chronic effects, it was decided to consider centriacinar damage over a single ozone season and also over 10 ozone seasons. Specifying exposure conditions within a season required extensive consultation because of variations in exposure pattern over time. Ambient ozone concentrations vary diurnally, daily and seasonally. Moreover, an individual's personal exposure depends on hls or her location and activity level at any point in time. The challenge was to represent typical exposure variations within a season in an easily comprehensible manner while specifying exposure levels in a form suitable for discussing dose-response functions. This task was particularly important because experts differ a s to the ozone exposure characteristics they deem instrumental in causing adverse health effects. It was important not to omit or misrepresent exposure information that might be deemed crucial. The solution was to specify seasonal means for daily maximum 8 h exposure values for individuals of 0.025, 0.050,0.075 and 0.100 parts per million (ppm). For each seasonal mean and for each of two populations in two geographic locations, associated data were provided on the distributions of these personal exposures, on the distributions of personal and ambient daily peak exposures, and on other characteristics of the exposure patterns. As described earlier, these data were to be as representative as possible of the actual range of conditions associated with a given seasonal mean value for each population of interest in each geographic location. The histograms illustrating patterns of exposure are explained in this part of the protocol, as are the dosimetry results. The assumptions made regarding exposure history, ambient conditions, and other air pollutants are also stated. Population a t risk: The populations considered for the encoding are all children aged 6 through 18 and all adult outdoor workers in Los Angeles and New York City. These populations include people with and without respiratory problems and allergies, smokers and nonsmokers, and people exhibiting different levels of activity. It was decided not to stratify the populations along any of these dimensions, although the possibility that one or more experts might wish to do so was not ruled out. (As it turned out, none did.) Therefore, the experts had to assume distributions for these and any other factors that they believed to exist in the populations of interest. Factors to consider: This part of the protocol serves to remind the experts of factors that may be relevant to their judgments: results reported in the literature, issues concerning extrapolation from animal studies, stages of response, time scales of effects, levels of repair, effects of varying the exposure concentrations over time (e.g., relative importance of cumulative exposure versus peak exposure), and possible interactions of ozone with other air pollutants. The experts are asked about their qualitative judgments concerning these factors. Health significance: This brief part of the protocol simply sets the stage for an exploration of the experts' opinions about the short- and long-term health consequences of lung damage resulting from longterm ozone exposure. Important questions include how significant the centriacinar lesions are in themselves and what implications they might have for a person's future health. Factors to keep in mind when making probability judgments: The purpose of this part of the protocol is to help the experts provide probabilities that accurately represent their judgments. This objective

4.3 FORMAL ELICITATION PROCESS

1

41

is accomplished primarily through review of some commonly found cognitive biases and simplifying heuristics and through some suggestions on how to try to mitigate their effects. The topics covered include (1) effects related to the order in which evidence is considered or probabilities assessed, (2) the effect of memory on judgment, (3) the tendency to overestimate the reliability of information, (4) the influence of the importance of events on probability judgments, and (5)the difficulty of discriminating between levels of uncertainty. T r a i n experts No formal training was provided to the experts prior to the encoding sessions. The risk assessment team trusted that the experts would read and study the encoding protocol, particularly the section dealing with factors to keep in mind when making probability judgments. At the beginning of the encoding sessions (which were conducted in the offices of the experts), two or three risk assessment team members explained the mechanics of providing probability judgments. Different response modes4 and stimuli5were explained and tried with the experts. What worked best for a particular individual was favored throughout the encoding session. Usually, a mixture of the possibilities was used with each expert. The end result in all cases was a set of cumulative probability distributions over response rate, conditional on a specified ozone concentration. P r e s e n t issues The training that the experts did receive was generally phrased in the context of the judgments that were sought from them, making it possible to explore the basis for their judgments. The encoding protocol and other materials (e.g.,relevant sections of the CD, recent papers, information on variations in ozone levels over time in Los Angeles and New York City) that had been provided insured that there was some amount of similar information available to all experts. Discussion of t h e analysis The encoding protocol was mailed to the experts several weeks before the encoding sessions were conducted. The letter attached to the protocol and the protocol encouraged the experts to prepare for the encoding session by reviewing the materials that had been provided and other materials that they had and thought would be helpful. Thus, the experts had time to think of issues and questions to discuss with risk assessment team members during their encoding sessions. E x p e r t s discuss analysis of issues a n d background information The workshop that was held between the two rounds of encoding sessions provided a forum for the experts to further discuss the issues and relevant information with each other and with ECAO and OAQPS staff who were present a t the workshop. One expert substantially changed some judgments as a result of the discussion a t the workshop. Conduct elicitations

As was mentioned earlier, judgments were obtained during two rounds of encoding sessions. Two or three members of the risk assessment team met privately with each expert to elicit judgments. "Two types of response mode were tried: direct and indirect. In the direct response mode, the expert responds to questions that require numbers as answers. In the indirect mode, the expert chooses between two propositions involving probability statements; the probabilities are adjusted until the expert finds it difficult to choose between the two propositions. GProbabilityaids used include a small probability wheel, ordinary playing cards, many-sided dice, coins, graph paper and a computer program that facilitates probability encoding.

42

1

4. ELICITATION OF EXPERT JUDGMENT

Judgments were not considered "final" until each expert "signed off' on the judgments recorded after the second round of encodings. Experts were free to change their judgments a t any time before "signing off." The judgments were about the response rate for the formation of mild and moderate lesions in the centriacinar region of the human lung, conditional on specific assumptions about the average ozone concentration to which the "populations of interest" (i.e., children and outdoor workers) would be exposed. An example of the judgments that were obtained is shown in Figure 4.1, which shows a set of cumulative probability distributions over the fraction of children in Los Angeles in whom mild lesions form after exposure to tropospheric ozone for one season (sometimes, for simplicity, called the response rate for the formation of mild lesions), according to Expert A. There are four probability distributions in Figure 4.1, one for each of four ozone concentrations: 0.025, 0.050, 0.075 and 0.010 ~ p r n It . ~is interesting to note that 0.025 ppm is considered to be a naturally occurring level of ozone, and that Expert A judges that ozone-induced mild lesions will occur in some children after being exposed at this level for one ozone season. Expert A's judgments are uncertain. Even a t background levels, Expert A judged a central 98 percent subjective probability interval of 0.15 to 0.75 for response rate. The central 90 percent subjective probability interval, obtained by interpolation, is almost as wide-0.18 to 0.72. Another way of looking at these judgments, which is more familiar to health scientists, is shown in Figure 4.2, which depicts the median, 0.05-fractile, and 0.95-fractile exposure-response relationships. Taken together, the 0.05- and 0.95-fractile exposure-response relationships define central 90 percent subjective probability intervals over the range shown for ozone concentration.

Consolidate and aggregate results Consolidation was not needed in the ozone chronic risk assessment because the problem was not divided into smaller parts to facilitate the elicitation process. The experts directly judged the probability of formation of lesion^.^

Review and communication of results The entire elicitation process and results were carefully documented. Experts received written records of their judgments after each round of encoding. After the second round of encoding, the experts were asked to "sign off' on their judgments to signify that what was recorded accurately reflected their feelings and beliefs. The experts were free to change any of their judgments until they signed off. The entire process from time of receipt of the encoding protocol to time of signoff covered several months and included a workshop attended by all of the experts. Thus, there was ample time for the experts to give careful thought to their judgments. Documentation included not only the quantitative judgments, but also extensive discussion of related qualitative judgments expressed during the encoding sessions and the workshop. Topics included extrapolation from animals to humans; dosimetry; definition of the health endpoint; time scale, sequence of

90be precise, these are seasonal mean daily maximum 8 h ozone concentrations. Such a measure is obtained by first finding the average value of the "worst" 8 h period for each day in the ozone season (i.e., the 8 h period with the highest average ozone concentration), and then computing the average of all of the daily averages for the entire ozone season. 'For an example in which it was necessary to divide the problem into two parts so that experts could reliably make probability judgments, see the lead risk assessment (Wallsten and Whitfield, 1986; Whitfield and Wallsten, 1989) that was conducted in 1984 to 1985. In that assessment, one of the health endpoints was intelligence quotient (IQ) decrement (i.e., reduced IQ). To obtain probability distributions over IQ decrement, it was necessary to define three components: mean IQ decrement among children not exposed to lead, mean IQ decrement among children exposed to lead, and population standard deviation for IQ levels. These components were then combined to obtain the desired probability distributions.

4.3

FORMAL ELICITATION PROCESS

1

43

AIBBl

"1 (0

s

e

0.6

OR

-&-

0.05 pprn

+

0.075 ppm

-e 0.1ppm

0.2

0.4

0.6

0.8

1.O

Response Rate

Fig. 4.1. The judgments of Expert A regarding the probability of formation of ozone-induced mild lesions among Los Angeles children after exposure to ozone for one season.

AIBBl

0.06

0.02

0.04

0.06

0.m

0.10

Ozone Concentration (ppm) Fig. 4.2.

Judgments of Expert A in exposure-response format.

events, and effect of time variation; degree of repair between ozone seasons; interaction with other pollutants; health significance of effects; populations a t risk; and research needs. No attempt was made to develop a consensus among the experts. In fact, one goal of the enterprise was to insure that experts had the opportunity to freely express their own judgments and that the judgments would not be altered in any way. For example, it would have been easy to "average" the judgments of the experts; however, the risk assessment team had agreed not to do this. Differences between the experts were considered to be important information- evidence of the lack ofknowledge and uncertainty about the probability of formation of lesions. All judgments were tabulated, and graphs (e.g.,Figure 4.3) were used to compare and contrast the judgments.

44

/

4. ELICITATION OF EXPERT JUDGMENT

lLCl0

ILCl

ABCDEF

ABCDEF

ABCDEF

ABCDFF

ABCOEF

0.075 ppm

0.100 ppn

0.025 ppm

aDSOv

0 . 0 7 5 ~ 0.100 ppm

L

. . _ _ I . .

0.025 ppm

O.mO ppm

O.o!,.T=: . , . ,

. ....... . . .

n . . . . .

OLClO

OLcl

w

(1)

1.o

T 0.8-

0.1

? -

-

8

4

ABCDEF

-

T a d

0.4-

0.2-

-r :

! i

!

0

-

1

on^----I;.

,

I

ABCDEF

ABCDEF

O&Sppm

0.OSOppm

,

. , ,,;, , , .*. ,4

ABCDEF

ABCOEF

ABCDEF

ABCDEF

0.075 Ppm

0.1OOW

O.DS

ppm

0.060 ppm

a .

,'

ABCDEF 0.0% ppll

ABCDEF

alw ppn

Fig. 4.3. Medians and 90 percent subjective confidence intervals for judgments about the rate for the formation of mild and moderate lesions in individuals Living in Los Angeles for 1 and 10 ozone seasons. Figure 4.3 displays the medians and central 90 percent subjective confidence intervals for judgments of Experts A through F regarding the probability of formation of mild and moderate lesions after 1 and 10 seasons of ozone exposure among children (and, in some cases, workers) living in Los Angeles (and, in some cases, New York City). In the Figure, symbols (e.g.,circles, squares, etc.) identify the median judgments of the experts. The upper and lower ends of the central 90 percent subjective confidence interval about a median judgment of an expert are indicated by short horizontal lines above and below, respectively, the median, and vertical lines (of different styles) conned the median to the upper and lower ends of its confidence interval. The five or six character codes identifying the judgments have the following meaning: Character 1: Character 2: Character 3: Character 4: Characters 5-6:

Expert (A, B, C, D, E or F) Lesion type (I = mild, 0 = moderate, B = both) City (L = Los Angeles, N = New York City, B = both) Population (C = children, W = outdoor workers, B = both) Number of ozone seasons (1,10, B = both)

For example, CIBCl denotes Expert C's judgments about mild lesions for both Los Angeles and New York C i t p children after exposure to ozone for one season. In a subsequent phase of the ozone risk assessment, these probabilistic exposure-response relationships were combined with exposure distributions to compute overall risk, which also was a probability distribution. Results were computed for each expert to clearly show differences (when they exist) attributable to the judgments of the experts. 8Expert C judged that populations of children in Los Angeles and New York City would respond identically, given the same seasonal mean daily maximum 8 h ozone concentrations.

5. Summary and Conclusions This Commentary provides a brief survey of methods for addressing the issue of estimation of uncertainty and propagation of uncertainty through a model. When uncertainty must be disclosed, all sources of uncertainty should be considered, including uncertainty in model parameters and uncertainty in model structure. These sources of uncertainty may be propagated through the preferred model (or through several alternative models) to produce an estimate of uncertainty for the assessment endpoint. Uncertainty may be propagated using algebraic formulae for relatively simple equations; however, for more complex equations, Monte Carlo simulation i s preferred by most practitioners of uncertainty analysis. Quantitative uncertainty analysis usually requires that the state of knowledge about the uncertain components of the mathematical model be described by probability distributions. I n the absence of site-specific data, these distributions a r e derived using professional judgment along with whatever evidence is available to corroborate the current state of knowledge about the model parameters, the model structure, and any dependencies t h a t exist among uncertain model components. In fact, under most circumstances, expert judgment must be employed because, even when site-specific data exist, they are seldom complete. Probability distributions obtained using judgment are subjective. They are employed to represent the state of knowledge about fixed quantities, such as uncertain model parameters, correlation coefficients, and uncertain correction terms accounting for potential bias due to uncertainty in model structure. They can also be assigned to describe the state of knowledge about the mean, variance and shape of model parameters that represent true distributions about which there is uncertainty. When the assessment endpoint is a true (but unknown) distribution of values, uncertainty due to the presence of stochastic variability and uncertainty due to lack of knowledge about fixed quantities should be evaluated separately. When relevant site-specific d a t a are unavailable, the most defensible method for obtaining subjective probability distributions is through the formal elicitation of expert judgment. Formal elicitation is used to encode what is known and not known about a n uncertain model component. Because of the expense involved, exploratory analyses a r e recommended to first identify the model components of dominant importance to the overall estimate of uncertainty. Expert elicitation then focuses on refining the quantification of the state of knowledge for these components. In the absence of formal expert elicitation, important assessments should be assigned to a t least two independent organizations, and t h e assessment should include resources for resolving discrepancies.

APPENDIX A

Equations Used to Calculate the Mean and Standard Deviation of Various Distributions Used in Analytical Error Propagation

Log-normal distribution: p = the mean of the logarithms a2 = the variance of the logarithms

However, in a situation where you are given only the arithmetic mean and arithmetic variance, then p and a2can be approximated with the following equations (Hoffman and Gardner, 1983):

(A.1)

where

a

the arithmetic mean of the distribution s = the standard deviation of the distribution =

Log-uniform distribution (Hoffman and Gardner, 1983):

where "rnin" and "max"are the lower and upper limits, respectively, of the distribution.

EQUATIONS FOR ANALYTICAL ERROR PROPAGATION

1

47

Asymmetrical log-triangular distribution (Johnson and Kotz, 1970):

where

H* a b

= =

=

the mode of the triangular distribution the minimum of the triangular distribution the maximum of the triangular distribution

Normal distribution: The mean value of the normal distribution is simply the value a t the 50th percentile. In a normal distribution, the median, mode and mean are the same. The variance of the normal distribution is the second central moment of the variable or the standard deviation squared. Uniform distribution: X = min

+ max 2

Asymmetrical triangular distribution (Johnson and Kotz, 1970):

1

%=-(H*+b +a) 3 s2 =

1 [(a)Z+ (b)l - (a)(b)+ (H*)' 18

-

(H*)(a + b)l

(A.10)

Other distributions: In addition to these distributions, others that may be used are the Poisson, Weibull, gamma and beta distributions, as well as any number of discrete distributions (Decisioneering, Inc., 1994; Palisade Corporation, 1991).

References BARTELL, S.M., BRECK, J.E., GARDNER, R.H. and BRENKERT, A.L. (1986). "Individual parameter perturbation and error analysis of fish bioenergetics models," Can. J . Fish. Aquat. Sci. 43, 160-168. BIOMOVS (1993). Biospheric Model Validation Study. Final Report, BIOMOVS Technical Report 15 (Swedish Radiation Protection Institute, Stockholm). BOGEN, K.T. (1990). Uncertainty in Environmental Health Risk Assessment (Garland Publishing, Inc., New York). BOGEN, K.T. (1992). RiskQ: A n Interactive Approach to Probability, Uncertainty, and Statistics for Use with Mathematica-Reference Manual (Lawrence Livermore National Laboratory, Livermore, California). BONANO, E.J., HORA, S.C., KEENEY, R.L. and VON WINTERFELDT, D. (1990). Elicitation and Use of Expert Judgment in Performance Assessment for High Level Radioactive Waste Repositories, U.S. Nuclear Regulatory Commission NUREGICR-5411 (Sandia National Laboratory, Albuquerque, New Mexico). BRENKERT, A.L., GARDNER, R.H., BARTELL, S.M. and HOFFMAN, F.O. (1988). "Uncertainties associated with estimates of radium accumulation in lake sediments and biota," pages 185 to 192 in Reliability of Radioactive Transfer Models, Desmet, G., Ed., Commission of the European Communities (Elsevier Applied Science, London). CACUCI, D.G. (1981). "Sensitivity theory for nonlinear systems. Part I and 11," J. Math. Phys. 22,2794. CLEMEN, R.T. and WINKLER, R.L. (1985). "Limits for the precision and value of information from dependent sources," Oper. Res. 33, 427-442. CONOVER, W.J. (1980). Practical Nonparametric Statistics, 2nd ed. (John Wiley & Sons, New York). COOKE, R.M. (1991). Experts i n Uncertainty, Opinion and Subjective Probability i n Science (Oxford University Press, New York). COOKE, R.M. and VOGT, F. (1990). PARFUM Parameter Fitting for Uncertain Models (University of Technology, Delft, Netherlands). DECISIONEERING, INC. (1994). Crystal Ball: User's Guide (Decisioneering, Inc., Denver, Colorado). DEVROYE, L. (1986). Non-Uniform Random Deviate Generation (Springer-Verlag, Inc., New York). DOWNING, D.J., GARDNER, R.H. and HOFFMAN, F.O. (1985). "An examination of response-surface methodologies for uncertainty analysis in assessment models," Technometrics 27, 151-163. DRAPER, D., HODGES, J.S., LEARNER, E.E., MORRIS, C.N. and RUBIN, D.B. (1987). A Research Agenda for Assessment and Propagation of Model Uncertainty, N-2683-RC (The RAND Corporation, Santa Monica, California). EPA(1986). U.S. Environmental Protection Agency. h r Quality Criteria for Ozone and Other Photochemical Oxidants, EPA-60018-841020a to 020e (U.S. Environmental Protection Agency, Washington). EPA (1989). U.S. Environmental Protection Agency. Risk Assessment Guidance for Superfund, Volume I: H u m a n Health Evaluation Manual (Part A), EPAl54011-891002 (U.S. Environmental Protection Agency, Waslungton). EVANS, J.S. (1990). Health Effects Models for Nuclear Power Plant Accident Consequence Analysis, U.S. Nuclear Regulatory Commission NUREGICR-4214, Rev. 1, P a r t 1 (National Technical Information Service, Springfield, Virginia). FERSON, S. and KUHN, R. (1992). "Propagating uncertainty i n ecological risk analysis using interval and fuzzy arithmetic," pages 387 to 481 in Computer Techniques in Environmental Studies N, Zannetti, P., Ed. (Elsevier Applied Science, London). FREY, H.C. (1993). "Separating variability and uncertainty in exposure assessment: Motivations and method," Paper No. 93-RA-116A.02 in Proceedings of the 86th Annual Meeting of the Air and Waste Management Association (Air and Waste Management Association, Pittsburgh, Pennsylvania).

GARDNER, R.H. (1988). TAM3: A Program Demonstrating Monte Carlo Sensitivity and Uncertainty Analysis, Biospheric Model Validation Study, BIOMOVS Technical Report 2 (Swedish Radiation Protection Institute, Stockholm). GARDNER, R.H. and TRABALKA, J.R. (1985). Methods of Uncertainty Analysis for a Global Carbon Dioxide Model, U.S. Department of Energy Technical Report DOE/OW21400-4 (National Technical Information Service, Sprin$eld, Virginia). GARDNER, R.H., O'NEILL, R.V., MANKIN, J.B. and CARNEY, J.H. (1981). "A comparison of sensitivity analysis and error analysis based on a stream ecosystem model," Ecol. Model. 12, 177-194. GARDNER, R.H., ROJDER, B. and BERGSTROM, U. (1983). PRISM: A Systematic Method for Determining the Effectof Parameter Uncertainties on Model Predictions, Studsvik Energiteknik AB Report/ NW-831555 (Nykoping, Sweden). GLAESER, H., HOFER, E., KLOOS, M. and SKOREK, T. (1994). "Uncertainty and sensitivity analysis of a post-experiment calculation in thermal hydraulics," Rel. Eng. Sys. Saf. 45, 19-33. GRIEWANK, A. and CORLISS, H., Eds. (1991).Automatic Differentiation ofAlgorithms: Theory, Implementation and Application (Society for Industrial and Applied Mathematics, Philadelphia). HAMBY, D.M. (1994). "A review of techniques for parameter sensitivity analysis of environmental models," Environ. Monit. Assess. 32, 125-154. HELTON, J.C. (1993a). "Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal," Rel. Eng. Sys. Saf. 42, 3271367. HELTON, J.C. (1993b). "Risk, uncertainty in risk, and the EPA release limits for radioactive waste disposal," Nucl. Technol. 101, 18-39. HELTON, J.C. (1994). "Treatment of uncertainty in performance assessments for complex systems," Risk Anal. 14, 483-511. HELTON, J.C. and BREEDING, R.J. (1993). "Calculation of reactor accident safety goals," Rel. Eng. Sys. Saf. 39, 129-158. HELTON, J.C., GARNER, J.W.,RECHARD, R.P., RUDEEN, D.K. and SWIFT, P..N. (1992). Preliminary Comparison with 40CFR Part 191, Subpart B for the Waste Isolation Pilot Plant: Volume 4: Uncertainty and Sensitiuity Analysis Results, SAND91-089314 (Sandia National Laboratory, Albuquerque, New Mexico). HELTON, J.C., GARNER, J.W., MARIETTA, M.G., RECHARD, R.P., RUDEEN, D.K and SWIFT, P.N. (1993). "Uncertainty and sensitivity analysis results obtained in a preliminary performance assessment for the waste isolation pilot plant," Nucl. Sci. Eng. 114, 286-331. HENRION, M. and WISHBOW, N. (1987). DEMOS User's Manual: Version Three (Carnegie Mellon University, Pittsburgh, Pennsylvania). HOFER, E. (1990)."On some distinctions in uncertainty analysis," in Methods for Treatment ofDifferent Types of Uncertainty, PSACDOC (90).11 (Organization for Economic Cooperation and Development, Nuclear Energy Agency, Paris). HOFFMAN, F.O. and GARDNER, R.H. (1983). "Evaluation of uncertainties in radiological assessment models," Chapter 11 of Radiological Assessment: A Textbook on Environmental Dose Analysis, Till, J.E. and Meyer, H.R., Eds. (U.S. Nuclear Regulatory Commission, Washington). HOFFMAN, F.O., MILLER, C.W. and NG, Y.C. (1984). "Uncertainties in radioecological assessment models," pages 331 to 349 in Proceedings of a Seminar on the Environmental Transfer to Man of Radionuclides Released from Nuclear Installations (Commission of European Communities, Luxembourg). HOFFMAN, F.O., BLAYLOCK, B.G., FRANK, M.F., HOOK, L.A., ETNIER, E.L. and TALMAGE, S.S. (1991). Preliminary Screening of Contaminants in the Off-Site Surface Water Environment Downstream of the U.S. Department of Energy Oak Ridge Reservation, ORNIdER-9 (Oak Ridge National Laboratory, Oak Ridge, Tennessee). HOFFMAN, F.O., BLAYLOCK, B.G., FRANK,M.L. and TI-IIESSEN, K.M. (1993). "A risk- based screening approach for prioritizing contaminants and exposure pathways at Superfund sites," Envimn. Monit. Assess. 28, 221-237. H O W , S.C. (1993). "Acquisition of expert judgment: Examples from risk assessment," J. Energy Eng. 118, 136-148.

50

/

REFERENCES

HORA, S.C. and IMAN, R.L. (1989). "Expert opinion in risk analysis: The NUREG-1150 methodology," Nucl. Sci. Eng. 102, 323-331. HORA, S.C., HORA, J.A. and DODD, N.G. (1992). "Assessment ofprobability distributions for continuous random variables: A comparison of the bisection and fixed value methods," Organ. Behavior Human Dec. Proc. 51, 133-155. IAEA (1989). International Atomic Energy Agency. Evaluating the Reliability ofPredictions Made Using Environmental Transfer Models, IAEA Safety Series 100 (International Atomic Energy Agency, Vienna). IAEA (1995). International Atomic Energy Agency. Validation of Models Using Chernobyl Fallout Data from the Central Bohemia Region of the Czech Republic-Scenario CB, First report of the VAMP Multiple Pathways Assessment Working Group, IAEA TECDOC-795 (International Atomic Energy Agency, Vienna). IMAN, R.L. and CONOVER, W.J. (1980). "Small sample sensitivity analysis techniques for computer models, with an application to risk assessment," Commun. Stat., P a r t A 9, 1749-1842. IMAN, R.L. and CONOVER, W.J. (1982). "A distribution-free approach to introducing rank correlation among input variables," Commun. Stat., P a r t B 11, 311-334. IMAN, R.L. and HELTON, J.C. (1988). "An investigation of uncertainty and sensitivity analysis techniques for computer models," Risk Anal. 8, 71-90. IMAN, R.L. and HELTON, J.C. (1991). "The repeatability of uncertainty and sensitivity analyses for complex probabilistic risk assessments," Risk Anal. 11, 591-606. IMAN, R.L. and SHORTENCARIER, M.J. (1984). A FORTRAN 77 Program a n d User's Guide for the Generation of Latin Hypercube a n d Random Samples for Use with Computer Models, U.S. Nuclear Regulatory Commission NUREGICR-3624 (Sandia National Laboratory, Albuquerque, New Mexico). IMAN, R.L., HELTON, J.C. and CAMPBELL, J.E. (1981a). "An approach to sensitivity analysis of computer models, P a r t 1. Introduction, input variable selection and preliminary variable assessment," J . Qual. Tech. 13, 174-183. IMAN, R.L., HELTON, J.C. and CAMPBELL, J.E. (1981b). "An approach to sensitivity analysis of computer models, Part 2. Ranking of input variables, response surface validation, distribution effect and techniques synopsis," J. Qual. Tech. 13, 232-240. JOHNSON, N.L. and KOTZ, S. (1970). Continuous Uniuariate Distributions, Vol. 2. (Houghton Mifflin Company, Boston). KANYAR, B. and NIELSEN, S.P. (1989). Users Guide for the Program TAMDYN, Biospheric Model Validation S t u d y , BIOMOVS Technical Report 4 (Swedish Radiation Protection I n s t i t u t e , Stockholm). KAPLAN, S, and GARRICK, B.J. (1981). "On the quantitative definition of risk," Risk Anal. 1, 11-27. KLEE, A.J. (1986). The MOUSE Manual ( U S . Environmental Protection Agency, Cincinnati, Ohio). LAVENUE, M.A. and RAO, R.B.S. (1992). A Modeling Approach to Address Spatial Variability within the Culebra Dolomite Transmissiuity Field, SAND92-7306 (Sandia National Laboratory, Albuquerque, New Mexico). LEIGH, C.D., THOMPSON, B.M., CAMPBELL, J.E., LONGSINE, D.E., KENNEDY, R.A. and NAPIER, B.A. (1992). User's Guide for GENZZ-S: A Code for Statistical a n d Deterministic Simulations of Radiation Doses to Humans from Radionuclides in the Enuironment, Report UC-721 (Sandia National Laboratory, Albuquerque, New Mexico). MARTZ, H.F. and WALLER, R.A. (1982). Bayesian Reliability Analysis (John Wiley & Sons, New York). MCKAY, M.D., CONOVER, W.J. and BECKMAN, R.J. (1979). "A comparison of three methods for selecting values of input variables in the analysis of output from a computer code," Technometrics 21,239-245. MCKEE, D.J., Ed. (1994). Tropospheric Ozone: Human Health a n d Agricultural Impacts (Lewis Publishers, Inc., Boca Raton, Florida). MCKONE, T.E. and BOGEN, K.T. (1992). "Uncertainties i n health risk assessment: An integrated case based on tetrachloroethylene in California groundwater," Reg. Tox. Phann. 15, 86-103. MEAD, R. and PIKE, D.J. (1975). "Areview of response surface methodology from a biometric viewpoint," Biometries 31, 803-85.1.

REFERENCES

1

51

MEYER, M.A. and BOOKER, J.M. (1991). Eliciting and Analyzing Expert Judgment, A Practical Guide (Academic Press, New York). MORGAN, M.G. and HENRION, M. (1990). Uncertainty, A Guide to Dealing with Uncertainty i n Quantitative Risk and Policy Analysis (Cambridge University Press, New York). MORTON, R.H. (1983). "Response surface methodology," Math. Sci. 8,31-52. MURRAY, D.M. and BURMASTER, D.E. (1993). "Review of RiskQ: An interactive approach to probability, uncertainty, and statistics for use with Mathematica,'' Risk Anal. 13,479-482. MYERS, R.H. (1971). Response Surface Methodology (Allyn and Bacon, Boston). NAS/NRC (1995). National Academy of SciencesJNational Research Council. Radiation Dose Reconstruction for Epidemiologic Uses (National Academy Press, Washington). NCRP (1989). National Council on Radiation Protection and Measurements. Screening Techniques for Determining Compliance with Environmental Standards, Releases ofRadionuclides to the Atmosphere, NCRP Commentary 3, rev. (National Council on Radiation Protection and Measurements, Bethesda, Maryland). NCRP (1993). National Council on Radiation Protection and Measurements. Limitation of Exposure to Ionizing Radiation, NCRP Report No. 116 (National Council on Radiation Protection and Measurements, Bethesda, Maryland). NCRP (1996). National Council on Radiation Protection and Measurements. Screening Models for Releases ofRadionuclides to Atmosphere, Surface Water, and Ground, NCRP Report No. 123 (National Council on Radiation Protection and Measurements, Bethesda, Maryland). NRC (1990). U.S. Nuclear Regulatory Commission. Severe Accident Risks: A n Assessment for Five U.S. Nuclear Power Plants, NUREG1150 (National Technical Information Service, Springfield, Virginia). NRC (1995). U.S. Nuclear Regulatory Commission. Probabilistic Accident Consequence Uncertainty Analysis, NUREGICR-6244 (National Technical Information Service, Springfield, Virginia). O'NEILL, R.V., GARDNER, R.H., HOFFMAii, F.O. and SCHWARZ, G. (1981). "Parameter uncertainty and estimated radiological dose to man from atmospheric 1-131 releases: A Monte Carlo approach," Health Phy. 40,760-764. ORTIZ, N.R., WHEELER, T.A., BREEDING, R.J., HORA, S., MEYER, M.A. and KEENEY, R.L. (1991). "Use of expert judgment in NUREG-1150," Nuc. Eng. Design 126,313-331. PALISADE CORPORATION (1991). 63 RISK: Risk Analysis and Simulation Add-In for Microsoft Excel: User's Guide (Newfield, New York). PRENTICE, R.L. and THOMAS, D.C. (1993). "Methodologic research needs in environmental epidemiology: Data analysis," Environ. Health Persp. 101, 39-48. RECHARD, R.P., TRAUTH, K.M., RATH, J.S., GUZOWSKI, R.V., HORA, S.C. and TIERNEY, M.S. (1992). The Use of Formal and Informal Expert Judgments When Interpreting Data for Performance Assessments, SAND-92-1148C (Sandia National Laboratory, Albuquerque, New Mexico). ROBERDS, W.J. (1990). Methods for Developing Defensible Subjective Probability Assessments, Transportation Research Board Record No. 1288 (National Academy Press, Washington). RUBINSTEIN, R.Y. (1981). Simulation and the Monte Carlo Method (John Wiley & Sons, New York). SALTELLI, A. and MARIOVET, J. (1990). "Non-parametric statistics in sensitivity analysis for model output: A comparison of selected techniques," Rel. Eng. Sys. Saf. 28, 229-253. SCAVIA, D., POWERS, W.F., CANALE, R.P. and MOODY, J.L. (1981). "Comparison of first-order error analysis and Monte Carlo simulation in time-dependent lake eutrophication models," Water Resour. Res. 17, 1051-1059. VON WINTERFELDT, D. and EDWARDS, W. (1986). Decision Analysis and Behavioral Research (Cambridge University Press, Cambridge, United Kingdom). WALLSTEN, T.S. and WHITFIELD, R.G. (1986).Assessing the Risks to Young Children of Three Effects Associated with Elevated Blood-Lead Levels, Argonne National Laboratory Report No. ANLIAA-32 (Argonne National Laboratory, Argonne, Illinois). WEBER, G., HOFER, E. and KRZYKACZ, B. (1992). ''Uncertainty and sensitivity analysis of fog formation rates calculated with the containment code FIPLOC-M," J . Aerosol Sci. 23, S269-S272. WHITFIELD, R.G. and WALLSTEN, T.S. (1989). "A risk assessment for selected lead-induced health effects: An example of a general methodology," Risk Anal. 9, 197-207.

52

1

REFERENCES

WHITFIELD, R.G., WALLSTEN, T.S., WINKLER, R.L., RICHMOND, H.M., HAYES, S.R. and ROSENBAUM, A.S. (1991).Assessing the Risk of Chronic Lung Injury Attributable to Ozone Exposure, Argonne National Laboratory Report No. EATS-2 ( h g o n n e National Laboratory, Argonne, Illinois). WILLIAMS, K.A. and HUDSON, C.R., I1 (1989). ORMONTE: An Uncertainty Analysis Code for Use with User-Developed Systems Models on Mainframe or Personal Computers; A User's Guide, ORNLI TM-10714 (Oak Ridge National Laboratory, Oak Ridge, Tennessee). WINKLER, R.L. (1968). "The consensus of subjective probability distributions," Manage. Sci. 15, B-61 to B-75. WINKLER, R.L. (1971). "Probabilistic prediction: Some experimental results," J. Am. Stat. Assoc. 66,675-685. WINKLER, R.L., WALLSTEN, T.S., WHITFIELD, R.G., HAYES, S.R. and ROSENBAUM, A.S. (1995). "An assessment of the risk of chronic lung injury attributable to long-term ozone exposure," Oper. Res. 43, 19-28. WORLEY, B.A. (1987). Deterministic Uncertainty Analysis, ORNL-6428 (Oak Ridge National Laboratory, Oak Ridge, Tennessee).

The NCRP The National Council on Radiation Protection and Measurements is a nonprofit corporation chartered by Congress in 1964 to:

1. Collect, analyze, develop and disseminate i n t h e public interest information and recommendations about (a) protection against radiation and (b) radiation measurements, quantities and units, particularly those concerned with radiation protection. 2. Provide a means by which organizations concerned with the scientific and related aspects of radiation protection and of radiation quantities, units and measurements may cooperate for effective utilization of their combined resources, and to stimulate the work of such organizations.

3. Develop basic concepts about radiation quantities, units and measurements, about the application of these concepts, and about radiation protection. 4. Cooperate with the International Commission on Radiological Protection, the International Commission on Radiation Units and Measurements, and other national and international organizations, governmental and private, concerned with radiation quantities, units and measurements and with radiation protection. The Council is the successor to the unincorporated association of scientists known as the National Committee on Radiation Protection and Measurements and was formed to carry on the work begun by the Committee in 1929. The Council is made up of the members and the participants who serve on the scientific committees of the Council. The Council members who are selected solely on the basis of their scientific expertise are drawn from public and private universities, medical centers, national and private laboratories, and industry. The scientific committees are composed of experts having detailed knowledge and competence in the particular area of the committee's interest.

NCRP Commentaries NCRP commentaries are documents that provide preliminary evaluations, critiques, reviews, results of exploratory studies, or extensions of previously published NCRP reports on an accelerated schedule when time for the normal Council review process is not available. They are approved for publication by the Board of Directors of the Council. Included in the series are: No.

Title

1 Krypton-85 in the Atmosphere-With

Specific Reference to the Public Health Significance of the Proposed Controlled Release at Three Mile Island (1980) Preliminary Evaluation of Criteria for the Disposal of Transuranic Contaminated Waste (1982) Screening Techniques for Determining Compliance with Environmental StandardsReleases of Radionuclides to the Atmosphere (19861, Revised (1989) Guidelines for the Release of Waste Water from Nuclear Facilities with Special Reference to the Public Health Significance of the Proposed Release of Treated Waste Waters at Three Mile Island (1987) Review of the Publication, Living Without Landfills (1989) Radon Exposure of the U.S. Population-Status of the Problem (1991) Misadministration of Radioactive Material in Medicine-Scientific Background (1991) Uncertainty in NCRP Screening Models Relating to Atmospheric Transport, Deposition and Uptake by Humans (1993) Considerations Regarding the Unintended Radiation Exposure of the Embryo, Fetus or Nursing Child (1994) Advising the Public about Radiation Emergencies (1994) Dose Limits for Individuals Who Receive Exposure from Radionuclide Therapy Patients (1995) Radiation Exposure and High-Altitude Flight (1995) A n Introduction to Efficacy in Diagnostic Radiology and Nuclear Medicine (Justification of Medical Radiation Exposure) (1995) A Guide for Uncertainty Analysis in Dose and Risk Assessments Related to Environmental Contamination (1996)

NCRP commentaries are distributed by the NCRP Publication Office. Information on prices and how to order may be obtained by directing an inquiry to: NCRP Publications 7910 Woodmont Avenue, Suite 800 Bethesda, MD 20814-3095