Pharmaceutical and Medical Device Validation by Experimental Design

Pharmaceutical

and Medical Device Validation by Experimental Design

Pharmaceutical

and Medical Device Validation by Experimental Design

Edited by

Lynn D. Torbeck

Torbeck & Associates, Inc. Evanston, Illinois, USA

Informa Healthcare USA, Inc. 52 Vanderbilt Avenue New York, NY 10017 © 2007 by Informa Healthcare USA, Inc. Informa Healthcare is an Informa business No claim to original U.S. Government works Printed in the United States of America on acid-free paper 10 9 8 7 6 5 4 3 2 1 International Standard Book Number-10: 1-4200-5569-0 (Hardcover) International Standard Book Number-13: 978-1-4200-5569-6 (Hardcover) This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish reliable data and information, but the author and the publisher cannot assume responsibility for the validity of all materials or for the consequence of their use. No part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC) 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Library of Congress Cataloging-in-Publication Data Pharmaceutical and medical device validation by experimental design / edited by Lynn D. Torbeck. p. ; cm. Includes bibliographical references and index. ISBN-13: 978-1-4200-5569-6 (hb : alk. paper) ISBN-10: 1-4200-5569-0 (hb : alk. paper) 1. Pharmaceutical technology– Validity–Research–Methodology. 2. Experimental design. I. Torbeck, Lynn D. [DNLM: 1. Technology, Pharmaceutical–methods. 2. Multivariate Analysis. 3. Quality Control. 4. Reproducibility of Results. 5. Research Design. 6. Technology, Pharmaceutical–standards. QV 778 P5353995 2007] RS192.P468 2007 615’.19–dc22 Visit the Informa Web site at www.informa.com and the Informa Healthcare Web site at www.informahealthcare.com

2007001418

Foreword

Imagine the following scenario: you have a seemingly impossible quality problem to solve. A new lyophilized product your company is about to launch is apparently unstable at commercial scale. Management is breathing down your neck for an immediate solution, since the research and development scale-up data indicated the formulation is stable. A crisis team was formed and has been working on a solution for almost six months. Seeing the difficulty, a colleague suggests a different approach to looking at the problem, calling together a subteam of four people that are the most knowledgeable about the production process. They use a few simple tools and determine the most probable cause of the problem in a couple of hours using existing data. The resulting action plan corrects the problem, stable lyophilized lots are produced, and the product is launched on schedule. Does this sound too good to be true? Fortunately, for those facing similar seemingly insuperable problems, the story is actually true. Not only is this specific story true, but also there are many more like it. Virtually every major biopharmaceutical company in the world has its own case study. These solutions were achieved thanks to a few simple process analysis tools and a systematic structured approach to visualizing data-rich experiments. iii

iv

Foreword

How do I know they work? Because I have personally used them, I have taught people to use them, and I have seen the results myself first hand. Read this book. Buy copies for the people in your company who work on difficult problems and train them in the methodology. Use the techniques to help find your own solutions to nearly impossible problems. Do not miss a chance to use the most powerful toolkit I have seen used in my 35 years in the industry. If you do not use these tools, you are also missing the opportunity to bring robust life saving therapeutics, as quickly as possible, to the people we really work for— the patients and their families. THE APPROACH DOE is an acronym for design of experiments, also called experimental design or multifactor experiments. These experimental approaches provide rich veins of data that can be mined for information that cannot be found any other way. The reason for design in the description is that DOE experiments must be scientifically sound and statistically valid, especially for use in the highly regulated biopharmaceutical industry. Scientific soundness is the cornerstone of the scientific peer review process, and (read world regulatory authority) Food and Drug Administration review and approval of such studies in support of product license approval. Statistical validity is necessary to ensure the integrity of the experiments and to appropriately interpret the significance of the data. In real terms, these are the same science and statistics that are applied to clinical trials; now the application is being extended to production process monitoring, quality control testing, validation and process analytical technology or PAT. The DOE concepts are also valuable tools capable of being used to exploit existing data and help solve seemingly impossible product quality problems. Scientists and managers are traditionally taught that experiments should only change one factor at a time (e.g., temperature), holding other factors constant (time, pH, etc.),

Foreword

v

in order to explain the individual effect on product quality. By contrast, the design of experiments trials are carefully worked out to assess many possible factors (time, temperature, and pH) at various levels (one hour vs. two hours at 25°C vs. 30°C at pH 6.5 vs. pH 7.5) to determine which one, or combination, has the greatest effect on the product’s quality characteristics, such as yield, impurities, and viral reduction. DOE measures not only the single factor effects, but also the cumulative and interaction effects of all the factors investigated on product quality. Most important is the fact that DOE is the only way to see interaction effects; one factor at a time experiments give you just that, one effect for each factor. INTERACTION EFFECTS DOE is the only technique that enables scientists and managers to find, see, and use interaction effects that can improve product quality and yield or help set process boundaries to prevent failure. The well entrenched views that only one factor at a time can be studied and the widely held management maxim that “if it ain’t broke, don’t fix it” are not only wrong but, in some cases, dangerous; a process can drift into failure, or periodically have an “unexplainable” failure, due to interaction effects. DOE permits scientists to conduct small scale, low cost process improvement experiments to model large scale, “unbroken” commercial processes without endangering current production runs and product supply. These same experiments can also generate data that drives continuous process improvement (read “make more profitable”) of a process that “ain’t broke.” LEAST COST/FASTEST ROUTE DOE is the least costly way to collect data. First, the basis for performing a series of designed experiments is usually reviewing the existing process database; the longer a process has run the richer the possibilities. Second, the experiments

vi

Foreword

are usually conducted using a small scale process simulation—a much less expensive approach than tying up the production line. Third, the scale and cost of down-sized runs permits a relatively larger number of runs in a relatively short period of time. The resulting body of data provides a roadmap for process improvements that can be verified by additional small scale or scaled up experiments prior to full scale transition that meets ongoing production requirements. VALIDATION DOE’s data will define critical process parameters for validation. The results of designed experiments are the identification of the individual parameters and those parameter interactions that have the most effect on product quality and yield. These are the critical process parameters that need to be assessed during process validation. Just as important, DOE identifies those parameters that do not impact product quality. These do not need to be validated but rather monitored and controlled within their defined process ranges. The savings of validating the DOE defined critical process parameters versus validating “everything we think may be critical” is substantial in time, monetary, and human resource terms. Furthermore, given the scientific soundness and statistical validity of DOE results, processes validated using properly applied DOE principles are nearly bullet proof with the world’s regulatory agencies. SUMMARY DOE is the fastest route to a profitable, reliable, robust, validated process. DOE’s requirement of a rigorous design methodology that passes peer review with the scientists most knowledgeable about the process ensures scientific soundness. The depth of DOE’s statistical foundation that enables the measurement of multiple effects and interactions in a single set of experiments proves DOE’s statistical validity. DOE is also a resource conservator, since it requires less time and

Foreword

vii

provides data quicker than single factor at a time experiments. The proven statistical validity of the DOE technique guarantees accurate analysis of the data and valuable information for management to turn data analysis and interpretation into action. Ronald C. Branning Genentech, Inc. South San Francisco, California, U.S.A.

Preface

Designed experiments were shown to be useful for validation almost 30 years ago. The famous chemist/statistician, W. J. Youden, illustrated the use of Plackett–Burman designs for ruggedness testing in the 1975 book, Statistical Manual of the Association of Official Analytical Chemists (1). In that short introduction, he noted that “… if the program is carefully laid out, a surprisingly small amount of work suffices.” The example given showed seven factors varied in eight runs. Each of the seven factors is estimated using all eight data values. This is the equivalent of having 56 data points, but only spending the money to buy eight. This efficiency is very attractive to laboratory managers who are charged with validating many analytical methods in a short time. Note that ruggedness testing is still required today by the Food and Drug Administration in the 1987 Guideline for Submitting Samples and Analytical Data for Method Validation (2). This editor gave his first talk on using designed experiments in 1978 in response to the then draft Current Good Manufacturing Practices. In 1984, Chao, St. Forbes, Johnson, and von Doehren (3) gave an example of granulation validation using a 23 full factorial. Since then, there have been many journal articles and conference talks showing the application

ix

x

Preface

of designed experiments to validation. A recent addition is the text by Lewis et al., Pharmaceutical Experimental Design (4). Yet, for all of the dispersed information and examples, there still is unease in the pharmaceutical industry about using designed experiments in general or for validation specifically. Typical questions include, “What will the Food and Drug Administration say if I use a designed experiment for validation?” or “Won’t that take a lot longer and cost a lot more money?” In answer to the first question, the Food and Drug Administration requires the industry to use carefully designed experiments in clinical and animal studies. The well-regarded double blind clinical trial is a designed experiment. These are usually done exactly by the book. It is hard to see the Food and Drug Administration objecting to the use of designed experiments in validation when it is required in other areas. Further, in 1985, Ed Fry, while with the Food and Drug Administration, said in his excellent article “Then, the processes that cause variability … must be identified. Experiments are conducted (that is validation runs) to ensure that factors that would cause variability, are under control …” (5). This book is the answer to the second question. Designed experiments are the most scientific, the most efficient, and the most cost effective way we know how to collect data. Designed experiments need not be complicated or statistically complex. Realistic case studies and examples illustrate the use of design of experiments for validation. The use of graphics illustrate the designs and results. Formulas are minimized and used only where necessary. Where possible, a step-by-step approach or procedure is given. Detailed protocols and reports with realistic data are given where appropriate. This book succeeds if the reader feels that it is obvious that design of experiments is the most logical and rational approach to use. A variety of examples and case studies are given to show the wide range of application. Assay and bioassay validation, process and equipment validation, etc. Not all cases are “end point” validation, but are further up-stream, and part of the life cycle validation discussed by Chapman (6). Each chapter stands alone. It is not necessary to read them in sequence. The reader is encouraged to delve into the

Preface

xi

chapters that seem most applicable and interesting. The text is intended to be a learn-by-doing-by-example. Find an example close to a intended project and mimic the approach. It is hoped by the authors and the editor that this text will encourage people to use designed experiments for validation and learn first hand the benefits. Lynn D. Torbeck REFERENCES 1.

Youden WJ. Statistical Manual of the Association of Official Analytical Chemists. Arlington, VA: AOAC, 1975.

2.

FDA, Guideline for Submitting Samples and Analytical Data for Method Validation, Rockville, Maryland, 1987.

3.

Chao A, Forbes S, Johnson, R, von Doehren P. Prospective process validation. In: Loftus B, Nash R, eds. Pharmaceutical Process Validation. New York: Marcel Dekker, 1984:125–148.

4.

Lewis G, Mathieu D, Phan-Tan-Luu R. Pharmaceutical Experimental Design. New York: Marcel Dekker, 1999.

5.

Fry E. The FDA’s viewpoint. Drug and Cosmetic Industry 1985; 137(1):46–51.

6.

Chapman K. A history of validation in the United States: part 1. Pharmaceutical Technology October 1991:82–96.

Contents

Foreword Ronald C. Branning . . . . iii Preface . . . . ix Contributors . . . . xv 1. Designing Experiments for Validation of Quantitative Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 1 T. Lynn Eudey 2. Validation of Chemical Reaction Processes . . . . . .47 Robert F. Dillard and Daniel R. Pilipauskas 3. The Role of Designed Experiments in Validation and Process Analytical Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .89 Lynn D. Torbeck and Ronald C. Branning 4. Experimental Design for Bioassay Development and Validation . . . . . . . . . . . . . . . . . .103 David M. Lansky 5. Use of Experimental Design Techniques in the Qualification and Validation of Plasma Protein Manufacturing Processes . . . . . . . . . . . . .119 Sourav Kundu xiii

xiv

Contents

6. Response Surface Methodology for Validation of Oral Dosage Forms . . . . . . . . . . . . . .141 Thomas D. Murphy 7. The Role of Designed Experiments in Developing and Validating Control Plans . . . . . .169 Wayne A. Taylor 8. Efficient and Effective Process Development Work: The Key to a Successful Validation Study . . . . . . . . . . . . . .209 Jeffrey T. Field Index . . . . 241

Contributors

Ronald C. Branning Genentech, Inc., South San Francisco, California, U.S.A. Robert F. Dillard Biostatistics and Data Management, Takeda Pharmaceuticals North America, Inc., Lincolnshire, Illinois, U.S.A. T. Lynn Eudey Department of Statistics, California State University East Bay, Hayward, California, U.S.A. Jeffrey T. Field J. T. Field Consulting Services, LLC, Woodbury, Connecticut, U.S.A. Sourav Kundu Technical Operations, Aventis Behring, Bradley, Illinois, U.S.A. David M. Lansky Lansky Consulting, LLC, d/b/a Precision Bioassay, Burlington, Vermont, U.S.A. Thomas D. Murphy T. D. Murphy Statistical Consulting, LLC, Morristown, New Jersey, U.S.A. Daniel R. Pilipauskas Global Manufacturing Services, Pfizer, Inc., New York, New York, U.S.A.

xv

xvi

Wayne A. Taylor Illinois, U.S.A.

Contributors

Taylor Enterprises, Inc., Libertyville,

Lynn D. Torbeck Torbeck & Associates, Inc., Evanston, Illinois, U.S.A.

1 Designing Experiments for Validation of Quantitative Methods T. LYNN EUDEY Department of Statistics California State University East Bay Hayward, California, U.S.A.

INTRODUCTION AND SCOPE Why Validate? What Are We Measuring? Development Versus Validation What Is Needed? Guidelines for Validation VALIDATION PROTOCOL Protocol Content EXAMPLE VALIDATION PROTOCOL PROTOCOL FOR THE VALIDATION OF SOP 123 FOR USE IN THE DETERMINATION OF MASS OF PRODUCT W §1. Purpose and Scope §2. Background §3. Validation Approach 1

2

Eudey

§4. Responsibilities and Documentation §5. Training Verification and Documentation §6. Test Work §6.1. Accuracy §6.2. Precision §6.3. Linearity §6.4. Range §6.5. Other Performance Characteristics

§7. Validation Matrix §8. Attachments and Forms TERMS AND DEFINITIONS Accuracy Precision and Ruggedness Linearity Robustness DESIGN OF EXPERIMENTS Terms and Definitions Factors, Levels, and Treatments Fixed Effects, Random Effects, Main Effects, and Interactions Nested and Crossed Factors Aliasing and Confounding

Experimental Designs Full Factorial Designs Fractional Factorial Designs Plackett-Burman Designs

Designs for Quantitative Method Validation STATISTICAL ANALYSES EXAMPLE VALIDATION DATA ANALYSIS Model for Accuracy and Precision Linearity Analysis Example Protocol Analysis Results Accuracy Precision Linearity Across-Assay Linearity

CONCLUSIONS REFERENCES

INTRODUCTION AND SCOPE Quantitative methods are assays that result in meaningful numeric measurements for a characteristic of a product. Quantitative methods are used in assessing whether final product meets specifications. They are also used to measure product quality (or quantity) in various stages of manufacturing and the results are often used in quality control charts. Validation is an objective process used to determine whether a quantitative method is performing as expected and is appropriate for its intended use. This chapter provides the motivation behind validation, some terms and definitions used in validation, a consolidated statistically sound approach to validation, along with appropriate statistical analysis, and reporting of validation results. A hypothetical but realistic example is presented and is used to illustrate the validation process. Motivation and some of the logistics of validation are presented here in the introductory section of the chapter. An example protocol is presented in the second section, followed by a section of terms and definitions. Design of experiments, presented in the fourth section, is used to ensure that the validation experiments represent the populations of all “runs” of the method that are being validated. Pragmatic limitations are discussed in this section. The fifth section is a continuation of the example and contains a hypothetical data set with an analysis. The sixth section discusses statistical analyses and illustrates an analysis of the example validation data. In this chapter, “method” refers to the procedure of interest—the method being validated. The term “assay” is defined 3

4

Eudey

as a single execution of this method, possibly abstract, while “run” refers to an actual single execution of the method. Often a procedure will call for multiple measures within an assay; these are referred to as “replicates.” The reportable value of an assay could be the result of one replicate or the average of multiple replicates (here, this is in the abstract sense; it is the formula, or formulae, which will be used to calculate the reported value). If the reportable value is defined as the average of three replicates, then the reported value would be the average of the observed three replicate values from one run of the method. Why Validate? Validation can be a method of quantifying the performance of a process; in this case measuring the performance of a quantitative method. In 1985, E. M. Fry wrote: “Validation has a quantitative aspect—it’s not just that you demonstrate that a process does what it purports to do; you actually have to measure how well its does that … then, the processes that cause variability … must be identified. Experiments are conducted (that is, validation runs) to ensure that factors that would cause variability, are under control (1).” The process in question here is a quantitative method. The question becomes: “How well does the method measure the parameter that it purports to be measuring?” Validation provides an objective measure of a method’s performance. Using samples with a known (or at least previously measured) value of a product parameter, validation can provide useful information about accuracy, precision, linearity, and other characteristics of the method’s performance outside of its daily use on unknown samples. In addition, validation can be used to identify sources of undesired variability. Validation of a method involves running assays on aliquots from the same sample a number of times and ideally is done over a well-balanced design of all the external factors effecting performance. If, for example, more than one machine could be used for the method in question then, ideally, all machines should be tested in the validation. In a similar manner, the validation should be run over a number of days

Designing Experiments for Validation of Quantitative Methods

5

(or otherwise changing environmental conditions) to show the method is rugged to typical environmental changes. Additionally, more than one analyst should run the validation assays if that is the normal practice or if the analyst position has a high turnover. What Are We Measuring? Validation is the process of measuring the performance of a previously developed quantitative method. The characteristics measured to assess performance are defined and explained in detail later in this chapter. Typically, for a quantitative method the characteristics of specificity, accuracy, precision, and linearity are measured. The range of the method can be defined, or verified, by the region where the aforementioned characteristics are acceptable. Although their determination is more appropriately a development task, the characteristics of robustness, limit of detection, and limit of quantitation can also be verified during validation. The discussion here centers on the quantitative characteristics of accuracy, precision, and linearity. Design of experiments is also useful in supporting the validation of specificity and studying robustness (refer to section on Validation Terms and Definitions). Development Versus Validation Although the validation approach can be used in development, for simplification the author assumes that the method is fully developed before validation. Validation is an objective way of verifying that the development stage has been successful and the method is performing to expectations. Typically, the range of the method as well as the method’s input parameters will be defined during development and verified during validation. Robustness of a method is defined as “a measure of its capacity to remain unaffected by small but deliberate variations in method parameters and provides an indication of its reliability during normal usage” [International Conference Harmonisation (ICH) Guideline Q2A (2)]. Intervals for method

6

Eudey

parameters (e.g., ranges for elements or conditions of sample preparation) should be honed in the development stage to ensure consistency in assay results and may be verified during validation. Ranges for factors internal to the method such as incubation time and temperature are also determined during development. The standard operating procedure (SOP) should state ranges for method factors where assay performance is fairly consistent or robust. Ideally, the range is specified as target (low and high) so that the operator is aware of the target for the parameter, as well as how much the parameter can vary, and the method will still have similar assay performance. For example, a method with an incubation time specified 25 minutes (24.5, 25.5) is less robust with respect to incubation time than a method with an incubation time specified 25 minutes (20, 30). Experimental design can play a key role in finding these ranges of robustness during method development, but validation is a step taken after the appropriate ranges are set. Thus, for validation, each assay is run per the SOP and the internal factors are generally considered constants. If there is an acceptable range for a specific input parameter then validation can be used to verify that the method is still robust over the SOP specified range. What Is Needed? Guidelines for Validation Agencies, such as the Food and Drug Administration (FDA), United States Pharmacopoeia (USP), and ICH issue guidelines for method validations. Torbeck (3) has provided a comprehensive list of guidelines and references in his course on assay validation. For purposes of this chapter the ICH guidelines Q2A and Q2B (2,4) will be used. There are some minor differences in nomenclature between the different guidelines, and some of these will be explained in the short section on terms and definitions. The purpose of a method validation is to show that the method performs to the expectations of the user. A priori expectations need to be set out in the form of acceptance criteria. These must be formally stated using a preapproved

Designing Experiments for Validation of Quantitative Methods

7

protocol. In addition to setting down a priori acceptance criteria for the method’s performance characteristics, the protocol is a document written to specify how the validation will be run, the layout of the experimental design, the form of documentation for training and for execution of the validation experiments, how the data will be collected and analyzed, and additionally provides a structure for writing a final report. A hypothetical protocol and highlights of a statistical report will be used to demonstrate this process. It is important to remember that the intent of validation is to show that a method is acceptable for its intended use. Keep in mind that validation is not an exploratory or development process.

VALIDATION PROTOCOL Protocol Content A validation protocol states the purpose of the validation method, the intended substance being tested, the definition of the reportable value, the validation approach, the specific directions for conducting the validation assays, the statistical approach for analyzing the resulting data, and the nonambiguous acceptance criteria. The protocol should allow no room for ambiguity in the execution of the protocol or in the acceptance criteria. The content of a protocol is described next. There are many ways to format a protocol; herein are suggestions for the content of a protocol. The following is a hypothetical example of a validation protocol. For purposes of this example we will assume that SOP 123 is the procedure used to measure mass (in μg) using a calibration curve from a standard solution (all samples are pipetted to 1 μL; hence, we are actually measuring concentration, but volume is considered constant at 1 μL, so the term “mass” will be used for the measurement). Validation performance characteristics are briefly defined in the context of a protocol. These terms are discussed in more detail in the following section on terms and definitions. Square brackets [ ] delineate general content of a section or parenthetic remarks.

8

Eudey

The quantitative method characteristics to be validated will depend on the nature of the method itself. The reader should refer to the table in ICH guideline Q2A, which “lists those validation characteristics regarded as the most important for the validation of different types of analytical procedures.” The design shown in the example validation matrix allows simultaneous assessment of accuracy, precision, and linearity. Ideally, the expected masses are obtained by an alternative physical methodology that is highly accurate and precise. In the case of using a commercial standard, the vendor should provide statistics on the accuracy and precision of the method used to obtain/measure the stated label concentration. When using a previously released lot, the expected masses will be based on the certificate of analysis concentration for the lot of Analyte B used in this validation. In the latter case, the measurement of accuracy is relative to the historical certificate of analysis value. The validation protocol needs to state that the accuracy measurement is relative to the historical value rather than to an independently obtained measurement. EXAMPLE VALIDATION PROTOCOL This section presents an example validation protocol. As mentioned before, square brackets [ ] are used to discuss content in general.

PROTOCOL FOR THE VALIDATION OF SOP 123 FOR USE IN THE DETERMINATION OF MASS OF PRODUCT W [The title page, or some other page prefacing the protocol, must include signatures, with dates of signing, of all appropriate personnel to show approval of the protocol, including, but not limited to, the author of the protocol, the personnel responsible for the execution of the validation assays and collection of the data, the personnel responsible for the data analysis, as well as appropriate quality assurance and management.]

Designing Experiments for Validation of Quantitative Methods

§1. Purpose and Scope The purpose of this protocol is to define the method performance characteristics to be observed, the design and execution of the validation experiments, the data analysis, and the acceptance criteria for the validation of SOP 123 for use on Product W. [The scope of the document defines the specific product and method combination being validated. It also states the department responsible for running the assays and the facility (or facilities) where the method is being validated. If more than one facility is involved then reproducibility of the method between laboratories needs to be validated.]

§2. Background [This section describes the background of the use of SOP 123 and a brief description of the method.] The current range of this method for Product W is 50 μg to 150 μg.

§3. Validation Approach [This section describes the general approach in validating the method, lists and defines the validation characteristics to be evaluated (refer to Section III. Terms and Definitions).] To validate SOP 123 for measuring mass of Product W, the quantitative method performance characteristics of accuracy, precision, linearity, and range will be assessed using the validation assays shown in the design matrix over two days and using two operators. As per ICH guideline Q2A, the validation experiments will be run from 40 μg to 180 μg. The test lot will be diluted out and concentrated up to specific expected masses using the mass cited on certificate of analysis for the lot of Product W selected for the validation. The points on the Product W curve will be as follows: 40 μg, 50 μg, 70 μg, 90 μg, 110 μg, 130 μg, 150 μg, and 180 μg. Accuracy in percent recovery is defined by the ratio of the observed test lot mass to the certificate of analysis mass for the test lot. Thus, accuracy is relative to the previously measured mass and not by comparison to an objective measure of mass of the test lot.

9

10

Eudey

Precision of the method is defined by the sum of intermediate precision (interday, interoperator, and interassay precision) and repeatability. Linearity of the method is assessed by the linear regression statistics of observed mass against expected mass. Current range of the method will be verified by acceptable Accuracy, Precision, and Linearity (see Acceptance Criteria). [Other procedural details such as test lot preparation to aliquot all the samples needed for the validation experiments should also be described briefly in this section of the protocol.] Referring to SOP 123, the reportable value is defined as the average of the three replicates. For this validation, accuracy, precision, and linearity of the replicate values will be assessed; consequently, the precision of the results reported is at the replicate level not the reportable value level. [Note this practice is done for two reasons: first, to conserve resources, and second, when it is not possible to repeat an additional assay with exactly the same sample preparation. The repeatability component of the precision is defined as within-assay variance. Using the replicate values yields a within-assay variance.]

§4. Responsibilities and Documentation [This section, or sections, of the protocol describes the responsibilities of the pharmaceutical or biotechnology company and the departments within the company in carrying out a validation as well as listing the documentation needed for reference including SOP 123.]

§5. Training Verification and Documentation [Record on a Training Record that the personnel executing the validation experiments are adequately trained in compliance with company quality policies. Additionally, document that these personnel have been trained on the validation protocol.]

§6. Test Work [This section describes the execution of the validation experiments, how this execution may differ from the SOP, how and where the data will be recorded, as well as specifying how each of the validation characteristics will be evaluated.] In assays A, B, C, and D use the method of SOP 123 to determine

Designing Experiments for Validation of Quantitative Methods

11

triplicate measures of mass for Product W. For each point on the Product W curve (40 μg, 50 μg, 70 μg, 90 μg, 110 μg, 130 μg, 150 μg, and 180 μg) record each triplicate measure on the test report form.

§6.1. Accuracy §6.1.1. Data Analysis For each point on the Product W curve, divide the observed mass by the expected mass and express as a percent. For each expected mass, the average percent accuracy will be reported with a 95% confidence interval [as per the ICH guideline Q2B]. §6.1.2. Acceptance Criteria [Acceptance criteria appropriate to the use of the method for accuracy are stated in the protocol before the data analysis. Often the acceptance criteria are based on design criteria for the development of the method or development data.] For each mass in the tested range observed, average accuracy must be between 90% and 110%. [These are hypothetical acceptance criteria.] §6.2. Precision §6.2.1. Data Analysis For each mass on the Product W curve, the data will be analyzed by variance components analysis (VCA) to estimate the precision components due to InterOperator, InterDay, InterAssay, and Repeatability. For each expected mass, the precision components will be recorded as a variance, standard deviation, and percent of total precision. Nonzero components will be reported with a two-sided 95% confidence interval for the standard deviation. Within each mass, total variance is defined as the sum of the variance components. Total precision will be expressed as a variance, standard deviation, and coefficient of variation (%CV) also called the percent relative standard deviation or %RSD. §6.2.2. Acceptance Criteria [Acceptance criteria appropriate to the use of the method for precision are stated in the protocol before the data analysis.

12

Eudey

Often, the acceptance criteria are based on design criteria for the development of the method or development data.] For each mass in the tested range observed, the total %CV must be less than 10%. [These are hypothetical acceptance criteria. Note that the precision acceptance criterion can be on the standard deviation or some other metric of variability, preferably a metric that is somewhat consistent in value over the range of the assay.]

§6.3. Linearity §6.3.1. Data Analysis Within each of the assays A, B, C, and D, least squares linear regression of observed mass will be regressed on expected mass. The linear regression statistics of intercept, slope, correlation coefficient (r), coefficient of determination (r2), sum of squares error, and root mean square error will be reported. Lack-of-fit analysis will be performed and reported. For each assay, scatter plots of the data and the least squares regression line will be presented. §6.3.2. Acceptance Criteria For each assay, the coefficient of determination must be greater than 0.975. [These are hypothetical acceptance criteria. Other metrics for linearity could be used for the acceptance criterion. Note that for precise assays, a significant lack-of-fit may not be a meaningful lack-of-fit, due to a slight but consistent curve or other artifact in the data. If lack-of-fit is used for the acceptance criterion, the requirement should be placed on absolute deviation or percent departure from linearity rather than statistical significance. Often the acceptance criteria are based on design criteria for the development of the method or development data.] §6.4. Range Range is defined by the interval where the method has demonstrated acceptable accuracy, precision, and linearity.

§6.5. Other Performance Characteristics [Other method performance characteristics should be addressed as appropriate. The reader is encouraged to refer to ICH Q2A

Designing Experiments for Validation of Quantitative Methods

13

to determine which performance characteristics need to be evaluated in the validation. Depending on the characteristics addressed, the validation may need more than one design of experiments.]

§7. Validation Matrix [A matrix of assays is presented to display the study design over the appropriate factors. For this example (Table 1), the factors are Operator, Day, and Assay. For this example, Assay is defined as the interaction between Operator and Day. Note that this matrix shows a hierarchical, or nested, model as Assay (sample preparation) is uniquely defined by Operator and Day.]

§8. Attachments and Forms [This section contains forms for documenting that all training requirements have been met, sample preparation instructions as necessary to the execution of the validation experiments, and data recording forms. Places for signatures for performance and review should be included on each form as appropriate to company quality and documentation standards. For examples, see Tables 2 and 3.] Table 1 Matrix of Validation Assays Assay A B C D

Day

Operator

Replicates

1 1 2 2

I II I II

3 3 3 3

Four assays with three replicates each. Day 1 is a separate day than Day 2.

Table 2 Training Verification to Include Training on Standard Operating Procedure (SOP) 123 and Training on This Validation Name

Department

List SOP

Training record complete

Verified by— initials/date

Add a section for comments, as well as a place for signatures for the personnel who reviewed the training records and conducted the validation training in accordance with company quality standards for training documentation.

14

Eudey

Table 3 Assay Recording Sheet, One for Each Assay Assay Data Recording Sheet for Validation of SOP 123 Copy 1 for Each Assay Date of assay: Assay validation ID: Operator: Sample lot number:

Sample Standard

Test lot

Expected mass (μg)

Response per replicate (absorbance, light units, etc.) 1

2

3

Replicate calibrated value of mass (μg) 1

2

3

40 50 70 90 110 130 150 180 40 50 70 90 110 130 150 180

Include rows/columns for standard curve and other data normally recorded during the execution of standard operating procedure (SOP) 123.

TERMS AND DEFINITIONS The following gives definitions of method validation characteristics following the references (2,4) ICH Q2A “Text on Validation of Analytical Procedures” and ICH Q2B “Validation of Analytical Procedures: Methodology.” ICH Q2A identifies the validation characteristics that should be evaluated for a variety of analytical methods. ICH Q2B presents guidelines for carrying out the validation of these characteristics.

Designing Experiments for Validation of Quantitative Methods

15

The validation characteristics of accuracy, precision, and linearity were presented in the example protocol of the previous section. Accuracy Accuracy is a measure of how close to truth a method is in its measurement of a product parameter. In statistical terms, accuracy measures the bias of the method relative to a standard. As accuracy is a relative measurement, we need a definition of “true” or expected value. Often, there is no “gold standard” or independent measurement of the product parameter. Then, it may be appropriate to use a historical measurement of the same sample or a within-method control for comparison. This must be accounted for in the design of experiments to be conducted for the validation and spelled out in the protocol. Accuracy is measured by the observed value of the method relative to an expected value for that observation. Accuracy in percent can be calculated as ratio of observed to expected results or as a bias of the ratio of the difference between observed and expected to the expected result. For example, suppose that a standard one-pound brick of gold is measured on a scale 10 times and the average of these 10 weights is 9.99 lbs. Then calculating accuracy as a ratio, the accuracy of the scale can be estimated at (9.99/10) × 100% = 99.90%. Calculating the accuracy as a bias then [(9.99 – 10)/10] × 100% = −0.10% is the estimated bias. In the first approach ideal accuracy is 100%, and in the second calculation ideal bias is 0%. Precision and Ruggedness Precision is a measure of how variable a method is in its measurement of a product parameter under normal usage. In statistical terms, precision is measured by the variance of the method. Additionally, the typical sources of variability are accounted for in assessing precision; these are random factors external to the method, such as analysts, days, and changing assay “hardware”

16

Eudey

(e.g., gels or columns). Variance components measure the contribution of each of these external factors to the variability of the method’s results on the same sample of product. To estimate the variance components, the design of validation experiments must be balanced with respect to these factors. Precision components are defined at three levels: reproducibility, intermediate precision, and repeatability. Reproducibility is the variability of the method between laboratories or facilities. However, as a laboratory is not “randomly selected” from a large population of facilities, laboratory is a fixed effect. Consequently, the assessment of reproducibility is a question of comparing the average results between laboratories. Additionally, the variation observed within laboratory should be compared to ensure that laboratory does not have an effect either on the average result of the method or on the variability of the method. To assess reproducibility, conduct the same set of validation experiments within each laboratory and compare both the accuracy results and the precision results. If the differences are meaningful, analysis of variance (ANOVA) tests can be conducted to determine whether there is a statistically significant laboratory effect on the mean or on the variance of the method. For simplicity, the validation discussed within this chapter will not consider reproducibility and only one laboratory is considered. Intermediate precision components typically include day, operator, and assay. Day is a random effect that captures random environmental changes that are not controlled and may have an effect on the assay result. Operator, usually modeled as a random effect, captures the variation in assay results due to change in personnel running the assay. Assay captures the variation one would expect from one complete run of the method to the next complete run of the method. Thus, assay captures the random variation due to slight perturbations in the sample preparation from assay run to assay run (when run by the same operator on the same day). Often, it is not possible for each operator to run more than one assay on each day; however, the validation experiments can be designed to estimate this variance component under a reasonable assumption.

Designing Experiments for Validation of Quantitative Methods

17

ICH Q2A defines repeatability as the variability of the assay results “under the same operating conditions over a short interval of time. Repeatability is also termed intra-assay precision.” A reportable value of an assay is often the average of a specified number of replicate values, where the replicates are processed using the same sample preparation. Thus, it is not possible to obtain a true repeat of the assay’s reportable value. Repeatability can be modeled as the within-assay variability of the replicates; however, it should be noted that this precision is at the level of the replicate and not at the level of the reportable value. If the reportable value is an average of K replicates the variance of the reportable values will be less than that of the replicates by a factor__ of 1/K (the standard deviation will be less by a factor of (1/√K ). Ruggedness is not mentioned in the ICH guidelines. The USP (5) defines ruggedness as “the degree of reproducibility of test results obtained by the analysis of the same samples under a variety of normal test conditions.” Thus, ruggedness also addresses the typical external changes in the execution of the method such as change in day, operator, or assay preparations. Ruggedness and intermediate precision are two sides of the same coin, but they measure different concepts. Intermediate precision is measured with a standard deviation or %RSD. Ruggedness is the lack of a factor effect. “Changing the instrument has no effect on the results” is a statement addressing ruggedness. Ruggedness is modeled as a fixed effect measuring the change in average results due to a change in instrument. Linearity A method is “linear” or operating in the linear region if the measurement of the product parameter is proportional to the quantity being measured. Although there seems to be some debate regarding this issue, “linearity” does not refer to the linear range of a calibration curve used within the method or assay for calculation of the reportable value, but rather to the proportionality of the method’s reportable value to the quantity being measured. Here too, it is important that the validation

18

Eudey

experiments are balanced both with respect to the variance components and also over the range of method validation. Thus, the levels of expected input should be evenly spaced across the range of the method with an equal number of observations at each level. Robustness Robustness refers to the effect on the assay response due to minor perturbations in sample preparations. In contrast to precision, robustness refers to the minor perturbations of factors internal to the assay (e.g., incubation time, temperature, or amount of reagent) whereas intermediate precision and ruggedness refer to factors necessary in the performance of the method that are external to the assay (day-to-day changes, operator changes, instrument changes). In method development, acceptable ranges for assay input parameters should be found where the assay response stays fairly constant. In these ranges the method is said to be robust. To quote for Q2B of the ICH Guidelines: “The evaluation of robustness should be considered during the development phase and depends on the type of procedure under study. It should show the reliability of an analysis with respect to deliberate variations in method parameters (2).” DESIGN OF EXPERIMENTS Terms and Definitions Factors, Levels, and Treatments Factors are the major categories of a design. The levels of each factor are the “values” or “versions” that the factor can assume. Treatments are the combinations of the factor levels that are assigned to the experimental units. In our validation experiment there are factors of “operator” and “day,” each having two different levels. There are four treatments in the example validation matrix (refer to Table 1 in the Example Protocol).

Designing Experiments for Validation of Quantitative Methods

19

Fixed Effects, Random Effects, Main Effects, and Interactions Factors are fixed if the “values” of the levels are given or do not vary. Factors are random if the value is, for pragmatic purposes, randomly selected from a population of all values. To illuminate in an example: if a small lab has two analysts that will always be running the assays then the factor “analyst” is considered fixed with two levels (one level for analyst 1, the other level for analyst 2); however, if the lab is large and/or there is a high turnover in analysts then for each run of an assay the “analyst” could be regarded as selected at random among a population of trained analysts. In a statistical model, fixed effects have an influence on the mean value or average of the method’s response while random effects have an influence on the variability of the method. Fixed effects are assessed in the context of accuracy. Random effects are assessed in the context of precision and become the intermediate precision components. In designing the validation design matrix the validation assays need to be balanced over both the fixed effects and the random effects. A mixed effects model (or design) occurs when both fixed effects and random effects are present (6). Only one lot of well-characterized product or reference standard should be used in the validation. Product lot or batch is not a factor in method validation. Factors that are typically assessed are instrument or machine, analyst or operator, day, and other factors external to the method itself that come into play in the typical running of an assay. Main effects of a factor are the differences in average response between the levels of the factor averaged over all other levels of the other factors. For example, if the factor machine has two levels, machine 1 and machine 2, the main effect, due to machine, would be the difference in average response from machine 1 and the average response from machine 2 (averaged over all other factors). An interaction occurs when the effect of one factor is dependent on the level of another factor. For example, if there is a significant increase in average response between the low and high levels of factor A when factor B is low and there is a significant decrease in

20

Eudey Interaction Between Time and Temperature

Response

250 200 150

27 degrees 31 degrees

100

35 degrees

50 0 2 hours 3 hours 4 hours Time (hours)

Figure 1 Interaction between time and temperature. The response profile to time depends on the temperature.

average response between the low and high levels of factor A when factor B is high then there is an interaction between factor A and factor B. Figure 1 shows an interaction between time and temperature. Nested and Crossed Factors Factor A is nested within factor B if the levels of factor A change with the level of factor B. For example, if batches or lots are tested by taking samples from the production line and then aliquots are taken from each sample, then the aliquots are nested within the samples which are nested within the batches (7). Factors C and D are crossed if all levels of factor C can occur with all levels of factor D. For example, in the sample protocol of section IIB, days and operators are crossed, as each of the two operators perform an assay on each of two the days. Aliasing and Confounding If each level of one factor only occurs at a specific level of another factor, then the two factors are said to be confounded. When confounding or aliasing occurs, the data analysis cannot distinguish between the effects of the two confounded factors.

Designing Experiments for Validation of Quantitative Methods

21

If a significant effect is apparent it is impossible to tell which factor is responsible for the effect. In designing experiments, the number of treatments or experimental trials can be reduced by intentionally aliasing one factor with another. For example, a main effect can be aliased with a high-order interaction, especially if it is a reasonable assumption that the interaction effect does not exist or is negligible. Experimental Designs In this section, three categories of experimental design are considered for method validation experiments. An important quality of the design to be used is balance. Balance occurs when the levels each factor (either a fixed effects factor or a random effects variance component) are assigned the same number of experimental trials. Lack of balance can lead to erroneous statistical estimates of accuracy, precision, and linearity. Balance of design is one of the most important considerations in setting up the experimental trials. From a heuristic view this makes sense, we want an equivalent amount of information from each level of the factors. Full Factorial Designs A full factorial design is one in which every level of each factor occurs for every combination of the levels of the other factors. If each of n factor has two levels, frequently denoted “+” for one level and “–” for the other level, then the number of treatments is two raised to the nth power. Full factorial designs when each factor has the same number of levels are referred to as kn factorial designs, where k is the number of levels and n is the number of factors, as kn is the number of treatments needed. If factor A has three levels, factor B has two levels, and factor C has three levels then a full factorial design has 3 × 2 × 3 = 18 treatments or experimental trials. The treatments of a design (or the experimental trials) can be represented in a design matrix. Each row of the matrix represents a treatment; each column represents the level of each factor. A full factorial design for three factors, factor A,

22

Eudey

Table 4 Full Factorial Design for Three Factors, Factors A, B, and C, Each with Two Levels, Low (−) and High (+) Treatment 1 2 3 4 5 6 7 8

Factor A

Factor B

Factor C

− + − + − + − +

− − + + − − + +

− − − − + + + +

Every level of each factor occurs with every level of the other factors and combination of the other two factors.

factor B, and factor C, each with three levels, is shown in the design matrix presented in Table 4. Fractional Factorial Designs Often a full factorial design requires too many resources and an experimental design with fewer treatments is required. Fractional factorials can reduce the number of experimental trials by aliasing one or more of the factors with other factors or interactions of factors. A fractional factorial of a full factorial design 2n cuts the number of trials or treatments by1/2, 1/4, or 1/2p. When the number of experimental trials is reduced from 2n by a factor of 1/2p. the design is referred to as a 2n–p fractional factorial. The full factorial design (23 factorial design with eight treatments) presented in Table 4 can be reduced to a 23–1 fractional factorial design with four experimental trials if factor C is aliased with the two-factor interaction of factors A and B. The resulting design is presented in Table 5. Note that the effect of factor C is confounded with the interaction of factors A and B. This can be observed in the design matrix by multiplying the levels of factors A and B to get the level of factor C (+ × + = +, + × − = −, and − × − = +). The resolution of a design is a categorization of the aliasing in the design. For resolution III design, main effects are not aliased with each other but each main effect is aliased

Designing Experiments for Validation of Quantitative Methods

23

Table 5 Fractional Factorial Design for Three Factors, Factors A, B, and C, Each with Two Levels, Low (−) and High (+) Treatment 1 2 3 4

Factor A

Factor B

Factor C

− + − +

− + + −

+ + − −

Factor C is aliased with the interaction of factors A and B (refer to text).

with two-factor interactions. In a resolution IV design, some main effects are aliased with three-factor interactions and two-factor interactions are aliased with each other (2–4). The design shown in Table 5 is resolution III, as main effect of factor C is aliased with the interaction between factors A and B. Also, A is aliased with B and C, and B is aliased with A and C. Resolution of design is discussed in more detail in most texts on experimental design including John (6), Box, Hunter, and Hunter (7), and Kuehl (8). Plackett-Burman Designs Plackett-Burman designs are a group of orthogonal array designs of resolution III, where the number of experimental trials is a multiple of four, but is not a power of two. In most Plackett-Burman designs each factor has exactly two levels. These designs are useful as they require fewer resources. Placket-Burman designs are used to estimate main effects only. The reader is referred to the references (6–8) on experimental design for listings of Plackett-Burman designs. Designs for Quantitative Method Validation Full factorial designs can be used in quantitative method validation. With very few factors this is a feasible design. A simple way to display the experiment necessary for the validation is to display the assay runs in a table or matrix. For instance, suppose a method is run on two different machines and the goal is to assess intermediate precision components. We have the random effects of operator and day and a fixed

24

Eudey

effect due to machine. If the factors operator and day each have two levels, the full factorial design is shown in Table 4 with factor A as operator, factor B as day, and factor C as machine. Eight assays will be run; the replicates of the assays will be used in estimating repeatability. All factors are crossed with each other. The full factorial design may not be practical. Perhaps it is not possible for one operator to run two assays on one day. Then a fractional factorial design can be used. The design shown in Table 5 is resolution III design with four experimental trials. As machine is aliased with the interaction between operator and day, the main effect due to machine can be estimated only if we assume there is negligible interaction between operator and day. Often, interactions can be assumed to be negligible based on prior knowledge or the science that is known about the situation. The design shown in Table 5 is also the design of the example protocol. In the example protocol the third factor is assay. The effect of assay is aliased with the interaction of operator and day, and assay is also nested within the interaction. Note that if the design of Table 4 were used, and two assays were run for each operator and day combination, then assay would still be nested within the interaction. Each time an assay is run, it is a different assay; hence, the number of levels of the factor assay is just the number of assays that are run. However, running two assays within each operator and day combination will give a better empirical estimate of interassay variability but assay effect cannot be estimated independently of the interaction. This is because assay takes on a different “value” each time the assay is run. (In contrast, in the previous example the third-factor machine is crossed with the factors operator and day; consequently, the machine effect can be estimated independently of operator and day effects.) For quantitative methods, the ICH guidelines recommend validating a range that extends 20% below the lower end of the range and 20% above the upper end of the range (for content uniformity the range is extended to 70–130% of the SOP range). For the linearity portion of the method, National Committee for Clinical Laboratory Standards [NCCLS (9)] recommends at

Designing Experiments for Validation of Quantitative Methods

25

least five levels evenly spaced across the range to be validated. For example in validating a method that measures concentration of an analyte, the sample being tested would be diluted out to (or concentrated to) five expected concentrations evenly spaced between 20% below the lower limit of the method’s stated range and 20% above the upper limit of the method’s stated range. In an assay that can measure multiple concentrations simultaneously, each assay would test these five diluted samples in K-replicates (within-assay replicates; the assay’s reportable value could be the average of these K replicates). If this procedure is repeated using I operators over J days then the method characteristics of accuracy, precision, and linearity can be assessed with one set of I × J assays. In essence, the factor concentration is crossed with the other factors as each of the five concentrations is run in all of the assays. In the example protocol, eight levels of mass are used.

STATISTICAL ANALYSES The experimental design selected, as well as the type of factors in the design, dictates the statistical model to be used for data analysis. As mentioned previously, fixed effects influence the mean value of a response, while random effects influence the variance. In this validation, the model has at least one fixed effect of the overall average response and the intermediate precision components are random effects. When a statistical model has both fixed effects and random effects it is called a mixed effects model. The statistical software package SAS® (SAS Institute Inc., Cary, North Carolina, U.S.A.) has a module [PROC MIXED (10)], which analyzes mixed effects models and provides estimates for both the fixed (shifts on the mean) and random effects (variance components estimates). Analysis of accuracy and precision can be accomplished using the same statistical model with a slightly different response variable. For accuracy, the response variable (here this is the replicate value and differs from the reportable value of the assay as run outside of the validation) is observed mass

26

Eudey

divided by expected mass as a percentage. For precision the response variable is observed mass. Both the accuracy and the precision results are reported for each expected mass. The linearity assessment uses the entire data set over all expected masses. Consequently, the root mean square error (RMSE) from the linearity analysis can be viewed as measurement of overall repeatability. ANOVA can be used to test whether the fixed effects are significant. For example, if there is a factor machine with two levels (representing two machines), an ANOVA can be used to estimate machine effect. If the difference between the average responses for each machine is not meaningfully different, and the variance components within each machine are similar, then the variance components can be analyzed averaging over the machines. If there is a meaningful difference between the two observed machine averages, then an F-test can be used to test whether machine effect is significant. Accuracy is estimated from the fixed effects components of the model. If the overall mean is the only fixed effect, then accuracy is reported as the estimate of the overall mean accuracy with a 95% confidence interval. As the standard error will be calculated from a variance components estimate including intermediate precision components and repeatability, the degrees of freedom can be calculated using Satterthwaite’s approximation (6). The software program SAS has a procedure for mixed model analysis (PROC MIXED); PROC MIXED has an option to use Satterthwaite’s degrees of freedom in calculating the confidence interval for the mean accuracy. An example program and output is shown later for the example protocol. Precision components are estimated using the random effects of the model. There are a number of ways to estimate the precision components (or variance components). Variance components are estimates of variances (variance = standard deviation squared) and consequently are restricted to being nonnegative. One approach is to conduct an ANOVA and solve for the variance components using the theoretical composition of the expected mean squares (called method of moments estimates or ANOVA estimates). The ANOVA method is shown in

Designing Experiments for Validation of Quantitative Methods

27

many textbooks, including Searle et al. (11) which provides an in-depth discussion of variance components estimation. Another method is to use maximum likelihood estimation (MLE). Both of these approaches have drawbacks. ANOVA estimates can be negative as they are a linear combination of sums of squares and there is nothing inherent in restricting the resulting estimates to be nonnegative. Maximum likelihood estimates can be biased as the estimators do not factor in the number of fixed effects being estimated in the model. A third approach is to use maximum likelihood and restrict the domain of the variance estimates to nonnegative values by “estimating variance components based on residuals calculated after fitting by ordinary least squares just the fixed effects part of the model” (11); this is known as restricted maximum likelihood estimation (REML). If the design is balanced (i.e., there are the same number of assays run per each day, per each operator, and so on) and the ANOVA estimates are nonnegative, then theoretically the REML estimates and the ANOVA estimates agree. SAS’s PROC MIXED has options for all of these methods of estimation. The example protocol analysis shows use of REML. For any of these methods, it is very important that the original design of experiments is balanced to get statistically sound estimates of variance components. The basic idea behind variance components estimation is that the variability of the data is parsed out to the random effects of the model. For example, operator-to-operator variability can be assessed roughly by calculating the standard deviation of the operator-specific averages. Thus, it is clear why balance is very important; if the data are not balanced then we are comparing averages from differing number of values, thus the average with the smaller sample size is more variable than that with the larger sample size (i.e., we are mixing apples with oranges). Total precision is defined as the sum of the precision components. This summation takes place on the variance scale (variance = standard deviation squared). In theory, variances from independent terms in a model add up to total variance. See the precision analysis in the validation analysis example of the next section.

28

Eudey

Linearity is analyzed using least squares regression of the observed responses against the expected responses, or the regression of the observed responses against the amount of analyte in the sample. The linearity here is of the reportable value versus the amount of analyte in the sample. Note, in the example protocol the responses are the replicate values for each assay at each mass; the average of three within-assay replicates is the reportable value (at each mass) of SOP 123 (in release testing, only one reportable value, the average of three replicates, is reported as the mass of the sample is unknown). If the entire range is assayed in each assay (as in the example validation), then linearity can be assessed within each assay or over all assays. If linearity is assessed using all the data then the RMSE can be used as a measure of overall precision as all the assays over all days and operators are represented. If linearity is assessed within each assay then the RMSE is another measure of repeatability (measurement of error variability within assay). When there is more than one observation (whether it be replicate values or more than one reportable value over a number of assays) at each level of the analyte, then a lack-offit analysis can be conducted. This analysis tests whether the average response at each level of the analyte is a significantly better model for average assay response than the linear model. A significant lack-of-fit can exist even with a high correlation coefficient (or high coefficient of determination) and the maximum deviation of response from the predicted value of the line should be assessed for practical significance. EXAMPLE VALIDATION DATA ANALYSIS The data are collected and recorded in tables (Tables 6A–D for hypothetical data). The validation characteristics of accuracy, precision, and linearity are analyzed from the data for this example validation. Model for Accuracy and Precision The statistical model used to analyze both accuracy and precision is a mixed effects model. Separately for each expected

Designing Experiments for Validation of Quantitative Methods

29

Table 6A Data from Assay A for Example Validation Operator 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Day

Replicate

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

Expected 40 40 40 50 50 50 70 70 70 90 90 90 110 110 110 130 130 130 150 150 150 180 180 180

Observed 38.7398 38.6092 39.0990 49.4436 48.7904 48.6924 70.6760 69.5980 70.2180 88.7400 87.7280 88.4780 106.3140 105.2700 106.3780 123.5940 122.5820 122.1900 138.5220 138.9480 138.9480 172.9140 172.2600 173.7300

mass of 40 μg, 50 μg, 70 μg, 90 μg, 110 μg, 130 μg, 150 μg, and 180 μg: Yijk = μ + Opi + Dayj + Assay (Op × Day)ij + εijk

(1)

Here, Y is the response variable. For accuracy, Y is observed mass divided by expected mass expressed as a percent, % accuracy (note that this is measuring percent recovery against an expected value). For precision, Y is observed mass. The term μ is a fixed effect representing the population overall average mass for each expected mass. The term Op is the random component added by operator i; Day is the random component added by day j; and Assay is nested in the interaction of operator and day. The term ε is the random error of replicate k of the assay performed by operator i on day j.

30

Eudey

Table 6B Data from Assay B for Example Validation Operator 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Day

Replicate

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

Expected 40 40 40 50 50 50 70 70 70 90 90 90 110 110 110 130 130 130 150 150 150 180 180 180

Observed 38.0584 38.1000 38.0450 47.6778 47.7412 48.1222 67.4260 66.9500 66.9500 86.7940 85.6180 86.1580 103.4300 101.6200 102.0960 118.4780 117.4940 118.2260 134.7020 132.1940 132.5440 179.2500 172.1000 174.5000

Average accuracy is estimated by the overall observed accuracy. The parameter being estimated is μ. The standard error of this estimate is a function of the variance components for operator, day, assay, and repeatability. SAS PROC MIXED is used to analyze the data. For accuracy, the SAS code used is: *SAS code for Accuracy; PROC MIXED DATA = example CL; CLASS oper day assay; MODEL acc = / solution cl ddfm = satterth; RANDOM oper day assay; BY exp; RUN

Designing Experiments for Validation of Quantitative Methods

31

Table 6C Data from Assay C for Example Validation Operator 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

Day

Replicate

Expected

Observed

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

40 40 40 50 50 50 70 70 70 90 90 90 110 110 110 130 130 130 150 150 150 180 180 180

41.7500 43.2800 43.5100 48.1538 47.9650 48.3110 67.9920 67.8340 68.0220 88.6140 87.1980 86.9780 104.1140 103.3580 103.6100 119.9580 119.4560 119.2360 136.9660 133.6340 134.3260 179.1500 178.2500 178.9100

In the code, “oper” reads in the level of operator, “day” reads in the level of day, “assay” reads in the assay, “acc” is Y for accuracy and is defined by the ratio of observed mass to expected mass (“exp”). Note that the “BY exp” the procedure is run separately for each level of the expected mass using “exp” as the variable name. In this manner, the precision of the accuracy and the variance components are estimated independently at each level of the expected mass. Thus, we obtain “picture” of the assay’s performance characteristics across the operational range. As assay is nested in the interaction between operator and day, the same analysis can be coded used “oper* day” in place of “assay” in the previuos code. The “CL” and “cl” in the procedure line and model statement are

32

Eudey

Table 6D Data from Assay D for Example Validation Operator 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

Day

Replicate

Expected

Observed

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

40 40 40 50 50 50 70 70 70 90 90 90 110 110 110 130 130 130 150 150 150 180 180 180

41.4500 41.7500 41.6000 48.5228 49.1456 49.3636 68.8860 70.0380 70.1620 86.9140 87.9740 88.5340 104.8500 104.5680 104.9740 120.3240 120.0440 121.3820 136.1720 136.7960 136.6380 173.2800 183.2000 175.3200

calls for confidence limits. In the model statement “ddfm = satterth” calls for Satterthwaite’s degrees of freedom (see Ref. 6 for a discussion of Satterthwaite’s approximation) to be used in calculating the confidence limits. Model (1) is used for precision analysis. For precision, Y is the observed mass (rather than observed mass divided by expected mass). Each random effects component (operator, day, assay, and error or repeatability) contributes to the total variance of Y. The total variance of Y is defined as the sum of 2 2 2 the variance components. If σ Oper , σ Day , σ Assay , and σε 2 are the variance components for operator, day, assay, and repeatability, respectively, then total variance is σ 2Oper+ σ 2Day+ σ 2Assay+ σ 2ε .

Designing Experiments for Validation of Quantitative Methods

33

For precision, the SAS code used is shown next. Note that the only change is the Y value; for precision, Y is observed mass, called “obt” in the code. *SAS code for Precision; PROC MIXED DATA = example CL; CLASS oper day assay; MODEL obt = / solution cl ddfm = satterth; RANDOM oper day assay; BY exp; RUN This program will give variance estimates for each of the precision components along with two-sided 95% confidence intervals for the population variance component for each expected mass. SAS PROC MIXED will provide ANOVA estimates, maximum likelihood estimates, and REML estimates; the default estimation, used here, is REML. Method of moments estimates (also known as ANOVA estimates) can be calculated directly from the raw data as long as the design is balanced. The reader is referred to Searle et al. (11) for a thorough but rather technical presentation of variance components analysis. The equations that follow show the ANOVA estimates for the validation example. First, a two-factor with interaction ANOVA table is computed (Table 7). Then the observed mean squares are equated to the expected mean squares and solved for the variance components (Table 8 and the equations that follow). The expected mean squares are equated to the observed mean squares (those from the data) and solved for the variance components. Thus, σˆ 2ε = MSE is the variance estimate MSA – MSE for repeatability, σˆ 2Assay = _________ is the variance estimate m 2 MSD – MSA __________ ˆ for assay, σ Day = is the variance estimate for day, a·m MSO – MSA and σˆ 2Oper = __________ is the variance estimate for operator. b·m To express as standard deviations, take the square root of each. The ANOVA estimates are applicable when the design is balanced and there are no missing data values; using this method with missing data or an unbalanced design will result in misleading erroneous variance estimates.

34

Eudey

Table 7 Analysis of Variance (ANOVA), Sum of Squares and Mean Square Formulae for a Two-Factor Mixed Model with Interaction for the Validation Matrix of Tables 4 or 5 to be Used in ANOVA Variance Component Estimation Source Operator Day

df

Mean square

Sum of squares

a–1

SSO = ∑b · m( y=i.. − y≡...)2

SSO MSO = ____

b–1

SSD = ∑a · m( y=.j. − y≡...)2

SSD MSD = ____

dfO dfD

SSA SSA = ∑∑m( y–ij. − y–i.. − y–.j. + y≡...)2 MSA = ____

Assay = operator (a – 1) × × day (b – 1)

dfA

Repeatability

ab(m – 1)

SSE = ∑∑∑( yijk − y≡...)2

Total

abm–1

SST = ∑∑∑( yijk − y≡...)2

SSE MSE = ____ dfE

Here, a bar over the Y indicates averaging over the missing subscript(s); the subscript “i” is an index for operator, the subscript “j” is an index for day, assay is indexed by the level of “i” and “j,” and “k” is an index for the replicate within each assay. In the example, the grand average is y≡..., the average for operator “i” is y=i. = ∑j ∑ yijk /6, the k average of the six replicates produced by operator “i” for i = 1, 2, and the average for assay “i, j” is the average of the three replicates of operator “i” on day “j.” The design must be balanced for these formulas to apply. Abbreviations: a, number of operators; b, number of days; m, number of replicates; df, degrees of freedom. Source: From Ref. 11.

Linearity Analysis For the example validation, linearity will be analyzed within each assay and across all four assays. For the second analysis, as mentioned before, the RMSE can be used as a measure of Table 8 Expected Mean Squares for the Two-Factor with Interaction Analysis of Variance Table Shown in Table 7 Source

Mean square

Expected mean square

Operator

SSO MSO = _____

σε +m⋅σ

2

Day

MSD =

σε +m⋅σ

2

Assay = operator × day

MSA =

σε +m⋅σ

2

Repeatability

MSE =

dfO SSD _____ dfD SSA _____ dfA SSE _____ dfE

2

2

2

σε 2

Assay

Assay

Assay

+b⋅m⋅σ

2

+a⋅m⋅σ

2

Oper

Day

Designing Experiments for Validation of Quantitative Methods

35

overall precision. For the first analysis, the RMSE is a measure of repeatability across all masses within each assay. The model for linearity is: Observed mass = α + β · Expected mass + ε

(2)

Here, α is the population intercept and β is the population slope of observed mass regressed on expected mass. The error term, ε, is the residual added to the population line to obtain the observed mass. Least squares estimation (traditional regression estimation) will yield estimates of α and β, as well as an estimate for the standard deviation of ε (RMSE). These estimates form the sample regression line for predicting observed mass (predicted mass or Yˆ ): Yˆ = αˆ + βˆ · Expected mass

(3)

The RMSE is standard deviation of the residuals (residual = observed mass – predicted mass). To test for lack-of-fit, a classification variable is added to the model representing the expected mass group: Observed mass = α + Mass class + β · Expected Mass + e

(4)

Note that this analysis can only be performed if there are replicate values for each level of expected mass (mass class in the previous model). The model testing lack-of-fit is shown next. An ANOVA F-test is used to test whether the classification variable is significant, indicating significant (but not necessarily a meaningful) lack-of-fit to the linear model. In essence, this test asks the question “Do the averages at each expected mass fit the data better than the linear regression model?” If the method is very precise it will not take a very large deviation of observed mass from predicted mass to result in significant lack-of-fit. As a result, the deviations themselves should be analyzed for a meaningful lack-of-fit. The two programs (SAS PROC GLM) for linearity are shown next for analyzing linearity within each of the four assays. The first gives the least squares regression estimates and the regression statistics; “obt” is the observed mass and “exp” is the expected mass for each of the eight masses assayed. The second program tests for lack-of-fit to the linear model; a new classification variable “expclass” has eight settings

36

Eudey

corresponding to the eight expected masses (40 μg, 50 μg, 70 μg, 90 μg, 110 μg, 130 μg, 150 μg, and 180 μg). * To give least squares estimates for linear regression; PROC GLM data = example; Title 'obt = obtained mass, exp = expected mass'; MODEL obt = exp; BY assay; OUTPUT out = robt p = yhat r = res; RUN; *To test lack-of-fit to linear model; PROC GLM data = example2; Title 'Lack of fit analysis'; CLASS expclass ; MODEL obt = exp expclass/solution; BY assay; RUN The output statement in the first program creates a file with the data, the predicted masses by the linear regression model (Yˆ), and the residuals of the observed values minus the predicted values for each replicate at each expected mass. These deviations can be assessed for meaningful lack-of-fit. Example Protocol Analysis Results This section shows the analysis results for the data of the validation example (Tables 6A–D). The results of the statistical analysis should be reported in a manner that is in accordance with company quality standards. For example, documentation of data auditing of the data listing from SAS against the raw data, a complete referencing to the electronic storage locations of the data, the SAS program, the program output, and a hardcopy write up of the statistical analysis approach should all be maintained with the validation report. For some companies, this usually results in a statistical report that is on the order of 150 to 200 pages in length (the entire SAS output is often included with the report). The tables and graphs reported as well as some examples of SAS output will be presented here. The report should enable another statistical analyst to reproduce the analysis at a later date.

Designing Experiments for Validation of Quantitative Methods

37

Accuracy The model used for accuracy (1) is shown in the previous section (Example Protocol section IIB); recall that accuracy for each data point is defined as Accuracy = (Observed mass / Expected mass) × 100%. The assay-specific averages for accuracy, the across-assay average, standard error of the across-assay average, Satterthwaite degrees of freedom, and 95% confidence intervals are shown in Table 9. Note that because Satterthwaite degrees of freedom are a linear combination of sums of squares, they can assume noninteger values. An Excel chart showing the average accuracy and 95% confidence limits for each of the expected masses is shown next (Fig. 2). The confidence intervals tend to be wide if one (or more) of the intermediate precision components has a large contribution to variability. The acceptance criteria for accuracy is that each observed average accuracy must be within the interval (90%, 100%) for the expected masses within SOP 123’s stated range of 50 to 150 μg. As all the observed averages are within the interval (90%,100%), the acceptance criteria for accuracy have been met. Precision The statistical model for precision is shown in equation (1). For precision, Y is defined as observed mass. For each of the eight expected masses the precision components are reported as variances for each component. The total variance is the sum of the component variances. The component percent of total is calculated on the variance scale as: (Variance component estimate / Total variance) × 100%. The square root of each variance estimate is taken and reported as the standard deviation. Confidence intervals were generated by SAS and reported on the variance scale; square roots of the confidence limits were calculated and reported as the confidence interval for the component standard deviation. The coefficient of variation (%CV) is calculated as: (Observed component std dev / Observed average) × 100%. Table 10 shows the precision results for the example validation. The acceptance criteria for

97.04 97.95 100.23 98.13 96.35 94.45 92.54 96.09

A 95.17 95.69 95.87 95.77 93.07 90.82 88.76 97.38

B 107.12 96.29 97.07 97.33 94.27 91.96 89.98 99.32

C

D 104.00 98.02 99.56 97.56 95.27 92.76 91.02 98.48

Assay accuracy averages (%)

Abbreviation: df, degrees of freedom.

40 50 70 90 110 130 150 180

Expected protein mass (μg) 100.83 96.99 98.18 97.20 94.74 92.50 90.58 97.82

Average accuracy (%)

Table 9 Accuracy Results for the Example Validation Data

4.875 0.589 1.029 0.532 0.700 0.763 0.800 1.081

Standard error (%) 1.13 3 3 1 3 3 3 1

df

53.12 95.11 94.91 90.45 92.51 90.07 88.03 84.11

Lower (%)

148.54 98.86 101.46 103.94 96.97 94.93 93.12 111.53

Upper (%)

95% Confidence interval on accuracy

38 Eudey

Designing Experiments for Validation of Quantitative Methods

39

Accuracy (%)

Accuracy Results

160 140 120 100 80 60 40 20 0 40

50

70

90

110

130

1501

80

Expected Mass

Figure 2 Excel chart showing average accuracy with 95% confidence limits.

precision were that %CV must be at most 10%. The acceptance criteria were met. Linearity The models used for analysis of linearity were shown in (2) and (4) before. Table 11 presents the summary statistics for the assay-specific regressions. Figures 3 to 6 show the assay scatter plots with regression lines. Table 12 shows the regression results for the lack-of-fit analysis; as each of the assays displayed a significant lack-of-fit, the maximum percent deviation is calculated as the maximum of: [(Observed Mass − Predicted mass)/Predicted mass] × 100%. In addition, the number of percent deviations over 10% is calculated for each expected mass. The acceptance criteria for linearity were that each and every coefficient of determination (r2) must be at least 0.975. (For this analysis, the simple r2 was calculated.) The acceptance criteria were met. Across-Assay Linearity An alternative approach to linearity analysis is to analyze the linearity of all the data. The SAS code for the overall analysis is the same as shown before except that the line “By Assay”; is deleted from the code. Table 13 shows the summary statistics for the least squares regression, Figure 7 shows a scatter plot (Text continues on page 44.)

40 40 40 40 40 50 50 50 50 50 70 70 70 70 70 90 90 90 90

Expected protein mass (μg) Oper Day Assay Residual Total Oper Day Assay Residual Total Oper Day Assay Residual Total Oper Day Assay Residual

Source 0.68 2.67 0.00 0.49 2.79 0.00 0.00 0.56 0.33 0.65 0.00 0.00 1.42 0.47 1.49 0.26 0.00 0.78 0.72

0.4568 7.1090 0.0000 0.2436 7.8094 0.0000 0.0000 0.3097 0.1112 0.4209 0.0000 0.0000 2.0025 0.2184 2.2209 0.0667 0.0000 0.6091 0.5219

5.85 91.03 0.00 3.12 100.00 0.00 0.00 73.57 26.43 100.00 0.00 0.00 90.17 9.83 100.00 5.57 0.00 50.85 43.58

Standard deviation (μg) Variance (μg2) Percent (%)

Table 10 Precision Results for the Example Validation Data

0.29 1.19 . 0.34 . . . 0.30 0.23 . . . 0.79 0.32 . 0.15 . 0.36 0.49

Lower 40.29 88.39 . 0.90 . . . 2.72 0.64 . . . 5.70 0.90 . 4.58E + 101 . 14.51 1.38

Upper

95% Confidence interval on σ . . . . 40.33 . . . . 48.49 . . . . 68.73 . . . .

Estimated mean

. . . . 6.93 . . . . 1.34 . . . . 2.17 . . . .

CV (%)

40 Eudey

Total Oper Day Assay Residual Total Oper Day Assay Residual Total Oper Day Assay Residual Total Oper Day Assay Residual Total 1.09 0.00 0.00 1.50 0.60 1.62 0.00 0.00 1.95 0.60 2.04 0.00 0.00 2.31 1.13 2.57 0.00 2.45 0.00 3.07 3.93

Abbreviation: CV, coefficient of variance.

90 110 110 110 110 110 130 130 130 130 130 150 150 150 150 150 180 180 180 180 180

1.1977 0.0000 0.0000 2.2487 0.3647 2.6134 0.0000 0.0000 3.8198 0.3553 4.1751 0.0000 0.0000 5.3345 1.2756 6.6101 0.0000 6.0013 0.0000 9.4323 15.4340

100.00 0.00 0.00 86.04 13.96 100.00 0.00 0.00 91.49 8.51 100.00 0.00 0.00 80.70 19.30 100.00 0.00 38.88 0.00 61.12 100.00

. . . 0.83 0.41 . . . 1.09 0.40 . . . 1.27 0.76 . . 0.98 . 2.15 .

. . . 6.28 1.16 . . . 7.78 1.14 . . . 10.26 2.16 . . 593.64 . 5.39 .

87.48 . . . . 104.22 . . . . 120.25 . . . . 135.87 . . . . 176.07

1.25 . . . . 1.55 . . . . 1.70 . . . . 1.89 . . . . 2.23

Designing Experiments for Validation of Quantitative Methods 41

42

Eudey

Table 11 Linearity Summary Statistics for the Example Validation Data Correlation Assay coefficient A B C D

0.9989 0.9956 0.9946 0.9960

Coefficient of determination

Residual Root mean sum of Intercept Slope square error squares

0.9978 0.9912 0.9893 0.9919

2.88 0.62 2.26 2.93

0.93 0.93 0.93 0.93

2.11 4.22 4.67 4.04

97.52 392.15 480.07 358.50

175

Observed Mass

150

y-hat =2.88 + 0.93 Exp

125 100 75 50 25 50

100 150 Expected Mass

200

Figure 3 Scatter plot of Assay A data with least-squares line.

200

Observed Mass

y-hat =0.62 + 0.93 Exp

150

100

50

50

100 150 Expected Mass

200

Figure 4 Scatter plot of Assay B data with least-squares line.

Designing Experiments for Validation of Quantitative Methods

43

200

Observed Mass

y-hat = 2.26 + 0.93 Exp 150

100

50

50

100

150

200

Expected Mass

Figure 5 Scatter plot of Assay C data with least-squares line. 200

Observed Mass

y-hat = 2.93 + 0.93 Exp 150

100

50

50

100 150 Expected Mass

200

Figure 6 Scatter plot of Assay D data with least-squares line. Table 12 Summary of Lack-of-Fit Analysis for Linearity of Example Validation Data

Assay A B C D

Lack-of-fit P-value

Maximum deviation (%)

Level of maximum deviation (μg)