Comprehensive Chemometrics Volume 1: Chemical and Biochemical Data Analysis

COMPREHENSIVE CHEMOMETRICS Chemical and Biochemical Data Analysis Related Titles of Interest JOURNALS Chemometrics an...

Author: Steven D. Brown | Romà Tauler | Beata Walczak

365 downloads 1349 Views 7MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

COMPREHENSIVE CHEMOMETRICS Chemical and Biochemical Data Analysis

Related Titles of Interest JOURNALS Chemometrics and Intelligent Laboratory Systems Analytica Chimica Acta Talanta BOOKS AND SERIES Chemometrics in Spectroscopy Chemometrics Tutorials Statistics in Spectroscopy Data Handling in Science and Technology Series Chemometrics: A Textbook Design and Optimization in Organic Synthesis Data Analysis and Signal Processing in Chromatography The Data Analysis Handbook Experimental Design: A Chemometric Approach Handbook of Chemometrics and Qualimetrics Multivariate Analysis of Data in Sensory Science Nature-inspired Methods in Chemometrics: Genetic Algorithms and Artificial Neural Networks Scientific Data Ranking Methods Statistical Design – Chemometrics Wavelets in Chemistry

COMPREHENSIVE CHEMOMETRICS Chemical and Biochemical Data Analysis

Editors-in-Chief

Steven D. Brown University of Delaware, Newark, DE, USA

` Tauler Roma Institute of Environmental Assessment and Water Research (IDÆA), Spanish Council of Scientific Research (CSIC), Barcelona, Spain

Beata Walczak University of Silesia, Katowice, Poland

AMSTERDAM BOSTON HEIDELBERG LONDON NEW YORK OXFORD PARIS SAN DIEGO SAN FRANCISCO SINGAPORE SYDNEY TOKYO

Elsevier Radarweg 29, PO Box 211, 1000 AE Amsterdam, The Netherlands Linacre House, Jordan Hill, Oxford OX2 8DP, UK First edition 2009 Copyright ª 2009 Elsevier B.V. All rights reserved The following article is a US Government work in the public domain and is not subject to copyright: CHAPTER 1.16 CONSTRAINED AND UNCONSTRAINED OPTIMIZATION No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without the prior written permission of the publisher Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone (+44) (0) 1865 843830; fax (+44) (0) 1865 853333; email: [email protected]. Alternatively you can submit your request online by visiting the Elsevier web site at http://elsevier.com/locate/permissions, and selecting Obtaining permission to use Elsevier material Notice No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. Because of rapid advances in the medical sciences, in particular, independent verfication of diagnoses and drug dosages should be made British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Library of Congress Catalog Number: 2008943480 ISBN: 978-0-444-52702-8 For information on all Elsevier publications visit our website at books.elsevier.com Printed and bound in Slovenia 09 10 11 12 11 10 9 8 7 6 5 4 3 2 1

Working together to grow libraries in developing countries www.elsevier.com | www.bookaid.org | www.sabre.org

Contents of Volume 1 Contributors to Volume 1

vii

Preface

ix

Editors in Chief

x

Contents of All Volumes

xi

Section Editors

xv

Statistics 1.01

An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management F. F. Pitard, Francis Pitard Sampling Consultants, Broomfield, CO, USA

1

1.02

Quality of Analytical Measurements: Statistical Methods for Internal Validation M. C. Ortiz, L. A. Sarabia, M. S. Sańchez, and A. Herrero, University of Burgos, Burgos, Spain

17

1.03

Proficiency Testing in Analytical Chemistry M. Thompson, University of London, London, UK

77

1.04

Statistical Control of Measures and Processes A. J. Ferrer-Riquelme, Technical University of Valencia, Valencia, Spain

97

1.05

Quality of Analytical Measurements: Univariate Regression M. C. Ortiz, M. S. Sańchez, and L. A. Sarabia, University of Burgos, Burgos, Spain

127

1.06

Resampling and Testing in Regression Models with Environmetrical Applications J. Roca-Pardinãs, University of Vigo, Vigo, Spain C. Cadarso-Sua´rez and W. Gonza´lez-Manteiga, University of Santiago de Compostela, Santiago de Compostela, Spain

171

1.07

Robust and Nonparametric Statistical Methods I. Gijbels and M. Hubert, Katholieke Universiteit Leuven, Leuven, Belgium

189

1.08

Bayesian Methodology in Statistics J. M. Bernardo, Universitat de Vale`ncia, Valencia, Spain

213

Experimental Design 1.09

Experimental Design: Introduction R. Phan-Tan-Luu, University Paul Cezanne, Marseille, France R. Cela, Universidad de Santiago de Compostela, Santiago, Spain

247

1.10

Screening Strategies R. Cela, Universidad de Santiago de Compostela, Santiago, Spain M. Claeys-Bruno and R. Phan-Tan-Luu, University Paul Cezanne, Marseille, France

251

v

vi Contents of Volume 1

1.11

The Study of Experimental Factors R. Carlson, University of Tromsø, Tromsø, Norway J. E. Carlson, Lulea˚ University of Technology, Lulea˚, Sweden

301

1.12

Response Surface Methodology L. A. Sarabia and M. C. Ortiz, University of Burgos, Burgos, Spain

345

1.13

Experimental Design for Mixture Studies D. Voinovich and B. Campisi, University of Trieste, Trieste, Italy R. Phan-Tan-Luu, University Paul Cezanne, Marseille, France

391

1.14

Nonclassical Experimental Designs R. Phan-Tan-Luu and M. Sergent, University Paul Cezanne, Marseille, France

453

1.15

Experimental Designs: Conclusions, Terminology, and Symbols R. Phan-Tan-Luu, University Paul Cezanne, Marseille, France R. Cela, Universidad de Santiago de Compostela, Santiago, Spain

501

Optimization 1.16

Constrained and Unconstrained Optimization B. W. Bader, Sandia National Laboratories, Albuquerque, NM, USA

507

1.17

Sequential Optimization Methods B. Dejaegher and Y. V. Heyden, Vrije Universiteit Brussel, Laarbeeklaan, Brussels, Belgium

547

1.18

Steepest Ascent, Steepest Descent, and Gradient Methods R. G. Brereton, University of Bristol, Bristol, UK

577

1.19

Multicriteria Decision-Making Methods M. Pavan, Joint Research Centre, European Commission, Ispra, Italy R. Todeschini, University of Milano–Bicocca, Milan, Italy

591

1.20

Genetic Algorithms R. Leardi, University of Genoa, Genoa, Italy

631

Index to Volume 1

655

Contributors to Volume 1 B. W. Bader Sandia National Laboratories, Albuquerque, NM, USA

Chapter 1.16 p. 507

J. M. Bernardo Universitat de Vale`ncia, Valencia, Spain

Chapter 1.08 p. 213

R. G. Brereton University of Bristol, Bristol, UK

Chapter 1.18 p. 577

C. Cadarso-Sua´rez University of Santiago de Compostela, Santiago de Compostela, Spain

Chapter 1.06 p. 171

B. Campisi University of Trieste, Trieste, Italy

Chapter 1.13 p. 391

J. E. Carlson Lulea˚ University of Technology, Lulea˚, Sweden

Chapter 1.11 p. 301

R. Carlson University of Tromsø, Tromsø, Norway

Chapter 1.11 p. 301

R. Cela Universidad de Santiago de Compostela, Santiago, Spain

Chapter 1.09 p. 247 Chapter 1.10 p. 251 Chapter 1.15 p. 501

M. Claeys-Bruno University Paul Cezanne, Marseille, France

Chapter 1.10 p. 251

B. Dejaegher Vrije Universiteit Brussel, Laarbeeklaan, Brussels, Belgium

Chapter 1.17 p. 547

A. J. Ferrer-Riquelme Technical University of Valencia, Valencia, Spain

Chapter 1.04 p. 97

I. Gijbels Katholieke Universiteit Leuven, Leuven, Belgium

Chapter 1.07 p. 189

W. Gonza´lez-Manteiga University of Santiago de Compostela, Santiago de Compostela, Spain

Chapter 1.06 p. 171

A. Herrero University of Burgos, Burgos, Spain

Chapter 1.02 p. 17

Y. V. Heyden Vrije Universiteit Brussel, Laarbeeklaan, Brussels, Belgium

Chapter 1.17 p. 547

M. Hubert Katholieke Universiteit Leuven, Leuven, Belgium

Chapter 1.07 p. 189

vii

viii Contributors to Volume 1

R. Leardi University of Genoa, Genoa, Italy

Chapter 1.20 p. 631

M. C. Ortiz University of Burgos, Burgos, Spain


M. Pavan Joint Research Centre, European Commission, Ispra, Italy

Chapter 1.19 p. 591

R. Phan-Tan-Luu University Paul Cezanne, Marseille, France

Chapter 1.09 p. 247 Chapter 1.10 p. 251 Chapter 1.13 p. 391 Chapter 1.14 p. 453 Chapter 1.15 p. 501

F. F. Pitard Francis Pitard Sampling Consultants, Broomfield, CO, USA

Chapter 1.01 p. 1

J. Roca-Pardinãs University of Vigo, Vigo, Spain

Chapter 1.06 p. 171

M. S. Sańchez University of Burgos, Burgos, Spain

Chapter 1.02 p. 17 Chapter 1.05 p. 127

L. A. Sarabia University of Burgos, Burgos, Spain


M. Sergent University Paul Cezanne, Marseille, France

Chapter 1.14 p. 453

M. Thompson University of London, London, UK

Chapter 1.03 p. 77

R. Todeschini University of Milano–Bicocca, Milan, Italy

Chapter 1.19 p. 591

D. Voinovich University of Trieste, Trieste, Italy

Chapter 1.13 p. 391

Preface Some 40 years ago, the first publications appeared on the use of computer-aided mathematics to analyze chemical data. With those publications, the modern field of chemometrics was launched. Both the speed and power of computers and the sophistication of analytical instrumentation have made great leaps in the intervening time. The ready availability of chemometric software, coupled with the increasing need for rigorous, systematic examination of ever-larger and more sophisticated sets of measurements from instrumentation, has generated great interest in reliable methods for converting the mountains of measurements into more manageable piles of results, and for converting those results into nuggets of useful information. Interest in application of chemometrics has spread well beyond chemists with a need to understand and interpret their measurements; now chemometrics is helping to make important contributions in process engineering, systems biology, environmental science, and other disciplines that rely on chemical instrumentation, to name only a few areas. As applications of chemometrics continue to grow, so too does the methodology of chemometrics itself. At 40, chemometrics is a scientific field with many mature areas, but it is also a field where change continues to occur at a rapid pace, driven both by advances in chemical instrumentation and measurement and by close connection of chemometrics with the machine learning, statistics, and signal processing research communities. The interfacial location of chemometrics, falling between measurements on the one side and statistical and computational theory and methods on the other, poses a challenge to the new practitioner: gaining sufficient breadth and depth to use chemometrics effectively. The four volumes of Comprehensive Chemometrics are the result of a meeting in Oxford in September 2005, where the editors planned a work that would cover all of the major areas of chemometric research and a wide sample of current applications. Our goal was to produce a reference work that would serve both the new and the experienced practitioner. We divided the coverage of methodology into sections: Statistics, edited by L. Sarabia; Experimental Design, edited by R. Phan-Tan-Luu; Optimization, edited by R. Leardi; Data Preprocessing, edited by J. Trygg; Linear Soft-Modeling, edited by A. de Juan; Unsupervised Data Mining, edited by D. Coomans; Linear Regression Modeling, edited by J. Kalivas; Non-Linear Regression, edited by L. Buydens; Classification Feature Selection, edited by B. Lavine; and Multivariate Robust Techniques, edited by P. van Espen. The Editors-in-Chief oversaw a section on applications, where several of the newer directions in chemometrics are explored in depth. What has resulted from this collaboration is a resource that captures the practice of chemometrics in the early twenty-first century. The four volumes in this work include about 100 chapters, making this the most wide-reaching and detailed overview of the field of chemometrics ever published. Comprehensive Chemometrics offers depth and rigor to the new practitioner entering the field, and breadth and varied perspectives on current literature to more experienced practitioners aiming to expand their horizons. Software and data sets, both of which are especially valuable to those learning the methods, are provided in many chapters. The coverage is not only comprehensive, but it is also authoritative; authors contributing to Comprehensive Chemometrics are among the most distinguished practitioners of the field. Comprehensive Chemometrics would not have been possible without the work of the Section Editors named above, who helped us identify authors and who were tireless in reviewing submissions. We also owe special thanks to Adrian Shell, Senior Acquisitions Editor at Elsevier, for supporting the project and seeing the project off, to Claire Byrne, our Developmental Editor, for her patience and persistence in seeing this project to completion, and to Hazel Harris, our Project Manager, for keeping the production schedule. Finally, we extend special thanks to all of our authors whose efforts have made the work the valuable reference that it is. Steven D. Brown, Roma´ Tauler and Beata Walczak September 2008

ix

Editors in Chief Steven D. Brown obtained the Ph.D. degree in analytical chemistry in 1978 from the University of Washington, working with Bruce Kowalski. The same year he was appointed Assistant Professor at the University of California, Berkeley, and he held a joint appointment at Lawrence Berkeley Laboratory. In 1981, he moved to Washington State University and, in 1986, to the Department of Chemistry and Biochemistry at the University of Delaware, where he is presently Willis F. Harrington Professor. He has served as Chair of the Department of Chemistry and Biochemistry for 5 years, as a Section president of the American Chemical Society, and as President of the North American Chapter of the International Chemometrics Society. He is one of the three Founding Editors of the Journal of Chemometrics and has served for 20 years, first as its North American Editor and then for 12 years as its Editor-inChief. He was winner of the first EAS Award in Chemometrics in 1986. His research interests concern a wide range of problems in chemometrics. The focus of his research has been the development of new instrumental methods through use of multivariate mathematical methods for multicomponent analysis, including calibration transfer, and the novel use of data fusion methods. Roma` Tauler Ferre´ obtained his PhD in Chemistry at the University of Barcelona in 1984. He is Research Professor at the Institute of Environmental Assessment and Water Research (IDÆA), CSIC, in Barcelona (Spain), since July 2003. Previously he was Associate Professor of Analytical Chemistry at the Analytical Chemistry Department of the University of Barcelona during 1987– 2003. He performed postdoct stays at Institut fu¨r Anorg. u. Anal. Chemie, Univ. of Innsbruck, Innsbruck (Austria) in 1985 and 1989 and a year’s sabbatical as a research scientist at the Center for Process Analytical Chemistry (CPAC), Department of Chemistry, University of Washington, Seattle, USA, in 1992. At present, he is the Editor in Chief of the journal Chemometrics and Intelligent Laboratory Systems and of this Major Reference Work on Chemometrics. He has published more than 200 research papers, most of them in the field of chemometrics and its applications, and in particular in the development and applications of new multivariate resolution methods. In the recent years he has focused more on the investigation of environmental problems. Beata Walczak graduated in chemistry from the Faculty of Mathematics, Physics and Chemistry, Silesian University, Katowice, Poland, in 1979. Since then, she has been working in the Institute of Chemistry, Silesian University, where now she is the head of the Department of Chemometrics. She has been involved in chemometrics from the early 1990s and her main scientific interest is in all aspects of data exploration and modeling (dealing with missing and censored data, dealing with outliers, data representativity, enhancement of instrumental signals, signal warping, data compression, linear and nonlinear projections, development of modeling approaches, feature selection techniques, etc.). She has authored and co-authored around 140 scientific papers and 250 conference papers, and has delivered many invited lectures at numerous international chemistry meetings. She acts as Editor of Chemometrics and Intelligent Laboratory Systems and ‘Data Handling in Chemistry and Technology’ (the Elsevier book series), and also as a member of the editorial boards of Talanta, Analytical Letters, Journal of Chemometrics and Acta Chromatographica. x

Contents of All Volumes Volume 1 Statistics Edited by L. Sarabia, University of Burgos, Burgos, Spain 1.01 1.02 1.03 1.04 1.05 1.06 1.07 1.08

An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management Quality of Analytical Measurements: Statistical Methods for Internal Validation Proficiency Testing in Analytical Chemistry Statistical Control of Measures and Processes Quality of Analytical Measurements: Univariate Regression Resampling and Testing in Regression Models with Environmetrical Applications Robust and Nonparametric Statistical Methods Bayesian Methodology in Statistics

Experimental Design Edited by R. Phan-Tan-Luu, University Paul Cezanne, Marseille, France 1.09 1.10 1.11 1.12 1.13 1.14 1.15

Experimental Design: Introduction Screening Strategies The Study of Experimental Factors Response Surface Methodology Experimental Design for Mixture Studies Nonclassical Experimental Designs Experimental Designs: Conclusions, Terminology, and Symbols

Optimization Edited by R. Leardi, University of Genoa, Genoa, Italy 1.16 1.17 1.18 1.19 1.20

Constrained and Unconstrained Optimization Sequential Optimization Methods Steepest Ascent, Steepest Descent, and Gradient Methods Multicriteria Decision-Making Methods Genetic Algorithms

Volume 2 Data Preprocessing Edited by J. Trygg, Umea˚ University, Umea˚, Sweden 2.01 2.02

Background Estimation, Denoising, and Preprocessing Denoising and Signal-to-Noise Ratio Enhancement: Classical Filtering xi

xii Contents of All Volumes

2.03 2.04 2.05 2.06 2.07 2.08 2.09 2.10 2.11

Denoising and Signal-to-Noise Ratio Enhancement: Wavelet Transform and Fourier Transform Denoising and Signal-to-Noise Ratio Enhancement: Derivatives Denoising and Signal-to-Noise Ratio Enhancement: Splines Variable Shift and Alignment Normalization and Closure Model Based Preprocessing and Background Elimination: OSC, OPLS, and O2PLS Standard Normal Variate, Multiplicative Signal Correction and Extended Multiplicative Signal Correction Preprocessing in Biospectroscopy Batch Process Modeling and MSPC Evaluation of Preprocessing Methods

Linear Soft-Modeling Edited by A. de Juan, University of Barcelona, Barcelona, Spain 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 2.20 2.21 2.22 2.23 2.24 2.25

Linear Soft-Modeling: Introduction Principal Component Analysis: Concept, Geometrical Interpretation, Mathematical Background, Algorithms, History, Practice Independent Component Analysis Introduction to Multivariate Curve Resolution Two-Way Data Analysis: Evolving Factor Analysis Two-Way Data Analysis: Detection of Purest Variables Two-Way Data Analysis: Multivariate Curve Resolution – Noniterative Resolution Methods Two-Way Data Analysis: Multivariate Curve Resolution – Iterative Resolution Methods Two-Way Data Analysis: Multivariate Curve Resolution – Error in Curve Resolution Multiway Data Analysis: Eigenvector-Based Methods Multilinear Models: Iterative Methods Multiset Data Analysis: ANOVA Simultaneous Component Analysis and Related Methods Multiset Data Analysis: Extended Multivariate Curve Resolution Other Topics in Soft-Modeling: Maximum Likelihood-Based Soft-Modeling Methods

Unsupervised Data Mining Edited by D. Coomans, James Cook University, Townsville, QLD, Australia 2.26 2.27 2.28 2.29 2.30 2.31

Unsupervised Data Mining: Introduction Common Clustering Algorithms Data Mapping: Linear Methods versus Nonlinear Techniques Density-Based Clustering Methods Model-Based Clustering Tree-Based Clustering and Extensions

Volume 3 Linear Regression Modeling Edited by J. Kalivas, Idaho State University, Pocatello, ID, USA 3.01 3.02 3.03 3.04 3.05 3.06 3.07 3.08 3.09

Calibration Methodologies Regression Diagnostics Validation and Error Preprocessing Methods Variable Selection Missing Data Robust Calibration Transfer of Multivariate Calibration Models Three-Way Calibration

Contents of All Volumes xiii

Non-Linear Regression Edited by L. Buydens, Institute for Molecules and Materials, Radboud University of Nijmegen, Nijmegen, The Netherlands 3.10 3.11 3.12 3.13 3.14

Model-Based Data Fitting Kernel Methods Linear Approaches for Nonlinear Modeling Other Methods in Nonlinear Regression Neural Networks

Classification Edited by B. Lavine, Oklahoma State University, Stillwater, OK, USA 3.15 3.16 3.17 3.18 3.19

Classification: Basic Concepts Statistical Discriminant Analysis Decision Tree Modeling in Classification Feed-Forward Neural Networks Validation of Classifiers

Feature Selection Edited by B. Lavine, Oklahoma State University, Stillwater, OK, USA 3.20 3.21 3.22 3.23

Feature Selection: Introduction Multivariate Approaches: UVE-PLS Multivariate Approaches to Classification using Genetic Algorithms Feature Selection in the Wavelet Domain: Adaptive Wavelets

Multivariate Robust Techniques Edited by P. van Espen, University of Antwerp, Antwerp, Belgium 3.24

Robust Multivariate Methods in Chemometrics

Volume 4 Applications Edited by S. D. Brown, University of Delaware, Newark, DE, USA; R. Tauler, Institute of Environmental Assessment and Water Research (IDÆA), Spanish Council of Scientific Research (CSIC), Barcelona, Spain and B. Walczak, University of Silesia, Katowice, Poland 4.01 4.02 4.03 4.04 4.05 4.06 4.07 4.08 4.09 4.10 4.11 4.12 4.13 4.14 4.15

Representative Sampling, Data Quality, Validation – A Necessary Trinity in Chemometrics Multivariate Statistical Process Control and Process Control, Using Latent Variables Environmental Chemometrics Application of Chemometrics to Food Chemistry Chemometrics in QSAR Spectroscopic Imaging Spectral Map Analysis of Microarray Data Analysis of Megavariate Data in Functional Genomics Systems Biology Chemometrics Role within the PAT Context: Examples from Primary Pharmaceutical Manufacturing Smart Sensors Chemometric Analysis of Sensory Data Chemometrics in Electrochemistry Chemoinformatics High-Performance GRID Computing in Chemoinformatics

This page intentionally left blank

Section Editors Lutgarde Buydens is head of the Chemometrics Department at the Institute for Molecules and Materials of the Radboud University of Nijmegen, the Netherlands. She originally studied pharmacy at the Brussels free University and did her Ph.D. with professor Massart on a QSAR subject. After a postdoc position at the University of Illinois in Chicago with the group of Professor William Dunn III, she started at the University of Nijmegen where she became a full professor in 1995. She received the first Elsevier Chemometrics award in 1992. Her research interests include the (further) development of chemometrical techniques within several application areas. Chemometrical techniques for combining different (spectroscopic imaging) data is her major research area. Professor Buydens is author of more than 200 publications in international scientific journals and co-author of 4 scientific books, including the 2 volumes of the Handbook of Chemometrics and Qualimetrics. Since 1998, she has been the editor of the journal Analytica Chimica Acta and has also been the co-editor of various books and special issues of international scientific journals. Danny Coomans was the first Ph.D. student of chemometrics of the late Professor D. L. Massart at the Vrije Universiteit Brussel, one of the founders of the field of chemometrics. He obtained the degree in 1982. He is still parttime associated with the department. He is full-time Professor in Statistics and Intelligent Data Analysis in the School of Mathematics, Physics and IT at the James Cook University, Townsville, Australia. His research interests are in multivariate statistics, statistical pattern recognition and data mining, waveletbased feature extraction, computer intensive methods, and distributed computing. From an applied point of view, his areas of expertise include chemometrics, environmetrics, and bioinformatics with special interest in the analysis of spectral databases, microarrays, QSAR data, and aspects of industrial laboratory quality control. He has published over 250 research papers and has contributed to monographs related to chemometrics.

Anna de Juan has been an Associate Professor in the Department of Analytical Chemistry at the University of Barcelona since 2003, teaching chemometrics at undergraduate and graduate levels. She holds a degree and Ph.D. in chemistry from the University of Barcelona and her expertise is in multivariate curve resolution (MCR) methods: theoretical development and application to bioanalytical and analytical problems. Since 2002 she has been a member of the Editorial Advisory Board of Chemometrics and Intelligent Laboratory Systems and since 2006 of Analytica Chimica Acta. In 2004, she received the 4th Chemometrics Elsevier Award together with Karl Booksh. She has published around 60 papers in international journals and books, especially on multivariate curve resolution developments and related methods and on applications to the study of bioanalytical processes, image analysis, and general analytical applications.

xv

xvi

Section Editors

John H. Kalivas is a Professor in the Department of Chemistry at Idaho State University, a primary undergraduate institution in Pocatello. He is author and co-author of over 80 professional papers, book chapters, and books. He is a member of Sigma Xi, the Society for Applied Spectroscopy, and the Council on Undergraduate Research and he serves on the Editorial Board of the Journal of Chemometrics, Applied Spectroscopy, and Analytical Letters. Dr. Kalivas received the B.S. degree (1978) in chemistry from California Polytechnic State University, San Luis Obispo, and the Ph.D. degree (1982) in chemistry from the University of Washington, Seattle.

Barry K. Lavine is an Associate Professor of Chemistry at Oklahoma State University in Stillwater, OK. He has published around 90 papers in chemometrics and is on the editorial board of several journals including the Journal of Chemometrics, Microchemical Journal, and Chemoinformatics. He is the Assistant Editor of Chemometrics for Analytical Letters. Lavine’s research interests encompass many aspects of the applications of computers in chemical analysis including pattern recognition, multivariate curve resolution, and multivariate calibration using genetic algorithms and other evolutionary techniques.

Riccardo Leardi graduated in pharmaceutical chemistry and technology in 1983. Since then, he has been working in the section of Analytic Chemistry of the Department of Pharmaceutical and Food Chemistry and Technologies of the Faculty of Pharmacy of the University of Genova, and his research field is chemometrics. His interests are mainly devoted to problems related to food, environmental and clinical data, and to experimental design and process optimization. In the last years, his research focused mainly on genetic algorithms and on three-way methods. He is author of around 80 papers and around 80 communications in national and international meetings; he has been an invited speaker in 13 international meetings and in several industries and research centers. He is Review Editor of Journal of Chemometrics and Editorial Adviser of Analytica Chimica Acta. In November 2002, he started his activity of chemometric consultancy.

Roger Phan-Tan-Luu was Professor at the University Paul Cezanne of Marseille, France. In 1970, he founded the Laboratory of Methodology of Experimental Research in which he developed both the methodological approach and the algorithms that enable construction of efficient designs. He was at the start of the establishment of trainings about the Methodology of Experimental Research in several universities and industries in France and abroad. He is author and co-author of numerous scientific articles dealing with experimental designs. He belongs to the scientific committees of several international congresses. In 1992, Roger Phan-Tan-Luu was awarded Doctor Honoris Causa of the University of Ume˚a (Sweden).

Section Editors

xvii

Luis A. Sarabia received his Ph.D. in Statistics from the University of Valladolid (Spain) in 1979. Since 1974, he is teaching Statistics and Mathematics mostly to graduate and postgraduate students of Chemistry. At present, his research is centred on Chemometrics as a member of the Chemometrics and Qualimetrics group of the University of Burgos. His research activities include development of software and implementation of nonparametric and robust statistical methods, genetic algorithms, neural networks, etc. He is also involved in multivariate/multiway regression methods, methodology of the experimental design, quality assurance, and validation. He is the author of about a hundred papers on these matters.

Johan Trygg obtained his Ph.D in Organic Chemistry from Ume˚a University in 2001 (S. Wold). After receiving his Ph.D. at Ume˚a University he was awarded the prestigious Knut & Alice Wallenberg scholarship, and spent 2 years abroad as postDoc, first at the Institute for Molecular Bioscience (IMB), Brisbane, Australia, and later at Imperial College, London, both positions in the field of metabonomics. Johan Trygg brings with him years of analysis of high-complexity systems using an ever-broadening adaptation of chemometrics analysis, and provides expertise on the analysis and interpretation of highly complex data sets. Johan Trygg is an Associate Professor of Chemometrics at Ume˚a University, Sweden. He is acting group leader of Chemometrics & Bioinformatics at Ume˚a Plant Science Centre (UPSC) at Ume˚a University.

Pierre van Espen was one of the first Ph.D. students at the University of Antwerp. He is currently Professor in the Department of Chemistry at the University of Antwerp and Profesor Invitado at the University of Havana, Cuba. He accomplished some pioneering work on the analysis of X-ray spectra by means of the development of software for the PDP 11/45 computer. At a later stage, this resulted in the AXIL (analysis of X-ray spectra by iterative least squares regression) package, which gained international acceptance and widespread use through the International Atomic Energy Agency (IAEA). Apart from the AXIL package, his work also comprises application and extension of chemometric methods for the analysis of data from X-ray spectrometry.


1.01 An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management F. F. Pitard, Francis Pitard Sampling Consultants, Broomfield, CO, USA ª 2009 Elsevier B.V. All rights reserved.

1.01.1 1.01.2 1.01.3 1.01.4 1.01.4.1 1.01.4.2 1.01.4.2.1 1.01.4.2.2 1.01.4.2.3 1.01.4.3 1.01.4.3.1 1.01.4.3.2 1.01.4.3.3 1.01.4.4 1.01.4.4.1 1.01.5 1.01.5.1 1.01.5.2 1.01.5.3 1.01.5.4 1.01.6 1.01.6.1 1.01.6.2 1.01.7 1.01.7.1 1.01.8 1.01.8.1 1.01.8.1.1 1.01.8.1.2 1.01.8.1.3 1.01.8.2 1.01.8.3 1.01.8.3.1 1.01.8.3.2 1.01.8.3.3 1.01.8.3.4 1.01.9 1.01.10 References

Introduction Scope Definitions and Notations Dividing a Complex Problem into its Basic Components Dividing the Small-Scale Variability Optimization of Sampling Protocols The in situ nugget effect The fundamental sampling error The grouping and segregation error The Practical Implementation of the Sampling Protocol The increment delimitation error The increment extraction error The weighting error The Preservation of Samples Integrity Sample preparation errors Exercises Challenging the Reader Exercise #1: A Worldwide Problem for Ore Grade Control in Open Pits: Blasthole Sampling Exercise #2: Correctness of Straight-Path Cross-Stream Sampling Systems Exercise #3: Correctness of Rotating Vezin Cross-Stream Sampling Systems Exercise #4: Correctness of Hammer Cross-Belt Samplers The Critical Importance of Sampling Courses Case #1: A Bad Protocol for Blastholes Followed by an Incorrect Implementation Case #2: An Incorrect Sampling System for the Final Tail of a Flotation Plant The Enemies and Their Link to Geostatistics Important Remark Large-Scale Variability Definition of the Variogram Selection of a given process parameter of interest Heterogeneity affecting the given process parameter of interest Measuring heterogeneity variability with the variogram Extrapolation of the Variogram to Time or Distance Zero The Importance of Large-Scale Variability Variability issues during exploration Variability issues during mining Variability issues within a processing plant Variability issues during trade with customers Conclusions Recommendations

2 2 3 3 3 3 3 4 4 5 5 5 5 5 5 6 6 7 7 7 9 9 9 10 10 10 10 11 11 11 12 12 12 13 13 13 13 13 15

1

2 An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management

1.01.1 Introduction Considering the example of the mining industry, out of many other possible examples, the key to total quality management (TQM) is to optimize the recovery of natural resources and make quality products to satisfy users on several fronts such as reliability, effectiveness, and minimal cost: it cannot and will not be done without a thorough understanding of theory of sampling (TOS). As TOS is not taught at universities, the resulting economic losses are plaguing the industry. Management decisions are based on precise and accurate sampling. It is of utmost importance to communicate the benefits of correct sampling to management, to a board of directors, to shareholders, to geologists, to drillers, to miners, to metallurgists, to chemists, to sales people, environmentalists, geostatisticians, and to statisticians. It is the only way to get necessary cash flow, more profit, and added share value. If stakeholders cannot see the value of correct sampling, it is the company’s responsibility to show them through education of the management team. Training of key personnel such as geologists, drillers, miners, metallurgists, chemists, and environmentalists is essential to obtain results. They all must monitor and verify the quality of data, so that geostatisticians and even conventional statisticians can perform reliable, believable risk assessments, to enable the management to make ultimately crucial financial decisions.

1.01.2 Scope The acquisition of a reliable database is always a valuable asset to a company. However, for such a success to happen, a correct and balanced strategy is needed, which is illustrated as a three-legged table in Figure 1. If one leg of the table is weak the entire table may collapse. Usually, the weakest leg is a style of management that does not sufficiently support good sampling practices, though it is the only way to allow statisticians to do their work in understanding variability in the deposit or in processes, and subsequently advise management for wise courses of action. However, implementing correct sampling is easier said than done. Similar to safety issues for which companies spend millions of dollars, it must be internally standardized through correctness, internal guidelines, sustained training, and enforcement auditing. It must also be monitored for its added value through improved natural resources recovery, improved conciliation between economic units (e.g., mine, mill, smelter, and refinery), and added stakeholder value. The collection of samples has only one objective in mind: understanding variability on a large scale. Unfortunately, for practical reasons a sample is often made of a relatively small amount of material. Therefore, as it is collected, another type of unwanted variability is always introduced, which is a

Company $ benefits, added stakeholder value, and market perception

Emphasis on causes of problems by proactive management

A strong commitment to good sampling and good laboratory practices

Figure 1 The concept of the three-legged table.

Capability to understand variability and to perform reliable statistical studies

An Introduction to the Theory of Sampling: An Essential Part of Total Quality Management

3

small-scale variability that may often overwhelm the large-scale variability that is important to measure. Many people often do not realize that the variability they are looking at has little to do with the variability they want to measure; it is essential to make the difference as clear as possible, and this is the true, indepth objective of TOS.

1.01.3 Definitions and Notations The length of this paper being limited, for definitions and notations, the reader is referred to textbooks from various authors listed in References.

1.01.4 Dividing a Complex Problem into its Basic Components The strength of the TOS is the use of a logical Cartesian approach, which divides small- and large-scale variability into basic components that can be further analyzed one by one. If everyone understood that basic concept they would not struggle to understand TOS. It does not matter how deep anyone goes into the mathematics of TOS; if this basic Cartesian concept is not understood, it will result in confusion, ineffectiveness, and failure to reach economic objectives, which is a common observation in many industries around the world today. 1.01.4.1

Dividing the Small-Scale Variability

The small-scale variability can be divided into four categories: 1. 2. 3. 4.

the optimization of sampling protocols, the practical implementation of sampling protocols, the preservation of samples integrity, and the minimization of analytical errors.

Analytical errors are not the subject of this paper, but can be explored in guidelines for good laboratory practices. 1.01.4.2

Optimization of Sampling Protocols

The optimization of sampling protocols involves a good understanding of small-scale heterogeneity, which affects any component of interest in the lot to be sampled. 1.01.4.2.1

The in situ nugget effect The TOS addresses sampling difficulties when sampling particulate materials. However, before the material is broken up, as distance between samples tends toward zero, a small-scale Constitution Heterogeneity may greatly affect the repeatability of a sample, with a given selected mass, drilled at any given place. The preliminary knowledge of local mineralogy may help in the selection of a minimum mass for the basic module of observation (e.g., selection of a drilling diameter and sample length) during exploration, ore reserves estimations, and for the capability to be selective at a certain cutoff grade and at any given place during the mining operation. Failure to address this problem early enough can result in stunning estimation failures. The mean m[NE] of the in situ Nugget effect (NE) is zero; however, its large variance s2[NE] may lead to severe data skewness and therefore unwanted local illusions followed by stunning economic consequences. It is suggested to make a great difference between m[NE] ¼ 0 when there is free access to an entire population of possible skewed data points in a given area, and m[NE] 6¼ 0 when too few skewed data points affected by a large variance


are available in the same given area. Too often, large mistakes are repeated over and over because too few data points belonging to a skewed distribution are available in a given area.

1.01.4.2.2

The fundamental sampling error As soon as the in situ material is broken up into fragments, the size distribution, shape, and density of fragments begin to play an important role in sampling. Typical differences between individual fragments lead to the concept of constitution heterogeneity in TOS. A good understanding of constitution heterogeneity is the key to optimize sample and subsample masses. Heterogeneity tests and sampling nomographs are used to make such optimization.1–4 It is critically important to understand the various approaches that were used in TOS to perform this task. Indeed, it would be a great mistake, as frequently happens, to believe that Dr. Pierre M. Gy created one universal formula to calculate the variance of the fundamental sampling error (s2[FSE]), which is the result of constitution heterogeneity for broken up materials.5 There are many different approaches depending on the heterogeneity of the constituents of interest. The main approaches are:

1. the general case where the constituent of interest is not liberated from its gangue, 2. the special case where the constituent of interest is liberated from its gangue and may show delayed comminution properties, 3. the case where the constituent of interest is finely disseminated inside another major constituent, and 4. the case where the main objective of sampling is to obtain a sample representative of the particle size distribution. Failure to distinguish these different cases leads to a total misunderstanding of TOS. However, there are more subtleties to understand: suggested formulas have their limitations well addressed in TOS. For example, if the variance of the FSE becomes very large, formulas become meaningless and a massive misuse of TOS may take place. It was never the objective of TOS to address out-of-control variances generated by unwise protocols for which skewness introduces unfortunate illusions and misconceptions. Let us be clear on this issue: TOS cannot help people who are not preventive in understanding the constitution heterogeneity of their materials. The mean m[FSE] of the FSE is zero; however, its large variance may lead to severe data skewness and therefore unwanted local illusions. Wisdom in sampling starts with a good understanding of FSE so that simplistic Gaussian statistics may indeed reasonably apply. The last thing anyone wants to deal with is a FSE introducing skewness in a generated database as it would most certainly result in estimation errors very few people understand.

1.01.4.2.3

The grouping and segregation error Gravity is the mother of segregation. Materials segregate as soon as they are crushed, transported, and stored in stockpiles, silos, bins, or tanks throughout processes. Increments collected at any given place within a lot of particulate material to be sampled (i.e., small-scale heterogeneity) can be very different, leading to a certain amount of distribution heterogeneity. Materials segregate because fragments have a different size, shape, and density. Because segregation is a transient phenomenon changing all the time, so does distribution heterogeneity. The most effective action to minimize the negative effect of distribution heterogeneity in sampling protocols is to collect as many and as small correct increments to make up a sample as practically possible. If problems generated by small-scale distribution heterogeneity are not carefully addressed, a substantial grouping and segregation error (GSE) may result. The mean m[GSE] of the GSE is zero; however, its large variance may lead to catastrophic local illusions. Experience proves that GSE is far more difficult to minimize than what many people may think. Actually, many standard practices suggested for homogenizing materials do the opposite and segregate materials even further. In sampling, beware of old traditions and ceremonial misconceptions.


1.01.4.3

5

The Practical Implementation of the Sampling Protocol

Now that nearly everything is known to minimize the irrelevant, harmful variability introduced by a sampling protocol, it is still necessary to find out what happens during the practical implementation of that protocol. During this critically important phase, new sources of annoying and harmful variability take place, perhaps the most dangerous ones. There are three independent components that should be carefully reviewed, which are defined in the TOS with no possible ambiguity. 1.01.4.3.1

The increment delimitation error For an increment making up a sample to be correct, every part of the lot to be sampled must have exactly the same chance of becoming part of the sample. Therefore, the boundaries of the selected increment must coincide with an isotropic volume of observation which is:

A sphere if the lot is considered three-dimensional. Nobody knows how to do this, and this is the reason why

grab sampling or in-stream stationary probe sampling leads to unsolvable sampling problems. They are nothing more than a form of gambling. A cylinder if the lot can be considered two-dimensional. This is possible when drilling ore bodies or drilling a copper cathode. A uniform slice across a flowing stream if the stream is considered one-dimensional. We know how to solve these problems under the discharge of a stream. We do not know how to solve these problems when taking a cut on the top of a conveyor belt using a cross-belt hammer sampler.

The mean m[DE] of delimitation error (DE) is not zero; therefore, this error is a dangerous bias generator. The variance of DE can be quite large as it depends on how severe the deviation from the ideal isotropic volume of observation is and how severe transient segregation taking place at the sampling point is; the combination of these two problems can be devastating. Rules of increment delimitation correctness listed in the TOS are not negotiable. 1.01.4.3.2

The increment extraction error The increment extraction error (EE) could have been called the increment recovery error, which is well known by geologists drilling a deposit. This error takes place when the sampling tool is selective on what it is taking. The mean m[EE] of EE is not zero; therefore, this error is a dangerous bias generator. The variance of EE can be quite large as it depends on how severe the recovery problem is. Sampling systems must be carefully designed to minimize this error. Rules of increment extraction correctness listed in TOS are not negotiable. 1.01.4.3.3

The weighting error A good sampling system must be reasonably proportional. For example, when the cutter of a cross-stream sampler crosses a falling stream, it must take an increment the mass of which is proportional to the stream flow rate at this instant. Excessive deviation from proportionality can lead to a bias. Proportional sampling is probably the future of sampling. The sum of the variances of DE, EE, and weighting error (WE) is part of the variance of the materialization error (ME). Readers familiar with TOS may notice that the author chose to include the WE in the family making up the ME. The reason is that WE is the direct effect of how the sampling system collects increments. If the sampling system is not proportional, WE is likely to be significant; whereas if the sampling system is proportional, WE may be negligible. However, the flow rate fluctuations of a stream, which is part of the largescale variability, may also greatly affect WE. It is important to regulate flow rates as much as possible so that WE can be kept negligible during the sample materialization phase.

1.01.4.4

The Preservation of Samples Integrity

1.01.4.4.1

Sample preparation errors The variance of the preparation errors (PE) is a separate component introduced between sampling stages. It is convenient to include PE in the ME, grouping together all the errors generated by the practical implementation


2

S ME =

ΣS n

2 DEn

+

ΣS

2 EEn

+

n

ΣS

2 WEn

+

n

ΣS

2 PEn

n

Selective process during sampling stages

Non-selective process during preparation stages

Figure 2 Expanding the materialization error to include all errors introduced by the practical implementation of the sampling protocol.

of a sampling protocol. PE can be the result of contamination, losses, alteration of physical or chemical composition, human errors, ignorance, carelessness, fraud, or sabotage. The mean of PE is not zero; therefore, these errors are dangerous bias generators. The variance of PE can be 2 quite large as it depends on many nonselective factors. Finally, the variance SME of the ME can be written as illustrated in Figure 2. The components of the ME are not well known by manufacturers of sampling equipment, or perhaps, a more accurate statement would be to say they do not make a clear difference between these four independent errors. This problem is also perpetuated by standards on sampling, most of which are obsolete, and reluctant to endorse the TOS. Furthermore, these standards usually do not distinctly address the difference between errors generated by the selective process (i.e., DE, EE, and WE) and those generated by the nonselective process (i.e., PE). The key to understanding problems created by these four errors can be summarized by the following statement: all the constituents of the lot to be sampled must be given an equal probability p of being selected and preserved as part of the sample, which leads to the notion of equiprobabilistic sampling, which in turn leads to the notion of correct sampling systems. Not respecting this cardinal rule almost always results in the presence of sampling biases that cannot be accounted for and totally ruins the previous efforts made to optimize the sampling protocol. The correctness or incorrectness of a sampling system is a structural property. If the sampling system is not correct by design (i.e., structure), devastating effects will result, regardless of results from bias tests that would at times tend to prove otherwise.

1.01.5 Exercises Challenging the Reader Solutions for the following exercises are beyond the scope of this paper, but they are well addressed in other documents from the author listed in References. 1.01.5.1 Exercise #1: A Worldwide Problem for Ore Grade Control in Open Pits: Blasthole Sampling Figure 3 illustrates seven areas in sampling blastholes. Each area can be the object of major deviations from sampling correctness. At each of these areas name the possible sampling error that may take place (e.g., DE, EE, WE, or PE). The reader has 20 min to sort them by name and to provide a solution. If it cannot be done, further training in sampling methodology is highly recommended.6 Biases taking place from blasthole drilling cost mining companies a fortune every year. Usually and unfortunately it is a well-hidden cost. Assuming DE, EE, WE, and PE are well taken care of, which is a huge assumption, geostatistical simulations using results from duplicate field samples can have access to some extent to such worrisome costs, and actually it would be advisable for geostatisticians to address such potential financial losses in feasibility studies if they had a deep knowledge of TOS. Someone may ask why do this if DE, EE, WE, and PE have been taken care of. Make no mistake, conciliation problems are not only the result of sampling errors introducing biases, they are also the result of unreasonable variances affecting the in situ Nugget effect, FSE, and GSE, and these indeed can have their effects simulated. But, nobody can make a simulation of economic losses generated by the bias generators DE, EE, WE, and PE.


7

F

E Segregation

D G

Ideal sample Actual sample

A

Former sub-drill

C

Current sub-drill

B

Figure 3 The nightmare of ore-grade control.

1.01.5.2

Exercise #2: Correctness of Straight-Path Cross-Stream Sampling Systems

Straight-path cross-stream sampling systems are critically important to assess the mineral content of the feed to a plant, concentrates, and tailings. Yet, most of these sampling systems are flawed by design and will never fulfill their mission. The sampler illustrated in Figure 4 is a typical bottom-dump straight-path cross-stream sampler and as drawn it is apparently correct. Eleven important points are illustrated where other sampling systems may not be built correctly and generate difficulties and sampling biases at these points. At each of these areas name the possible sampling error that may take place (e.g., DE, EE, WE, or PE). The reader has 20 min to sort them by name and to provide a solution. If it cannot be done, further training in sampling methodology is highly recommended.7 Biases taking place at such sampling points can make it impossible to perform correct metallurgical accounting, and therefore it is very unlikely that the subsequent process can be optimized. Nobody can optimize an operation with a false database. 1.01.5.3

Exercise #3: Correctness of Rotating Vezin Cross-Stream Sampling Systems

Rotating Vezin cross-stream sampling systems are critically important for sampling small streams, or performing secondary or tertiary sampling in large sampling stations. Yet, most of these sampling systems are flawed by design and will never fulfill their mission. The sampler illustrated in Figure 5 is a well-designed Vezin sampler. Eleven important points are illustrated where other sampling systems may not be built correctly and generate difficulties and sampling biases at these points. At each of these areas, name the possible sampling error that may take place (e.g., DE, EE, WE, or PE). The reader has 20 min to sort them by name and to provide a solution. If it cannot be done, further training in sampling methodology is highly recommended.7 Biases taking place at such sampling points can make it impossible to perform correct metallurgical accounting, and therefore it is very unlikely that the subsequent process can be optimized. Nobody can optimize an operation with a false database. 1.01.5.4

Exercise #4: Correctness of Hammer Cross-Belt Samplers

Hammer cross-belt samplers are very popular, but they are not as correct as manufacturers may claim. The sampler illustrated in Figure 6 is a typical design. Seven important points are illustrated where severe


1

11

3 6 7

5

4

2

9 10

8

Figure 4 A primary sampler for the feed of a plant.

2

3

6 9

Falling stream

4

5 7

10

11

8 1

Figure 5 A rotating Vezin sampler.


9

7 6

4

1 2 3

5 Figure 6 A typical hammer cross-stream sampler.

difficulties may arise and generate sampling biases. At each of these areas name the possible sampling error that may take place (e.g., DE, EE, WE, or PE). The reader has 20 min to sort them by name and to provide a solution. If it cannot be done, further training in sampling methodology is highly recommended.7 Biases taking place at these sampling points can be financially devastating.

1.01.6 The Critical Importance of Sampling Courses All possible problems created by each point addressed in the above four exercises should be solved within minutes. Yet, because of a massive ignorance of TOS, these problems are the object of unnecessary doubts and arguments, time-consuming meetings, endless debates with manufacturers and engineering firms, and very expensive bias tests followed by doubtful statistics. Someone may wonder why doubtful statistics are used to interpret bias tests. The reason has something to do with a subtle property of segregation. Biases in sampling are the result of one form or another of segregation. However, the prime property of segregation is being a transient phenomenon changing all the time. Too many people think of a sampling bias as a constant bias as often observed for analytical biases. There is no such thing as a constant bias in sampling. A sample can be biased one way today, another way tomorrow, and can remain unbiased for a while. Therefore, a bias test for a sampling system can only declare there is indeed a bias; however, it cannot state that there is no bias. Furthermore, it is almost common practice to make a sampling bias test by comparing two bad samples. Each point addressed in the above exercises can lead to devastating money losses for the unaware company. Let us review a few well-documented cases that were presented at WCSB1 (World Conference on Sampling and Blending1) in Denmark in 2003.

1.01.6.1

Case #1: A Bad Protocol for Blastholes Followed by an Incorrect Implementation

Over a 10-year period a loss of US$134 000 000 was the result of a bad sampling and subsampling protocol used in a copper mine in Chile.8 The increase in the recovery of natural resources could be measured only after a better protocol was implemented for several consecutive months.

1.01.6.2

Case #2: An Incorrect Sampling System for the Final Tail of a Flotation Plant

Over a period of 20 years at a large copper mine in Chile, a stunning loss of US$2 000 000 000 was the result of using a flawed sampling system for the Final Tail of a copper–molybdenum flotation plant.8 Another company that treated the tailings for that period of time was quite prosperous. The first reaction of management after a flawless sampling station was installed was denial.


A book could be written about many other examples around the world, but usually companies are very silent about such a catastrophic economic outcome. This is where confidentiality agreements become convenient. There is no doubt denial can become a form of fraud.

1.01.7 The Enemies and Their Link to Geostatistics François-Bongarçon rightly said in 2003 that ‘‘TOS and geostatistics are inseparable’’. So far, we made a list of the enemies responsible for slowly inflating one of the most annoying sources of variability plaguing geostatistics. When calculating a variogram, it is often observed that as the distance between samples becomes practically nil a certain amount of variability remains, which is random and discontinuous in nature; it is called V[j ¼ 0], j being the lag between samples. For simplicity, we chose to call it V[0]. Figure 7 makes a summary of 2 refers to the partial variance introduced by the protocol. It them, expressed as variances, and the variance SHE 1 is indeed annoying when an exploration or ore-grade control database is affected by a certain amount of variability that cannot be explained by the structural properties of the deposit. It is indeed annoying when a process control database is affected by a certain amount of variability that cannot be explained by what is happening in the process. It is indeed annoying when an expensive environmental assessment database is affected by a certain amount of variability that cannot be explained by what is happening in the environment. 1.01.7.1

Important Remark

V[0] is a variance. Some sampling errors such as DE, EE, WE, and PE are bias generators. A variogram does not see a bias, though the mean of DE, EE, WE, and PE is not zero. However, because these sampling biases fluctuate considerably, the variogram may show an increase for V[0], which cannot be explained by the sampling protocol or the analytical variance.

1.01.8 Large-Scale Variability The study of the total variability in any lot to be sampled can be broken up into several components, and the variogram helps us take a close look at each of these components. This is well done in geostatistics9 for the study of mineral deposits, and in chronostatistics10,11 for the study of variability in a process. 1.01.8.1

Definition of the Variogram

The scientifically sound definition of the variogram as well as its limitations is beyond the scope of this paper. For these issues the reader is referred to the References section. Nevertheless, we shall proceed with a simplistic definition. Also, in what follows, the variability of differences between samples with a small lag must remain

ΣS n

ΣS n

ΣS

2

FSEn

n

2

S HE1

2

DEn

n

2

WEn

2

GSEn

ΣS

V[0]

2

S NE

ΣS

ΣS n

2

n

ΣS n

2

AEn

2

PEn

EEn

Figure 7 V[0] can be inflated by many factors; they are the enemies in sampling and they quickly add up.


11

reasonably constant through time if the variogram is used to study a parameter within a process. This limitation suggests taking precautions when selecting a series of chronological data, such as making reasonably sure along the selected chronology that no important changes have been applied to the process in the way it works. As such, it is likely that the suggested applications lose power when the basic sampling interval in the selected chronology is too long (e.g., days, weeks, or months). Aware of this limitation, the practitioner can use common sense not to mix data that do not belong to each other. 1.01.8.1.1

Selection of a given process parameter of interest The variability of parameters controlling or making up a process stream is most of the time a stochastic phenomenon. Such phenomenon is a hybrid in which we can find random and chaotic elements taking place at very short time intervals, but in which we can also find functional and continuous elements. Therefore, a certain parameter of interest (e.g., the proportion of a given size fraction in a stream feeding a SAG mill) characterizing a process stream can be described as follows: f ½t ¼ fL þ fA ½t þ fB ½t þ fC ½t

ð1Þ

where: fL is a constant term, such as the average proportion of a given size fraction of a stream feeding a SAG mill over f [t]. fA[t] is a random component characterized by neighboring fragments that are different and the resulting proportion of a given size fraction changing in a chaotic and unpredictable way at very short time intervals, regardless of the fact that these fragments may originate at the same place earlier in the process. fB[t] is a nonrandom, nonperiodic component, essentially continuous, which is an inherent property of the feed heading to the plant, for example, the slowly changing hardness of the ore. fC[t] is a periodic component characterized by the fact that people and machinery often work in a cyclic way, for example, a segregating stockpile introducing a cycle in the proportion of coarse fragments and fines feeding a SAG mill. 1.01.8.1.2

Heterogeneity affecting the given process parameter of interest Taking into account the fact that heterogeneity of a constant, such as fL, is obviously zero, the total heterogeneity affecting the parameter of interest in the stream is fully characterized by fA[t], fB[t], and fC[t]. 1.01.8.1.3

Measuring heterogeneity variability with the variogram The most natural way to compare two values such as f1 (e.g., the proportion of a given size fraction in the stream at time t1) and f2 (e.g., the proportion of a given size fraction in the stream at time t2), t1 and t2 being separated by a time lag called j, is to measure their difference d: d ¼ f1 – f2

ð2Þ

However, what is mostly relevant is the average difference d between many measurements N a given interval j apart: d¼

1 X fmþj – fm N –j m

ð3Þ

Such average difference converges toward zero; therefore, the squared difference should be used instead. Furthermore, differences account for variability twice; therefore, the convention has been taken to calculate the semi-variogram, leading to the following formula used for an absolute variogram: V ½j ¼

2 1 X fmþj – fm 2½N – j m

ð4Þ

A relative, dimensionless variogram could be calculated as well, making it easier to compare variograms from different experiments: V ½j ¼

X 2 1 fmþj – fm 2 2½N – j fL m

ð5Þ


The study of many variograms shows that they are usually made of four major components: V ½j ¼ VA ½j þ VB ½j þ VC ½j þ VD ½j

ð6Þ

where: VA[j] is a very short-range term, random, and discontinuous. At the limit, when j ¼ 0, this leads to a very important term in chronostatistics called V[j ¼ 0], simplified in further discussions to V[0], which is the variability from sampling, subsampling, and measurement that does not exist in the process. VB[j] is a long-range term, usually nonrandom, and continuous. This variability is the one that needs to be controlled or tamed in a process. VC[j] is a periodic term, continuous, tied to ways people work or to ways some process machines work. This is a source of variability usually poorly understood, leading to process overcorrections, and therefore losses of process efficiency. The economic impact of this source of variability, if misunderstood, is enormous. VD[j] is a random residual variability tied to the variogram precision when the variance V[j] is calculated with too few pairs N–j. VD[j] tends toward zero when the number of pairs increases. It is not a good practice to calculate any point on a variogram with less than 20 pairs. Actually, 30 pairs or more is strongly recommended.

1.01.8.2

Extrapolation of the Variogram to Time or Distance Zero

An accurate extrapolation of the variogram to time or distance zero is required for many of its applications. The most effective solution is to extrapolate either the first- or the second-order integral of the variogram4. Indeed, W[j] and W9[j] smooth out residual noise due to VD[j], cycles, and other features interfering with good extrapolation. However, as demonstrated in the sampling theory,1–4 the variogram and its integrals all intercept the variance axis at the same place V[0].

1.01.8.3

The Importance of Large-Scale Variability

The ultimate goal of sampling is to understand the variability on a large scale. It is extremely important in a very large number of areas. Let us give a few examples:

To find new natural resources To quantify natural resources To show reasonable continuity of natural resources To understand conciliation problems during mining To optimize processes at mines and plants To improve the quality of products To secure fair money return from products To diminish penalties To curtail fraud To minimize environmental liability To lift the market perception of the company’s fiscal health To improve profitability trends, etc.

The list could go on and on, but let us be more specific in some well-known areas and see what could interfere in a negative way to understand large-scale variability. 1.01.8.3.1

Variability issues during exploration During exploration it is important to measure the anisotropy of a geological unit. We need to find out whether grade trends may be different: North–South than East–West or than along a vertical axis. Variogram ranges need to be defined in different directions. Density of drilling needs to be optimized in different directions. Ore continuity and zone of influence need to be defined in different directions. Ultimately, a reliable geological model must be created. With a large V[0] damaging the database, these critically important tasks leading to a reliable feasibility study become weak.


13

1.01.8.3.2

Variability issues during mining During mining it is important to select a logical grade control drilling pattern and drilling density. The selection of a logical Kriging technique depends on quality data,8 and so does the selection of realistic and economic cutoff grades. All this may affect the selection of a long-term pit design. A reasonable acceptance level for conciliation differences between ore-grade control and the prediction from the geological model is crucial. With a large V[0] damaging the database, these critically important tasks leading to a reasonable and economic recovery of natural resources become weak, which necessarily result in financial losses.

1.01.8.3.3

Variability issues within a processing plant Within a processing plant, believable metallurgical accounting needs to monitor performance. Metallurgists need to control key process-operating parameters. Process trends need to be tamed in due time. Process cycles, always very costly, need to be identified and either eliminated or minimized. Reliable variographic control charts must be constantly updated at many places. Overcorrections of the process must be prevented. With a large V[0] damaging the database, these critically important tasks leading to a reasonable and economic recovery of natural resources become weak, which necessarily result in financial losses.

1.01.8.3.4

Variability issues during trade with customers Customers like a fair price, but hate bad surprises on product quality. Penalty application is a common way of doing business: ‘what costs me must cost you!’ The quality of a product cannot be controlled after the fact, and the key for success is to implement the many things that lead to a good product, all the way from geology, mining, and the processing plants. With a large V[0] damaging the database, these critically important tasks to keep quality of products within client specifications become weak, which necessarily result in further financial losses.

1.01.9 Conclusions There is no such a thing as continuous improvement without a clear understanding of TOS followed by a strong commitment to correct sampling practices. Many statisticians believe that interleaved sampling can solve everything if followed by a careful statistical analysis of the results. There is nothing wrong in their approach; however, there are two possible outcomes: 1. The variance analysis shows that there is no problem and all parties involved may feel comfortable enough. 2. The variance analysis shows that there is a problem. Now, without a clear understanding of TOS it is very unlikely they will find the causes and cure them effectively in the most economic way. The strategy that is suggested in TOS by dividing the complex problem into its basic components is the inescapable way to reach solutions quickly and effectively.

1.01.10 Recommendations The objective of this paper is only to help people looking at TOS for the first time to get organized. It is well known that newcomers often get discouraged as the way TOS is frequently presented can be intimidating. If the reader gets well organized, looking at the challenge with a clear Cartesian spirit, TOS is actually a lot simpler than what people may think.12,13 Follow the strategy illustrated in Figure 8 and quick progress is likely to occur. With emphasis on causes of sampling problems, the difficult climb to continuous improvement may take place. With emphasis on effects of sampling problems, as it is often the case, there is no progress possible. After a commitment has been taken for continuous improvement, it is still necessary to create a road map so that economic benefits can quickly be measured. A typical road map is illustrated in Figure 9.


If cause eliminated:

standardize If cause not eliminated:

re-analyze

Compulsory action on causes of sampling problems

Short courses, workshops, and training

Analyze existing data and find structural problems

Lost opportunities with emphasis on effects of problems Continuous improvement of mining process with emphasis on causes of problems Figure 8 Make a commitment for continuous improvement.

Director of standards of mining process: The synergy necessary for mining process efficiency

Selection of standards useful to a mining company Implementation of company's guidelines

Selection and offering of short courses, workshops, and training

Guidelines of best practices Selection of world experts

Identification of structural sampling problems and continuous improvement of mining process

Compulsory actions

Communication with QA/QC and Laboratory Managers

Figure 9 A typical road map suggested for the mining industry.

Accountability

Communication with top management at a company’s operations


15

References 1. Gy, P. Sampling of Particulate Materials, Theory and Practice. Developments in Geomathematics; Elsevier Scientific Publishing: Amsterdam, 1979 and 1983; Vol. 4. 2. Gy, P. Heterogeneite – Echantillonnage – Homogeneisation: Ensemble coherent de theories; Masson Editeur: Paris; 1988, ISBN: 2-225-81313-2. ISSN: 0765-0221. 3. Gy, P. Sampling of Heterogeneous and Dynamic Material Systems: Theories of Heterogeneity, Sampling and Homogenizing; Elsevier: Amsterdam, 1992. 4. Pitard, F. Pierre Gy’s Sampling Theory and Sampling Practice, 2nd ed.; CRC Press: Boca Raton, FL, 1993. 5. Pitard, F. Effects of Residual Variances on the Estimation of the Variance of the Fundamental Error. Chemometr. Intell. Lab. 2004, 74(1), 149–164. In the text, specifically referred to equations (2), (3), (4), (5). 6. Pitard, F. Blasthole Sampling for Grade Control: The Many Problems and Solutions. Sampling 2008, The Australasian Institute of Mining and Metallurgy, Perth, Australia, 2008; May 27–28, 2008. 7. Pitard, F. Sampling Correctness – A Comprehensive Guideline. Second World Conference on Sampling and Blending 2005, Publication Series No 4/2005; The Australian Institute of Mining and Metallurgy, 2005; ISBN: 1-920-80628-8. 8. Carrasco, P.; Carrasco, P.; Jara, E.; The Economic Impact of Correct Sampling and Analysis Practices in the Copper Mining Industry. Chemometr. Intell. Lab. 2004, 74(1), 209–231. 9. François-Bongarçon, D.; Theory of Sampling and Geostatistics: An Intimate Link. Chemometr. Intell. Lab. 2004, 74(1), 143–148. 10. Pitard, F. Practical Statistical Tools for Managers, Metallurgical Plant Design and Operating Strategies Conference,Sydney April 15–16, The Australasian Institute of Mining and Metallurgy, Sydney, April 15–16, 2002. 11. Pitard, F. Chronostatistics – A Powerful, Pragmatic, New Science for Metallurgists. Metallurgical Plant Design and Operating Strategies (MetPlant 2006), The Australasian Institute of Mining and Metallurgy, Perth, W.A. September 18–19, 2006. 12. Esbensen, K. H.; Minkkinen, P. Guest Editors. In Chemometrics and Intelligent Laboratory Systems, Vol. 74, 1, 2004, Special issue: 50 years of Pierre Gy’s Theory of Sampling, Proceedings First World Conference on Sampling and Blending (WCSB1). Tutorials on Sampling: Theory and Practice. 13. Lars, P. Pierre Gy’s Theory of Sampling (TOS) – In Practice: Laboratory and Industrial Didactics Including a First Foray into Image Analytical Sampling, Phd Thesis, ACABS Research Group, Aalborg University Esbjerg, Niels Bohrs Vej 8, Dk-67000 Esbjerg, Denmark, 2005.


Biographical Sketch

Mr. Francis F. Pitard is a consulting expert in Sampling, Statistical Process Control, and Total Quality Management for 23 years. He is President of Francis Pitard Sampling Consultants (www.fpscsampling.com) and Technical Director of Mineral Stats Inc. (www.mineralstats.com) in Broomfield, CO, USA. He provides consulting services in many countries. Mr. Pitard has 6 years of experience with the French Atomic Energy Commission and 15 years with Amax Extractive R&D. He has taught Sampling Theory, SPC, and TQM to approximately 6000 people and to more than 275 companies around the world through Continuing Education Offices of the Colorado School of Mines, for the Australian Mineral Foundation, for the Mining Department of the Universidad de Chile, and the University of Witwatersrand. He has degrees in chemistry from the Gay-Lussac Institute in Paris and from the Academy of Paris.

1.02 Quality of Analytical Measurements: Statistical Methods for Internal Validation M. C. Ortiz, L. A. Sarabia, M. S. Sańchez, and A. Herrero, University of Burgos, Burgos, Spain ª 2009 Elsevier B.V. All rights reserved.

1.02.1 1.02.2 1.02.2.1 1.02.2.2 1.02.2.3 1.02.2.4 1.02.2.5 1.02.2.6 1.02.2.7 1.02.2.8 1.02.3 1.02.3.1 1.02.3.2 1.02.3.3 1.02.3.4 1.02.3.5 1.02.3.6 1.02.3.7 1.02.3.8 1.02.4 1.02.4.1 1.02.4.2 1.02.4.3 1.02.4.4 1.02.4.5 1.02.4.6 1.02.5 1.02.5.1 1.02.5.2 1.02.5.3 1.02.5.4 1.02.5.5 1.02.5.6 References

Introduction Confidence and Tolerance Intervals Confidence Interval Confidence Interval on the Mean of a Normal Distribution Confidence Interval on the Variance of a Normal Distribution Confidence Interval on the Difference in Two Means Confidence Interval on the Ratio of Variances of Two Normal Distributions Confidence Interval on the Median Joint Confidence Intervals Tolerance Intervals Hypothesis Test Elements of a Hypothesis Test Hypothesis Test on the Mean of a Normal Distribution Hypothesis Test on the Variance of a Normal Distribution Hypothesis Test on the Difference in Two Means Test Based on Intervals Hypothesis Test on the Variances of Two Normal Distributions Hypothesis Test on the Comparison of Several Independent Variances Goodness-of-Fit Tests: Normality Tests One-Way Analysis of Variance The Fixed Effects Model Power of the ANOVA for the Fixed Effects Model Uncertainty and Testing of the Estimated Parameters for the Fixed Effects Model The Random Effects Model Power of the ANOVA for the Random Effects Model Confidence Intervals of the Estimated Parameters for the Random Effects Model Statistical Inference and Validation Trueness Precision Statistical Aspects of the Experiments to Determine Precision Consistency Analysis and Incompatibility of Data Accuracy Ruggedness

Symbols 1– 1– CC CC

F1 ;2 H0

confidence level power limit of decision capability of detection F distribution with v1 and v2 degrees of freedom null hypothesis

H1 N(,) NID(,2)

s s2

18 23 23 23 25 26 27 28 28 29 31 31 34 36 36 38 39 41 43 45 46 48 49 53 53 54 54 54 55 58 58 63 63 72

alternative hypothesis normal distribution with mean and standard deviation independent random variables equally distributed as normal with mean and variance 2 sample standard deviation sample variance

17

18 Quality of Analytical Measurements: Statistical Methods for Internal Validation

t X

"

2 R r

Student’s t distribution with v degrees of freedom sample mean significance level, probability of type I error probability of type II error bias (systematic error) random error

w 2 ;w2

mean degree(s) of freedom, d.f. standard deviation variance reproducibility (as standard deviation) repeatability (as standard deviation) (chi-squared) distribution with degrees of freedom

1.02.1 Introduction The set of operations to determine the value of an amount (measurand) suitably defined is called the measurement. The method of measurement is the sequence of operations that is used in the performance of measuremens. It is documented with enough details so that the measurement may be done without additional information. Once a method is designed or selected, it is necessary to evaluate its characteristics of operation and to identify the factors that can change these characteristics and to what extent they can change. If, in addition, the method is developed to solve a particular analytical problem, it is necessary to verify that the method is fit for purpose.1 This process of evaluation is called validation of the method. It implies the determination of several parameters that characterize the performance of the method: capability of detection, selectivity, specificity, ruggedness, and accuracy (trueness and precision). In any case, they are the measurements themselves which allow evaluation of the performance characteristics of the method and its fit for purpose. In addition, in the use stage of the method, they turn out to be the obtained measurements that will be used to make decisions on the analyzed sample, for example whether the amount of an analyte fulfills a legal specification and consequently whether the material from which the sample is taken is valid. Therefore, it is necessary to suitably model the data that a method provides. In what follows we will consider that the data provided by the analytical method are real numbers; other possibilities exist, for example, the count of bacteria or impacts in a detector take only (discrete) natural values. In addition, sometimes, the data resulting from an analysis are qualitative, for example, detection of the presence of an analyte in a sample. With regard to the analytical measurement, it is admitted that the value (measurment), x, provided by the method of analysis consists of three terms, the true value of the parameter , a systematic error (bias) , and a random error " with a zero mean, in an additive way: x ¼þþ"

ð1Þ

All the possible measurements that a method can provide when analyzing a specific sample constitute the population of the measurements. Ideally, this supposes to admit that it has limitless samples and that the method of analysis remains unalterable. In these conditions, the model of the analytical method, Equation (1), is mathematically a random variable, X, with mathematical expectation þ and variance equal to the variance of ", that is, a random variable X of mean E(X) ¼ þ and variance V(X) ¼ V("). A random variable, and thus the analytical method, is described by its probability distribution FX(x), that is, the probability that the method provides measurements less than or equal to x for any value x. Symbolically this is written as FX(x) ¼ pr{X x} for any real value x. In most of the applications, it is assumed that FX(x) is differentiable, which implies, among other things, that none of the possible results of the method has positive probability, that is, the probability of obtaining exactly a specific value is zero. In the case of a differentiable probability distribution, the derivative of FX(x) is the probability density function R (pdf) fX(x). Any function f (x) such that (1) it is positive, f (x) 0, and (2) the area under the function is 1, R f ðxÞ dx ¼ 1, is the pdf of a random variable. The probability that the random variable X takes values in the interval [a, b] is the area under the pdf over the interval [a, b], that is Z

b

prfX P½a; b g ¼ a

f ðxÞ dx

ð2Þ

Quality of Analytical Measurements: Statistical Methods for Internal Validation

19

and the mean and variance of X are written as Z E ðX Þ ¼

xf ðxÞ dx

ð3Þ

ðx – E ðX ÞÞ2 f ðxÞ dx

ð4Þ

R

Z V ðX Þ ¼ R

In general, the mean and the variance do not characterize in a unique way a random variable and therefore the method of analysis. Figure 1 shows the pdf of four random variables with the same mean 6.00 and standard deviation 0.61. These four distributions, uniform or rectangular (Figure 1(a)), triangular (Figure 1(b)), normal (Figure 1(c)), and Weibull (Figure 1(d)), are rather frequent in the scope of analytical determinations, and they appear in the EURACHEM/CITAC Guide1 (Appendix E) and also they are used in metrology.2 If the only available information regarding a quantity X is the lower limit, l, and the upper limit, u, but the quantity could be anywhere in between, with no idea of whether any part of the range is more likely than another part, then a rectangular distribution in the interval [l,u] would be assigned to X. This is so because this is the pdf that maximizes the ‘information entropy’ of Shannon, in other words the pdf that adequately characterizes the incomplete knowledge about X. Frequently, in reference material, the certified concentration is expressed in terms of a number and unqualified limits (e.g., 1000 2 mg l1). In this case, a rectangular distribution should be used (Figure 1(a)). When the available information concerning X includes the knowledge that values close to c (between l and u) are more likely than those near the bounds, the adequate distribution is a triangular one (Figure 1(b)), with the maximum of its pdf in c. If a good estimate, m, and associated standard uncertainty, s, are the only information available regarding X, then, according to the principle of maximum entropy, a normal probability distribution N(m,s) (Figure 1(c)) would be assigned to X (remember that m and s may have been obtained from repeated applications of a measurement method). Finally, the Weibull distribution (Figure 1(d)) is very versatile; it can mimic the behavior of other distributions such as the normal or exponential. It is adequate for the analysis of reliability of processes, and in chemical analysis it is useful in describing the behavior of the figures of merit of a long-term procedure, for

(a) 1.2 1.0 0.8 0.6 0.4 0.2 0.0

(b) 1.2 1.0 0.8 0.6 0.4 0.2 0.0 4

5

6

7

8

(c) 1.2 1.0 0.8 0.6 0.4 0.2 0.0

4

5

6

7

8

4

5

6

7

8

(d) 1.2 1.0 0.8 0.6 0.4 0.2 0.0 4

5

6

7

8

Figure 1 Probability density functions of four random variables with mean 6 and variance 0.375. (a) Uniform in [4.94, 7.06]. (b) Symmetric triangular in [4.5, 7.5]. (c) Normal (6, 0.61). (d) Weibull with shape 1.103 and scale 0.7 shifted to give a mean of 6.


example, the distribution of the capability of detection CCb3 or in the determination of ammonia in water by UV–vis spectroscopy during 350 different days.4 In the four cases given in Figure 1, the probability of obtaining values between 5 and 7 has been computed from Equation (2). For the uniform distribution (Figure 1(a)) this probability is 0.94, whereas for the triangular distribution (Figure 1(b)) it is 0.88, for the normal distribution (Figure 1(c)) 0.90, and for the Weibull distribution (Figure 1(d)) 0.93. Therefore, the proportion of the values that each distribution accumulates in the interval [5,7] orders the distributions as uniform, Weibull, normal, and triangular, although the triangular and normal distributions tend to give values symmetrically around the mean and the Weibull distribution does not. If another interval is considered, say [5.4,6.6], the distributions accumulate probabilities of 0.57, 0.64, 0.67, and 0.54, respectively, in which the difference between the values is more than before and, in addition, order the distributions as normal, triangular, uniform, and Weibull. If for each of those variables, value b is determined so that there is a fixed probability, p, of obtaining values below b (i.e., the value b such that p ¼ pr{X < b} for each distribution X), the results of Table 1 are obtained. For example, 5% of the times, a uniform distribution gives values less than b ¼ 5.05, less than 4.97 if it is a triangular distribution, and so on. In the table, the extreme values among the four distributions for each value of p have been marked, and great differences are observed caused as much by the form in which the values far from 6 are distributed (i.e., very different for the normal, the triangular, or the uniform distribution) as by the asymmetry of the Weibull distribution. Therefore, the mean and variance of a random variable constitute very limited information on the values provided by the random variable, unless additional information is at hand about the form of its density (pdf). For example, if one knows that the distribution is uniform or symmetrical triangular or normal, the random variable is completely characterized by its mean and variance. In practice, the pdf of a method of analysis is unknown. We only have a finite number, n, of these measurements, which are the results obtained when applying repeatedly (n times) the same method to the same sample. These n measurements constitute a statistical sample of the random variable X defined by the method of analysis. Figure 2 shows 100 results obtained when applying four methods of analysis, named A,B,C, and D, to aliquot parts of a sample to determine an amount whose true value (unknown) is ¼ 6. Clearly, the four methods behave differently. From the experimental data, the (sample) mean and variance are computed as Pn

i¼1 xi

x ¼ Pn s2 ¼

ð5Þ

n – xÞ2 n–1

i¼1 ðxi

ð6Þ

x and s2 are estimates of the mean and variance of the distribution of X. These estimates with the data in Figure 2 are shown in Table 2. Table 1 Values of b such that p ¼ pr{X < b} where X is one of the random variables in Figure 1 Random variable p

Uniform

Triangular

Normal

Weibull

0.01 0.05 0.50 0.95 0.99

4.96 5.05 6.00b 6.95a 7.04a

4.71 4.97a 6.00b 7.03 7.29

4.58a 4.99 6.00b 7.01 7.42

5.34b 5.37b 5.83a 7.22b 8.12b

a b

Minimum b among the four distributions. Maximum b among the four distributions.


(a) 40

(b) 40

30

30

20

20

10

10

0

21

0 3

4

5

6

7

8

9

10

(c) 40

(d) 40

30

30

20

20

10

10

0

3

4

5

6

7

8

9

10

3

4

5

6

7

8

9

10

0 3

4

5

6

7

8

9

10

Figure 2 Histograms of 100 measurements obtained with four different analytical methods on aliquot parts of a sample. (a) method A; (b) method B; (c) method C; (d) method D.

Table 2 Some characteristics of the distributions in Figure 2 Method

Mean, x Variance, s2 pr{5 < X < 7} pr{X < 6} pr{5 < N(x,s) < 7} pr{N(x,s) < 6}

A

B

C

D

6.66 0.25 0.76 0.09 0.75 0.09

6.66 1.26 0.54 0.24 0.55 0.27

6.16 1.26 0.63 0.46 0.62 0.44

6.16 0.25 0.94 0.35 0.94 0.37

According to the model of Equation (1), E ðX Þ ¼ þ . x , that is, the mean estimates the true value plus the bias . The bias estimated for methods A and B is 0.66 whereas for methods C and D it is 0.16. The bias of a method is one of its performance characteristics and must be evaluated during the validation of the method. In fact, technical guides, for example, that by the International Organization for Standardization (ISO), define that a method better fulfills the trueness if it has less bias. To estimate the bias, it is necessary to have samples with known concentration (e.g., certified material, spiked samples). The value of the variance is independent of the true content, , of the sample. For this reason, to estimate the variance, it is only necessary to have replicated measurements on aliquot parts of the same sample. Table 2 shows that methods B and C have the same variance, 1.26, which is 5 times greater than that of methods A and D, 0.25. The dispersion of the data that provides a method is the precision of the method and constitutes another performance characteristic to be determined in the validation of the method. In agreement with model (1), a measure of the dispersion is the variance V(X), which is estimated by means of s2. In some occasions, for evaluating trueness and precision, it is more descriptive to use statistics other than mean and variance. For example, when the distribution is rather asymmetric, as in Figure 1(d), it is more reasonable to use the median than the mean. The median is the value in which the distribution accumulates 50% of the probability, 5.83 for the pdf in Figure 1(d) and 6.00 for the other three distributions, which are symmetric around their mean. In practice, it is frequent to see the presence of anomalous (outliers) data that influence the mean and above all the variance, which is improperly increased; in these cases, it is advisable to use robust estimates of the centrality and dispersion values.5–7 In Chapter 1.07 of the present book, there is a detailed description of these robust procedures.


Figure 2 and Table 2 show that the two characteristics, trueness and precision, are independent in the sense that a method with better trueness (small bias), cases C and D, can be more, case D, or less, case C, precise. Analogously, A and B have an appreciable bias but A is more precise than B. A method is said to be accurate when it is precise and fulfills trueness. The histograms are an estimate of the pdf and allow evaluation of the performance of each method in a more detailed way than when considering only trueness and precision. For example, the probability of obtaining values in any interval can be estimated with the histogram. The third row in Table 2 shows the estimated frequencies for the interval [5,7]. Method D (better trueness, better precision) provides 94% of the values in the interval, whereas method B (worse trueness and precision) provides only 54% of the values in the interval. Trueness and precision interact, according to data in Table 2: The effect of increasing the precision, from B to A, when the bias is ‘small’ is an increase of 22% of results of a measurement to be in the interval [5,7], whereas when the bias is high, the increase is about 31%. This interaction should be taken into account when optimizing a process and also in the ruggedness analysis, which is another performance characteristic to be validated according to most of the guides. As can be seen in the last row of Table 2, if the method that provides more results below 6 is needed, C would be the method selected. The previous analysis shows the usefulness of knowing the pdf of the results of a method of analysis. As in practice we have only a limited number of results, two basic strategies are possible to estimate it: (1) to demonstrate that the experimental data are compatible with a known distribution (e.g., normal) and use the corresponding pdf; (2) to estimate the pdf by a data-driven technique based on a computer-intensive method such as the kernel method8 described in Section 1.03.2.4 of the present book, or by using other methods such as adaptive or penalized likelihood.9,10 The data of Figure 2 can be adequately modeled by a normal distribution, according to the results of four tests (chi-square, Shapiro–Wilks, skewness, kurtosis) (see Section 1.02.3.8), and the procedure will be detailed in the following. The last two rows in Table 2 show the probabilities of obtaining values in the interval [5,7] or less than 6 with the fitted normal distribution. When comparing these values with those computed with the empirical histograms (compare rows 3 and 5, and rows 4 and 6), evidently, there are no appreciable differences. In the validation of an analytical method and during its later use, methodological strategies of statistical type are needed to make decisions from the available experimental data. The knowledge of these strategies supposes a way to think and to act that, subordinated to the chemical knowledge, objectives both the analytical results and their comparison with those of other researchers and/or other analytical methods. Ultimately, a good method of analysis is a serious attempt to come close to the true value of the measurement. The difference between the true value, always unknown, and the one experimentally obtained can be estimated. For this reason, the result of a measurement has to be accompanied by an evaluation of (un)certainty or its degree of reliability. This is done by means of a confidence interval. When it is required to establish the quality of an analytical method, its capability of detection, precision, etc. have to be compared with those corresponding to other methods. This is formalized with a hypothesis test. Confidence intervals and test of hypothesis are the basic tools in the validation of analytical methods. In this introduction, the word sample has been used with two different meanings. Usually, there is no confusion because the context allows one to distinguish whether it is a sample in the statistical or chemical sense. In chemistry, according to the International Union for Pure and Applied Chemistry (IUPAC) (Section 18.3.2 of Incze´dy et al.11), ‘sample’ should be used only when it is a part of a selected material of a great amount of material. This meaning coincides with that of a statistical sample and implies the existence of sampling error, that is, error caused by the fact that the sample can be more or less representative of the amount in the material. For example, suppose that we want to measure the amount of plaguicide that remains in the ground of an arable land after a certain time. For this, we take several samples ‘representative’ of the ground of the parcel (statistical sampling) and this introduces an uncertainty in the results characterized by a variance (theoretical) 2s . Afterward, the quantity of plaguicide in each chemical sample is determined by an analytical method, which has its own uncertainty, characterized by 2m , in such a way that the uncertainty in the quantity of plaguicide in the parcel is 2s þ 2m provided that the method gives results independent of the location of the sample. Sometimes, when evaluating whether a method is adequate for a task, the sampling error can be an important part of the uncertainty in the result and, of course, should be taken into account to plan the experimentation. The modeling of sampling errors is studied in Chapter 1.01.


23

When the sampling error is negligible, for example when a portion is taken from a homogeneous solution, the IUPAC recommends to use words such as test portion, aliquot, or specimen.

1.02.2 Confidence and Tolerance Intervals There are important questions when evaluating a method, for example, what is the maximum value that it provides? In fact, given the random character of the results, the question cannot be answered with just a number. In order to include the degree of certainty in the answer, the question should be reformulated: What is the maximum value, U, that will be obtained 95% of the times that the method is used? The answer is the tolerance interval and to build it, the probability distribution must be known. Let us suppose that we know that it is a N(,), then a possible answer is U ¼ þ z0.05 because then the probability that the analytical method gives values greater than U is pr{method > U} ¼ pr{N(,) > þ z0.05}, which, according to the result in Appendix, is equal to pr{N(0,1) > z0.05} ¼ 0.05. For any 100 (1 )%, the value U ¼ þ z

ð7Þ

is the maximum value provided by the method, with a probability that the aforementioned assertion is false. Analogously, the results that will be obtained the 100 (1 )% of times will be above the following value, L L ¼ – z

ð8Þ

Then, the statement ‘the method provides values greater than L’ will be false 100 % of the times. Finally, the interval [L,U] contains 100 (1 )% of the values proportioned by the method ½L; U ¼ – z=2 ; þ z=2

ð9Þ

or, in other words, the statement that the method gives values between L and U is false 100 % of the times. 1.02.2.1

Confidence Interval

We have already noted that estimation of solely the mean, x, and the variance, s2, from n independent results provides very limited information on the performance of the method. The objective now is to make affirmations of the type ‘in the sample, the amount of the analyte, (unknown), is between L and U ( P [L,U]) with a certain probability that the statement is true’. In general, to obtain a confidence interval for a random variable, X, from a sample x1,x2,. . .,xn consists of obtaining two functions l(x1,x2,. . .,xn) and u(x1,x2,. . .,xn) such that prfX P½l; ug ¼ prfl X ug ¼ 1 –

ð10Þ

1 is the confidence level and is the significance level, that is to say, the statement that the value of X is between l and u will be false 100 % of the times. In the following, this idea will be specified in some interesting cases. 1.02.2.2

Confidence Interval on the Mean of a Normal Distribution

Case 1: Suppose that we have a random variable that follows a normal distribution with known variance. This will be the case, for example, of using an already validated method of analysis. In this case, we know that " in Equation (1) is normally distributed and its variance. Taking into account the pffiffiffiproperties of the normal distribution (see Appendix), the sample mean, X, is a random variable N ð; = nÞ; thus, by Equation (10), the following holds: pr – z=2 pffiffiffi X þ z=2 pffiffiffi ¼ 1 – n n

ð11Þ


that is, 100 (1 )% of the values of the sample mean are in the interval in Equation (11). A simple algebraic manipulation (subtract and X, multiply by –1) gives p ffiffi ffi p ffiffi ffi pr X – z=2 X þ z=2 ¼ 1– n n

ð12Þ

Therefore, according to Equation (10), the confidence interval on the mean that is obtained from Equation (12) is X – z=2 pffiffiffi ; X þ z=2 pffiffiffi n n

ð13Þ

Analogously, the confidence intervals at level 100 (1 )% for the maximum and minimum values of the mean are computed from pr X þ z pffiffiffi ¼ 1 – n pr X – z pffiffiffi ¼ 1 – n

ð14Þ ð15Þ

When measuring n aliquot parts of a test sample, we obtain n values x1,x2,. . .,xn; the sample mean x is the value that the random variable X takes and an estimate of the quantity . Example 1: Suppose that the analytical method follows a N(,4) and we have a sample of size 10 made up of values 98.87, 92.54, 99.42, 105.66, 98.70, 97.23, 98.44, 103.73, 94.45, and 101.08. pffiffiffiffiffi

The mean is 99.01 and, pffiffiffiffiffi

thus, the interval at 95% confidence level for this sample is [99.01 1.96 4= 10 , 99.01 þ 1.96 4= 10 ] equal to [96.54,101.48]. When considering different samples of size 10, different intervals will be obtained at the same 95% confidence level. The endpoints of these intervals are nonrandom values, and the unknown mean value, which is also a specific value, will or will not belong to the interval. Therefore, the affirmation ‘the interval contains the mean’ is a deterministic assertion that is true or false for each of the intervals. What one knows is that it is true for 100 (1 )% of those intervals. In our case, as 95% of the constructed intervals will contain the true value, we say, with a confidence level of 95%, that the interval [96.54,101.48] contains . This is the interpretation with the frequentist approach adopted in this chapter, that is to say that the information on random variables is obtained by means of samples of them and that the parameters to be estimated are not known but are fixed amounts (e.g., the amount of analyte in a sample, , is estimated by the measurement results obtained by analyzing it n times). With a Bayesian approach to the problem, a probability distribution is attributed to the amount of analyte and once fixed an interval of interest [a,b], the ‘a priori’ distribution of , the experimental results, and the Bayes’ theorem are used to calculate the probability ‘a posteriori’ that belongs to the interval [a,b]. It is shown that, although in most practical cases the uncertainty intervals obtained from repeated measurements using either theory may be similar, their interpretation is completely different. In Chapter 1.08 of this book, Bayesian statistical techniques are described. The works by Lira and Wo¨ger 12 and Zech13 are devoted to compare both approaches from the point of view of the experimental data and their uncertainty. Case 2: Suppose a normally distributed random variable with unknown variance that must be estimated together with the mean from n experimental data. The confidence interval is computed as in Case 1, but now the random variable X follows a Student’s t distribution with n 1 degrees of freedom (d.f.) (see Appendix); thus, the interval at the 100 (1 )% confidence level is given from s s pr X – t =2; pffiffiffi X þ t =2; pffiffiffi ¼ 1 – n n

ð16Þ


25

where t/2, is the upper percentage point of the Student t distribution with ¼ n 1 d.f. and s is the standard deviation computed with the sample. Analogously, the one-sided intervals at the 100 (1 )% confidence level come from s pr X þ t ; pffiffiffi ¼ 1 – n s pr X – t ; pffiffiffi ¼ 1 – n

ð17Þ ð18Þ

Example 2: Suppose that the distribution of the analytical method is a normal but its standard deviation is not known. With the data of Example 1, the sample standard deviation, s, is computed as 3.91. As t0.025,9 ¼ 2.262 (see Appendix), the confidence interval at 95% level is [99.01 2.26 1.24, 99.01 þ 2.26 1.24] ¼ [96.21,101.81]. The 95% confidence interval on the minimum of the mean is made, according to Equation (18), by all the values greater than 96.74 ¼ 99.01 1.83 1.24. The corresponding interval on the maximum will be made by the values less than 101.28. The length of the confidence intervals from Equations (12)–(15) tends toward zero when the sample size tends to infinity. This permits the computation

2 d. pffiffiffi of the sample size needed to obtain an interval of given length, It will suffice to consider d =2 ¼ z=2 = n, and take as n the nearest integer greater than 2z=2 =d . For example, if we want a 95% confidence interval with length, d, less than 2, in the hypothesis of Example 1, we will need a sample size greater than or equal to 62. The same argument can

2 be applied when the standard deviation is unknown. However, in this case, to compute n by 2t=2; s=d it is necessary to have an initial estimation of s, which, in general, is obtained in a pilot test with size n9, in such a way that in the previous expression the d.f., , are n9 1. An alternative is to enunciate the length of the interval in standard deviation units (remember that the standard deviation is unknown). For instance, in Example 2, if we want d ¼ 0.5s, we will need a sample size greater than (4z/2)2 ¼ 61.5; note the substitution of t/2, by z/2, which is mandatory because we do not have the sample size needed to compute t/2,, which is precisely what we want to estimate.

1.02.2.3

Confidence Interval on the Variance of a Normal Distribution

In this case, the data come from a N(,) distribution with and unknown, and we have a sample with values x1,x2,. . .,xn. The distribution of the random variable ‘sample variance’ S2 is related to the chi-square distribution, w2 (see Appendix). As a consequence, the confidence interval at 100 (1 )% level for the variance 2 is obtained from pr

( ) ðn – 1ÞS 2 ðn – 1ÞS 2 2 ¼ 1– w2=2; w21 – =2;

ð19Þ

where w2=2; is the critical value of a chi-square distribution with ¼ n 1 d.f. at level /2. As in the previous case for the sample mean, we should distinguish between the random variable sample variance S2 and one of its concrete values, s2 computed with Equation (6), that takes this variable when we have the sample x1,x2,. . .,xn. The intervals on the maximum and minimum of the variance at 100 (1 )% confidence level are obtained from Equations (20) and (21), respectively. (

ðn – 1ÞS 2 pr 2 w1 – ;

)

2

(

ðn – 1ÞS 2 pr 2 w2;

¼ 1–

ð20Þ

¼ 1–

ð21Þ

)


Example 3: Knowing that the n ¼ 10 data of Example 2 come from a normal distribution with both mean and variance unknown, the 95% confidence interval on 2 is found from Equation (19) as [7.22,50.83] because s2 ¼ 15.25, w20:025;9 ¼ 19:02, and w20:975;9 ¼ 2:70. If the analyst is interested in obtaining a confidence interval for the maximum variance, the 95% upper confidence interval is found from Equation (20) as [0,41.22] because w20:95;9 ¼ 3:33, that is, the upper bound for the variance is 41.22 with a probability of error equal to 0.05. To obtain confidence intervals for the standard deviation, it suffices to take the square root of the aforementioned intervals because this operation is a monotonically increasing transformation of the values; therefore, the intervals at 95% confidence level for the standard deviation are [2.69,7.13] and [0,6.42]. 2 2 The size, n, of thesample needed pffiffiffiffiffiffiffiffiffiffi ffi so that 2 s / is between 1 k and 1 þ k is given by the nearest integer greater than 1 þ 1=2 z=2 1 þ k þ 1 =k . For example, for k ¼ 0.5, such that the length of the confidence interval verifies 0.5 < s2/2 < 1.5, we need n ¼ 40 data (at least). Just for comparative purposes, we will admit that with the sample of size 40 we obtain the same variance s2 ¼ 15.25, w20:025;39 ¼ 58:12, and w20:975;39 ¼ 23:65; hence, the interval at 95% confidence level is [10.23,25.15], which verifies the required specifications.

1.02.2.4

Confidence Interval on the Difference in Two Means

Case 1. Known variances: Consider two independent random variables distributed as N1(1,1) and N2(2,2) with unknown means and known variances 21 and 22. We wish to find a 100 (1 )% confidence interval on the difference in means 1 2. Let x11,x12,. . ., x1n1 be a random sample of n1 observations from N1 and x21,x22,. . ., x2n2 be a random sample of n2 observations from N2. The 100 (1 )% confidence interval on 1 2 is obtained from the equation 8 sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi9 < 2 2 21 22 = 1 2 ¼ 1– pr X 1 – X2 – z=2 þ 1 – 2 X1 – X 2 þ z=2 þ : n1 n2 n1 n2 ;

ð22Þ

where X1 and X 2 are the random variables of the sample mean, which take the values x1 and x2 . The expressions analogous to Equations (14) and (15) for the one-sided intervals are obvious. Case 2. Unknown variances: The approach to this question is similar to the previous case, but here even the variances 21 and 22 are unknown. However, it can be reasonable to assume that both variances are equal, 21 ¼ 22 ¼ 2, and that the differences observed in their estimates with the data of both samples are not significant. Later, in the section dedicated to hypothesis tests, the methodology to decide about this question will be explained. An estimate of the common variance 2 is given by the pooled sample variance, which is an arithmetic average of both variances weighted by the corresponding d.f., sp2 ¼

ðn1 – 1Þs12 þ ðn2 – 1Þs22 n 1 þ n2 – 2

ð23Þ

The 100 (1 )% confidence interval is obtained from the following equation: rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 1 1 1 ¼ 1– pr X1 – X2 – t =2; s p þ 1 – 2 X1 – X 2 þ t =2; s p þ n1 n2 n1 n2

ð24Þ

where ¼ n1 þ n2 2 are the d.f. of the Student’s t distribution. The one-sided intervals at 100 (1 )% confidence level have an obvious expression deduced from Equation (24) by substituting t/2, for t, . If a fixed length is desired for the confidence interval, the computation explained in Section 1.02.2.2 can be immediately adapted to obtain the needed sample size. Example 4: We want to verify that a substance is sufficiently stable to remain unchanged in composition when it is stored for a month. Two series of measurements (n1 ¼ n2 ¼ 8) were made before and after a storage period. The results were x1 ¼ 90:8, s12 ¼ 3:89 and x2 ¼ 92:7, s22 ¼ 4:02, respectively. Compute a 95% confidence interval on the difference of means.


27

The two-sided interval when assuming equal variances (sp2 ¼ 3:96) (Equation (24)) is (90.8 92.7) 2.1448 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1.99 1=8 þ 1=8 ¼ ½ – 4:034; 0:234. As zero belongs to the interval, we can conclude that the substance is stable at 95% confidence level. When the assumption 21 ¼ 22 is not reasonable, we can still obtain an interval on the difference 1 2 by pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi using the fact that the statistic ½X1 – X 2 – ð1 – 2 Þ s12 =n1 þs22 =n2 is distributed approximately as a t with d.f. given by ¼

2

2 s1 =n1 þ s22 =n2 2 2 ðs12 =n1 Þ ðs22 =n2 Þ n1 – 1 þ n2 – 1

ð25Þ

The 100 (1 )% confidence interval is obtained from the following equation: 8 sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi9 < 2 2 s s s2 s2 = 1 2 pr X 1 – X2 – t =2; þ 1 – 2 X1 – X 2 þ t =2; 1 þ 2 ¼ 1 – : n1 n2 n1 n2 ;

ð26Þ

Example 5: We want to compute a confidence interval on the difference of two means with unknown and nonequal variances, with the results that come from an experiment carried out with four aliquot samples by two different analysts. The first analyst obtains x1 ¼ 3:285, whereas the second one obtains x2 ¼ 3:257. The variances were s12 ¼ 3:3310 – 5 and s22 ¼ 9:1710 – 5 respectively. Assuming that 21 6¼ 22 , Equation (25) gives ¼ 4.9, so the d.f. to apply Equation (26) are 5 and t0.025,5 ¼ 2.571. Thus, the 95% confidence interval is pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð3:285 – 3:257Þ 2:571 ð3:3310 – 5 =4Þ þ ð9:1710 – 5 =4Þ ¼ ½0:014; 0:042. That is, at 95% of confidence, the two analysts provide unequal measurements because zero is not in the interval. The confidence intervals for the maximum and the minimum are obtained by considering the last or the first term respectively in Equation (26) and replacing t/2, by t, . Case 3. Confidence interval for paired samples: Sometimes we are interested in evaluating an effect (e.g., the reduction of a polluting agent in an industrial spill by means of a catalyst) but it is impossible to have two homogeneous populations of samples without and with treatment to obtain the two means of the recoveries, because the amount of polluting agent is not controllable. In these cases, the solution is to determine the polluting agent before and after applying the procedure to the same spill. The difference between both determinations is a measure of the effect of the catalyst. The (statistical) samples obtained in this way are known as paired samples. Formally, the problem consists of having two paired samples of size n, x11,x12,. . ., x1n and x21,x22,. . ., x2n , and compute the differences between any pair of data, di ¼ x1i– x2i, i ¼ 1,2,. . .,n. If these differences follow a normal distribution, the 100 (1 )% confidence interval is obtained from sd sd pr d – t =2; pffiffiffi d þ t =2; pffiffiffi ¼ 1 – n n

ð27Þ

where d and sd are the mean and standard deviation of the differences di and ¼ n 1 are the d.f.

1.02.2.5

Confidence Interval on the Ratio of Variances of Two Normal Distributions

This section approaches the question of giving a confidence interval on the ratio 21/22 of the variances of two distributions N1(1,1) and N2(2,2) with unknown means and variances. Let x11,x12,. . ., x1n1 be a random sample of n1 observations from N1 and x21,x22,. . ., x2n2 be a random sample of n2 observations from N2. The 100 (1 )% confidence interval on the ratio of variances is given by the following equation: S12 21 S12 pr F1 – =2;1 ;2 2 2 F=2;1 ;2 2 ¼ 1 – S2 2 S2

ð28Þ


where F1 – =2;1 ;2 and F=2;1 ;2 are the critical values of an F distribution with 1 ¼ n2 1 d.f. in the numerator and 2 ¼ n1 1 d.f. in the denominator. Appendix contains a description of the F distribution. We can also compute one-sided confidence intervals. The 100 (1 )% upper or lower confidence limit on 21/22 is obtained from Equations (29) and (30), respectively: 2 S2 pr 12 F;1 ;2 12 ¼ 1 – 2 S2 S12 21 pr F1 – ;1 ;2 2 2 ¼ 1 – S2 2

ð29Þ ð30Þ

Example 6: In this example, we compute a two-sided 95% confidence interval for the ratio of the variances in Example 4. The resulting interval is [0.20 (3.89/4.02), 4.99 (3.89/4.02)] ¼ [0.19,4.83]. As 1 belongs to this interval, we can admit that both variances are equal. 1.02.2.6

Confidence Interval on the Median

This case is different from the previous cases, because the confidence interval is a ‘distribution-free’ interval, that is, there is no distribution assumed for the data. As it is known, a percentile (pct) is the value xpct such that 100 pct% of the values are less than or equal to xpct. It is possible to compute confidence intervals on any pct, but for values of pct near one or zero we need sample sizes, n, that are very large because the values n pct and n (1 pct) must be greater than 5. For the median (pct ¼ 0.5), it suffices to consider samples of size 10 or more. The fundamentals of these confidence intervals are based on the binomial distribution whose details are outside the scope of this chapter and can be found in Sprent.14 We use the data of Example 1 to show step by step how a 100 (1 )% confidence interval on the median is computed. The procedure consists of three steps: 1. To sort the data in ascending order. In our case, 92.54, 94.45, 97.23, 98.44, 98.70, 98.87, 99.42, 101.08, 103.73, and 105.66. The rank of each datum is the position that it occupies in the sorted list, for example, the rank of 98.44 is four. 2. To calculate the rank,prlffiffi,ffi of the the interval. It is the nearest integer

value that will be the lower endpoint of pffiffiffiffiffi less than 1=2 n – z=2 n þ 1 . In our case, this value is 0.5(10 1.96 10 þ 1Þ ¼ 2:40, thus rl ¼ 2. the value that will be the upper endpoint of the interval. It is the nearest integer 3. To calculate the rank, ru, of pffiffiffiffiffi pffiffiffi

greater than 1=2 n þ z=2 n – 1 . In our case, this value is 0.5(10 þ 1.96 10 – 1Þ ¼ 7:60, then ru ¼ 8. Hence, the 95% confidence interval on the median is made by the values that are between positions 2 and 8, that is, [94.45,101.08]. 1.02.2.7

Joint Confidence Intervals

Sometimes it is necessary to compute confidence intervals for several parameters but maintaining a 100 (1 )% confidence that all of them contain the true value of the corresponding parameter. For example, if two parameters are statistically independent, we can assure a 95% joint level of confidence by taking separately the corresponding 100 (1 )1/2% confidence intervals because (1 )1/2 (1 )1/2 ¼ (1 ). In general, if there are k parameters, we will compute the 100 (1 )1/k% confidence interval for any of them. However, if the sample statistics used are not independent, the above computation is not valid. The Bonferroni inequality states that the probability P that all the affirmations are true at 100 (1 – )% confidence k , where 1 i is the confidence level considered for the ith level is greater than or equal to 1 – i i¼1 interval (usually i ¼ /k). For example, if a joint 90% confidence interval is needed for the mean of two distributions, according to Bonferroni inequality i ¼ /2 ¼ 0.10/2 ¼ 0.05; thus, each should be the corresponding 95% confidence interval.


1.02.2.8

29

Tolerance Intervals

In the introduction to Section 1.02.2, the tolerance intervals of a normal distribution have been calculated knowing its mean and variance. Remember that the confidence interval [l,u] contains 100 (1 )% of the values of the distribution or, equivalently, pr{X ˇ [l,u]} ¼ . Actually, the values of the parameters that define the probability distribution are unknown; this uncertainty should be transferred into the endpoints of the interval. There are several types of tolerance regions, but in this chapter we will restrict ourselves to two common cases. Case 1. -content tolerance interval: Given a random variable X, an interval [l,u] is a -content tolerance interval at the confidence level if the following is fulfilled: prfprfX P½l; ug g

ð31Þ

To put in words, [l,u] contains at least 100 % of the X values with confidence level. For the case of an analytical method, this is to say that we have to determine, based on a sample of size n, the interval that will contain 95% ( ¼ 0.95) of the results and this assertion must be true 90% of the times ( ¼ 0.90). Evidently, the -content tolerance intervals can be one-sided, which means that the procedure will provide 95% of its results above l (respectively, below u) 90% of the times. We leave to the reader the corresponding formal definitions. One-sided and two-sided -content tolerance intervals can be computed either by controlling the center or by controlling the tails, and for both continuous and discrete random variables (a review can be seen in Patel15). Here we will limit to the case of a normally distributed X with unknown mean and variance, of which we have a sample of size n, with which the mean x and standard deviation s are estimated. We want to obtain a two-sided -content tolerance interval controlling the center, that is, an interval such that prfprfX P½x – ks; x þ ks g g

ð32Þ

To determine k, several approximations have been reported; consult Patel15 for a discussion on them. The approach by Wald and Wolfowitz16 is based on determining k1 such that 1 1 pr N ð0; 1Þ pffiffiffi þ k1 – pr N ð0; 1Þ pffiffiffi – k1 ¼ n n

ð33Þ

sffiffiffiffiffiffiffiffiffiffiffiffi n–1 k ¼ k1 w2;n – 1

ð34Þ

Therefore

w2,n1 is the point exceeded with probability when using the chi-square distribution with n 1 d.f. Example 7: With the data in Example 1, x ¼ 99:01, s ¼ 3.91, k1 ¼ 2.054, and w20:95;9 ¼ 3:33; thus, according to Equation (34), k ¼ 3.377 and, as a consequence, the interval [99.01 3.38 3.91, 99.01 þ 3.38 3.91] ¼ [85.81,112.23] contains 95% of the results of the method 95% of the times that the procedure is repeated with a sample of size 10. Remember the following points: (1) This tolerance interval is for individual results and pffiffinot ffi for mean values as the confidence intervals. (2) The standard deviation of the mean is estimated as s= n ¼ 1:24, whereas the standard deviation of the individual results of the method is estimated to be 3.91. (3) The length of the -content tolerance interval does not tend toward zero when increasing the sample size, as does with the confidence intervals. Now, the value of k in Equation (34) tends to z(1–)/2, which is the length of the theoretical interval that, in our example, would be [91.35,106.67] with z0.025 ¼ 1.96. Case 2. The interval [l,u] is called a -expectation tolerance interval if E fprfX P½l; ugg ¼

ð35Þ

Unlike the -content tolerance interval, condition (35) only demands that, on average, the probability that the random variable takes values between l and u is .


As in the previous case, we limit ourselves to obtain intervals of the form [ x – ks; x þ ks]. When the distribution of the random variable is normal and we have a sample of size n, the solution was obtained for the first time by Wilks17 and is k ¼ tð1 – Þ=2;

rffiffiffiffiffiffiffiffiffiffiffi nþ1 n

ð36Þ

where tð1 – Þ=2; is the upper (1-)/2 percentage point of the t distribution with ¼ n 1 d.f. For the same data as before, the expectation tolerance interval at 95% is [99.01 2.37 3.91, 99.01 þ 2.37 3.91] ¼ [89.73,108.29] because t0:025;9 ¼ 2:262. This interval is shorter than the -content tolerance interval because it only assures the mean of the probabilities that the interval contains the value of the random variable X. In fact, the interval [89.73,108.29] contains 95% of the values of X only 64% of the times, conclusion drawn by applying Equation (32) with k ¼ 2.37. Also, note that when the sample size tends to infinity, the value of k in Equation (36) tends toward z(1–)/2. Case 3. It is also possible to obtain tolerance intervals independent of the distribution (provided it is continuous) of variable X. These intervals are based on the rank of the observations, but they demand very large sample sizes, which makes them quite useless in practice. For example, to guarantee that the -content tolerance interval [l,u] is made as l ¼ x(1) and u ¼ x(n) (i.e., the endpoints are the smallest and the greatest values in the sample), it is necessary that n fulfills approximately the equation logðnÞ þ ðn – 1Þlogð Þ ¼ logð1 – Þ – logð1 – Þ.18 If we need, as in Example 7, ¼ ¼ 0.95, the value of n has to be 89. Nevertheless, Willinks19 has used the Monte Carlo method to compute shorter ‘distribution-free’ tolerance intervals (-content and -expectation); this is of utility in the calculation of the uncertainty proposed in Draft Supplement2 but it still requires sample sizes that are rather large in the scope of the chemical analysis. A complete theoretical development on the tolerance intervals (including their estimation by means of Bayesian methods) is in the book by Guttman.20 The tolerance intervals are of interest to show that a method is fit for purpose because when establishing that the interval [ x – ks; x þ ks] will contain in average 100 % of the values provided by the method (or 100 % of the values with confidence level), we are including precision and trueness. It suffices that the tolerance interval [ x – ks; x þ ks] is included in the specifications that the method should fulfill, so that it is valid for that purpose. Note that a method with high precision (small value of s) but with a significant bias can get to fulfill the specifications in the sense that a high proportion of its values are within the specifications. In addition in the estimation of s, the repeatability can be introduced as the intermediate precision or the reproducibility to consider the scope of application of the method. The use of the tolerance interval solves the problem of introducing the bias as a component of the uncertainty, in clear contradiction with the model of Equation (1). With the aim of developing analytical fit for purpose methods, the Societe´ Française des Sciences et Techniques Pharmaceutiques (SFSTP) has proposed21–24 the use of the -expectation tolerance intervals in the validation of quantitative methods. In four case studies, it has shown the validity of -expectation tolerance intervals as an adequate way to conciliate both the objectives of the analytical method in routine analysis and those of the validation step, and it proposes them as a criterion to select the calibration curve.25 Also, it has analyzed their adequacy to the guides26 that establish the performance criteria that should be validated and their usefulness in the problem of the transference of an analytical method.27 Gonza´lez and Herrador28 have proposed their computation from the estimation of uncertainty of the analytical assay. In all these cases, the expectation tolerance intervals based on the normality of data are used, that is, using Equation (36). To avoid dependence on the underlying distribution and the use of the classic distribution-free methods, Rebafka et al.29 have proposed the use of a bootstrap technique to calculate -expectation tolerance intervals, whereas Fernholz and Gillespie30 have studied the estimation by means of bootstrap of the -content tolerance intervals. Nevertheless, the confidence intervals will have to be used when evaluating the trueness and precision of a method with no need to have as objective the fulfillment of external requirements but their possible comparison with other methods and the quantification of the uncertainty and the bias of the results that it provides. There are other aspects of the determination of the uncertainty that are of practical interest, for example the problem that raises the fact that any uncertainty interval, particularly an expanded uncertainty interval, should


31

be restricted to the range of feasible values of the measurand. Cowen and Ellison31 analyzed how to modify the interval when the data are close to a natural limit in a feasible range such as 0 or 100% mass or mole fraction.

1.02.3 Hypothesis Test This section is dedicated to the introduction of a statistical methodology to decide whether an affirmation is false, for example, the affirmation ‘this method of analysis applied to this sample of reference provides the certified value’. If on the basis of the experimental results it is decided that it is false, we will conclude that the method has bias. The affirmation is habitually called hypothesis and the procedure of decision making is called hypothesis testing. A statistical hypothesis is an asseveration on the probability distribution that follows a random variable. Sometimes one has to decide on a parameter, for example, whether the mean of a normal distribution is a specific value. In other occasions it may be required to decide on other characteristics of the distribution, for example, whether the experimental data are compatible with the hypothesis that they come from a normal or uniform distribution. 1.02.3.1

Elements of a Hypothesis Test

As the results that provide the analytical methods are modeled by means of a probability distribution, it is evident that both, the validation of a method and its routine use, imply to make decisions that are formulated in a natural way as problems of hypothesis testing. In order to describe the elements of a hypothesis test, we will use a concrete case. Example 8: For an experimental procedure, we need solutions with pH values less than 2. The preparation of these solutions provides pH values that follow a normal distribution with ¼ 0.55. pH values obtained from 10 measurements were 2.09, 1.53, 1.70, 1.65, 2.00, 1.68, 1.52, 1.71, 1.62, and 1.58. Is the pH of the resulting solution adequate to proceed with the experiment? We may express this formally as H0 : ¼ 2:00 ðinadequate solutionÞ H1 : < 2:00 ðvalid solutionÞ

ð37Þ

The statement ‘ ¼ 2.00’ in Equation (37) is called the null hypothesis, denoted as H0, and the statement ‘ < 2.00’ is called the alternative hypothesis, H1. As the alternative hypothesis specifies values of that are less than 2.00 it is called one-sided alternative. In some situations, we may wish to formulate a two-sided alternative hypothesis to specify values of that could be either greater or less than 2.00 as in H0 : ¼ 2:00 H1 : 6¼ 2:00

ð38Þ

The hypotheses are not affirmations about the sample but about the distribution from which those values come, that is to say, is the value, unknown, of the pH of the solution that will be the same as the value provided by the procedure if the bias is zero (see the model of Equation (1)). In general, to test a hypothesis, the analyst must consider the experimental objective and decide upon a null hypothesis for the test, as in Equation (37). Hypothesistesting procedures rely on using the information in a random sample; if this information is inconsistent with the null hypothesis, we would conclude that the hypothesis is false. If sufficient evidence does not exist to prove falseness, the test defaults to the conclusion of the test is not to reject the null hypothesis but this does not actually prove that it is correct. It is therefore critical to choose carefully the null hypothesis in each problem. In practice, to test a hypothesis, we must take a random sample, compute an appropriate test statistic from the sample data, and then use the information contained in this statistic to make a decision. However, as the

32 Quality of Analytical Measurements: Statistical Methods for Internal Validation Table 3 Decisions in hypothesis testing The unknown truth Researcher’s decision

H0 is true

H0 is false

Accept H0 Reject H0

No error Type I error

Type II error No error

decision is based on a random sample, it is subject to error. Two kinds of potential errors may be made when testing hypothesis. If the null hypothesis is rejected when it is true, then a type I error has been made. A type II error occurs when the researcher fails to reject the null hypothesis when it is false. The situation is described in Table 3. In Example 8, if the experimental data lead to rejection of the null hypothesis H0 being true, our (wrong) conclusion is that the pH of the solution is less than 2. A type I error has been made and the analyst will use the solution in the procedure when in fact it is not chemically valid. If, on the contrary, the experimental data lead to acceptance of the null hypothesis when it is false, the analyst will not use the solution when in fact the pH is less than 2 and a type II error has been made. Note that both types of error have to be considered because their consequences are very different. In the case of type I error, a nonsuitable solution is accepted, the procedure will be inadequate, and the analytical result will be wrong with the later damages that it may cause (e.g., the loss of a client, or a mistaken environmental diagnosis). On the contrary, the type II error implies that a valid solution is not used with the corresponding extra cost of the analysis. It is clear that the analyst has to specify the risk that assumes to make these errors, and this is done in terms of the probability that they occur. The probabilities of occurrence of type I and II errors are denoted by specific symbols:

¼ prftype I errorg ¼ pr reject H0 =H0 is true

¼ prftype II errorg ¼ pr accept H0 =H0 is false

ð39Þ

In Equation (39), the symbol ‘/’ indicates that the probability is calculated under that condition. In the example we are following, will be calculated by means of a normal distribution of mean 2 and standard deviation 0.55. The probability of the test is called the significance level, and the power of the test is 1 , which measures the probability of correctly rejecting the null hypothesis. Statistically, in the example, one wants to decide about the value of the mean of a normal distribution with known variance and one-sided alternative test). p ffiffiffiffiffi With these data, the statistic is pffiffiffi hypothesis (a one-tail (Table 4, second row) Zcalc ¼ ðx – Þ=ð= nÞ ¼ ð1:708 – 2:0Þ= 0:55= 10 ¼ – 1:679: In addition, the analyst must assume the risk , say 0.05, which means that the decision rule that is going to apply to its experimental results will accept an inadequate solution 5% of the times. Therefore, the critical or rejection region is defined (Table 4, second row) as the set CR ¼ {Zcalc < 1.645}, that is, the null hypothesis will be rejected for the samples of size 10 that provide values of the statistic less than –1.645. In the example, the value Zcalc belongs to the critical region; thus, the decision is to reject the null hypothesis (i.e., the prepared solution is adequate) at 5% significance level. Given the present facilities of computation, instead of the CR, the statistical packages calculate the so-called p-value, which is the probability of obtaining the computed value of the statistic under the null hypothesis H0. In our case, p-value ¼ pr{Z 1.679} ¼ 0.0466. When the p-value is less than the significance level , the null hypothesis is rejected because this is equivalent to saying that the value of the statistic belongs to the critical region. However, which is the power of the decision rule (statistic and critical region) that has been used? Equation (39) implies that to calculate it is necessary to specify what is exactly understood by the alternative hypothesis. In our case, what is understood by pH smaller than 2. From a mathematical point of view, the answer is clear: any number smaller than 2, for example 1.999 9; from the point of view of the analyst, this mathematical answer does not have sense. Sometimes on the basis of the previous knowledge, in other cases because of the regulatory stipulations or simply by the detail of the working standardized procedure, the analyst


33

Table 4 Some parametric hypothesis tests Null hypothesis

Alternative hypothesis

Statistic

Critical region – 0 pffiffiffi = n

1 2 3

¼ 0 ¼ 0 ¼ 0

6¼ 0 < 0 > 0

Zcalc ¼

4 5 6

¼ 0 ¼ 0 ¼ 0

6¼ 0 < 0 > 0

tcalc ¼

7 8

1 ¼ 2 1 ¼ 2

1 6¼ 2 1 > 2

ð1 – 2 Þ ffi Zcalc ¼ qffiffiffiffiffiffiffiffiffiffiffiffi 21 22 n1 þ n2

9

1 ¼ 2

1 6¼ 2

ð1 – 2 Þ tcalc ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffi ð Þ sp n11 þ n12

10

1¼ 2

1 > 2

11

d ¼ 0

d 6¼ 0

12

d ¼ 0

d > 0

13

2 ¼ 20

2 6¼ 20

14

2 ¼ 20

2 > 20

15

21 ¼ 22

21 6¼ 22

16

21 ¼ 22

21 > 22

– 0 pffiffiffi s= n

Zcalc < – Z=2 or Zcalc >Z=2 fZcalc < – Z g fZcalc >Z g

t < – t=2;n – 1 or tcalc >t=2;n – 1

calc t < – t;n – 1

calc tcalc > t;n – 1

Zcalc < – z=2 or Zcalc > z=2 fZcalc > – z g

tcalc < – t=2;n1 þn2 – 2 or tcalc > t=2;n1 þn2 – 2 tcalc >t;n1 þn2 – 2

n

o

tcalc ¼

d¯ pffiffiffi ð Þ sd = n

w2calc ¼

ðn – 1Þs12 20

n o w2calc < w21 – =2;n – 1 or w2calc >w2=2;n – 1 n o w2calc > w2;n – 1

Fcalc ¼

s12 s22

n o Fcalc < F1 – =2;n1 – 1;n2 – 1 or Fcalc >F=2;n1 – 1;n2 – 1

Fcalc > F;n1 – 1;n2 – 1

tcalc < – t=2;n – 1 or tcalc >t=2;n – 1

tcal > t;n – 1

The values z are the percentiles of a standard normal distribution such that ¼ pr{N(0,1) > z}. The values t, are the percentiles of a Student’s t distribution with degrees of freedom such that ¼ pr{t > t,}. ðn1 – 1Þs12 þ ðn2 – 1Þs22 is the pooled variance. ( ) sp2 ¼ n1 þ n2 – 2 ¯ ( ) d is the mean of the differences di ¼ x1i – x2i between the paired samples; sd is its standard deviation. n o The values w2; are the percentiles of a w2 distribution with degrees of freedom such that ¼ pr w2 > w2; . The values F, 1,2 are the percentiles of an F distribution with 1 degrees of freedom for the numerator and 2 degrees of freedom for the denominator, such that ¼ pr{F > F,1,2}.

can decide the value of pH that is considered less than 2.00, for example, a pH less than 1.60. This is the same as assuming as ‘pH equal to 2’ any smaller value such that its distance to 2 is less than 0.40. In these conditions jj pffiffiffi ¼ pr N ð0; 1Þ < z – n

ð40Þ

Where jj ¼ 0.40 in our problem and when replacing it in Equation (40), we have ¼ 0.26 (calculations can be seen in Example A9 of Appendix). That is to say, whatever the decision taken, the decision rule leads to throw a valid solution away 26% of the times. Evidently, this is an inadequate rule. A simple analysis of Equation (40) explains the situation. To decrease , we should decrease the value pffiffiffi z – ðjj=Þ n. This may be done by decreasing z (i.e., increasing the significance level ) or increasing pffiffiffi ðjj=Þ n. As both the procedure precision, , and the difference of pH that we wish to detect are fixed, the only possibility left is to increase the sample size. Solving Equation (40) in n we have n.

2 z þ z ðjj=Þ2

ð41Þ

The values of and for sample sizes of 10, 15, 20, and 25, maintaining and fixed, are given in Figure 3. As can be seen, and exhibit opposite behavior and, unless the sample size is increased, it is not


0.50 0.45

β = pr{Type II error}

0.40 0.35

n = 10

0.30 n =15

0.25

n = 20

0.20

n = 25

0.15 0.10 0.05 0

0

0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 α = pr{Type I error}

Figure 3 Simultaneous (opposite) behavior of and for different sample sizes, n ¼ 10, 15, 20, and 25.

possible to simultaneously decrease the probability of both errors. In our case, Equation (41) gives n ¼ 20.5 for ¼ ¼ 0.05, thus n ¼ 21 because the sample size must be an integer. The dotted lines in Figure 3 intersect in values of 0.263, 0.126, 0.058, and 0.025, which correspond to the sample sizes considered while maintaining the significance level ¼ 0.05. Again, we see that for a given , the risk decreases with the increase in n. Equation (40) also allows the analyst to decide the standard deviation (precision) necessary to obtain a decision rule according to the risks and that he/she wishes to admit. For example, if one has to make a decision on the validity of the prepared solution with 10 results and the analyst wishes ¼ ¼ 0.05, the only pffiffiffiffiffi option according to Equation (40) is to increase jj=. Solving 0.05 ¼ pr{N(0,1) < 1.645 ð0:40=Þ 10}, one obtains ¼ 0.3845. This means that the procedure should be improved from the current value of 0.55 until 0.38. If only five results were allowed, the standard deviation would have to decrease to 0.27 for maintaining both the significance level and the power of the test. Finally, there is an aspect in Equation (40) that does not have to pass unnoticed. Maintaining , , and n fixed, it is possible to modify the pH value that can be distinguished from 2.00 if the analyst simultaneously increases the precision of the method so that the ratio jj= stays constant. Said otherwise, without changing any of the specifications of the hypothesis test, by diminishing we can discriminate a value of pH nearer to 2. Qualitatively this argument is obvious – if a procedure is more precise, more similar results are easier to distinguish, so that with a more precise procedure different values will appear that would be considered equal with a less precise procedure. Equation (40) quantifies this relation for the hypothesis test we are doing. In summary, a hypothesis test includes the following steps: (1) defining the null, H0, and the alternative, H1, hypothesis according to the purpose of the test and the properties of the distribution of the random variable, which, according to Equation (1), is the distribution of the values provided by a method of measurement; (2) deciding on the probabilities and , that is, the risk for the two types of error that will be assumed for the decision; (3) computing the needed sample size; (4) obtaining the results, computing the corresponding test statistic and evaluating whether it belongs to the critical region CR; and (5) writing the analytical conclusion, which should entail more than reporting the pure statistical test decision. The conclusion should include the elements of the statistical test, the assumed distribution, , , and n. Care must be taken in writing the conclusion; for example, it is more adequate to say ‘there is no experimental evidence to reject the null hypothesis’ than ‘the null hypothesis is accepted’. Table 4 summarizes the tests most frequently used in the validation of analytical procedures and in the analysis of their results. 1.02.3.2

Hypothesis Test on the Mean of a Normal Distribution

Case 1. Known variance: Admit that the data follow a normal N(,) distribution with unknown . The corresponding tests are in rows 1–3 in Table 4. The test statistic is always the same, but, depending on


35

whether the alternative hypothesis is two-sided (row 1 in Table 4) or one-sided (rows 2 and 3), the critical region is different. The value z/2 verifies /2 ¼ pr{N(0,1) > z/2} or the analogous result for z. For the twotail test, the relation among n, , and is given by n.

2 z=2 þ z

ð42Þ

ðjj=Þ2

whereas for the one-tail tests, Equation (41) must be used. Case 2. Unknown variance: In this case, both the mean, , and the standard deviation, , of the normal distribution are unknown. The hypothesis tests are in row 4 of Table 4 for the two-tail case and in rows 5 and 6 for the one-tail tests. The statistic tcalc should be compared to the values t,n1 and t/2,n1 of the Student’s t distribution with n 1 d.f. The equation that relates , , and n is

¼ pr – t=2;n – 1 tn – 1 ðÞ t=2;n – 1

ð43Þ

pffiffiffi where ¼ ðjj=Þ n is the noncentrality parameter of a noncentral t() distribution, which in Equation (43) has n 1 d.f. Note the analogy with the ‘shift’ of the N(0,1) in Equation (40). The discussion about the relative effect of the sample size and the precision is similar to the case in which the variance is t ð Þ if H1: < 0 and known. The corresponding equations for one-tail tests are ¼ pr – t ;n – 1 n – 1 ¼ pr tn – 1 ðÞ t;n – 1 if H1: > 0. To compute n from Equation (43), the standard deviation is needed. To solve this additional difficulty, the comments in Case 2 of Section 1.02.2.2 are valid and can be applied here also. Usually, ¼ 2 or 3. Let us compare the solutions with known and unknown variance by supposing the data of Example 8, but considering that the variance is unknown. We wish to detect differences in pH of 0.73 (the same / as in Example 8). By using a sample of size 10, the probability is 0.32 instead of the previous 0.26 (calculations can be seen in Example A10 of Appendix). This increase in the probability of type II error is due to the less information we have about the problem; now the standard deviation is unknown. Case 3. The paired t-test: In Case 3 of Section 1.02.2.4 the experimental procedure and the reasons for considering paired samples have been already explained. To decide on the effect of a treatment, the null hypothesis is that the mean of the differences is zero, that is, H0: d ¼ 0 and the two-sided alternative H1: d 6¼ 0. This is the test shown in row 11 of Table 4, where there is only a one-tail test (row 12) because, if needed, it suffices to consider the opposite differences di ¼ x2i x1i instead of di ¼ x1i x2i, i ¼ 1,. . .,n. The statistic and the critical region are analogous to those of Case 2 (test on the mean with unknown variance). Example 9: Table 5 shows the results of the recoveries obtained in 10 different places with two solid-phase extraction (SPE) cartridges after fortification of wastewater with a sulfonamide. We want to decide whether cartridge A is more efficient than cartridge B and to compute the risk of the text. For answer these questions, it is important to define that we consider ‘different’ those differences between the means of recoveries that are greater than 2%. We use a paired t-test on the mean of the differences between the recoveries obtained with the two cartridges on the same sample (those of cartridge A minus those of cartridge B). By considering these differences, we eliminate the effect that the wastewater could have on the performance of the two cartridges. The test is carried out as follows: H0 : d ¼ 0 ðthe differences of recoveries are nullÞ

ð44Þ

H1 : d > 0 ðcartridge A provides recoveries greater than cartridge BÞ Table 5 Recovery of sulfonamides spiked in wastewater obtained by using two different extraction cartridges Place

1

2

3

4

5

6

7

8

9

10

Cartridge A (%) Cartridge B (%)

77.2 74.4

74.0 70.0

75.6 70.2

80.0 77.2

75.2 75.9

69.2 60.0

75.4 77.0

74.0 76.0

71.6 70.0

60.4 55.0

See Example 9 for more details.


pffiffiffiffiffi

pffiffiffi Following row 12 of Table 4, the statistic is tcalc ¼ d=ðsd = nÞ ¼ 2:69= 3:526= 10 ¼ 2:412 and the critical region CR ¼ {tcalc > t,n – 1}. The critical value t0.05,9 is equal to 1.833; thus the statistic belongs to the critical region. Therefore, the null hypothesis is rejected for ¼ 0.05 and we can conclude that cartridge A is more efficient than cartridge B, because the mean of the differences is positive.

To evaluate the power (1 ) of the test, the equation ¼ pr tn – 1 ðÞ t;n – 1 with d ¼j – 0 j= ¼ jj= ¼ 2=3:53 ¼ 0:57 provides 1 ¼ 0.496 6 for ¼ 0.05 and n ¼ 10 (calculations can be seen in Example A11 of Appendix). Hence, 50% of the times the conclusion of accepting that there is no difference between recoveries is wrong. In this case, the risk of a type II error is very large; in other words, the power is very poor when we want to discriminate differences of 2% in recovery because the ratio d is small. 1.02.3.3

Hypothesis Test on the Variance of a Normal Distribution

The variance is a measure of the dispersion of the data used to evaluate the precision of a procedure of analysis; thus, decisions have to be made on this parameter frequently. The corresponding hypothesis tests are in rows 13 and 14 of Table 4. Example 10: A validated procedure has a repeatability of 0 ¼ 1.40 mg l–1 when measuring concentrations around 400 mg l–1. After a technical revision of the instrument, the laboratory is interested in testing the hypothesis H0 : 2 ¼ 20 ¼ 1:96 ðthe repeatability has not changedÞ H1 : 2 > 1:96 ðthe repeatability got worseÞ

ð45Þ

The analyst decides that a repeatability is admissible until 2.0 times the initial one, 1.40 mg l–1, and assumes the risks ¼ ¼ 0.05. The sample size needed to guarantee the requirements of the analyst is formally obtained from a one-tail hypothesis test on the variance where k ¼ pr w2n – 1 < 2

ð46Þ

k is the value such that ¼ pr w2n – 1 > k and ¼ =0 . As ¼ 2.0, Equation (46) gives that for n ¼ 14, ¼ 0.040 2, whereas for n ¼ 13, ¼ 0.051 1 (calculations can be seen in Example A12 of Appendix). Therefore, he/she decides to do 14 determinations on aliquot parts of a sample with 400 mg l–1 obtaining a variance of 3.1 (s ¼ 1.76). The statistic is w2calc ¼ ð14 – 1Þ3:10=1:96 ¼ 20:56. As the critical region is made by the values CR ¼ fw2calc >w20:05;13 ¼ 22:36g, he/she concludes that there is no sufficient experimental evidence to say that the precision has worsened. In this case, the acceptance of the null hypothesis, that is, to maintain the repeatability below 2.0 times the initial one, will be erroneous 5% of the times because was fixed at 5%. The decision rule is equally protected against type I and II errors. 1.02.3.4

Hypothesis Test on the Difference in Two Means

Case 1. Known variances: We assume that X1 is normal and has unknown mean 1 and known variance 21, and that X2 is also normal with unknown mean 2 and known variance 22. We will be concerned with testing the hypothesis that the means 1 and 2 are equal. The two-sided alternative hypothesis is in line 7 of Table 4 and the one-sided in line 8 when we have a random sample of size n1 of X1 and another random sample of size n2 of X2. Example 11: A solid-phase microextraction (SPME) procedure to extract triazines from wastewater has been carried out. The results must be compared with previous ones where extraction was made by means of SPE. The repeatability of both procedures is known to be 5.36% for SPME procedure and 3.12% for SPE. The mean for 10 samples (Table 6) is 85.9% for recovery obtained with SPME and 81.8% for SPE. At a 0.05 significance level, is the recovery of SPME procedure greater than that of SPE?


37

Table 6 Recoveries of triazines from wastewater using solid-phase microextraction (SPME) and solid-phase extraction (SPE) Recovery (%) SPME SPE

91 86

85 82

90 85

81 86

79 79

78 82

84 80

87 77

93 79

91 82


As the standard repeatability of both procedures is known, a test to compare two means with normal distribution and known variances is adequate. H0 : SPME ¼ SPE ðthe recovery is the same for both proceduresÞ H1 : SPME > SPE ðthe recovery using SPME procedure is greater than that using SPEÞ

ð47Þ

pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi The statistic is Zcalc ¼ ðxSPME – xSPE Þ= 2SPME =n1 þ 2SPE =n2 ¼ ð85:9 – 81:8Þ= 28:73=10 þ 9:73=10 ¼ 2:091 following line 8 of Table 4. For a significance level ¼ 0.05, CR ¼ {Zcalc > z ¼ 1.645}. As the statistic 2.091PCR, the null hypothesis is rejected and we conclude that the mean of the recovery with SPME is greater than that with SPE. If a difference in recovery of 3% is sufficient in the analysis, what is the risk for this hypothesis test? A simple modification of Equation (40) shows that 8 9 > > < j j = ffi ¼ pr N ð0; 1Þ > z – qffiffiffiffiffiffiffiffiffiffiffiffi > 21 22 > : ; n1 þ n2

ð48Þ

By substituting our data in Equation (48), one obtains ¼ 0.55. That means that in 55% of the cases, we will incorrectly accept that the recovery is the same for both procedures. It is also possible to derive formulas to estimate the sample size required to obtain a specified for given and . For the one-sided alternative, the sample size n ¼ n1 ¼ n2 is n.

2 z þ z 2 21 þ22

ð49Þ

Again, with the data of the problem at hand and ¼ 0.05, Equation (49) gives the number 46.30, that is, 47 aliquot samples should be analyzed for each procedure so that ¼ ¼ 0.05. For the two-sided alternative, the sample size n ¼ n1 ¼ n2 is n.

2 z=2 þ z 2 21 þ22

ð50Þ

Case 2. Unknown variances: As in Section 1.02.2.4 (Case 2), there are two possibilities: (1) the unknown variances are equal 21 ¼ 22 ¼ 2, although for chance reasons the numerical values differ, and (2) both variances are different 21 6¼ 22. The question of deciding between (1) and (2) will be approached in Section 1.02.3.6. Let X1 and X2 be two independent normal random variables with unknown but equal variances. The statistics and the critical region for the two-tail test are in line 9 of Table 4 and in line 10 we can see the one-tail case. For the two-sided alternative, with risks and , we consider further that the two means are different if their difference is at least a quantity ¼ j1 2j. As the variances are unknown, the comments in Case 2 of Section 1.02.2.2 are also applicable. If we have samples from a pilot experiment with respective sizes n19 and n29, and sp2 is the pooled variance computed with them (see the statistic in lines 9 and 10 of Table 4), the sample size needed n ¼ n1 ¼ n2 is n.

2 t=2; þ t; 2 =2sp2

ð51Þ


where ¼ n19 þ n29 2 are the d.f. of the Student’s t distribution. If the aforementioned is not possible, the difference to be detected should be expressed in standard deviation units, then ¼ j1 2j ¼ k and the following expression applies:

2 z=2 þ z n. k2 =2

ð52Þ

where z/2 and z are the corresponding upper percentage points of the standard normal distribution. Example 12: An experimenter wishes to compare the mean of two procedures, stating that they are to be considered different if they differ in 2 or more standard deviations (k ¼ 2). In addition, he/she wants to assume ¼ 0.05 and ¼ 0.10. As z0.025 ¼ 1.96 and z0.10 ¼ 1.282, Equation (52) gives n ¼ 5.26, then six samples must be considered for each procedure. If he/she wishes to distinguish 1 standard deviation (k ¼ 1), then n ¼ 21.02, that is, he/she should have 22 data from each procedure. Although it is preferable to always take equal sample sizes, it can happen that it is more expensive or laborious to collect the data of X1 than that of X2. In this case, there are weighted sample sizes to be considered.32 In the case where equality of variances 21 and 22 cannot be admitted, there is no completely justified solution for the test. However, approximations exist with good power and easy to use tests, such as the Welch test. This method consists of substituting the known variances in the expression of Zcalc (Case 1 of this section) by their sample estimates, in such a way that the statistic becomes x1 – x2 t calc ¼ qffiffiffiffiffiffiffiffiffiffiffiffi s12 s22 n1 þ n2

ð53Þ

which follows a Student’s t with the d.f. in Equation (25). The critical region for the two-tail test is CR ¼ {tcalc < –t/2, or tcalc > t/2,}. The critical region for the one-tail test is CR ¼ {tcalc < –t, } if H1: < 0 and CR ¼ {tcalc > t,} if H1: > 0. As the variances are different, it seems reasonable to take the sample sizes, n1 and n2, also different. If 2 ¼ r 1, similar to Equation (52), one obtains n1 .

z=2 þ z

2

k2 ðrþ1Þ

ð54Þ

Once n1 is determined, n2 is obtained as n2 ¼ r n1. The computation of the sample sizes with different variances when pilot samples are at hand can be found in Schouten.32

1.02.3.5

Test Based on Intervals

The problem of deciding ‘the equality’ of the means of two distributions discussed in the previous section shows the fact that the result of interest (the two means are equal) is obtained by accepting the null hypothesis. Hence, the type II error becomes very important. To compute it, it is necessary to decide which is the least difference between the means that is to be detected, ¼ j1 2j. A more natural framework is to define the null and alternative hypotheses in such a way that the decision of accepting the equality of means is made by rejecting the null hypothesis, that is, the test should be posed as H0 : j1 – 2 j ðthe means are differentÞ H1 : j1 – 2 j < ðthe means are equalÞ


39

Contrary to the tests so far, the hypothesis of this test, called interval hypothesis, is not made by one point but an interval. The two one-sided tests (TOST) procedure consists of decomposing the interval hypotheses H0 and H1 into two sets of one-sided hypothesis: H01 : 1 – 2 – H11 : 1 – 2 > –

and H02 : 1 – 2 H12 : 1 – 2 <

The TOST procedure consists of rejecting the interval hypothesis H0 (and thus concluding equality of 1 and 2) if and only if both H01 and H02 are rejected at a chosen level of significance . If two normal distributions with the same unknown variance, 2, are supposed and two samples of size n1 and n2 are taken from each one, respectively, the two sets of one-sided hypothesis will be tested with ordinary onetail test (row 10 of Table 4). Thus, the critical region is 8 9 > > > : sp n1 þ n1 ; sp n11 þ n12 1 2

ð55Þ

where sp2 is the pooled variance and ¼ n1 þ n2 2 its d.f. The TOST procedure turns out to be operationally identical to the procedure of declaring equality only if the usual confidence interval at 100 (1 2)% on 1 2 is completely contained in the interval [,]. The expression that relates the sample sizes with and is n1 .

z þ z=2

2 f 2

ð56Þ

where f ¼ (r þ 1)/r and n2 ¼ r n1. When the sample sizes are equal, f ¼ 2. As is unknown in Equation (56), it should be adapted as in Case 2 of Section 1.02.3.4. When comparing Equation (56) with those corresponding to the two-tail t-test on the difference of means, one observes that it is completely analogous by exchanging the two risks (see Equations (50) and (52)). That is, the significance level and the power of the t-test become the power and significance level, respectively, of the TOST procedure, which completely agrees with the exchange of the hypotheses. The tests based on intervals have a great tradition in statistics, see for example the book (very technical) by Lehmann33. The TOST procedure is a particular case that has also been used under the name bioequivalence test.34 Mehring35 has proposed some technical improvements to obtain optimal interval hypothesis tests, including equivalence testing. It is shown that TOST is always biased, in particular, the power tends to zero for increasing variances independently of the difference of means. As a result, an unbiased test36 and a suitable compromise between the most powerful test and the shape of its critical region37 have been proposed. In chemistry the use of TOST has been proposed to verify the equality of two procedures.38,39 Kuttatharmmakull et al.40 provide a detailed analysis of the sample sizes necessary in a TOST procedure to compare methods of measurement. There are different versions of TOST for ratio of variables and for proportions; the details of the equations for these cases can be consulted in Section 8.13 of the book by Martin Andre´s and Luna del Castillo41 and in the book by Wellek.42 The latter is a comprehensive review of inferential procedures that enable one to ‘prove the null hypothesis’ for many areas of applied statistical data analysis. 1.02.3.6

Hypothesis Test on the Variances of Two Normal Distributions

Suppose that two procedures follow normal distributions X1 and X2 with unknown means and variances. We wish to test the hypothesis on the equality of the two variances, that is, H0: 21 ¼ 22. In practice, this is a relevant problem because this hypothesis is related to the equality of the precision of the two procedures, and also as a


previous check to decide about the equality of variances before applying the test on the equality of means (Case 2 of Section 1.02.3.4) or to compute a confidence interval on the difference of means (Case 2 of Section 1.02.2.4). Assume that two random samples of size n1 of X1 and of size n2 of X2 are available and let s21 and s22 be the sample variances. To test the two-sided alternative, we use the statistic and CR of line 15 of Table 4. The probability can be computed as a function of the ratio of variances l 2 ¼ 21 =22 that is to be detected, by the equation ¼ pr

F=2;n1 – 1;n2 – 1 F1 – =2;n1 – 1;n2 – 1 < F < n – 1;n – 1 1 2 l2 l2

ð57Þ

where with n1 1 and n2 1 d.f. and F=2;n1 –1;n2 –1 its upper /2 point, so that

Fn1 –1;n2 –1 denotes an F-distribution pr Fn1 –1;n2 –1 > F=2;n1 –1;n2 –1 ¼ =2. Similarly, F1–=2;n1 –1;n2 –1 is the upper 1 – ð=2Þ point. Example 13: Aliquot samples have been analyzed in random order under the same experimental conditions to carry out a stability test. The results are given in Table 7 and must be compared for assessing test material stability. (1) Is there experimental evidence of instability in the material? (2) What is the probability of accepting the null hypothesis when it is in fact wrong? To answer these questions, the analyst considers that the material is not stable if the mean of the test sample differs from the mean of the control sample in 2 standard deviations. (3) What should be the sample size if 1 standard deviation is needed for fit for purpose of this analysis (and ¼ ¼ 0.05)? 1. As we only know the estimates of the variance, a t-test to compare means before and after the storage of the analyte has been carried out. The first step is to test if the variances can be considered equal by using a two-tail F-test: H0: 21 ¼ 22 H1: 21 6¼ 22

Following row 15 in Table 4, Fcalc ¼ s12 =s22 ¼ 50:76=26:22 ¼ 1:93 and CR ¼ Fcalc < F1 – =2;n1 – 1;n2 – 1 or Fcalc > F=2;n1 – 1;n2 – 1 g with F0.025,8,8 ¼ 4.43 and F1– 0:025;8;8 ¼ 1=F0:025;8;8 ¼ 1=4:43 ¼ 0:225 7. Hence, there is no experimental evidence to say that the variances differ. Therefore, a hypothesis t-test on the difference of the two means with equal variances is being formulated (Case 2 of Section 1.02.3.4). The statistic and the CR are given in line 9 of Table 4. H0: 1 ¼ 2 ðthe test material is stableÞ H1: 1 6¼ 2 ðthe material is not stableÞ

pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 The ‘pooled’ variance, s , is 38.49, so s ¼ 6:20, with 9 þ 9 2 ¼ 16 d.f. t ¼ ð x – x Þ= sp 1=n1 þ 1=n2 p calc 1 2 p pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ ð45:41 – 43:46Þ= 6:20 1=9 þ 1=9 ¼ 0:67 and t0:025;16 ¼ 2:120. Therefore, the critical region is CR ¼ftcalc < – 2:120 or tcalc > 2:120g. Hence, there is no evidence to reject the null hypothesis, that is, with these data there is no experimental evidence of instability. 2. Power of the test: With the condition imposed by the analyst, Equation (52) with k ¼ 2 provides 0.05. 3. In this case, the analyst is interested in computing the sample size under the assumption that only 1 standard deviation is admissible for fit for purpose of this analysis. Therefore, k ¼ 1 and from Equation (52) n ¼ 25.99, so n1 ¼ n2 ¼ 26. The sample size is greater than the former because he/she is interested in distinguishing a quantity that is less than this in previous point 2. Table 7 Data for analysis of stability (arbitrary units) Control sample Test sample

46.31 43.12


44.90 43.00

44.12 44.75

36.07 39.66

39.20 37.74

36.39 37.50

50.71 54.79

47.85 53.08

45.60 55.07


41

Incidentally, the null hypothesis of the F-test has been accepted. When a standard deviation twice that of the control samples is to be detected, Equation (57) gives a probability for this test of 0.56. That means that 56% of the times the null hypothesis will be wrongly accepted, and in this case, we have accepted the null hypothesis. When the F-test is used as a previous step to the one of equality of means, and this one will be used with ¼ 0.05, it is common to fix ¼ 0.10 for the F-test. Equation (57) would provide n ¼ n1 ¼ n2 ¼ 24 with ¼ 0.11, and by maintaining that a change of 2 times the standard deviation of the control samples is to be detected (all calculations of can be seen in Example A13 of Appendix). In general, the F-tests on the equality of variances are very conservative and large sample sizes are needed to assure an adequately small probability of type II error.

1.02.3.7

Hypothesis Test on the Comparison of Several Independent Variances

When the hypothesis of equality of variances of several groups of data coming from normal and independent distributions is to be tested, a good practice is to draw the data for a visual inspection of their dispersion. Example 14: Table 8 shows the results of the determination of the acetic degree by means of an acid–base titrimetry, employing sodium hydroxide as the titrant. These data are adapted from the practice ‘Analysis and comparison of the acetic grade of a vinegar’ included in Ortiz et al.,43 and each series is a replicated determination carried out by a group of students over the same vinegar sample. The means and variances obtained by each group are also included in Table 8. Figure 4 shows the plot of the results obtained by different groups of students. The most commonly used tests to compare several variances are the Cochran’s, the Bartlett’s, and the Levene’s tests. In all the cases, we wish testing the hypothesis H0 : 21 ¼ ¼ 2i ¼ ¼ 2k

ð58Þ

Ha : At least one 2i is different

P The sample size of each group is denoted as ni, i ¼ 1,2,. . .,k, and N ¼ ki¼1 ni . Case 1. Testing the equality of several variances according to Cochran’s test: The null hypothesis is that the variances within each of the k groups of data are the same. This test detects if one variance is greater than the rest. The statistic is max s 2 Gcalc ¼ Pk i 2 i¼1 si

ð59Þ

The critical region at significance level is given by

CR ¼ Gcalc > G;k;

ð60Þ

where G;k; is the value tabulated in Table 9 for d.f. In the case ni ¼ n for all i, is ¼ n1. Table 8 Determination of acetic degree of a vinegar by means of an acid–base titration

Mean xi Variance si2

Group 1

Group 2

Group 3

Group 4

Group 5

6.028 6.028 5.998 6.089 6.059

5.974 6.004 6.005 5.852 5.944

5.886 5.970 5.880 5.910 5.910

6.132 6.120 6.131 6.072 6.071

5.916 6.123 6.034 6.004 6.152

6.040 4 1.203 10–3

5.955 8 3.997 10–3

5.911 2 1.267 10–3

6.105 2 0.969 10–3

6.045 8 8.993 10–3



Acetic degree

6.2 6.1 6.0 5.9 5.8 1

2

3 Group

4

5

Figure 4 Data for testing the equality of variances.

Table 9 Critical values for Cochran’s test for testing homogeneity of several variances at 5% significance level , degrees of freedom k

1

2

3

4

5

6

7

8

9

10

2 3 4 5 6 7 8 9 10

0.9985 0.9669 0.9065 0.8412 0.7808 0.7271 0.6798 0.6385 0.6020

0.9750 0.8709 0.7679 0.6838 0.6161 0.5612 0.5157 0.4775 0.4450

0.9392 0.7977 0.6841 0.5981 0.5321 0.4800 0.4377 0.4027 0.3733

0.9057 0.7457 0.6287 0.5441 0.4803 0.4307 0.3910 0.3584 0.3311

0.8772 0.7071 0.5895 0.5065 0.4447 0.3974 0.3595 0.3286 0.3029

0.8534 0.6771 0.5598 0.4783 0.4184 0.3726 0.3362 0.3067 0.2823

0.8332 0.6530 0.5365 0.4564 0.3980 0.3939 0.3185 0.2901 0.2666

0.8159 0.6333 0.5175 0.4387 0.3817 0.3384 0.3043 0.2768 0.2541

0.8010 0.6167 0.5017 0.4241 0.3682 0.3259 0.2926 0.2659 0.2439

0.7880 0.6025 0.4884 0.4118 0.3568 0.3154 0.2820 0.2568 0.2353

k, number of groups; , degrees of freedom. Adapted from Sachs, L. Applied Statistics. A Handbook of Techniques; New York: Springer-Verlag, 1982.

With the data of Example 14, Gcalc ¼ 8.993 10–3/16.429 10–3 ¼ 0.5474 and G0:05;5;5 – 1 ¼ 0:5441; thus, at 0.05 significance level, the null hypothesis should be rejected and the variance of group 5 should be considered different from the rest. Case 2. Bartlett’s test is appropriate to detect groups of similar variance within each of the k groups of data but that differs from one group to another. The statistic is defined using the following equations: w2calc ¼ 2:3026

q c

ð61Þ

k X

ðni – 1Þlog si2 q ¼ ðN – kÞlog sp2 –

ð62Þ

i¼1

hP c ¼1þ

k 1 i¼1 ni – 1

i

–

1 N –k

3ðk – 1Þ

ð63Þ

In Equation (62), ‘log’ meansh the decimal logarithm and sp2 is the pooled variance that, analogous to Equation i P k 2 (23), for k variances is sp2 ¼ i¼1 ðni – 1Þsi =ðN – k Þ. The critical region is n o CR ¼ w2calc > w2;k – 1

ð64Þ

In Example 14, c ¼ 1.10, q ¼ 3.43, and w2calc ¼ 7:19, which does not belong to the critical region defined in Equation (64) because w20:05;4 ¼ 9:49. Cochran’s and Bartlett’s tests are very sensitive to the normality assumption. The Levene’s test, particularly when it is based on the medians of each group, is more robust to the lack of normality of data.


43

Case 3. Levene’s test: Consider, in the jth group of replicates, the absolute deviations of the values xij from the mean of their repeats groups. lij ¼ xij – xi ; i ¼ 1; 2; . . . ; ni

ð65Þ

Consider the data arranged as in Table 8 and compute the usual F statistics for the deviations lij Pk Fcalc ¼

i¼1

Pk

j ¼1

2 ni ðli – lÞ k–1

Pni ðlij – li Þ2 i¼1

ð66Þ

N –k

where li is the mean of the ith group and lis the global mean. The critical region at 100 (1 )% confidence level is

CR ¼ Fcalc > F;k – 1;N – k

ð67Þ

Note that the numerator of Equation (66) is the pooled variance of the deviations and the denominator is the global variance of these deviations. Computing the differences in Equation (65) with the data of Table 8, Fcalc ¼ (2.205 10–3)/(0.905 10–3) ¼ 2.44. As F0.05,4,20 ¼ 2.866, there is no evidence to reject the null hypothesis (the variances are equal). The Levene’s test using group medians instead of means is more recommendable. The adaptation is simple; one has to consider the absolute value of the differences but to the median, x˜i , of each group lij ¼ xij ˜– x i ; i ¼ 1; 2; . . . ; ni

ð68Þ

The statistics is again that of Equation (66) and it is tested in the same way as before. With the same data of Table 8, but applying the transformation in Equation (68), one obtains Fcalc ¼ (2.146 10–3)/(1.360 10–3) ¼ 1.58, and the conclusion is the same. The variance of the five groups should be considered as equal. Frequently it happens that the three tests do not agree in the result, as is the case here. But the joint interpretation clarifies the situation: In the data of Example 14 , the variance of group 5 is greater than the variance of other groups. When the Levene’s test is applied, a large difference between both statistics is observed in the statistics when using the median. This makes one think that the increase in the variance of the last group is caused by some data being different from the others; this is graphically seen in Figure 4.

1.02.3.8

Goodness-of-Fit Tests: Normality Tests

The test on distribution or goodness-of-fit tests are designed to decide whether the experimental data are compatible with a predetermined probability distribution, generally characterized by one or several parameters, such as the normal, the Student’s t, or the F distribution. Almost all the inferential procedures proposed in this chapter are based on normality; thus, in most of the cases, it is necessary to verify whether the data are compatible with this assumption. In this section, we will show the chi-square test that is used for any distribution and the D’Agostino test that is recommendable for evaluating the normality of a data set. Case 1. Chi-square test: The test is designed to detect frequencies inadequate for a specified probability distribution F0. Given a sample x1,x2,. . .,xn from a random variable, one is interested in testing the hypothesis H0 : The distribution of the random variable is F0 H1 : This is not the case

ð69Þ

To compute the statistics, the n sample values are grouped into k classes (intervals). Denote by Oi, i ¼ 1,. . .,k, the frequency observed in each class and by Ei the expected frequency for the same class provided the distribution is exactly F0. Then, w2calc ¼

k X ðOi – Ei Þ2 i¼1

Ei

ð70Þ


The critical region at (1 ) 100% of confidence is n o CR ¼ w2calc > w2;k – p – 1

ð71Þ

n o where w2;k – p – 1 is the value such that pr w2k – p – 1 > w2;k – p – 1 ¼ , where p is a number that depends on the distribution F0, for instance p ¼ 2 for a normal, p ¼ 1 for a Poisson, and p ¼ 0 for a uniform distribution. The test requires that the expected frequencies are not too small. Therefore, the data are regrouped into bigger classes. In the practice of chemical analysis, the sample sizes are not large and when grouping the data the d.f. of the chisquare statistics are few, the critical value of Equation (71) becomes large, and it is necessary to have a large discrepancy between the estimated and observed frequencies to reject the null hypothesis. That means that the test is very conservative. Example 15: To show the validity of the use of the crystal violet (CV) as an internal standard in the determination by HPLC-MS-MS of malachite green (MG) in trout, a sample of trout was spiked with increasing concentrations of MG between 0.5 and 5.0 mg kg–1 and in all of them with 1 mg kg–1 of CV. The area of the CV-specific peak (transition 372 > 356) resulted: 1326, 1384, 1419, 1464, 1425, 1409, 1387, 1449, 1311, 1338, 1350, and 1345 abundance per count. To verify whether the CV is constant and independent of the concentration of MG, we can test the hypothesis H0 : The distribution of the random variable is uniform H1 : This is not the case

Table 10 shows the calculation of both observed and expected frequencies under the uniform distribution in the interval [1311,1464], the endpoints being respectively the minimum and maximum values in the sample. By summing up the values of the last column of Table 10, the statistics is w2calc ¼ 0:51, which does not belong to the critical region because it is not greater than w20:05;5–0–1 ¼ 9:49. Therefore, there is no evidence to reject the hypothesis that the data come from a uniform distribution. Case 2. Normality D’Agostino: The problem of verifying the normality of a set of data has been extensively treated. When the empirical and theoretical histograms are compared, the most commonly used tests are those of the chi-square and the Kolmogoroff–Smirnov. However, there are many characteristics that are specific for the pdf of a normal; for example, the skewness and the kurtosis, which are statistics related to the higher than two order moments for the normal pdf. A very powerful test is that of D’Agostino. H0 : The distribution of the random variable is normal H1 : This is not the case

To apply the test, the data are sorted in increasing order, so that x1 x2 xn. The statistic is n þ 1 Xn ixi – x i¼1 i 2 ffi ¼ rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi P

Pn

2 n 2 – n n x x =n i i¼1 i i¼1 Pn

Dcalc

i¼1

ð72Þ

Table 10 w2 test for uniform distribution applied to asses the validity of crystal violet as internal standard; data of Example 15 Class

Observed frequency (Oi)

Expected frequency (Ei)

ðOi – Ei Þ2 Ei

[1311, 1341.6) [1341.6, 1372.2) [1372.2, 1402.8) [1402.8, 1433.4) [1433.4, 1464)

3 2 2 3 2

2.40 2.40 2.40 2.40 2.40

0.15 0.07 0.07 0.15 0.07


45

Table 11 Significance limits for the D’Agostino normality test Significance level Sample size

¼ 0.05

10 12 14 16 18 20 22 24 26 28 30

DL 0.2513 0.2544 0.2568 0.2587 0.2603 0.2617 0.2629 0.2639 0.2647 0.2655 0.2662

¼ 0.01 DU 0.2849 0.2854 0.2858 0.2860 0.2862 0.2863 0.2864 0.2865 0.2866 0.2866 0.2866

DL 0.2379 0.2420 0.2455 0.2482 0.2505 0.2525 0.2542 0.2557 0.2570 0.2581 0.2592

DU 0.2857 0.2862 0.2865 0.2867 0.2868 0.2869 0.2870 0.2871 0.2872 0.2873 0.2873

Adapted from Martıń Andre´s, A.; Luna del Castillo, J. D. Bioestadı´stica para las ciencias de la salud; Spain: Norma Capitel Madrid, 2004.

Index i in Equation (72) refers to the ordered data. Table 11 shows some of the critical values of the statistics D,n with the two values, DL,n and DU,n, for each sample size n and significance level . The critical region of the test is

CR ¼ Dcalc < DL;n or Dcalc > DU;n

ð73Þ

For further details, consult the work by D’Agostino and Stephens.44 As with the confidence intervals, a Bayesian approach exists for the construction of the hypothesis tests that many statisticians prefer because of its internal coherence. Consult Chapter 1.08 and the basic references on this subject; for a recent comparative analysis of both approaches, see Moreno and Giroń.45

1.02.4 One-Way Analysis of Variance Sometimes, more than two means must be compared. One can think in comparing, say, five means, by applying the test of comparison of two means of Section 1.02.3.4 to each of the 10 pairs of means that can be formed taking them from two in two. This option has the serious problem of the requirement of enormous sample sizes, because to test the null hypothesis ‘the five means are equal’ with ¼ 0.05 and assuming that the 10 tests are independent, each one of the hypothesis ‘the means xi and xj are equal’ should be tested with a significance level of 0.0051 to obtain (1 0.0051)10 0.95. The appropriate procedure for testing the equality of several means is the analysis of variance (ANOVA). The ANOVA has many more applications; it is particularly useful in the validation of a model fit to some experimental data and, hence, in an analytical calibration (Chapter 1.05), or in the analysis of response surfaces (Chapter 1.12). Table 12 shows how the data are usually arranged in a general case, with a factor to k levels (e.g., five different extraction cartridges) and in each one n data (e.g., four determinations with each cartridge). Each of the N ¼ k n values xij (i ¼ 1,2,. . .,k, j ¼ 1,2,. . .,n) is the result obtained when using the ith cartridge withPthe jth aliquot sample. In general, in each level i, a different number of replicates are available ni, with N ¼ ki¼1 ni . To make the notation easier, we will suppose that all ni are equal, that is, ni ¼ n for each level. Suppose that the data in Table 12 can be described by the model. xij ¼ þ i þ "ij

with i¼ 1; 2; . . . ;k; j ¼ 1; 2; . . . ;n

ð74Þ

46 Quality of Analytical Measurements: Statistical Methods for Internal Validation Table 12 Arrangement of data for an ANOVA Factor Level 1

Level 2

Level 3

x11 x12 x13 .. . x1n

x21 x22 x23 .. . x2n

x31 x32 x33 .. . x3n

Level k .. .

xk1 xk2 xk3 .. . xkn

where is a parameter common to all treatments, called the overall mean, i is a parameter associated with the ith level, called the factor effect, and "ij is the random error component. In our example, is the content of the sample and i is the variation in this quantity caused by the use of the ith cartridge. Note that in the model of Equation (74), the factor effect is additive; this is an assumption that may be unacceptable in some practical situations. We would like to test certain hypotheses about the treatment effects and to estimate them. For the hypothesis testing the model errors are assumed to be normal and independently distributed random variables, with mean zero and variance 2, NID(0,2). The variance 2 is assumed to be constant for all levels of the factor. The model of Equation (74) is called the one-way ANOVA, because only one factor is studied. The analysis for two or more factors can be seen in the Chapter 1.12 about factorial techniques. Furthermore, the data of Table 12 are required to be obtained in random order so that the environment in which the factor varies is as uniform as possible. There are two ways for choosing the k factor levels in the experiment. In the first case, the k levels are specifically chosen by the researcher, as the cartridges in our example. In this case, we wish to test the hypothesis about the size of i, and conclusions will apply only to the factor levels considered in the analysis and they cannot be extended to similar levels that were not considered. This is called the ‘fixed effects model’. Alternatively, the k levels could be a random sample from a larger population of levels. In this case, we would like to be able to extend the conclusions based in the sample to all levels in the population, independent of whether they have been explicitly considered in the analysis or not. Here i are random variables and information about the specific values included in the analysis is useless. Instead, we test the hypothesis about the variability. This is called the ‘random effects model’. This model is used to evaluate the repeatability and reproducibility of a method and also the laboratory bias when the method of analysis is being tested by a proficiency test. In the same experiment, and provided there are at least two factors, fixed and random effects can simultaneously appear.46,47 1.02.4.1

The Fixed Effects Model

In this model, the effect of the factors is defined as a deviation with respect to the overall mean, so that k X

i ¼ 0

ð75Þ

i¼1

From the individual data, the mean value per level is defined as Pn xi ¼

j ¼1 xij

n

; i ¼ 1; 2; . . . ; k

ð76Þ

and the overall mean is Pk x ¼

Pn

j ¼1 xij

i¼1

N

ð77Þ


47

A simple calculation gives k X n X

k k X n X X

2

2 xij – x ¼ n ðxi – xÞ2 þ xij – xi

i¼1 j ¼1

ð78Þ

i¼1 j ¼1

i¼1

Equation (78) shows that the total variability of the data, measured by the sum of squares of the difference of each datum and the overall mean, can be partitioned into a sum of squares of differences between level means and the overall mean and a sum of squares of differences of individual values and their level mean. The term

2 P P P n ki¼1 ðxi – xÞ2 measures the differences between levels, whereas ki¼1 nj¼1 xij – xi can be due to random error alone. It is common to write Equation (78) as SST ¼ SSF þ SSE

ð79Þ

where SST is the total sum of squares, SSF is the sum of squares due to change levels of factor, which is called sum of squares between levels, and SSE is the sum of squares due to error, which is called sum of squares within levels. There are N individual values, thus SST has N 1 d.f. Similarly, as there are k factor levels, SSF has k 1 d.f. Finally, SSE has N k d.f. We are interested in testing H0 : 1 ¼ 2 ¼ 3 ¼ ¼ k ¼ 0 ðthere is no effect due to the factorÞ H1: i 6¼ 0 for at least one i

Because of the assumption that the errors "ij are NID(0,2), the values xij are NID( þ i,2), and therefore SST/2 is distributed as a w2N – 1 . The Cochran’s theorem guarantees that, under the null hypothesis, SSF/2 and SSE/2 are independent chi-square distributions with k 1 and N k d.f., respectively. Therefore, under the null hypothesis, the statistic Fcalc ¼

SSF k–1 SSE N –k

¼

MSF MSE

ð80Þ

46 follows an Fk–1,N–k distribution, whereas under the alternative hypothesis, it follows aP noncentral F. The k 2 quantities MSF and MSE are called mean squares. Their means are E ðMSF Þ ¼ 2 þ n i¼1 i =ðk – 1Þ and 2 E ðMSE Þ ¼ , respectively. Therefore, under the null hypothesis, both are unbiased estimates of the residual variance, 2, whereas under the alternative hypothesis, the expected value of MSF is greater than 2. Therefore, if the null hypothesis is false, the numerator is significantly greater than the denominator of Equation (80) and the critical region at significance level is


ð81Þ

Usually, the test procedure is summarized in a table, called ANOVA table (Table 13). Example 16: To investigate the influence of fibers composition on an SPME procedure, an experiment was performed using five different fibers. The data shown in Table 14 are the results (after extraction) of four replicated analyses, with each fiber, of a sample spiked with 1000 ppb of triazine. All the analyses were carried out in random order and maintaining the rest of experimental conditions controlled. Table 13 Skeleton of an ANOVA of fixed effects Source of variation

Sum of squares

d.f.

Mean squares

E(MS)

Fcalc

Factor (between levels)

SSF

k–1

MSF

2

MSF MSE

Error (within levels)

SSE

N–k

MSE

Total

SST

N–1

2 þ

n

Pk

2 i¼1 i

k–1

48 Quality of Analytical Measurements: Statistical Methods for Internal Validation Table 14 Effect of the type of fiber in a solid-phase microextraction of triazine (ppb) Type of fiber

Replicates

Means xi Variances si2

Fiber 1

Fiber 2

Fiber 3

Fiber 4

Fiber 5

490 478 492 499

612 609 599 589

509 496 489 500

620 601 580 603

490 502 495 479

489.75 76.25

602.25 108.92

498.50 69.67

601.00 268.67

491.50 93.67

In the last two rows of Table 14, the means and variances for each fiber are given. Before doing the ANOVA, the hypothesis of equality of variances should be tested: H0: 21 ¼ 22 ¼ ¼ 2k H1 : At least one 2i is different

With the variances in Table 14, the statistics of the Cochran’s test (Equation (59)) is Gcalc ¼ 268.67/ 617.168 ¼ 0.435. As G0.05,k,n1 ¼ 0.5981 (see Table 9), the statistic does not belong to the critical region (Equation (60)) and there is no evidence to reject the null hypothesis at 5% significance level. The statistics of the Bartlett’s test is w2calc¼1.792 (Equation (61)) and the critical value is w20.05,4¼9.488, so there is no evidence to reject the null hypothesis either (Equation (64)). The same happens with the Levene’s test; computing the absolute values, according to Equation (65), of the data of Table 14, Fcalc ¼ 14.70/44.01 ¼ 0.33. As F0.05,4,15 ¼ 3.06, there is no evidence to reject the null hypothesis on the equality of variances. By using the median instead of the mean (Equation (68)), Fcalc ¼ 15.13/44.23 ¼ 0.34, and the conclusion is the same. From the analysis of the equality of variances, we can conclude that the variances of the five levels should be considered as equal. The ANOVA of the experimental data gives the results in Table 15. Considering the critical region defined in Equation (81), as Fcalc ¼ 114.54 is greater than the critical value F0.05,4,15 ¼ 3.06, we reject the null hypothesis and hence the conclusion is that the effect of the factor ‘fiber composition over extraction’ is significant at 0.05 level. 1.02.4.2

Power of the ANOVA for the Fixed Effects Model

The power of the ANOVA is computed by the following expression: n o 1 – ¼ pr Fk – 1;N – k; > F;k – 1;N – k

ð82Þ

where F;k – 1;N – k is the critical value of Equation (81), Fk – 1;N – k; is a noncentral F distribution with k 1 and N k d.f. of the numerator and denominator, respectively, and is the noncentrality parameter, whose value is given by Pk ¼n

2 i¼1 i 2

ð83Þ

Table 15 Results of the ANOVA for data of Table 14 Source

Sum of squares

d.f.

Mean squares

Fcalc

Between fibers Error (within fibers) Total

56 551.3 1 851.5 58 402.8

4 15 19

14 137.8 123.4

114.54


49

The noncentrality parameter P depends on the number of replicates and also on the difference in means that we wish to detectP in terms of ki¼1 i2 . Furthermore, the error variance is usually unknown. In such cases, we must choose ratios ki¼1 i2 =2 that we wish to detect. As the power, 1 , of the test increases with , we next ask what is the minimum subject to the condition that two of the i differ by or more. The minimum is obtained if two of the i differ by and the remaining k 2 equal the mean of these two;46 therefore, k X

i2 ¼

i¼1

2 2

ð84Þ

For example, with the data of Example 16 (Table 14), we are now interested in the risk to affirm that the type of composition fiber is not significant for the recovery. The answer consists of evaluating the probability by Equation (82). Suppose that we want to discriminate P effects greater than twice the MSE, that is, ki¼1 i2 =2 2, so ¼ n 2 ¼ 8, F0:05;4;15 ¼ 3:06 and ¼ 0.54 (calculations can be seen in Example A14 of Appendix). To put in words, 54 out of 100 times we will accept the null hypothesis (there is no effect of fiber) when it is wrong. This is not good enough for a useful decision rule. According to Equation (84), our value means that we want to discriminate a difference at least equal to 2 between the two types of fiber. Another question related to the above equation is to know the sample size before starting an experiment. In many situations, we would like to know what the sample size should be to assess whether both risks and will be good enough. Also, the following question can be answered: How P many replicates should be carried out in the experiment for ¼ ¼ 0.05 maintaining the approximation ki¼1 i2 =2 3? Note that, in this case, the analyst considers ‘effect of fiber type’ if it is greater than 3 times p MS ffiffiffi E, which is equivalent, using Equation (84), to detect a difference between two fibers at least equal to ¼ 6 2:5. To calculate the sample size, a table must be made to write as a function of n in Equation (82) with k, , and fixed at 5, 0.05, and 3 n, respectively. Following the results shown in Table 16, we need n ¼ 8 replicates with each fiber to achieve 0.05, but in practice n ¼ 7 suffices. 1.02.4.3

Uncertainty and Testing of the Estimated Parameters for the Fixed Effects Model

It is possible to derive estimators for the parameters and i (i ¼ 1,. . .,k) in the one-way ANOVA modeled by Equation (74). The normality assumption on the errors is not needed to obtain an estimate by least squares; however, the solution is not unique, so the constraint of Equation (75) is imposed. Using this constraint we obtain the estimates ˆ ¼ x and î ¼ xi – x; i ¼ 1; . . . ; k

ð85Þ

where x and xi have been defined in Equations (77) and (76), respectively. If the number of replicates, nP i, in each level is not equal (unbalanced ANOVA), then the constraint of Equation (75) should be changed by ki¼1 ni i ¼ 0 and use the weighted average of xi instead of the unweighted average in Equation (85). Now, if we assume that the errors are NID(0,2) and ni ¼ n, i ¼ 1,. . .,k, the estimates of Equation (85) are also the maximum likelihood ones. For unbalanced designs, the least squares solution is biased and the maximum likelihood is better. The reader interested in this subject should consult statistical monographs that describe this matter at a high level, such as Milliken and Johnson48 and Searle.49 The mean of the ith level is i ¼ þ i, i ¼ 1,. . .,k. In our case, with a balanced design an estimator of i would be ˆ i ¼ ˆ þ î ¼ xi and, as errors are NID(0,2), then xi is NID(i,2/n). Using MSE as an estimator of 2, Equation (16) gives the confidence interval at (1 ) 100% level: "

rffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffi# MSE MSE ; xi þ t =2;N – k xi – t =2;N – k n n

ð86Þ

Table 16 Probability of type II error, , as a function of the number n of replicates in the ANOVA of the fiber types n

4 0.347

5 0.203

6 0.111

7 0.058

8 0.029

9 0.014


A (1 ) 100% confidence interval on the difference in any two level means, say i – j , would be "

xi – xj – t=2;N – k

rffiffiffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffiffiffi#

2MSE 2MSE ; xi – xj þ t=2;N – k n n

ð87Þ

With the data in Example 16 (Tablep 14), a 95% confidence interval on the difference between fibers 1 and 2 is ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi given by [(489.75 602.25) 2.131 2 123:43=4] ¼ [–129.24,–95.76]. Finally, the (1 ) 100% confidence interval on the global mean is "

rffiffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffiffi# MSE MSE ; x þ t =2;N – k x – t =2;N – k nk nk

ð88Þ

Rejecting the null hypothesis in the fixed effect model of the ANOVA implies that there are differences between the k levels, but the exact nature of the differences is not specified. To treat this question, two procedures are used: the orthogonal contrasts and the multiple test. Case 1. Orthogonal contrasts: For example, with the data of Table 14, we would like to test the hypothesis H0: 4 ¼ 5. The linear relation related with this hypothesis is x4 – x5 ¼ 0; this linear combination is called a contrast. A contrast is tested by comparing its sum of squares to the mean square error. The resulting statistic would be distributed as an F, with 1 and N k d.f. Each contrast is defined by the coefficients of the linear combination, P in the previous case (0,0,0,1,–1). Two contrasts C ¼ (c1,c2,. . .,ck) and D ¼ (d1,d2,. . .,dk) are orthogonal if ki¼1 ci di ¼ 0. There are many ways to choose the orthogonal contrast coefficients for a set of levels. Usually, something in the experiment should suggest which comparison will be of interest. In order to show the procedure, we are going to raise with the data of Table 14 a fictitious case of orthogonal contrasts with purely didactic purpose; in each problem, its peculiarities and the previous knowledge of the analyst will suggest the contrast to be studied. The comparisons between the means per fiber and the associated orthogonal contrasts proposed are H0 : 4 ¼ 5

C1 ¼ – x4 þ x5

H0 : 1 þ 3 ¼ 4 þ 5

C2 ¼ x1 þ x3 – x4 – x5

H0 : 1 ¼ 3

C3 ¼ x1 – x3

H0 : 42 ¼ 1 þ 3 þ 4 þ 5

C4 ¼ – x1 þ 4 x 2 – x3 – x4 – x5

The sum of squares associated with each contrast C is SSC ¼

n

P

k i i¼1 ci x

2 ð89Þ

Pk

2 i¼1 ci

For example, SSC1 ¼ 4 (–1 601.00 þ 1 491.50)2/2 ¼ 23 981 with 1 d.f. These sums of squares are incorporated into the ANOVA table as shown in Table 17. Now, to test each of the hypotheses written in the table, it suffices to compare the corresponding Fcalc with the critical value F0.05,1,15 ¼ 4.54. Except for C3, the rest of the contrasts are significant according to Equation (81). Thus, we should reject the hypothesis that fibers 4 and 5 give the same recovery. The hypothesis that the mean of the Table 17 ANOVA table with orthogonal contrasts for composition of fibers for SPME Source

Sum of squares

d.f.

Mean squares

Fcalc

Between fibers C1: 4 ¼ 5 C2: 1 þ 3 ¼ 4 þ 5 C3: 1 ¼ 3 C4: 42 ¼ 1 þ 3 þ 4 þ 5 Error (within fibers) Total

56 551.3 23 981.0 10 668.0 153.1 21 550.0 1 851.5 58 402.8

4 1 1 1 1 15 19

14 137.8 23 981.0 10 668.0 153.1 21 550.0 123.4

114.54 194.28 86.43 1.24 174.59


51

recovery for fibers 1 and 3 is the same as for fibers 4 and 5 is also rejected. Also, fiber 2 differs significantly from the mean of the other four, whereas there is no experimental evidence to reject that fibers 1 and 3 provide the same recovery. It is evident that the possibilities of the analysis of the experimental results are very ample. Case 2. Comparison of several means: Many different methods have been described that were specifically designed for the comparison of several means. Here, we will describe the method of Newman–Keuls. The hypothesis test is the following: H0 : All the differences two by two are equal to zero H1 : At least one difference is nonnull

The procedure consists of the following steps: 1. To sort the means per level, xi , i ¼ 1,2,. . .,k, in decreasing order, xr ð1Þ xr ð2Þ xr ðkÞ . The subindex r(i) refers to the rank of the corresponding mean, that is, the position that it occupies in the ordered list. For example, the means of Table 14 have the following ranks: r(1) ¼ 2, r(2) ¼ 4, r(3) ¼ 3, r(4) ¼ 5, and r(5) ¼ 1, respectively. That means that the first one, which is 489.75, has rank 5, that is, is in the fifth position in the decreasing ordered list. Table 18 shows the ordered means and the ranks in the second column. 2. To create a table with the differences between the means from greatest to lowest being t equal to the difference of ranks plus one. Table 19 contains all the possible contrasts two by two of the means, grouped in each column by the number, t (number of means that separates them plus one), in the ordered list. We wish to test each of the following hypotheses: H0 : xrðiÞ – xrðiþk – 1Þ ¼ 0 H1 : xrðiÞ – xrðiþk – 1Þ > 0

The statistic is Rt ¼ q ðt; kðn – 1ÞÞ

rffiffiffiffiffiffiffiffiffiffi MSE n

ð90Þ

The value of the statistic depends, as always, on the significance level , on t, and on the d.f. N-k of MSE. Further, the first term in Rt changes with the difference of ranks, t. The corresponding values are written in the last row in Table 19 and some of their tabulated values are in Table 20. Table 18 Newman–Keuls procedure for multiple comparison test; data of SPME fibers Levels

Rank

Mean

Homogeneous groups

2 4 3 5 1

1 2 3 4 5

602.25 601.00 498.50 491.50 489.75

Table 19 Skeleton for using the corresponding tabulated values for the Newman– Keuls procedure t: difference of ranks plus one k

k–1

...

2

xr ð1Þ – xr ðkÞ

xr ð1Þ – xr ðk – 1Þ xr ð2Þ – xr ðkÞ

... ... .. .

xr ð1Þ – xr ð2Þ xr ð2Þ – xr ð3Þ ... xr ðk – 1Þ – xr ðkÞ

q ðk; kðn – 1ÞÞ

q ðk – 1; kðn – 1ÞÞ

...

q ð2; kðn – 1ÞÞ

The subscript r(i) indicates the ith rank. k is the number of levels in the ANOVA and q are the tabulated values at significance level .

52 Quality of Analytical Measurements: Statistical Methods for Internal Validation Table 20 Values of q(t,), the upper percentage points of the Studentized range for ¼ 0.05 t

2

3

4

5

6

7

8

9

10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

17.969 6.085 4.501 3.926 3.635 3.460 3.344 3.261 3.199 3.151 3.113 3.081 3.055 3.033 3.014 2.998 2.984 2.971 2.960 2.950

26.98 8.33 5.91 5.04 4.60 4.34 4.16 4.04 3.95 3.88 3.82 3.77 3.73 3.70 3.67 3.65 3.63 3.61 3.59 3.58

32.82 9.80 6.82 5.76 5.22 4.90 4.68 4.53 4.41 4.33 4.26 4.20 4.15 4.11 4.08 4.05 4.02 4.00 3.98 3.96

37.08 10.88 7.50 6.29 5.67 5.30 5.06 4.89 4.76 4.66 4.57 4.51 4.45 4.41 4.37 4.33 4.30 4.28 4.25 4.23

40.41 11.74 8.04 6.71 6.03 5.63 5.36 5.17 5.02 4.91 4.82 4.75 4.69 4.64 4.59 4.56 4.52 4.49 4.47 4.45

43.12 12.44 8.48 7.05 6.33 5.90 5.61 5.40 5.24 5.12 5.03 4.95 4.88 4.83 4.78 4.74 4.70 4.67 4.65 4.62

45.50 13.03 8.85 7.35 6.58 6.12 5.82 5.60 5.43 5.30 5.20 5.12 5.05 4.99 4.94 4.90 4.86 4.82 4.79 4.77

47.36 13.54 9.18 7.60 6.80 6.32 6.00 5.77 5.59 5.46 5.35 5.27 5.19 5.13 5.08 5.03 4.99 4.96 4.92 4.90

49.07 13.99 9.46 7.83 6.99 6.49 6.16 5.92 5.74 5.60 5.49 5.39 5.32 5.25 5.20 5.15 5.11 5.07 5.04 5.01

t is the difference of ranks plus one. is the degrees of freedom of MSE. Adapted from Sachs, L. Applied Statistics. A Handbook of Techniques; New York: Springer-Verlag, 1982.

The values of q ðt ; kðn – 1ÞÞ are tabulated. Table 20 shows some of them. The critical region is:

CR ¼ xrðiÞ – xrðiþk – 1Þ Rt

ð91Þ

The results obtained when applying the method of Newman–Keuls to the data of Example 16 are given in Table 21. The first column contains the means to be compared, for example, 1–2 indicates that the comparison is between x1 and x2 ; the second column contains the differences (without sign) between them; the values of t (difference of ranks plus one) are in the third column, as x1 has rank 5 and the rank of x2 is 1 (the ranks are in Table 18), the value of t is 5. With the value of q (Table 20) and Equation (90), the critical value (Rt) is computed for each of the tests. The critical value Rt allows the analyst to decide whether the estimated difference is significant or not. The decision of rejecting or not rejecting the null hypothesis is shown in the last column of Table 21.

Table 21 Results of the Newman–Keuls test to data of SPME fibers Contrast levels

Differences xi – xj

t

q0.05,t,15

Critical values, Rt

CR ¼ xr ðiÞ – xr ðiþk – 1Þ Rt

1–2 1–3 1–4 1–5 2–3 2–4 2–5 3–4 3–5 4–5

112.5 8.75 111.25 1.75 103.75 1.25 110.75 102.5 7.0 109.5

5 3 4 2 3 2 4 2 2 3

4.37 3.67 4.08 3.01 3.67 3.01 4.08 3.01 3.01 3.67

4.37 5.555 ¼ 24.27 3.67 5.555 ¼ 20.39 4.08 5.555 ¼ 22.66 3.01 5.555 ¼ 16.72 3.67 5.555 ¼ 20.39 3.01 5.555 ¼ 16.72 4.08 5.555 ¼ 22.66 3.01 5.555 ¼ 16.72 3.01 5.555 ¼ 16.72 3.67 5.555 ¼ 20.39

Reject H0 No evidence to reject H0 Reject H0 No evidence to reject H0 Reject H0 No evidence to reject H0 Reject H0 Reject H0 No evidence to reject H0 Reject H0

H0: the difference is null, x¯ix¯j ¼ 0; H1: x¯ix¯j 6¼ 0.


53

Usually, the result of this multiple comparison is presented as in the last column of Table 18. The symbols ‘’ aligned indicate that the corresponding means are all equal two by two. For example, the means x2 and x4 by one side and any other pair among x1 , x3 , and x5 on the other side, as can be seen in Table 21. It is possible to conclude that there are two fiber groups; as far as the recovery is concerned, fibers 2 and 4 provide results that are significantly equal and greater than the others. The other group is made by the other three that are similar to each other but different from the two previous ones. 1.02.4.4

The Random Effects Model

In many cases, the factor of interest is a random variable as well, so that the chosen levels are in fact a sample of this random variable and we want to extract conclusions about the population from which the sample comes. For example, in the case of validating an analytical method, several laboratories will apply it to aliquot samples so that it is possible to decide what part of the variability of the results is attributable to the change of laboratory and what to the repetition of the procedure inside the same laboratory. These are the concepts of reproducibility and repeatability. The same happens in the analytical control of processes: It is necessary to distribute the variability observed between the one due to the measurement procedure and the one assignable to the process. The linear statistical model is xij ¼ þ i þ "ij

with i ¼ 1; 2; . . . ; k; j ¼ 1; 2; . . . ; n

ð92Þ

where i and "ij are independent random variables. Note that the model is identical in structure to the fixed effect case (Equation (74)), but the parameters have a different interpretation. If the V( i) ¼ 2 , then the variance of any observation is

V xij ¼ 2 þ 2

ð93Þ

The variances of Equation (93) are called variance components, and the model, Equation (92), is called components of variance or the random effects model. To test hypotheses in this model, we require that the "ij are NID(0,2), all of the i are NID(0,2 ), and i and "ij are independent. The sum of squares equality SST ¼ SSF þ SSE still holds. However, instead of testing the hypothesis about individual levels effects, we test the hypothesis H0 : 2 ¼ 0 H1 : 2 > 0

If 2 ¼ 0, all levels are identical; if 2 > 0, then there is variability between levels. Thus, under the null hypothesis, the ratio Fcalc

SSF MSF k ¼ –1 ¼ SSE MSE N –k

ð94Þ

is distributed as an F with k 1 and N k d.f. The expected means of MSF and MSE are E ðMSF Þ ¼ 2 þ n2

ð95Þ

E ðMSE Þ ¼ 2

ð96Þ


ð97Þ

and

Therefore, the critical region is

1.02.4.5

Power of the ANOVA for the Random Effects Model

The power of the random effects ANOVA is obtained from F;k – 1;N – k 1 – ¼ pr Fk – 1;N – k > 2

ð98Þ

54 Quality of Analytical Measurements: Statistical Methods for Internal Validation 2

where 2 ¼ 1 þ n 2 . As 2 is usually unknown, we may either use a prior estimate or define the value of 2 that we are interested in detecting in terms of the ratio 2 =2 . An application to determine the number of replicates in a proficiency test can be seen in Example A15 of Appendix. 1.02.4.6

Confidence Intervals of the Estimated Parameters for the Random Effects Model

In general, the mean value per level xi does not have more statistical meaning than being a sample of the random factor. But sometimes, as in the case of the proficiency tests, this mean value is of interest for each laboratory participant. The variance of the mean value per level is theoretically equal to V ðxi Þ ¼ 2 þ 2 =n. From Equations (95) and (96), MSF =n (with k – 1 d.f.) estimates the variance of the mean per level. As a consequence, the 100 (1 )% confidence interval is "

rffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffi# MSF MSF ; xi þ t =2;k – 1 xi – t =2;k – 1 n n

ð99Þ

When calculating the variance for the global mean, it is necessary to consider the variability contributed by the factor, as the factor always acts/. For example, when evaluating an analytical method, the results without the variability attributable to the factor laboratory are not conceivable. The variance of the overall mean is P V ðxÞ ¼ ki¼1 V ðxi Þ=k2 , which is estimated by MSF =nk, with k 1 d.f., in such a way that the 100 (1 )% confidence interval is " x – t =2;k – 1

rffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffi# MSF MSF ; x þ t =2;k – 1 nk nk

ð100Þ

The ANOVA of random effects is a model of practical interest because it allows attributing real meaning to many statements that seem evident. For example, when in a proficiency test the samples are distributed to laboratories, it is insisted that they must be homogeneous. Strictly speaking, in most of the occasions, it is impossible to assure homogeneity, but it is enough that the variability attributable to the change of sample is significantly smaller than the one attributable to the procedure of analysis. This can be guaranteed by means of an ANOVA of random effects.

1.02.5 Statistical Inference and Validation 1.02.5.1

Trueness

The trueness is a key concept; several international organizations are unifying its definition. For example, the definition ‘‘The closeness of agreement between the average value obtained from a large series of test results and an accepted reference value’’ has been adopted by the IUPAC (Incze´dy et al.,11 Chapter 18). The definition of the ISO7 exactly coincides with it, and it is the definition accepted by the European Union in the Decision 2002/657/EC3 as far as the operation of the analytical methods and the interpretation of results are concerned. The trueness is usually expressed in terms of bias, which combines all the components of the systematic error, denoted by in Equation (1). The decisions on the trueness of a method are problems of hypothesis testing on the central value of a distribution; in case the random error can be assumed to have a zero mean, they are in fact tests on the mean because, according to Equation (1), the expected mean value for a series of measurements will be þ and the problem is reduced to test whether is zero or not (equivalently, to test whether x is significantly equal to ). To use one or another test depends only on the information available about the distribution of the random error – its type (normal, parametric, or unknown) and, in the case of normality, whether the variance is known or not. Some common cases are given below: 1. To decide whether an analytical procedure fulfills trueness by the use of a reference sample whose value is assumed to be true. If normal data with known variance 2 are supposed, then the tests of Section 1.02.3.2 will be used.


55

2. To decide whether an analytical procedure has bias specifically positive (or negative) by the use of a reference sample whose value is assumed to be true. If normal data are assumed, the one-tail test versions of Cases 1 and 2 of Section 1.02.3.2 will be of use. When a hypothesis test is to be posed, one can think about omitting some known data, for example the variance. The effect is a loss of power, that is, with the same value of and the same sample size, the probability of type II error will be greater. Said otherwise, if it is desired to have the same power, larger sample sizes are needed to acquire the same experimental evidence; a calculation on this matter is in Case 2 of Section 1.02.3.2. The same applies about the use of one-tail tests (2, 3, 5, and 6 in Table 4) respect to the respective two-tail tests (1 and 4); or about the use of nonparametric tests that do not impose any type of distribution a priori. 3. In other occasions, the question of trueness is considered comparatively between two methods: ‘To decide if the difference in means between both, when they are applied to the same sample of reference, is significant or not’. It is the two-tail test. The one-tail case is ‘to decide whether one has bias of specific sign against the other’. In these tests (Section 1.02.3.4), two experimental means are compared, one coming from applying n1 times the first method to the reference sample (aliquot parts) and the other one coming from applying n2 times the second method. Under the normality assumption, we will have to know whether the variances of both methods are known (tests 7 and 8 of Table 4) or it is necessary to estimate them from the sample. In addition, in the second case we have to decide whether they are equal (tests 9 and 10 of Table 4) or different (Equation (53)). 4. Sometimes it is impossible to use reference samples similar enough; a solution is the use of the ‘test on the difference of means with paired data’ (Case 3 of Section 1.02.3.2). For example, suppose that we wish to introduce a new method of analysis on line, faster, to indirectly determine the content of an analyte in wastewater of a company, and we need to decide whether it maintains the trueness at the same level as the previous method, which is sufficiently proved. Once the new method is ready and validated with reference samples, real samples have to be measured. The difficulty is that we cannot be sure about the amount that is to be found because this may vary from day to day. In order to eliminate the ‘factor sample’, paired designs are used: Both methods are applied to aliquot parts of the same sample, and two series of paired results x1i and x2i are obtained when applying the old and new methods, respectively. To compute the mean here has no sense because in that case we would be introducing variability due to the change of sample in each series. The correct procedure is to subtract both values di ¼ x1i x2i; now the differences are caused exclusively by the change of method and its mean will estimate the bias attributable to the new method. It will be enough to apply a test on the mean; thus, the normality and independence hypotheses must be evaluated on the differences, as well as the standard deviation, which is also estimated with them. This test for paired data is frequently used to evaluate the improvement achieved in a procedure by a technical variation, as is the case of Example 9. The effect of the change on the trueness must always be evaluated in the range of concentrations in which one is going to use the procedure. An alternative to the use of this test is the analysis of the pairs of data by a linear regression; in this case, the regression method used should consider the existence of error in both axes (see Section 1.05.10). The normality (Section 1.02.3.8), independence of the determinations, and, if it is necessary the equality of variances hypotheses will have to be tested with the appropriate tests (Section 1.02.3.7). Also, it is important to remember that the presence of outlier data tends to greatly increase the variance so that the tests become insensitive, that is to say, a larger experimental evidence is needed to reject the null hypothesis. The nonparametric alternative has, in general, a high cost in terms of power for the same significance level (or in terms of sample size). For this reason, it will not have to be used unless it is strictly necessary. In addition, some nonparametric tests also assume hypothesis on the distribution of the values, for example, to be symmetric or unimodal.

1.02.5.2

Precision

The other very important criterion in the validation of a method is the precision. In the ISO 5725,7 the IUPAC (Incze´dy et al.,11 Sections 2 and 3), and the European norm,3 we can read ‘‘Precision, the closeness of agreement between independent test results obtained under stipulated conditions’’.


The precision usually is expressed as imprecision. The lesser the dispersion of the random component in Equation (1), the more precise the procedure. It must be remembered that the precision depends solely on the distribution of the random errors and is not related to the value of reference or that assigned to the sample. In a first approach, it is computed as a standard deviation of the results; nevertheless, even the ISO 5725-5 recommends the use of a robust estimation. Two measures, limits in a certain sense, of the precision of an analytical method are the reproducibility and the repeatability. Repeatability is defined as ‘‘precision under repeatability conditions. Repeatability conditions means conditions where independent test results are obtained with the same method on identical test items in the same laboratory by the same operator using the same equipment with short intervals of time’’. The repeatability limit, r, is the value below which is, with a probability of 95%, the absolute value of the difference between two results of a test, obtained under repeatability conditions. The repeatability limit is given by r ¼ z=2

pffiffiffi 2 r

ð101Þ

where z/2 is the critical value of the standard normal distribution. r is the repeatability expressed as standard deviation. Reproducibility is defined as ‘‘precision under reproducibility conditions. Reproducibility conditions means conditions where test results are obtained with the same method on identical test items in different laboratories with different operators using different equipment’’. The reproducibility limit, R, defined in Equation (102), is the value below which is, with a probability of 95%, the absolute value of the difference between two results of a test, obtained under reproducibility conditions. R ¼ z=2

pffiffiffi 2 R

ð102Þ

where R is the standard deviation computed under reproducibility conditions. If n < 10, a correction factor7 should be applied to Equation (6) when r (or R) is estimated. The ISO introduces the concept of intermediate precision when only some of the factors described in the reproducibility conditions are varied. A particular interesting case is when the ‘internal’ factors of the laboratory (analyst, instrument, day) are varied, which in Commission Decision3 is called intralaboratory reproducibility. One of the causes of the ambiguity when defining precision is the laboratory bias. When the method is applied only in a laboratory, the laboratory bias is a systematic error of that laboratory. If the analytical method is evaluated in general, the laboratory bias becomes a part of the random error: to change the laboratory contributes to the variance, which is expected for a determination done with that method in any laboratory. The most eclectic position is the one described in the ISO 5725 that declares ‘‘The laboratory bias is considered constant when the method is used under repeatability conditions, but is considered as a random variable if series of applications of the method are made under reproducibility conditions.’’ On the basis of these points, we can deduce that to evaluate the precision of an analytical method is equivalent to estimating the variance of the random error in the results and that the discrepancies that an appear when establishing the sources of variability must be explicitly identified, for example the laboratory bias. The precision of two methods can be compared by a hypothesis test on the equality of variances, under the normality assumption that it is a (Snedecor) F-test (Section 1.02.3.6). Another usual problem is to decide whether the variance observed can be considered significantly equal or not to an external value, using a w2 test (Section 1.02.3.3). It is common that the lack of control on a concrete aspect of an analytical procedure is the origin of a great variability. If the experimental conditions are not stable, we will have an additional variability in the determinations. The F-test permits one to decide whether the precision improves significantly when an optimization is carried out. In fact, many improvements in the procedures are due to the identification of the causes of the variability in the results and their quantification. This aspect of control and improvement of the precision is dealt with some detail in the section dedicated to the ruggedness of the chemical analyses.


57

The technique used to construct an ANOVA with random effects (Section 1.02.4.4) is also the adequate technique to split the variance that presents each experimental data in addends that are specially adapted to estimate the repeatability and the reproducibility of an analytical method when interlaboratory test comparison has been carried out. In the following, the use of an ANOVA to estimate reproducibility and repeatability in a proficiency test is briefly explained. There is no doubt that a good analytical procedure has to be insensitive to the change of laboratory. To decide about this question, k laboratories apply a procedure to aliquot samples; each laboratory makes n determinations. In the terminology of the ANOVA, we have a random factor (the laboratory) at k levels and n replicates in each level. In general, it is not necessary to have the same number of replicates in all the levels. We denote by xij the experimental results, where i ¼ 1,. . .,k identifies the laboratory and j ¼1,. . .,n the replicate. Figure 5 is a skeleton of Equations (93)–(96) and shows how to compute an estimate of the variance of the random variable " in Equation (93). If the analytical procedure is well defined, the k estimates si2 are expected to be approximately equal and to gather the variability due to the use of the analytical method by only one laboratory. In these conditions, the pooled variance sp2 is a joint (‘pooled’) estimate of the same variance, that is, by definition, the repeatability of the method expressed as standard deviation (ISO 5725) sr ¼

pffiffiffiffiffiffiffiffiffiffi V ð"Þ sp

ð103Þ

From the same data we can obtain k estimates of the bias i (Figure 5, top) and then the variance of the laboratory bias, considering this bias as a random variable. Taking into account the quantities estimated by the variances described in Figure 5, one obtains the following expression for the interlaboratory variance: V ðÞ sx2 –

Variance n s x2 with k –1 degrees of freedom It estimates V(ε) + n V(Δ), and thus the reproducibility is sR = V(ε) + V(Δ)

sp2

ð104Þ

n

If the variability due to the change of laboratory is not significant, it also estimates the intralaboratory variance ⇑

Variance of the means, V(Δ)

∑ i =1( xi − x ) 2 k

s x2 =

k−1 ⇑

Mean estimated per laboratory

Determinations made by each laboratory xij = μ + Δi + εij

Variance estimated per laboratory

Pooled variance

x1

x2

x3

...

xk

⇑ x11 x12 x13 . . . x1n

⇑ x21 x22 x23 . . . x2n

⇑ x31 x32 x33 . . . x3n

... ... ... ... ... ... ...

⇑ xk1 xk2 xk3 . . . xkn

⇓

⇓

⇓

s 12

s 22

s 32

⇓ ...

s k2

⇓ 2 + (n – 1) s 2 + … + (n – 1) 2 – 1) (n s sk 2 1 ≈ V(ε) s p2 = k(n – 1) ⇓

Intralaboratory variance = s p2 k(n–1) degrees of freedom It estimates V(ε) Figure 5 ANOVA of random effects for a interlaboratory study.

sp estimates the repeatability, sr


which, linked to Equation (1), provides the following estimate of the reproducibility as standard deviation (ISO 5725): sR

pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi V ðÞ þ V ð"Þ

ð105Þ

In the ANOVA, the null hypothesis is that 1 ¼ 2 ¼ ¼ k ¼ 0 (i.e., there is no effect of the factor), and the alternative is that at least one laboratory has nonnull bias (there is effect of the factor). The conclusion of the ANOVA is obtained by deciding whether both variances, n sx2 and sp2 , can be considered significantly equal. To decide it, an F-test is applied. The logic is clear: If there is no effect (laboratory effect), V() should be significantly zero and, thus, both variances are equal, or in other words, they estimate the same quantity. In practice, the expression for the computation of the power of the ANOVA with random effects (Equation (98)) is useful in deciding the number of laboratories that should participate, k, and the number of replicated determinations, n, that each one must do. It is essential to remember that an ANOVA requires normal distribution of the residuals and equality of the variances s21,s22,. . .,sk2. When the number of replicates is two (n ¼ 2), a common way of the interlaboratory analysis is the use of the graph of Youden50 to show the trueness and precision of each laboratory. Actually, the Youden’s graph is none other than the graphical representation of an ANOVA shown in Kateman and Pijpers51 (p 112). In addition to being used for comparing the quality of the laboratories, the graph of Youden can be used to compare two methods of analysis in terms of the laboratory bias they have. An approach for the comparison of two methods in the intralaboratory situation has been proposed by Kuttatharmmakull et al.52 Instead of the reproducibility, as included in Figure 5 and ISO guidelines, the (operator þ instrument þ time) different intermediate precision is considered in the comparison. In the case of precision, the effect of outlier data is really devastating; hence, a very delicate task of analysis to detect those outlier data is essential. In general, more than one test is needed (usual ones are those of Dixon, and Grubbs and Cochran), especially to accept the hypotheses of the ANOVA made for the determination of repeatability and reproducibility. In view of the difficulties, the AMC5,6 advises the use of robust methods to evaluate the precision and trueness and for proficiency testing. This path is also followed in the new ISO norm about reproducibility and repeatability. 1.02.5.3

Statistical Aspects of the Experiments to Determine Precision

The analysis of data implies three steps: 1. Critical examination of the data, in order to identify outliers or other irregularities, and to verify the suitability of the model. 2. To compute for each level of concentration the preliminary values of precision and the mean values. 3. To establish the final values of precision and means, including the establishment of a relation between precision and the level of concentration when the analysis indicates that such relation can exist. The analysis includes a systematic application of statistical tests for detecting outliers, and a great variety of such tests are available from the literature and could be used for this task.

1.02.5.4

Consistency Analysis and Incompatibility of Data

From the data collected in a specific number of levels, a decision must be taken about certain individual results or values that seem to be ‘different’ from those of the rest of laboratories or that can vary the estimations. Specific tests are used for the detection of these outlier numerical results. Case 1. Elimination of data: It is the classic procedure based on detecting and, if this is the case, eliminating the outlier data. The tests are of two types. The test of Cochran is related to the interlevels variability of the factor


59

and should be applied first; its objective is to detect an anomalous variance in one or several of the levels of the factor. The test of Cochran has already been described in Section 1.02.3.7. Later the test of Grubbs will be applied. It is basically a test on the intralevel variability to discover possible outlier individual data. It can be used (if ni > 2) for those levels in which the test of Cochran has led to the suspicion that the interlevel variation is attributable to an individual result. It is applied in two stages: 1. Detection of a unique outlying observation (single Grubbs’ test) In a data set xi (i ¼ 1,2,. . .,n) sorted in increasing order, to prove whether the greatest observation, xn, is incompatible with the rest, the following statistic is computed: Gn;calc ¼

xn – x s

ð106Þ

On the contrary, to verify whether the smallest observation, x1, is significantly different from the rest, the statistic G1 is computed as G1;calc ¼

x – x 1 s

ð107Þ

In Equations (106) and (107), x and s are, respectively, the mean and standard deviation of xi. To decide whether the greatest or smallest value is significantly different from the rest at % significance level, the values obtained in Equations (106) and (107) are compared to the corresponding critical values written down in Table 22. The decision includes two ‘anomaly levels’: (a) If Gi,calc < G0.05,i, with i ¼ 1 or i ¼ n, accept that the corresponding x1 or xn is similar to the rest. (b) If G0.05,i < Gi,calc < G0.01,i, with i ¼ 1 or i ¼ n, the corresponding x1 or xn is considered an straggler. (c) If G0.01,i < Gi,calc, with i ¼ 1 or i ¼ n, the corresponding x1 or xn is incompatible with the rest of data of the same level (statistical outlier). Table 22 Critical values for Grubbs’ test One largest or one smallest

Two largest or two smallest

n

¼ 0.05

¼ 0.01

¼ 0.05

¼ 0.01

4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1.481 1.715 1.887 2.020 2.126 2.215 2.290 2.355 2.412 2.462 2.507 2.549 2.585 2.620 2.651 2.681 2.709

1.496 1.764 1.973 2.139 2.274 2.387 2.482 2.564 2.636 2.699 2.755 2.806 2.852 2.894 2.932 2.968 3.001

0.0002 0.0090 0.0349 0.0708 0.1101 0.1492 0.1864 0.2213 0.2537 0.2836 0.3112 0.3367 0.3603 0.3822 0.4025 0.4214 0.4391

0.0000 0.0018 0.0116 0.0308 0.0563 0.0851 0.1150 0.1448 0.1738 0.2016 0.2280 0.2530 0.2767 0.2990 0.3200 0.3398 0.3585

Adapted with permission from ISO-5725–2. Accuracy, Trueness and Precision of Measurement Methods and Results; Ge`neve, 1994; pp. 22, Table 5.


2. Detection of two outlying observations (double Grubbs’ test) Sometimes it is necessary to verify that two extreme data (very large or very small) incompatible with the others do not exist. In the case of the two greatest observations, xn and xn–1, the statistic G is computed as G¼

sn2 – 1;n s02

ð108Þ

2 s1;2 s02

ð109Þ

Pn – 2 2 P P 2 1 where s02 ¼ ni¼1 ðxi – xÞ2 and sn1;n ¼ n2 i¼1 xi – n – 2 i¼1 xi . Similarly, it is possible to decide jointly on the two smallest observations, x1 and x2, by means of the following statistic: G¼

2 P P 2 where s1;2 ¼ ni¼3 xi – n –1 2 ni¼3 xi . The decision rule is analogous to that of the case of an extreme value but with the corresponding critical values in Table 22. In general, the norms, for example ISO 5725,7 propose to inspect the origin of the anomalous results and, if assignable cause does not exist eliminate the incompatible ones and leave the straggler ones indicating their condition with an asterisk. Example 17: With didactic purpose to apply the test of Grubbs and to verify the effect of outliers, the data of Table 23 have been considered as a unique series of 20 results. The greatest value is 15.93 and the lowest is 13.03, as s ¼ 0.60 and x ¼ 13:60, Equation (106) gives G20,calc ¼ 3.889 and Equation (107) gives G1,20 ¼ 0.942. By consulting the critical values in Table 22, G0.05,20 ¼ 2.709 and G0.01,20 ¼ 3.001; therefore, according to the decision rule in Case 1 (single Grubbs’ test), the value 15.93 should be considered different from the rest. Applying again the test, with 19 data, the greatest value now is 13.92 and the lowest is still 13.03, with G19,calc ¼ 1.804 and G1,calc ¼ 1.785. As the tabulated values are G0.05,19 ¼ 2.681 and G0.01,19 ¼ 2.968, there is no evidence to say that any of the extreme values is different from the rest. Table 24 contains the mean and standard deviation, with and without the value 15.93. A large effect is observed on the standard deviation, which is reduced in more than 50%. Table 23 Data of Example 17 Series 1

Series 2

Series 3

Series 4

Series 5

13.50 13.40 13.47 13.49

13.50 13.51 13.35 13.35

13.70 13.71 13.76 13.80

13.04 13.03 15.93 13.04

13.48 13.47 13.92 13.46

Table 24 Robust and nonrobust estimates of the centrality and dispersion parameters (data of Table 23)

Nonrobust procedures Mean, x Standard deviation, s

With all data (n ¼ 20)

Without 15.93 (n ¼ 19)

13.60 0.60

13.47 0.25

13.49

13.48

13.50 0.26 0.27

13.48 0.21 0.24

Robust procedures Median H15, centrality parameter MAD H15, dispersion


61

The test of Grubbs can also be applied to the mean values by level. In practice, the test of Grubbs is also used to restore the equality of variances in the ANOVA when the homogeneity of variances is rejected (Section 1.02.3.7). The work by Ortiz et al.43 contains a complete analysis with sequential application of the tests of Cochran, Bartlett, and Grubbs. Case 2. Robust methods: The procedure described in the previous section is focused on the detection of anomalous data within a set of results. Nevertheless, the elimination of these data is not recommendable when the variability of the analytical procedure is to be evaluated because the procedure is sensitive to the present values, that is to say, it depends on the data that have been eliminated (Equations (106)–(109) can lead to elimination of data in successive stages because of the reduction of the variance), and because the attainable real variance is underestimated. As previously indicated, the values of repeatability (r) and reproducibility (R) are determined by means of an ANOVA whose validity depends on whether the hypotheses of normality and homogeneity of variances are fulfilled. The robust methodology proposed in this section avoids these limitations. Its technical details can be found in Hampel et al.53 and Huber.54 An alternative to the procedures based on the elimination of outlier data, as exposed in the ISO 5725-5 norm, consists in using the H15 estimator proposed by Huber (c ¼ 1.5 and ‘Proposal 2 Scale’, Huber54), recommended by the Analytical Methods Committee5,6 and accepted in the Harmonized Protocol.55 It is an estimator whose influence function is a monotone function that limits the influence of the anomalous data by ‘moving them’ toward the position of the majority, but maintaining for them the maximum influence. This is carried out by transforming the original data by means of the function m:s:c: ðx Þ ¼ max½m – cs; minðm þ cs; x Þ

ð110Þ

where m and s are the central and dispersion parameters, which must be iteratively estimated. The function in Equation (110) is represented in Figure 6. The estimate is exactly the generalization of the maximum-likelihood estimate. It is asymptotically optimum for high-quality data, that is, data with little contamination and not very different from data following a Student’s t distribution with three d.f. Remember that Hampel et al.53 have shown that the Student’s t distributions with between 3 and 9 d.f. reproduce high-quality experimental data, and that for t3 the efficiency of the mean and standard deviation is 50 and 0%, respectively. Therefore, in practice there is the need for a robust estimate of high-quality empirical data (as those provided for the present analytical methods). The H15 estimator provides enough protection against high concentration of data that are abnormally large but near to the correct data. Nevertheless, the clearly anomalous data are not rejected with the H15 estimator, and they maintain the maximum influence but bounded. This produces an avoidable loss of the efficiency of the H15 estimator between 5 and 15% when the proportion of anomalous data present is also between 5 and 15% Ψ(x) m+cs

m

m–cs

m–cs

Figure 6 Function m.s.c.

m

m+cs


(rather usual percentages in routine analyses). In order to avoid this limited weakness, robust estimators as the median and the median of absolute deviation (MAD) (Equation (111)) are necessary at least in the first step of the calculation, to identify surely most of the ‘suitable’ data. MAD ¼ median

jxi – medianfxi gj 0:6745

ð111Þ

The robust procedure obtained when adapting the H15 estimator to the problem of the estimation of repeatability and reproducibility as posed in ISO norm consists of two stages and it has been followed in an identical way to the proposal in Sanz et al.56 As in the parametric procedure, it uses the mean of the table of data and the standard deviation. Therefore, once the robust procedure is applied, the data necessary to estimate the reproducibility or the intermediate precision are at hand. In order to verify the utility of these robust procedures, with the same data of Table 23 considered as a unique series of 20 values, the median and the centrality parameter of the H15 estimator have been written down in Table 24. These are very similar to the nonrobust estimates, as much with 20 as with 19 data. Nevertheless, when comparing the robust parameters of dispersion, MAD and H15, they do not differ when considering 20 or 19 data and are similar to the standard deviation obtained after applying the method of Grubbs and repeating the calculations without the outlier. For this reason, it is a good strategy to apply systematically robust procedures together with the classic ones. The difference in the results is an indication of the presence of outlier data, in which case the robust estimations will have to be used. The effect, and therefore the advantage, of the robust procedures is much more remarkable when an ANOVA of random effects is evaluated, for example, to estimate the reproducibility and repeatability of a method by means of an interlaboratory test as the one described in Figure 5. To show this, we will use the data of Table 23, this time considering its structure of levels of the factor (k ¼ 5) and replicates (n ¼ 4). The values of reproducibility and repeatability should not be accepted if the homogeneity of variances in the ANOVA assumption is not fulfilled. In this case, it is necessary to verify whether some of the levels have outlier data. The first column of Table 25 shows that the ANOVA with all the data is not acceptable because of the lack of equality of variances (rejection in the tests of variance homogeneity). In addition, it is observed that the anomaly in the data causes that the estimates sR and sr are equal and very different from the robust estimates. Once the value 15.93 of series 5 is removed, the ANOVA (column 2 of Table 25) is significant but still there is lack of variance homogeneity. Nevertheless, the new estimates of sR and sr are more similar to those obtained with the robust procedure. The lack of equality of variances forces one to eliminate series 4, which has a very different variance (smaller than the others), and later the value 13.92 of series 5. The final result of this sequential process is the third column of Table 25. The ANOVA is significant, the homogeneity of variances can be accepted, and the estimates of the reproducibility and the repeatability are 0.153 and 0.058, similar to the ones that would be obtained with the robust procedure without series 4. The values sR and sr can be too small due to the elimination of data, with the risk of having underestimations that are not realistic and thus the laboratories cannot fulfill them. For this reason, it is advised to avoid reduction of the sample and to maintain the initial robust estimates.5–7 Table 25 Robust and nonrobust estimates of the repeatability and reproducibility with data of Table 23 ANOVA

With all data (n ¼ 20)

Without 15.93 (n ¼ 19)

Without series 4 and 13.92 (n ¼ 15)

Nonrobust procedure Fcalc (p-value) SSF (d.o.f.) SSE (d.o.f.) sR sr p-value, Cochran’s test p-value, Bartlett’s test p-value, Levene’s test

0.22 (0.92) 0.094 (4) 0.431 (15) 0.657 0.657 8.9 10–9 3.9 10–8 0.53

17.02 ( 0 lim prfjXn – X j > "g ¼ 0

n!1

This means that the probability of the set where Xn differs from X is getting smaller and smaller. Furthermore, we say that Xn converges to X ‘almost surely’ if n o pr lim jXn – X j ¼ 0 ¼ 1 n!1

Almost sure convergence implies that the set of ideal measurements such that the outcomes (real values) of Xn are getting closer and closer to X has probability one. It can be proven that almost sure convergence implies convergence in probability, which then implies convergence in distribution. The following three fundamental results for convergence are the most widely used in practice. The ‘weak law of large numbers’ states that if X1,X2,. . .,Xn,. . . are independent and identically distributed random variables with finite mean , then X1 þ þ Xn ! in probability: n

If the random variables also have a finite variance (a weaker condition is also possible), then we have the ‘strong law of large numbers’, X1 þ þ Xn ! almost surely: n

The ‘central limit theorem’ says that for independent (or weakly correlated) random variables X1,X2,. . .,Xn, with the same distribution, pffiffiffi nðX – Þ ! N ð0; 1Þin distribution

where and 2 are the mean and variance of the random variables Xn. This means that the distributional shape of X is more and more like that of a normal standard random variable as n increases. 7. Some Computational Aspects The accessibility of the personal computers allows doing the statistical calculations without tables. It is advisable to use software specially dedicated to Statistics, but in the initial stages of the learning it is worthwhile to make the calculations manually, so that the intuition necessary to avoid the errors derived from a nonreflexive and automatic use of the software is acquired. The basic distributions can be programmed with the algorithms in Abramowitz and Stegun,60 normal distributions are in Chapter 3.12, the Student’s t and F are in Chapter 3.12, and finally, the chi-square distribution is in Section Chapter 3.12. Chapter 5 (appendices) of the book by Meier and Zu¨nd61 shows the necessary numerical approximations and programs in BASIC for these distributions. To compute the noncentral F, the needed numerical approximation can be consulted in Johnson and Kotz,62 and Evans et al.63 All the calculations in this chapter have been made with the Statistics Toolbox for MATLAB.64 What follows is a list of basic command instructions used. Note that all the commands referring to cumulative distribution functions, equations (A8–A11), compute the cumulative probability until the corresponding value of the distribution. However, along the text and in tables A1–A4, the calculated probability is always the upper percentage point, that is, the cumulative probability above the corresponding value.


71

Normal distribution ¼ prfN ð;Þ < zg

ðA8Þ

– z ¼ norminv(, , ) Example A1: ¼ 0.05, ¼ 0, ¼ 1; then norminv(0.05, 0, 1) gives z ¼ –1.645. – ¼ normcdf(z, , ) Example A2: z ¼ 1.645, ¼ 0, ¼ 1; then normcdf(1.645, 0, 1) gives ¼ 0.95 Student’s t distribution with degrees of freedom

¼ pr t < t;

ðA9Þ

– t ¼ tinv(,v) Example A3: ¼ 0.05, ¼ 5; then tinv(0.05,5) gives t ¼ –2.015. – ¼ tcdf(t,v) Example A4: t ¼ 1.645, ¼ 5; then tcdf(1.645,5) gives ¼ 0.9196. n o ¼ pr w2 < w2;

ðA10Þ

w2 distribution with degrees of freedom

– x ¼ chi2inv(,) Example A5: ¼ 0.05, ¼ 5; then chi2inv(0.05,5) gives x ¼ 1.1455. – ¼ chi2cdf(x,) Example A6: x ¼ 9.24, ¼ 5; then chi2cdf(9.24,5) gives ¼ 0.9001. F1 ;2 distribution 1 and 2 degrees of freedom

¼ pr F1 ;2 < F;1 ;2

ðA11Þ

– x ¼ finv(,1,2) Example A7: ¼ 0.95, 1 ¼ 5, 2 ¼ 15; then finv(0.95,5,15) gives x ¼ 2.9013. – ¼ fcdf(x,1,2) Example A8: x ¼ 2.90, 1 ¼ 5, 2 ¼ 15; then fcdf(2.90,5,15) gives ¼ 0.9499. Power for the z-test, Equation (40)

Example A9. With the data of Example 8, jj ¼ 0.40, ¼ 0.55, n ¼ 10, ¼ 0.05, normcdf(norminv(0.95,0,1) – 0.40 sqrt(10)/0.55) gives 0.2562. Power for the t-test, Equation (43)

Example A10. With the same data as for the z-test, the sentence includes the noncentral t-distributionp‘nctcdf’ ffiffiffiffiffi and the t distribution ‘tinv’, both of them with n – 1 ¼ 9 d.f. and noncentrality parameter ð0:40=0:55Þ 10. nctcdf(tinv(0.95,9),9, 0.40 sqrt(10)/0.55) gives ¼ 0.3165. pffiffiffiffiffi Example A11. With data of Example 9, ¼ 0.05, n ¼ 10, and the noncentrality parameter is 0:57 10. nctcdf(tinv(0.95,9),9, 0.57 sqrt(10)) gives ¼ 0.5034.


Power for the chi-square test, Equation (46)

Example A12. We have ¼ 2, ¼ 0.05, and n ¼ 14 or n ¼ 13 to obtain a value of 0.05. chi2cdf(chi2inv(0.95,13)/(2 2),13) gives ¼ 0.0402. chi2cdf(chi2inv(0.95,12)/(2 2),12) gives ¼ 0.0511. (note that the d.f. equal 14 – 1 ¼ 13 or 13 – 1 ¼ 12). Power for the F-test, Equation (57)

Example A13. Data: ¼ 0.05, n1 ¼ n2 ¼ 9, l ¼ 1/2 ¼ 2 are those of question (3) in Example 13. fcdf(finv(0.975,8,8)/(2 2),8,8)-fcdf(finv(0.025,8,8)/(2 2),8,8) gives ¼ 0.5558. If we look for the sample size n ¼ n1 ¼ n2 so that 0.10, trying some values, we get fcdf(finv(0.975,22,22)/(2 2),22,22)-fcdf(finv(0.025,22,22)/(2 2),22,22) that gives ¼ 0.1115 and fcdf(finv(0.975,23,23)/(2 2),23,23)-fcdf(finv(0.025,23,23)/(2 2),23,23) that gives ¼ 0.0981. Consequently, n ¼ 24. Power for the ANOVA with fixed effects, Equation (82)

Example A14. ¼ 0.05, 1 ¼ 4, 2 ¼ 15, noncentrality parameter ¼ n

P

i2 =2 ¼ 4 2 ¼ 8.

ncfcdf(finv((0.95),4,15),4,15,8) gives ¼ 0.5364 Note that in the ANOVA the levels of factor are k ¼ 5 and n ¼ 4 replicates per level. Power for the ANOVA with random effects, Equation (98)

Example A15. Suppose that 10 laboratories participate in a proficiency test to evaluate a method. Assumed risks are ¼ ¼ 0.05 and it is desirable to detect at least an interlaboratory variance equal to the intralaboratory variance, that is, 2 ¼ 1 þ n2 =2 ¼ 1 þ n 1. With these data k ¼ 10;n ¼ 4;fcdf(finv(0.95,k – 1,k (n – 1))/(1 þ 1 n),k – 1,n (k – 1)) gives ¼ 0.0973 k ¼ 10;n ¼ 5;fcdf(finv(0.95,k – 1,k (n – 1))/(1 þ 1 n),k – 1,n (k – 1)) gives ¼ 0.0494 Thus, each laboratory must do five determinations.

References 1. EURACHEM/CITAC. Guide CG4 In Quantifying Uncertainty in Analytical Measurement, 2nd ed.; Ellison, S. L. R., Rosslein, M., Williams, A., Ed.; 2000. ISBN 0–948926–15-5. Available from the Europchem Secretariat. (see http://www.eurochem.org). 2. Draft Supplement 1 to the ‘Guide to the Expression of Uncertainty in Measurement’. Evaluation of measurement data. Propagation of distributions using a Monte Carlo method (2004). 3. Commission Decision 12 August 2002, Brussels. Off. J. Eur. Commun. L 221 (17 August 2002) 8-36. Implementing Council Directive 96/23/EC concerning the performance of analytical methods and the interpretation of results. 4. Aldama, J. M. Practicum of Master in Advanced Chemistry; University of Burgos: Burgos, Spain, 2007. 5. Analytical Methods Committee. Robust Statistics – How Not to Reject Outliers, Part 1. Basic Concepts. Analyst 1989, 114, 1693–1697. 6. Analytical Methods Committee. Robust Statistics – How Not to Reject Outliers, Part 2. Inter-laboratory Trials. Analyst 1989, 114, 1699–1702. 7. ISO 5725. Accuracy Trueness and Precision of Measurement Methods and Results, Part 1. General Principles and Definitions, Part 2. Basic Method for the Determination of Repeatability and Reproducibility of a Standard Measurement Method, Part 3. Intermediate Measures of the Precision of a Standard Measurement Method, Part 4. Basic Methods for the Determination of the Trueness of a Standard Measurement Method, Part 5. Alternative Methods for the Determination of the Precision of a Standard Measurement Method, Part 6. Use in Practice of Accuracy Values. Gene`ve, 1994. 8. Analytical Methods Committee. Technical brief No. 4. Ed. M. Thompson. 2006. www.rsc.org/amc/. 9. Silverman, B. W. Density Estimation for Statistics and Data Analysis; Chapman and Hall: London, Great Britain, 1986. 10. Wand, M. P.; Jones, M. C. Kernel Smoothing; Chapman and Hall: London, Great Britain, 1995. 11. Incze´dy, J.; Lengyel, T.; Ure, A. M.; Gelencse´r, A.; Hulanicki, A. Compendium of Analytical Nomenclature IUPAC, 3rd ed.; Pot City Press Inc.: Baltimore 2nd printing, 2000; p 50. 12. Lira, I.; Wo¨ger, W. Comparison Between the Conventional and Bayesian Approaches to Evaluate Measurement Data. Metrologia 2006, 43, S249–S259. 13. Zech, G. Frequentist and Bayesian Confidence Intervals. Eur. Phys. J. Direct 2002, C12, 1–81. 14. Sprent, P. Applied Nonparametric Statistical Methods; Chapman and Hall, Ltd: New York, 1989. 15. Patel, J. K. Tolerance Limits. A Review. Commun. Stat. Theory Methods 1986, 15 (9), 2716–2762. 16. Wald, A.; Wolfowitz, J. Tolerance Limits for a Normal Distribution. Ann. Math. Stat. 1946, 17, 208–215.


73

17. Wilks, S. S. Determination of Sample Sizes for Setting Tolerance Limits. Ann. Math. Stat. 1941, 12, 91–96. 18. Kendall, M.; Stuart, A. The Advanced Theory of Statistics, Inference and Relationship. Charles Griffin & Company Ltd: London, 1979; pp 547–548; Section 32.11; Vol. 2. 19. Willink, R. On Using the Monte Carlo Method to Calculate Uncertainty Intervals. Metrologia 2006, 43, L39–L42. 20. Guttman, I. Statistical Tolerance Regions; Charles Griffin and Company: London, 1970. 21. Huber, Ph.; Nguyen-Huu, J. J.; Boulanger, B.; Chapuzet, E.; Chiap, P.; Cohen, N.; Compagnon, P. A.; Dewe´, W.; Feinberg, M.; Lallier, M.; Laurentie, M.; Mercier, N.; Muzard, G.; Nivet, C.; Valat, L. Harmonization of Strategies for the Validation of Quantitative Analytical Procedures. A SFSTP Proposal – Part I. J. Pharm. Biomed. Anal. 2004, 36, 579–586. 22. Huber, Ph.; Nguyen-Huu, J. J.; Boulanger, B.; Chapuzet, E.; Chiap, P.; Cohen, N.; Compagnon, P. A.; Dewe´, W.; Feinberg, M.; Lallier, M.; Laurentie, M.; Mercier, N.; Muzard, G.; Nivet, C.; Valat, L.; Rozet, E. Harmonization of Strategies for the Validation of Quantitative Analytical Procedures. A SFSTP Proposal – Part II. J. Pharm. Biomed. Anal. 2007, 45, 70–81. 23. Huber, Ph.; Nguyen-Huu, J. J.; Boulanger, B.; Chapuzet, E.; Cohen, N.; Compagnon, P. A.; Dewe´, W.; Feinberg, M.; Laurentie, M.; Mercier, N.; Muzard, G.; Valat, L.; Rozet, E. Harmonization of Strategies for the Validation of Quantitative Analytical Procedures. A SFSTP Proposal – Part III. J. Pharm. Biomed. Anal. 2007, 45, 82–86. 24. Feinberg, M. Validation of Analytical Methods Based on Accuracy Profiles. J. Chromatogr. A 2007, 1158, 174–183. 25. Rozet, E.; Hubert, C.; Ceccato, A.; Dewe´, W.; Ziemons, E.; Moonen, F.; Michail, K.; Wintersteiger, R.; Streel, B.; Boulanger, B.; Hubert, Ph. Using Tolerance Intervals in Pre-Study Validation of Analytical Methods to Predict In-Study Results. The Fit-forFuture-Purpose Concept. J. Chromatogr. A 2007, 1158, 126–137. 26. Rozet, E.; Ceccato, A.; Hubert, C.; Ziemons, E.; Oprean, R.; Rudaz, S.; Boulanger, B.; Hubert, Ph. Analysis of Recent Pharmaceutical Regulatory Documents on Analytical Method Validation. J. Chromatogr. A 2007, 1158, 111–125. 27. Dewe´, W.; Govaerts, B.; Boulanger, B.; Rozet, E.; Chiap, P.; Hubert, Ph. Using Total Error as Decision Criterion in Analytical Method Transfer. Chemom. Intell. Lab. Syst. 2007, 85, 262–268. 28. Gonza´lez, A. G.; Herrador, M. A. Accuracy Profiles from Uncertainty Measurements. Talanta 2006, 70, 896–901. 29. Rebafka, T.; Cle´mençon, S.; Feinberg, M. Bootstrap-Based Tolerance Intervals for Application to Method Validation. Chemom. Intell. Lab. Syst. 2007, 89, 69–81. 30. Fernholz, L. T.; Gillespie, J. A. Content-Correct Tolerance Limits Based on the Bootstrap. Technometrics 2001, 43 (2), 147–155. 31. Cowen, S.; Ellison, S. L. R. Reporting Measurement Uncertainty and Coverage Intervals Near Natural Limits. Analyst 2006, 131, 710–717. 32. Schouten, H. J. A. Sample Size Formulae with a Continuous Outcome for Unequal Group Sizes and Unequal Variances. Stat. Med. 1999, 18, 87–91. 33. Lehmann, E. L. Testing Statistical Hypothesis; Wiley & Sons: New York, 1959. 34. Schuirmann, D. J. A Comparison of the Two One-Sided Tests Procedure and the Power Approach for Assessing the Equivalence of Average Bioavailability. J. Pharmacokinet. Biopharm. 1987, 15, 657–680. 35. Mehring, G. H. On Optimal Tests for General Interval Hypothesis. Commun. Stat. Theory Methods 1993, 22 (5), 1257–1297. 36. Brown, L. D.; Hwang, J. T. G.; Munk, A. An Unbiased Test the Bioequivalence Problem. Ann. Stat. 1998, 25, 2345–2367. 37. Munk, A.; Hwang, J. T. G.; Brown, L. D. Testing Average Equivalence. Finding a Compromise Between Theory and Practice. Biom. J. 2000, 42 (5), 531–552. 38. Hartmann, C.; Smeyers-Verbeke, J.; Penninckx, W.; Vander Heyden, Y.; Vankeerberghen, P.; Massart, D. L. Reappraisal of Hypothesis Testing for Method Validation: Detection of Systematic Error by Comparing the Means of Two Methods or of Two Laboratories. Anal. Chem. 1995, 67, 4491–4499. 39. Limentani, G. B.; Ringo, M. C.; Ye, F.; Bergquist, M. L.; McSorley, E. O. Beyond the t-Test. Statistical Equivalence Testing. Anal. Chem. 2005, 77, 221A–226A. 40. Kuttatharmmakull, S.; Massart, D. L.; Smeyers-Verbeke, J. Comparison of Alternative Measurement Methods: Determination of the Minimal Number of Measurements Required for the Evaluation of the Bias by Means of Interval Hypothesis Testing. Chemom. Intell. Lab. Syst. 2000, 52, 61–73. 41. Andre´s Martin, A.; Luna del Castillo, J. D. Bioestadı´stica para las ciencias de la salud; Norma-Capitel: Madrid, 2004. 42. Wellek, S. Testing Statistical Hypotheses of Equivalence; Chapman & May/CRC Press LLC: Boca Raton, FL, 2003. 43. Ortiz, M. C.; Herrero, A.; Sanllorente, S.; Reguera, C. The Quality of the Information Contained in Chemical Measures (electronic book); . Servicio de Publicaciones Universidad de Burgos: Burgos, 2005 Available on. http://web.ubu.es/investig/grupos/ cien_biotec/QA4/index.htm. 44. D’Agostino, R. B.; Stephens, M. A.; Eds.; Goodness-of-Fit Techniques. Marcel Dekker Inc.: New York, 1986. 45. Moreno, E.; Giroń, F. J. On the Frequentist and Bayesian Approaches to Hypothesis Testing (with discussion). Stat. Oper. Res. Trans. 2006, 30 (1), 3–28. 46. Scheffe´, H. The Analysis of Variance; Wiley & Sons: New York, 1959. 47. Anderson, V. L.; MacLean, R. A. Design of Experiments. A Realistic Approach; Marcel Dekker Inc.: New York, 1974. 48. Milliken, G. A.; Johnson, D. E. Analysis of Messy Data: Designed Experiments; Wadsworth Publishing Co, Belmont, NJ, 1984; Vol. I. 49. Searle, S. R. Linear Models; Wiley & Sons, Inc: New York, 1971. 50. Youden, W. J. Statistical Techniques for Collaborative Tests; Association of Official Analytical Chemists: Washington, DC, 1972. 51. Kateman, G.; Pijpers, F. W. Quality Control in Analytical Chemistry; Wiley & Sons: New York, 1981. 52. Kuttatharmmakull, S.; Massart, D. L.; Smeyers-Verbeke, J. Comparison of Alternative Measurement Methods. Anal. Chim. Acta 1999, 391, 203–225. 53. Hampel, F. R.; Ronchetti, E. M.; Rousseeuw, P. J.; Stahel, W. A. Robust Statistics. The Approach Based on Influence Functions; Wiley-Interscience: Zurich, 1985. 54. Huber, P. J. Robust Statistics; Wiley & Sons: New York, 1981. 55. Thompson, M.; Wood, R. J. Assoc. Off. Anal. Chem. Int. 1993, 76, 926–940.

74 Quality of Analytical Measurements: Statistical Methods for Internal Validation 56. Sanz, M. B.; Ortiz, M. C.; Herrero, A.; Sarabia, L. A. Robust and Non Parametric Statistic in the Validation of Chemical Analysis Methods. Quı´m. Anal. 1999, 18, 91–97. 57. Garcıá, I.; Sarabia, L.; Ortiz, M. C.; Aldama, J. M. Usefulness of D-optimal Designs and Multicriteria Optimization in Laborious Analytical Procedures. Application to the Extraction of Quinolones from Eggs. J. Chromatogr. A 2005, 1085, 190–198. 58. Garcıá, I.; Sarabia, L.; Ortiz, M. C.; Aldama, J. M. Robustness of the Extraction Step When Parallel Factor Analysis (PARAFAC) is Used to Quantify Sufonamides in Kidney by High Performance Liquid Chromatography-Diode Array Detection (HPLC-DAD). Analyst 2004, 129 (8), 766–771. 59. Massart, D. L.; Vandeginste, B. G. M.; Buydens, L. M. C.; de Jong, S.; Lewi, P. J.; Smeyers-Verbeke, J. Handbook of Chemometrics and Qualimetrics: Part A; Elsevier: Amsterdam, 1997. 60. Abramowitz, M.; Stegun, I. A. Handbook of Mathematical Functions; Government Printing Office, 1964. 61. Meier, P. C.; Zu¨nd, R. E. Statistical Methods in Analytical Chemistry, 2nd ed.; Wiley & Sons: New York, 2000. 62. Johnson, N.; Kotz, S. Distributions in Statistics: Continuous Univariate Distributions – 2; Wiley & Sons: New York, 1970; p 191 (Equation (5)). 63. Evans, M.; Hastings, N.; Peacock, B. Statistical Distributions, 2nd ed.; Wiley & Sons: New York, 1993; pp 73–74. 64. Statistics Toolbox for use with Matlab, version 5.3, The MathWorks, Inc., 2006.


Biographical Sketches

Marıá Cruz Ortiz received her Ph.D. in Chemistry from the University of Valladolid (Spain) in 1988. She is a member of the staff of Analytical Chemistry in the University of Burgos (Faculty of Sciences) since 1989. She does research and teaching on Analytical Chemistry and Chemometrics. Her research activity has focused on experimental design, optimization, pattern recognition, quality assurance, and validation of analytical methods according to official regulations and multivariate and/or multiway regression models. All of these applied to problems in Food Chemistry, typification, etc. has resulted about 100 papers. She is at present the head of an active research group, the Chemometrics and Qualimetrics group of the University of Burgos.

Luis A. Sarabia received his Ph.D. in Statistics from the University of Valladolid (Spain) in 1979. Since 1974, he is teaching Statistics and Mathematics mostly to graduate and postgraduate students of Chemistry. At present, his research is centred on Chemometrics as a member of the Chemometrics and Qualimetrics group of the University of Burgos. His research activities include development of software and implementation of nonparametric and robust statistical methods, genetic algorithms, neural networks, etc. He is also involved in multivariate/multiway regression methods, methodology of the experimental design, quality assurance, and validation. He is the author of about a hundred papers on these matters.

75


Ma Sagrario Sańchez received her Ph.D. in Mathematics from the University of Valladolid in 1997. She is working at the University of Burgos since 1991 and as a member of the permanent staff since 2002. The teaching activities are mostly directed to students of the degree in Chemistry and Science and Food Technology, and postgraduate courses. She is also a permanent member of the Chemometrics and Qualimetrics Group since its foundation. Her main research activities are developed into the areas of interest of the group, which include modeling and analysis of n-way data, modeling of categories, design of experiments, optimization, etc. by using classical methods as well as computationally intensive methods (such as neural networks or evolutionary algorithms).

Ana Herrero, after completing her undergraduate studies in Chemistry from the University of Valladolid (Spain) in 1991, received her Ph.D. at the University of Burgos (Spain) in 1996. She is working at the University of Burgos since 1992, teaching and researching in Analytical Chemistry. Her research involves Chemometrics and she is a member of the Chemometrics and Qualimetrics group of the University of Burgos. Experimental design, optimization, pattern recognition, multivariate regression analysis, quality assurance, and validation of analytical methods are among the fields she is working on.

1.03

Proficiency Testing in Analytical Chemistry

M. Thompson, University of London, London, UK ª 2009 Elsevier B.V. All rights reserved.

1.03.1 1.03.1.1 1.03.1.2 1.03.1.3 1.03.2 1.03.2.1 1.03.2.2 1.03.2.3 1.03.2.4 1.03.2.5 1.03.2.6 1.03.2.7 1.03.2.8 1.03.2.9 1.03.2.10 1.03.3 1.03.3.1 1.03.3.2 1.03.3.3 1.03.4 1.03.4.1 1.03.4.2 References

Overview of Proficiency Testing in Analytical Chemistry Introduction: The Purpose and Organization of Proficiency Testing Proficiency Testing in the Context of Analytical Quality Assurance The Development of Proficiency Testing and Normative Documents z-Scoring Requirements for Scoring Assigned Values and Their Uncertainties Robust Statistics Kernel Densities Mixture Models Choice of p in Calculating z-Scores Interpretation and Use of z-Scores Scope of Proficiency Testing and Problems with z-Scoring Skewed Data Sets Lognormal Data Sets Validation of Test Materials Introduction Testing for Homogeneity Testing for Stability Further Information from Proficiency Testing Results Review of the Operation of a Scheme Information about Analytical Methods

Symbols Fi; j h HA H0 H15

k l

m n N(0, 1) pi

the F statistic with i and j degrees of freedom. smoothing parameter in estimation of a kernel density. an alternative hypothesis. a null hypothesis. the ‘Huber H15 proposal’ for estimating simultaneously a robust mean and variance. multiplier for a standard deviation to define various confidence intervals. correction factor for a variance that accommodates downweighting of some observations. index for an iterative procedure. number of observations in a dataset. the standard normal deviate. the proportion of the ith component in a mixture model.

Pˆ i xj uA var( ) x x˜ i

xA xL y z zL 2f ˆ i

78 78 79 79 79 79 80 80 81 84 85 87 88 89 90 90 90 90 93 93 93 94 94

the likelihood of datum j belonging to component i of a mixture model. uncertainty on an assigned value. the variance of a variable. result of an analytical measurement. ‘pseudo-value’ used in the calculation of a robust statistic. The index i refers to a stage in an iterative procedure. assigned value a limiting value for arbitrarily dichotomizing a metric. a probability density. z-score defined by z ¼ ðx – xA Þ=p a z-score based on a customised target value. chi-squared distribution with f degrees of freedom. estimate, usually robust, of a sample mean. The index i refers to a stage in an iterative procedure.

77

78 Proficiency Testing in Analytical Chemistry

all

an

ˆ i

a standard deviation regarded as the maximum allowable between-sample component in a test for sufficient homogeneity in a proficiency test material. within-sample (repeatability) component of standard deviation in a randomized replicated experiment. estimate, usually robust, of a sample standard deviation. The index i refers to a stage in an iterative procedure.

f p ˆ rob sam

ðÞ

a target value based on fitness for purpose. standard deviation for proficiency (informally, ‘target value’). a robust estimate of a sample standard deviation. between-sample component of standard deviation in a randomized replicated experiment. the standard normal density.

1.03.1 Overview of Proficiency Testing in Analytical Chemistry 1.03.1.1

Introduction: The Purpose and Organization of Proficiency Testing

Since 1980, proficiency testing has grown from a small collection of schemes organized on a self-help basis to a minor industry, for the most part with third-party accreditation, in which nearly all analytical laboratories participate. As a result, proficiency testing practice has been subjected to a deep scrutiny, and procedures have been studied, refined, and documented. Protocols relevant to the conduct of schemes have been issued by ISO (International Organization for Standardization), IUPAC (International Union of Pure and Applied Chemistry), ILAC (International Laboratory Accreditation Cooperation), and AOAC International (Association of Analytical Communities International). Regardless of all this activity, the basic plan for proficiency testing has remained unchanged. Proficiency testing in analytical chemistry laboratories comprises the regular distribution, by the scheme provider, of portions of an effectively homogeneous material for analysis in the participating laboratories by an unspecified method. The concentration of the analyte is unknown to the participant at the time of analysis. The participant reports the result to the scheme provider by a deadline. The scheme provider then issues a report containing the preferred value of the analyte concentration, a summary of the results of all the participants, and a score that gives an indication of the accuracy achieved by each individual result. This sequence of events, known as a ‘round’, is repeated at regular intervals several times per year. A round can comprise several test materials and call for the determination of a number of analytes per material. The participant usually pays for this service. The primary purpose of proficiency testing is to check the accuracy of the results obtained by a laboratory and thereby allow the participant to detect and subsequently remedy faulty procedures. The scoring should be designed solely to support that requirement. The basis of proficiency testing is therefore the distribution of materials typically encountered, and the use by the participant of routine methods, so that the outcome is, as closely as possible, representative of normal working practice. Ideally, the proficiency test material should be analyzed as part of a routine run. (A ‘run’ is a set of analyses carried out under repeatability conditions, i.e., with the same analyte, same method, same equipment and analyst, and in a short period of time, so that factors affecting accuracy remain constant.) There are, however, secondary uses of proficiency testing scores that tend to undermine this ethos. Accreditation agencies require laboratories to participate in a proficiency test if a relevant one is available, and may take an interest in the scores obtained and the documented efforts of the laboratory to reduce notable inaccuracies. Moreover, laboratories are often asked for an account of their recent scores, so that potential customers can gage their efficiency. As a result, there is an undue pressure on participants to excel, and this may result in proficiency test materials receiving special (rather than routine) treatment. Such practice is contrary to the ethos of proficiency testing and must be avoided by participants. Moreover, scheme providers and accreditation agencies should take steps they can to discourage it. It is easy to imagine that these problems could be circumvented by the distribution of proficiency test materials that were disguised as routine samples.


79

But it only takes a moment’s thought to see that organizing such a scheme would present considerable difficulty and be prohibitively expensive on the scale required for analytical chemistry.

1.03.1.2

Proficiency Testing in the Context of Analytical Quality Assurance

Aspects of quality assurance bearing directly on the accuracy of analytical results can be considered under four principal headings that should be considered as a sequence: 1. Fitness for purpose – specifying appropriate accuracy. 2. Method validation – determining the accuracy that can be obtained in a laboratory and checking that it fulfills the fitness-for-purpose specification. 3. Internal quality control – ensuring that the accuracy obtained at validation time is maintained in every successive run of analysis. 4. Proficiency testing – regular checking that the previous steps are fulfilling their purpose. Proficiency testing is essentially an occasional check on the whole analytical system, where it has the special advantage of allowing participants to compare their results with those of peer laboratories and with an external reference. It should not be seen as an alternative to the internal quality control, which should be carried out in every run of analysis. There is an important corollary here: an unexpected inaccuracy in a proficiency test means that the participant’s validation procedure and/or internal quality control are faulty as well as the analytical method used. Proficiency testing is not a method for training analysts, for certifying reference materials, for validating analytical methods, nor for diagnosing problems, although it can sometimes throw some light on these aspects. 1.03.1.3

The Development of Proficiency Testing and Normative Documents

Proficiency testing developed independently in various application sectors of analytical chemistry, initially in biomedical analysis, then in food analysis and later in environmental analysis. (Schemes in the industrial sector undoubtedly existed from an early date, but are usually not openly documented.) The first attempt to produce a universally applicable format was the International Harmonized Protocol1 recognized by IUPAC, ISO, and AOAC International. This was followed by an ISO Guide to proficiency testing generally (i.e., not just chemical testing)2 and a book dealing with putting proficiency testing into practice in analytical chemistry.3 ILAC Guidelines for assessing the competence of proficiency test providers appeared in 2000 and is currently (2008) under revision.4 An ISO Standard on statistical methods for proficiency testing appeared in 20055 and a revised International Harmonized Protocol for analytical chemistry in 2006.6 There are now a large number of schemes available, many of them accredited.7

1.03.2 z-Scoring 1.03.2.1

Requirements for Scoring

Most schemes convert a participant’s result x into a score reflecting its accuracy. The ideal score would be universal: a particular score of (say) 2.5 should always convey the same information about accuracy. A score that approaches the ideal is the z-score, given by z¼

x – xA p

ð1Þ

where p is the ‘standard deviation for proficiency’ (known informally as the ‘target value’), which is provided in advance by the scheme provider. The ‘assigned value’ xA is the best available estimate of the true concentration of the analyte. The provider releases the value of xA only after the deadline for submission of results. This assigned value (with a suitably small uncertainty) could conceivably be found by a procedure comparable in effort and cost to the certification of a reference material. Alternatively, it could be a value


determined by a national metrological reference laboratory by a procedure such as isotope dilution mass spectrometry. Both these options are usually too costly for use in routine proficiency testing. Therefore, the assigned value is, at present, nearly always identified as the consensus of the participants’ results, although this may be inadequate if there is a general bias in the results because of the widespread use of a biased analytical method. Consequently, there is considerable emphasis on the identification of a ‘consensus’. The value of p should represent fitness for purpose and should be determined so that z-scores within the range 2 are regarded as fit, whereas scores outside the range 3 suggest that some investigation is called for, where necessary followed by remedial action. More detail on selecting a suitable value for p is given in Section 1.03.2.6. Some metrologists advocate a scoring system in which the score is based not only on the participant’s result, but also on the estimated uncertainty of the individual result. The premise of the idea is that analysts should always report (or, at least, be able to report) their uncertainty. Such a scoring system suffers from a number of problems and is not recommended by the IUPAC Harmonized Protocol. In most schemes, the provider specifies the uncertainty that is broadly fit for purpose in the application sector, and it is the task of the participants to provide a result falling within that uncertainty. The required accuracy is explicit in the scheme, and participants’ estimates of uncertainty are irrelevant. If a different criterion of uncertainty is needed of a participant by a particular customer, the analyst should calculate a special score for that specific purpose, based on the customer’s criterion, as discussed in Section 1.03.2.5: however, that is not a legitimate concern of a proficiency testing scheme.

1.03.2.2

Assigned Values and Their Uncertainties

In the determination of an assigned value, attention is usually focused on the exact meaning of ‘consensus’ and how best to estimate it from the participants’ results. The usual estimates of central tendency (mean, median, and mode) are all potential candidates for consensus, but each presents some interesting problems. The arithmetic mean is clearly unsatisfactory because results of a round nearly always contain outliers, often extreme outliers. A robust mean (including the median) is often suitable, but its utility depends on the distribution of results (outliers aside) being largely symmetric. In practice, results from a round of a proficiency test are often somewhat skewed and sometimes multimodal. The dominant mode is an attractive choice intuitively, but it requires a complicated procedure to quantify the mode from the data of limited digit resolution sampled from an underlying continuous distribution. Clearly, the choice of estimator calls for inspection of the data as a graphical presentation and the exercise of judgment. (Accreditation agencies are apt to question this call for judgment: they prefer to see a rigidly applied procedure. However, the possibilities inherent in data from a round of a proficiency test are so large that it is impracticable to lay down adequately a fixed protocol for determining a consensus. Hence, this is the appropriate time to emphasize that statistics is a tool designed to assist judgment, not to replace it. Accordingly, all proficiency schemes must have access to a statistical expert.) The uncertainty on the consensus uA is also of interest: if the uncertainty is too large, the z-scores produced will reflect this variation to an undue extent. Thus, uA < p =3 is preferred, so that the z-scores are affected to a rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffi maximum extent of 2p þ uA2 =2p ¼ 10=9, about 5% relative. This criterion can usually be achieved if there are more than about 15 participants reporting valid results. Some metrologists advocate the use of a qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi modified score defined by ðx – xA Þ= 2p þ uA2 , which allows scoring to proceed even when uA is high. This strategy has the defect that it allows the calculation of apparently meaningful scores when an unduly large part of the variation is derived from uA and not from the variation in the accuracy of the participants: there is no indication in the modified score that suggests the possibility of such a problem.

1.03.2.3

Robust Statistics

The Harmonized Protocol recommends the use of a robust mean as consensus when the results appear to be largely symmetric, outliers aside. This is the most frequent outcome of a round of a test. An example is shown in


81

Frequency

15

10

5

0 20

24 21 22 23 Result for alumina (% mass fraction)

25

Figure 1 Outcome of a round of a proficiency test (Al2O3 in a rock) where the participants’ results (histogram) approximate to a slightly contaminated sample from a normal distribution. The parameters of the fitted normal distribution (red line) were estimated by the H15 procedure.

Figure 1. The H15 estimates of Huber are recommended but other robust estimates may well be suitable.8 Robust statistics accommodate extreme results by downweighting them. They are calculated by a procedure rather than a formula. The H15 procedure is as follows: 1. Obtain initial estimates ˆ 0 ; ˆ 0 2 of the mean and variance. These could be simple mean and variance or (better) estimates based on the median and 1.5 MAD (median absolute difference from the median). 2. Select a value 1 k 2 depending on the expected proportion of outliers. The usual value of k ¼ 1.5 is suitable for 5–10% of outliers. 3. Transform the raw results xi , i ¼ 1, . . ., n, to ‘pseudo-values’ x˜i as follows, where m ¼ 0 initially: 8 xi þ kˆ m ; xi > ˆ m þ kˆ m > > < x˜i ¼ xi – kˆ m ; xi < ˆ m – kˆ m > > : xi ; ˆ m – kˆ m < x i < ˆ m þ kˆ m :

ð2Þ

4. Calculate ˆ mþ1 ¼ meanðx˜Þ; ˆ 2mþ1 ¼ l varðx˜Þ, where l is a factor determined by the value of k. 5. Increment m in the formula for x˜ and repeat the iteration until the statistics have converged to a required degree of accuracy. Robust estimation also providesp a ffiffimethod of estimating uncertainty on the consensus. The standard error of an ffi arithmetic mean is given by = n, so it is convenient to regard the standard uncertainty of the consensus as pffiffiffi uA ¼ ˆ rob = n. This value is usually a slight underestimate, as robustification is equivalent to downweighting pffiffiffi some of the data, so that n is too great a denominator. However, a more accurate value can be obtained by the bootstrap. In practice, there is no consequential difference between the two estimates.9 The main purpose of uA is the comparison with p =3, so a rough estimate will suffice. An Excel add-in and a Minitab macro for calculating H15 robust statistics can be found in the AMC web site.10

1.03.2.4

Kernel Densities

Kernel densities provide a useful means of judging whether a set of results from a round of a proficiency test is unimodal or multimodal and, in either case, estimating the values of the identified mode, modes, or shoulders. The judgment of whether a data set is multimodal depends on having a good graphical representation of the distribution of the data: the number of modes apparent depends on aspects of the presentation. A dotplot will show many modes. A histogram will show a number depending on the width of the bins used. The automatic choice of width provided by statistical packages may prove misleading in this context. In a sense, there is no


‘true’ number of modes – it depends on the degree of smoothing, which in turn depends on what the scientist expects the data to be. However, the kernel density has been found to be serviceable in proficiency testing because, in that instance, there is a rational approach to the choice of smoothing parameter.11 A kernel density is produced by replacing each data point xi on the line by a probability density function (the kernel), usually the normal distribution Nðxi ; h2 Þ, where h is a parameter that controls the degree of smoothing. The densities are added together at each point on the line and normalized to unit area to give the kernel density, n x – x 1 X i

fˆ¼ h nh i¼1

ð3Þ

Density

where ðÞ is the standard normal density.12 The procedure is illustrated in Figures 2 and 3. The kernel density is a smooth representation of the data, but we can estimate it only at discrete values of x. The density can be calculated directly by adding the individually calculated kernels at each point. However, as the kernel density is a convolution of the data with a normal distribution, it can also be obtained via the Fourier transform.10 The number of modes shown depends on the selected value of h. The kernel density is more disperse than the original data (as to be expected with smoothing) with a variance varðx Þ þ h2 . The inflation is not at all serious if h is less than about 0.75 of the original standard deviation of x, when it amounts to 25%. This points to a suitable benchmark for h in proficiency testing: because p is set to represent a dispersion that the data should conform to, we can use h ¼ 0.75p as a starting value to investigate the parameter space.

0.0 Measurement axis

Density

Figure 2 A kernel density (red line) derived from four data points by the addition of normal kernels (black lines) with standard deviation h.

0 Measurement axis Figure 3 A kernel density (red line) derived from the same data as shown in Figure 2, but with kernels of standard deviation 2h.


83

An Excel add-in and a Minitab macro for producing kernel densities can be found in the AMC Website.10 Figures 4–6 illustrate an example of this procedure applied to a proficiency test for polyunsaturated fatty acids in cooking oil. The data can be downloaded from the AMC Website.10 The original data displayed as a dotplot (Figure 4) are difficult to interpret in terms of modality, but show an unexpected large robust standard deviation of 0.96% mass fraction. A plot of the number of modes observed versus the value of h used to make the kernel density (Figure 5) shows that a two-mode model has a good range of stability. Normal kernel densities with 0.2 h 0.4 (Figure 6) show a marked tendency for two modes, at 39.3 and 40.6. As will be seen (Section 1.03.2.5), this is a useful model of the data. The uncertainties of the modes can be quantified as their standard errors, which can be estimated by the bootstrap.11 However, the latter is a somewhat lengthy process, likely to take several minutes on modern personal computers. The bootstrap standard errors corresponding to the modes of the example data are 0.09 and 0.21, respectively.

39

40 Analytical result (% mass fraction)

41

Figure 4 Dotplot of data from a proficiency test round: polyunsaturated fatty acids in cooking oil.

8

Number of modes

7 6 5 4 3 2 1 0 0.0

0.1

0.2

0.3 0.4 0.5 Value of h

0.6

0.7

0.8

Figure 5 Plot of number of modes vs. h. Data from Figure 4.

0.6

Density

0.5 0.4 0.3 0.2 0.1 0.0 35

36

37

38 39 40 41 Result (% by mass)

42

43

44

Figure 6 Kernel densities obtained by using h values of 0.2 (black line), 0.3 (green line), and 0.4 (red line). Data from Figure 4.


When we have additional information about the various analytical methods used by the participants, it is sometimes possible to identify one of several modes as a consensus. In that circumstance, we need to estimate the analyte concentration where the mode is situated. This can be done by using quadratic interpolation between (say) the six discrete points surrounding the mode on the concentration axis of the kernel density. In the instance illustrated, we know that the two modes originated from the use of two different calibration strategies by participants, only one of which corresponds with the proper definition of the measurand. As a consequence, one can ascertain that the major mode at 39.30 0.09 is the correct consensus. A novel mathematical procedure akin to kernel density estimation, ‘Cofino statistics’, has been advocated for use in proficiency tests.13 Although it provides a good estimate of a consensus, expert opinion is that it suffers from certain difficulties.14 First, it is couched in terms of an unusual notation and, second, the variance associated with the consensus is ambiguous.

1.03.2.5

Mixture Models

Another method of modeling a multimodal data set is to regard it as a mixture of two or more component normal distributions with unknown parameters and in unknown proportions. The ‘EM’ (expectation maximization) algorithm is a rapid and easily coded procedure for estimating the unknown parameters.15–17 The procedure starts with initial estimates of the means (or of component membership), but quite rough estimates are usually good enough to lead to convergence in a small number of iterations. It is often convenient to start with the position of modes (or shoulders) estimated visually from a kernel distribution. The normal mixture model has a density y given by the sum of m separate normal components in proportions pi,i ¼ 1, . . .,m, so that we have y¼

m X

m X

pi ¼ 1

ð4Þ

exp – ðx – i Þ2 =22i pffiffiffiffiffi fi ðxÞ ¼ 2i

ð5Þ

i¼1

pi fi ðxÞ;

i¼1

where fi(x) is the normal distribution

We estimate the parameters in the ‘M-step’ from the data xj , j = 1, . . . ,n, where the weights Pî ðx j Þ are likelihoods of datum j belonging to component i: Pn pî ¼

j ¼1

Pî xj

n

ˆ j ¼1 xj Pi xj

ˆ i ¼ Pn ˆ j ¼1 Pi xj

2 Pn xj – i Pî xj j ¼1 for individual variances ˆ i 2 ¼ Pn ˆ j ¼1 Pi ðx j Þ

2 Pm Pn x – Pî xj j i i¼1 j ¼1 ˆ 2 ¼ for a pooled variance n Pn

ð6Þ

The parameter estimates for each component are simply the mean ˆ i and variance ˆ i 2 weighted by the likelihood of each data point belonging to that component. We now estimate the weights in the ‘E-step’ as

pî f i xj

Pî xj ¼ Pm i¼1 pî f i xj

ð7Þ


85

The steps are used alternately until a sufficient degree of convergence is obtained. The algorithm can start at either step. A simple and effective way of starting is to cut the data at the midpoints between the modes of a kernel density. Then for component i, set Pî xj ¼ 1 if xj falls in the appropriate range and Pî xj ¼ 0 otherwise. The algorithm is then started at the M-step. Alternatively, the algorithm can be started at the Estep using ‘guestimated’ parameters for each component. Goodness of fit of the data to models with different numbers of components can be investigated until a satisfactory but parsimonious model is found. Adding an extra component will always improve the fit but, unfortunately, there is no simple method of determining whether one model is significantly better than another. The likelihood ratio method is regarded as unreliable in this context. A Minitab macro for producing one-dimensional mixture models can be found in the AMC Website.10 We can apply the method to the example data shown in Figures 4–6. This gives us a solution (Figure 7) with two components with the following statistics:

Mean

Pooled standard deviation

Proportion

Standard error

39.29 40.58

0.34 0.34

0.65 0.35

0.07 0.09

The statistics are close to those derived from the kernel density. However, no judgment about the degree of smoothing is required, and the computation is much quicker. The kernel density is by no means superseded, however: it provides a useful first look at the data. That the two-component mixture model comprises a good fit can be judged from a quantile–quantile plot of the data versus the model (Figure 8). If one component is identified (from additional evidence such as methods used by each participant) as comprising ‘correct’ results, the mean ˆ i of that component can be taken as the consensus and its standard error pffiffiffiffiffiffi with the uncertainty. The standard error is given by ˆ i = npi , where ˆ i is the estimated standard deviation, pi the estimated proportion of the ith component, and n the total number of results.

1.03.2.6

Choice of p in Calculating z-Scores

Ideally a specific z-score, say –1.7, should convey the same information about accuracy, irrespective of the identity or concentration of the analyte, the composition of the test material, the method of analysis, or the laboratory in which it is conducted. The ‘target value’ p can therefore be regarded as a scaling (for the ‘error’ (xxA)) that is designed to approach the ideal. Early thoughts on the subject were that p should be identified 0.9 0.8 0.7 Density

0.6 0.5 0.4 0.3 0.2 0.1 0.0 36.6

37.6

40.6 38.6 39.6 Analytical result

41.6

42.6

Figure 7 Two-component normal mixture model (red line), showing separate components (blue lines), from the results of a proficiency test (data from Figure 4).


Quantile of data

1.0

0.5

0.0 0.0

0.5 Quantile of model

1.0

Figure 8 Quantile–quantile plot of proficiency test results and the two-component model (Figure 7).

with a robust standard deviation of the results of a round. For the usual slightly contaminated normal distribution of results, this would provide a set of z-scores that would resemble the standard normal deviate N(0,1), but with somewhat more than 5% of the z-scores falling outside the range 2. That choice of p certainly harmonizes the outcomes of diverse proficiency tests, but provides the wrong information. It has the unfortunate effect of causing a false sense of security: up to 95% of participants will receive an apparently satisfactory score, within the range 2, regardless of whether or not the uncertainties of the results are appropriate for the application. A much better strategy is to equate the value of p with the standard uncertainty that is regarded as fit for purpose. All of the examples in this chapter are based on this principle. The strategy demands a clear definition of fitness for purpose and a methodology for locating the corresponding uncertainty. We can start by noting that an analytical method is not fit for purpose if it is too expensive to use or if its use results in an unduly high proportion of incorrect (and therefore costly) decisions. Experienced analysts are accustomed to balancing these losses as a matter of professional judgment and indeed, in the long term, methods that fall short in either of these aspects will be replaced with more cost-effective methods. We have recently seen that it is possible to employ decision theoretic methods18 to make the judgment more rigorous. To carry out this, we need to find how each cost affecting the customer is determined by the uncertainty of the measurement.19–22 We have to consider both the cost of measurement and the extra cost of an incorrect decision. There is an inverse relationship between the cost of analysis and the accuracy achieved. Indeed, in the absence of specific information to the contrary, it is a good rule of thumb to assume an inverse square relationship. For example, if we need to reduce the uncertainty by a factor of 2, the cost will increase by a factor of 4. The probability of a mistaken decision (for instance, whether the concentration of a toxin in a food exceeds the legal limit) also depends on the uncertainty – the greater the uncertainty, the greater the chance of an incorrect decision. The consequence cost of an incorrect decision may be twofold. Incorrect rejection of a batch of product implies a loss amounting to the cost of production and disposal, or the cost of reprocessing, an already compliant material. Incorrect acceptance of noncompliant material gives rise to the cost of compensation to down-line stakeholders multiplied by the probability of detection. The total long-term loss to the customer is the sum of these two losses, measurement costs and consequence costs, one decreasing and the other increasing in relation to rising uncertainty. The sum must therefore have a minimum cost, and the uncertainty that is fit for purpose (Figure 9). Although some of these contributions may be difficult to cost exactly, a workable value for consequence cost might be the cost of insuring against the occurrence of an incorrect decision. Even when the necessary data are unavailable, the idea provides a useful conceptual framework for fitness for purpose considerations. All of the above presupposes that the scheme provider can specify a value of p that is suitable for a whole application sector, that is, for all of the participants on all occasions. That is not invariably the case. Sometimes, an analyst’s customer may have a fitness criterion that differs from that of the closest relevant scheme. In such circumstances, it would be correct for the participant to calculate modified z-scores zL ¼ ðx – xA Þ=f , using the

87

Cost


0 0

uf Uncertainty

Figure 9 Cost of measurement (black line) and long-term average cost of incorrect decisions (green line) as a function of uncertainty of measurement. The expectation of total cost (red line) shows a minimum cost defining an uncertainty uf that is ‘fit for purpose’.

scheme’s assigned value xA but a customer-specific fitness criterion f. Such modified z-scores can be interpreted in exactly the same way as normal z-scores.

1.03.2.7

Interpretation and Use of z-Scores

If a participant was performing exactly according to the specified p, the z-scores in the long term would resemble the standard normal deviate z N(0,1).23,24 About 95% of the results would fall in the range 2 and about 0.3% would fall outside 3. The latter outcome would be so unusual that it could safely be regarded as indicating a change, which called for investigation, in the performance of the system. It is therefore sensible to interpret z-scores in terms of N(0,1). Needless to say, z-scores, either from different participants in a round or from a single participant over time, never exactly fulfill N(0,1). Individual participants may operate with any combination of (1) a reproducibility variance greater or less than p, (2) a bias that shifts the center of the distribution, and (3) a heavy tailed distribution. Those participants with a variance greater than p, and/or those with a bias, will produce in the long term more than 5% of scores outside the range 2 and more than 0.3% of scores outside the range 3. (Of course, many of the results from maladapted participants will fall inside these ranges in individual rounds: proficiency testing cannot guarantee to detect questionable performance.) Those with variance less than p and no bias will, of course, produce less than the expected number of suspect scores. This is exactly the desired outcome, so it is correct to interpret scores on the basis of N(0,1), even though we know that that is not a realistic expectation. The point is emphasized here because it is widely misunderstood among analytical chemists. The target value p is meant to prescribe the uncertainty that is fit for purpose, not to describe the results that are produced by the participants, and the normal distribution is nearly always appropriate for this purpose. There are a number of statistically sound methods of combining z-scores to obtain a broader perspective on accuracy, but all require caution in their interpretation. P pffiffiffiffi For our hypothetical ideal participant, the rescaled sum of m normally distributed z-scores z= m is distributed as N(0,1) and can be interpreted like z. It has the disadvantage that large scores of opposite sign could compensate and create (for the unwary) a false impression of satisfactory performance. The sum of the squared z-scores P z2 interpreted as 2m would avoid this problem, but is unduly sensitive to extreme results. If used at all, such combination scores should be restricted to successive scores for the same analyte. Combining scores from several different analytes may enable a persistently poor outcome for a particular analyte to escape detection. A visual presentation of the scores is far more informative and less likely to be misinterpreted. Successive scores from the same analyte can be plotted on a Shewhart control chart with control lines at 2 and 3 and interpreted by using the usual Westgard rules25,26 (Figure 10). Alternatively, a cusum chart or a zone chart27


3 2

z-Score

1 0 –1 –2 –3 0

10 Round number

20

Figure 10 Shewhart control chart applied to successive z-scores produced by a participant. All of the scores are ‘in control’ and apparently call for no action.

Weights 8 3

z-Score

2 1 0 –1 –2 –3

4 2

2

1

9

6 3 4

1

7

1

1 2

1

4 5 6

1 6

2

2

3

4

4 8 2

7 12 Round number

17

Figure 11 Zone chart (or ‘J-chart’) applied to successive z-scores produced by a participant (same data as Figure 10).

could be used to pick up a persistent bias. The useful but relatively little known zone chart (otherwise known as a ‘J-chart’) combines the capabilities of the Shewhart and cusum charts. It is constructed by dividing a standard control chart into zones with widths of one standard deviation each side of the mean (Figure 11). Results as they occur are assigned to the appropriate zone and given a corresponding weight as shown in Figure 11. While successive results fall on the same side of the mean, the weights are cumulated. If the nth result falls on the opposite side of the mean to its predecessor, the cumulated weight is reset to zero before cumulation resumes with the nth weight. When the total weight amounts to 8 or greater, the system is deemed to be out of control. In Figure 11, the chart flags an ‘out-of-control’ condition at observation 7, by virtue of the accumulated J-score of 9 produced by six successive positive z-scores. There is a strong temptation among non-scientists to classify or rank laboratories on the basis of proficiency test scores. This tendency should be resisted, because it can lead to serious misunderstanding. Arbitrarily chosen decision limits such as 3 should not be regarded as class boundaries. Ranking is even more invidious. The rank of a laboratory consistently performing according to fitness for purpose could vary greatly from round to round.

1.03.2.8

Scope of Proficiency Testing and Problems with z-Scoring

Proficiency testing and scoring as described above is limited to measurements on interval or ratio scales. It therefore cannot accommodate results on nominal or ordinal scales, results such as ‘less than xL’ (a limit value), ‘not detected’, or ‘absent’. Although some results of that kind may be clearly fit for purpose, they cannot be


89

converted into z-scores. Some providers have adopted the expedient of replacing such results with numerical results such as zero, or one-half of the detection limit, but this practice has the danger of biasing the data set. Attempts at a Bayesian interpretation of these low results have not been satisfactory so far because the outcome is very sensitive to any realistic prior distribution. Other schemes simply reject such results and do not convert them into scores. Attempts to devise a hybrid scoring system that can accommodate both types of data have not been successful so far. An extreme version of this problem occurs in proficiency tests where the result is qualitative, that is, the result is either ‘present’ or ‘absent’. Scoring in such tests comprises little more than tabulating the proportions of true negative, false negative, true positive, and false positive. Needless to say, the information content of a single result is small compared with that of a result on an interval or ratio scale, and many results have to be accumulated for a useful conclusion to be drawn.

1.03.2.9

Skewed Data Sets

Sometimes in a round of a proficiency test, the results are unimodal but display a strong positive skew. To ensure the correct treatment of such results, it is essential to understand the underlying causes of the skew. One commonly encountered cause occurs when some or all of the participants are reporting results near the detection limit of their analytical method. Results less than the detection limit are often not reported or, if they are reported, not displayed by the scheme provider. As the participants’ methods will have a variety of detection limits, and the results are low-censored, the outcome is a positively skewed distribution. An example is shown in Figure 12. Another contribution to skewed results near the detection limit is encountered when the analytical result is the quotient of two intermediate results, each with a substantial relative standard deviation (RSD). This sometimes happens when the results are corrected for recovery. Even if the two parent distributions are normal, the quotient will have a positive skew. With parent RSDs up to about 15%, the skew is mild and can be safely ignored. With parent RSDs of 20%, however, the skew is pronounced and, as resulting RSD will be around 30%, the final results will be at the detection limit and the data set will be censored to some degree. Scheme providers often circumvent these problems by ensuring that the concentration of the analyte is more than about 5 times the detection limit expected from the majority of participants. However, measuring nearlimit concentrations (even those of exactly zero) is a legitimate concern for many analysts – a substantial proportion of the work involves just such materials – so this solution is hardly supportive of participants needs. Unfortunately, no simple general solution has emerged. Under these circumstances, it is tempting for providers to log-transform positively skewed data sets and then interpret the outcome on the basis of a normal distribution. That action is inappropriate in most instances, because the distributions are not lognormal and, indeed, may contain negative values before they are censored. In addition, action limits based on the lognormal distribution will be asymmetric in the original (untransformed) domain of the results, and impute a different importance to low and high errors of equal magnitude. High errors would seem to be diminished in importance, whereas low errors would tend to attract unduly high scores. It has to be mentioned that a satisfactory method of dealing with low results and the resulting skews has yet to be devised. Although research in this area is being conducted, it is not clear that an acceptable solution will emerge. If participants reported the uncensored result and its true uncertainty, all would be clear. But this is unlikely to happen. Even the definition of uncertainty in this area is ambiguous. As in many other instances, a judgment call is required from the scheme provider. Such judgments are credible only if both analytical and statistical expertise are available.

0

1 2 Result for niobium (ppm, mass fraction)

Figure 12 Dotplot of the results from a round of a proficiency test (niobium in a rock) showing a marked skew. In this instance, the skew probably resulted from the use of methods with poor detection by some participants.


1.03.2.10

Lognormal Data Sets

We must also consider those unusual instances (in chemical measurements at least) where the measurement process is such that the errors are predominantly multiplicative. The quantitative measurement of genetically modified food constituents via the polymerase chain reaction is one such process. This process naturally gives rise to a lognormal distribution of results (although the binomial and normal distributions also make a (probably minor) contribution to the final result). A study of two such schemes has concluded that log-transformation before the formation of z-scores is the correct approach in that case.28

1.03.3 Validation of Test Materials 1.03.3.1

Introduction

Test materials distributed by the scheme should be of a matrix typically encountered in the relevant analytical sector and, in the long term, should contain analyte concentrations representing both acceptable and unacceptable levels. Materials must be sufficiently stable and homogeneous, so that differences in composition among the units received by the participants are insignificantly small in comparison with the uncertainty of the measurement process. Such a requirement calls for regular (or at least occasional) testing by the scheme provider. Because of this restriction, it is usually impossible to distribute materials in their native state. For example, rock samples have to be distributed as a fine powder to ensure a sufficient degree of homogeneity, and vegetable samples as a puree. 1.03.3.2

Testing for Homogeneity

To ensure that participants receive portions of test material with only inconsequential differences in composition (a condition known as ‘sufficient homogeneity’), it is necessary to test the material after it has been divided into containers for distribution. As it is universally recognized that it is the analyst’s responsibility to take a representative test portion from the laboratory sample, there is no need to test for heterogeneity within individual containers, only between containers. The test comprises the selection of 10 or more distribution units at random from the complete set and the analysis of each in duplicate, in a random sequence, under repeatability conditions, by a method with appropriate precision (i.e., with a standard deviation an < 0:3p ; the results must be recorded with sufficient digit resolution to support the statistical analysis). The between-unit variance sam is then found by analysis of variance and compared with a maximum-allowable variance all to test whether it is significantly greater. Two precautions, however, are needed to make this test meaningful. First, the randomization has to be strictly conducted, so that the data contain no systematic features that could falsely enhance or reduce the appearance of heterogeneity. It is worthwhile to give the analysts explicit instructions on how to do that: experience has shown that implicit instructions are often misunderstood or ignored. It is also necessary to check every data set for systematic effects by inspection of a simple display of result versus sample number. Undesirable features that may invalidate the test are a trend or discontinuity (Figure 13), bias between the first and second results (Figure 14), or insufficient digit resolution of data (Figure 15). Second, a statistical approach more sophisticated than the simple F-test of the hypothesis H0 : sam ¼ 0 versus HA : sam > 0

ð8Þ

is required to avoid an undue probability of rejection of good test material or rejection of the whole data set as incapable of supporting the required decision.29 The problem here is that, given an analytical method of sufficiently high precision, almost any material can be shown to be significantly heterogeneous. This does not necessarily mean, however, that the material is unsuitable for use in the proficiency test. In addition, as homogeneity testing is expensive (often requiring as much as one-third of the whole cost of the proficiency test), it is desirable to minimize the false rejection rate. For example, if sam is acceptable but only slightly less

91

Result


1

2

3

4 5 6 7 8 9 Sample identity number

10 11 12

Result

Figure 13 Data from a homogeneity test showing the first (red) and second (green) results from each distribution unit. The data show a trend, probably because the data were analyzed in the order of their code numbers rather than in a random order. This feature would tend to suggest falsely that the material was significantly heterogeneous.

1

2

3


10 11 12

Result

Figure 14 Data from a homogeneity test showing the first (red) and second (green) results from each distribution unit. The data show a bias between the first and second results, indicating that the analysis was conducted in two sequences, with each unit analyzed once in each sequence. This feature would tend to disguise significant heterogeneity.

1

2

3


10 11 12

Figure 15 Data from a homogeneity test showing a single outlying result, normally attributable to an analytical mistake.


than all, about one-half of all sufficiently homogeneous materials would be rejected. To deal with these problems, it is necessary to test a different hypothesis, namely H0 : sam all versus HA : sam > all

ð9Þ

where all is the maximum-allowable value, determined independently by the practical requirements of the proficiency test (i.e., not derived from the data). To carry out this test, we calculate ˆ an and ˆ sam in the normal way from the mean squares in the analysis of variance, and reject the material when ˆ 2sam >

2all 2n – 1 ˆ 2an Fn – 1;n – 1 þ 2 n–1

ð10Þ

1. 2n1 is the value exceeded with a probability of 0.05 by a chi-squared random variable with n 1 degrees of freedom. 2. Fn1, n is the value exceeded with a probability of 0.05 by a random variable with the F-distribution with n 1 and n degrees of freedom. 3. n is the number of units analyzed in duplicate. (Duplicate analysis is assumed in Equation (10).) 4. A suitable value for the allowable variance is 2all 2p =9 (i.e., all p =3 where p is the target value), which ensures that any heterogeneity between distribution units has a negligible effect on the z-scores.

Result

As each homogeneity test requires at least 20 replicate analyses, the occurrence of an analytical outlier in a data set is not rare. The first edition1 of the Harmonized Protocol stated that no outlying data should be excluded from the analysis of variance, because outliers comprised valid evidence of heterogeneity. However, a single analytical outlier (Figure 15) has the (perhaps) unexpected effect of making rejection of the null hypothesis less probable, because it has a bigger effect on the within-sample mean square than the between-sample mean square. Moreover, a single outlier must have occurred because of an analytical lapse, as the analyst should have ensured that both test portions were representative of the distribution unit. More recent opinion, therefore, is that a single outlier should be rejected before the calculation of the mean squares. Cochran’s test for unequal variances at 95% confidence is suitable for this purpose. (There is a potential exception to this rule. Some trace analytes occur only as rare discrete particles containing a high concentration of the analyte, even in ‘homogenized’ material. This could give rise to a valid solitary outlier. An example would be the determination of platinum in an ore.) However, a pair of apparently outlying results from the same distribution unit (Figure 16) must never be rejected: it is a true indication of heterogeneity. The occurrence of two outlying results from different distribution units suggests that the analysis was unreliable and that the whole data set should be discarded.

1

2

3


10 11 12

Figure 16 Data from a homogeneity test showing a pair of outlying results from a single unit. This is a strong indication of heterogeneity.


93

It should be noted that homogeneity tests have a low statistical power unless an unrealistically large number of distribution units are tested. The test should therefore be regarded as a screen to ensure that serious problems are avoided. 1.03.3.3

Testing for Stability

Materials distributed in proficiency tests should be sufficiently stable over the period during which the assigned value is designed to be definitive. The term ‘sufficiently stable’ implies that any changes that occur during the relevant period have an inconsequential effect on the interpretation of the results of a round. Normally, the period in question is the interval between preparation of the material and the deadline for return of the results, although the period may be longer if the provider intends subsequently to offer unused distribution units for sale. Before the stability test, the material should be packed in containers in which it is to be distributed. It is not usually necessary or even feasible that the material for every round in a series should be tested. However, each newly considered material/analyte combination should be tested once before it is used in a proficiency test and occasionally thereafter. Stability tests comprise a comparison of the material before and after the ‘appropriate’ treatment. This usually requires a batch of the distribution units to be randomly divided into two equal subsets. The ‘experimental’ subset is subjected to the appropriate treatment, whereas the ‘control’ subset is kept under conditions of maximum stability, for example, low temperatures and low oxygen tension. Ideally, the stability test should involve exposing the experimental subset to the most extreme conditions likely to be encountered during the distribution and storage. More realistically, because of the low power of the statistical test, the experimental subset should be kept under conditions of accelerated decomposition, for example, higher temperatures than normal. After the treatment, the two subsets are analyzed by a method of high precision. Although simple in principle, stability tests must be carefully designed to avoid confounding any change in the material with variation in the efficacy of the analytical method used. Analysis of the control material at the beginning of the test period and experimental material at the end automatically includes any run-to-run analytical difference in the results and may well lead to an incorrect conclusion. The recommended approach is, if at all possible, to analyze the experimental and control subsets together in a random order, within a single run of analysis, that is, under repeatability conditions. Any highly significant difference between the mean results of the two subsets can then safely be regarded as evidence of instability. However, it must be emphasized that such tests are of low statistical power unless inordinately large numbers of test units are involved or highprecision analytical methods are employed. Although, in principle, a distinction must be drawn between statistically significant instability and consequential instability, because of the low statistical power of the test given practicable conditions of testing, a significant result is unlikely. For instance, given an analytical method with repeatability standard deviation an and two subsets of 10 distribution units each, an instability change of a magnitude of an would be found significant in only 56% of the experiments by using the two-sample t-test at 95% confidence. Thus, if it were deemed that a change in the z-score of 0.05 would be inconsequential, then an instability amounting to a change in analyte concentration of 0.05p could be tolerated. However, an analytical method with an 0:05p is unlikely to be available.

1.03.4 Further Information from Proficiency Testing Results Although proficiency tests are designed solely for the immediate benefit of individual participants, there are certain further uses of the resultant data that can be used for more general purposes, without requiring changes that would compromise the primary purpose. 1.03.4.1

Review of the Operation of a Scheme

The scheme provider can examine several aspects of the scheme that reflect its efficacy. First, on the assumption that participants are trying to achieve fitness for purpose, we should see the robust standard


deviation of the results ˆ rob p or, in immature schemes, ˆ rob approaching p in successive rounds. Efficacy can be quantified as the proportion of participants that obtain a ‘satisfactory’ score, say within the range 2, or the ratio ˆ rob =p , which should be approximately 1. Providers should review the operation of a scheme that seems to be inefficacious, although they usually have no powers to compel participants to compliance. They should, however, advise participants if the lack of compliance seems to be the result of factors such as inept choice of the analytical method. Providers should also occasionally compare for significant bias a consensustype assigned value with an independent result from a national reference laboratory. 1.03.4.2

Information about Analytical Methods

If participants report the methods they have used to obtain the results, some performance indicators of the methods can sometimes be deduced. For example, if many participants followed closely a particular fully documented analytical procedure, a robust standard deviation of the results from a round would be akin to a reproducibility standard deviation obtained from a collaborative trial. This in turn can be regarded as a guide to the uncertainty that might pertain to results from individual laboratories, although, of course, results from some laboratories may not reach the implied accuracy. Over the course of several (say 5–10) rounds of the scheme, a good picture of the capability of the method would be assembled, at little cost. In some instances where there are several distinct methods available (or clearly defined variants of a single method), and enough participants use each of them, it is possible to compare the mean result of each method in a round and search for significant bias among them. It must be stressed that the within-method variability is of such magnitude that it is difficult to achieve a worthwhile result with this kind of comparison, unless (1) the methods are well defined, (2) large numbers of results are available for each method, and (3) the bias is substantial. As a consequence, few satisfactory studies of this kind have been reported.30,31

References 1. Thompson, M.; Wood, R. The International Harmonized Protocol for the Proficiency Testing of (Chemical) Analytical Laboratories. Pure Appl. Chem. 1993, 65, 926–940. 2. International Organisation for Standardisation. ISO Guide 43: Proficiency Testing by Interlaboratory Comparisons – Part 1: Development and Operation of Proficiency Testing Schemes; International Organisation for Standardisation: Geneva, Switzerland, 1994. 3. Lawn, R. E.; Thompson, M.; Walker, R. F. Proficiency Testing in Analytical Chemistry; The Royal Society of Chemistry: Cambridge, 1997. 4. International Laboratory Accreditation Cooperation. ILAC-G13:2000. Guidelines for the Requirements for the Competence of Providers of Proficiency Testing Schemes. Available at www.ilac.org. 5. International Organisation for Standardisation. ISO 13528: Statistical Methods for Use in Proficiency Testing by Interlaboratory Testing; International Organisation for Standardisation: Geneva, Switzerland, 2005. 6. Thompson, M.; Ellison, S. L. R.; Wood, R. The International Harmonized Protocol for the Proficiency Testing of Analytical Chemistry Laboratories. Pure Appl. Chem. 2006, 78, 145–196. 7. Nogueira, J. M. F.; Nieto-de-Castro, C. A.; Cortez, L. EPTIS: The New European Database of Proficiency Testing Schemes for Analytical Laboratories. Trends Anal. Chem. 2001, 20, 457–461 (see also www.eptis.bam.de). 8. Analytical Methods Committee. Robust Statistics – How Not to Reject Outliers – Part 1: Basic Concepts. Analyst 1989, 114, 1693–1697. 9. Thompson, M. The Variance of a Consensus. Accred. Qual. Assur. 2006, 10, 574–575. 10. AMC Software and AMC Datasets. Available at www.rsc.org. 11. Lowthian, P. J.; Thompson, M. Bump-Hunting for the Proficiency Tester: Searching for Multimodality. Analyst 2002, 127, 1359–1364. 12. Silverman, B. W. Density Estimation for Statistics and Data Analysis; Chapman and Hall: London, 1986. 13. Cofino, W. P.; van Stokkum, I. H. M.; Wells, D. E.; Ariese, F.; Wegener, J. W. M.; Peerboom, R. A. L. A new Model for the Inference of Population Characteristics from Experimental Data Using Uncertainties Application to Interlaboratory Studies. Chemom. Intell. Lab. Syst. 2000, 53, 37–55 14. Fearn, T. Comments on ‘Cofino Statistics’. Accred. Qual. Assur. 2004, 9, 441–444. 15. Dempster, A. P.; Laird, N. M.; Rubin, D. B. Maximum Likelihood from Incomplete Data via the EM Algorithm. J. R. Stat. Soc. Ser. B 1977, 39, 1–38. 16. Aitkin, M.; Wilson, G. T. Mixture Models, Outliers and the EM Algorithm. Technometrics 1980, 22, 325–331. 17. Thompson, M. Using Mixture Models for Bump-Hunting in the Results of Proficiency Tests. Accred. Qual. Assur. 2006, 10, 501–505. 18. Lindley, D. V. Making Decisions, 2nd ed.; Wiley: London, 1984. 19. Thompson, M.; Fearn, T. What Exactly Is Fitness for Purpose in Analytical Measurement? Analyst 1996, 121, 275–278.


95

20. Fearn, T.; Fisher, S.; Thompson, M.; Ellison, S. R. L. A Decision Theory Approach to Fitness for Purpose in Analytical Measurement. Analyst 2002, 127, 818–824. 21. Ramsey, M. H.; Lyn, J. A.; Wood, R. Optimised Uncertainty at Minimal Overall Cost to Achieve Fitness for Purpose in Food Analysis. Analyst 2001, 126, 1777–1783. 22. Ramsey, M. H.; Taylor, P. D.; Lee, J. C. Optimised Contaminated Land Investigation at Minimal Overall Cost to Achieve Fitness for Purpose. J. Environ. Monit. 2002, 4, 809–814. 23. Analytical Methods Committee. Understanding and Acting Upon Scores Obtained in Proficiency Testing Schemes. AMC Technical Briefs 2002, No. 11. Available at www.rsc.org. 24. Analytical Methods Committee. Proficiency Testing: Assessing z-Scores in the Longer Term. AMC Technical Briefs 2004, No. 16. Available at www.rsc.org. 25. Thompson, M. Harmonised Guidelines for Internal Quality Control in Analytical Chemistry Laboratories. Pure Appl. Chem. 1995, 67, 649–666. 26. Montgomery, D. C. Introduction to Statistical Quality Control; Wiley: Hoboken, NJ, 2005. 27. Analytical Methods Committee. The J-Chart: A Simple Plot that Combines the Capabilities of Shewhart and Cusum Charts, for Use in Analytical Quality Control. AMC Technical Briefs 2003, No. 12. Available at www.rsc.org. 28. Thompson, M.; Ellison, S. L. R.; Owen, L.; Mathieson, K.; Powell, J.; Key, P.; Wood, R.; Damant, A. P. Scoring in Genetically Modified Organism Proficiency Tests Based on Log-Transformed Results. J. AOAC Int. 2006, 89, 232–239. 29. Fearn, T.; Thompson, M. A New Test for ‘Sufficient Homogeneity’. Analyst 2001, 126, 1414–1417. 30. Thompson, M.; Owen, L.; Wilkinson, K.; Wood, R.; Damant, A. A Comparison of the Kjeldahl and Dumas Methods for the Determination of Protein in Foods, Using Data from a Proficiency Testing Scheme. Analyst 2000, 127, 1666–1668. 31. Lowthian, P. J.; Thompson, M.; Wood, R. The Use of Proficiency Tests to Assess the Comparative Performance of Analytical Methods: The Determination of Fat in Foodstuffs. Analyst 1996, 121, 977–982.


Biographical Sketch

Michael Thompson is Emeritus Professor of Analytical Chemistry in the University of London. He is the author of a book on proficiency testing (Proficiency Testing in Analytical Chemistry) and the International Harmonized Protocol on Proficiency Testing in Analytical Chemistry. He has also published about 30 research papers on proficiency testing and is statistical adviser to five international proficiency testing schemes. He is the Editor of AMC Technical Briefs. He is the recipient of the SAC Gold Medal (1995), the L. S. Theobald Lectureship (2006) (both from the Royal Society of Chemistry), the Harvey W. Wiley Award (2007) (from AOAC International), and Honorary Life Membership of the International Association of Geoanalysts (2008).

1.04

Statistical Control of Measures and Processes

A. J. Ferrer-Riquelme, Technical University of Valencia, Valencia, Spain ª 2009 Elsevier B.V. All rights reserved.

1.04.1 1.04.2 1.04.3 1.04.4 1.04.5 1.04.6 1.04.6.1 1.04.7 1.04.7.1 1.04.8 1.04.9 1.04.9.1 1.04.9.1.1 1.04.9.1.2 1.04.9.1.3 1.04.9.1.4 1.04.9.2 1.04.9.2.1 1.04.9.2.2 1.04.9.3 1.04.9.3.1 1.04.9.4 1.04.10 References

Introduction: Basics of Process Monitoring Phases in Process Monitoring Shewhart Control Charts CUSUM Control Charts EWMA Control Charts Performance Measures of Control Charts Example 1 Control Charts for Autocorrelated Processes Example 2 Integration of SPC and Engineering Process Control Multivariate Control Charts Original Variables-Based MSPC Schemes Hotelling T 2 control charts for monitoring process mean Multivariate Shewhart-type control charts for monitoring process variability MEWMA control charts Fault diagnosis Latent Variables-Based MSPC Schemes PCA-based MSPC: exploratory data analysis and offline process monitoring (Phase I) PCA-based MSPC: online process monitoring (Phase II) Potential of Latent Variables-Based MSPC in Industry Example 3 Monitoring Measurement Systems: Applications of SPC in Analytical Laboratory Methods Software

Symbols A

A3 ,B3 , B4 , and c4

A2 ,D3 ,D4 , and d2

b dimension of the principal component subspace coefficients for the calculation of the control limits of the

BðK=2;ðm – K – 1Þ=2Þ;

x – s control chart

c

coefficients for the calculation of the control limits of the

Ct

x – R control chart at Nð0;2 Þ

series of independent and identically normally distributed (iind) random disturbances with zero mean and variance

ContðDModX;xnew;k Þ

100 102 102 104 105 106 106 108 108 111 112 112 113 114 114 115 115 116 119 120 121 121 123 124

sample mean of the SPE sample 100(1–)% percentile of the beta distribution with K/2 and (m–K–1)/2 degrees of freedom correction factor for DModX CUSUM control statistic at stage t (definition 1) number of nonconformities in samples of size n at stage t (definition 2) contribution of kth variable to the

2

97

98 Statistical Control of Measures and Processes

ContðSPE;xnew;k Þ

Contðtnew;a ;xnew;k Þ

D

Dt

E enew,k

Et ei et

FðK;ðmn – m – Kþ1ÞÞ;

f(x; ) H

K

L

m

MR ¼

m P t¼2

n P

MRð2Þt =ðm – 1Þ

DModX in a new observation contribution of kth variable to the SPE in a new observation contribution of kth variable to the ath score at a new observation magnitude of the shift in the process parameter number of nonconforming units in samples of size n at stage t (m K) residual matrix residual corresponding to the kth variable in a new observation EWMA control statistic at stage t residual vector (ith row of matrix E) residual at stage t 100(1–)% percentile of the F distribution with K and mn–m–Kþ1 degrees of freedom probability distribution of x decision value (threshold) in the CUSUM control chart reference value in the CUSUM control chart distance of the control limits from the centerline number of samples average of the moving ranges sample size (K A) loading matrix

p

¼ p

m P

Dt =nm ¼

t¼1

m P

pˆ t =m

t¼1

pak

pˆ t

m ¼ P Rt =m R

population fraction nonconforming average sample fraction nonconforming loading of the kth variable at the ath principal component sample fraction nonconforming at stage t average range

t¼1

rk S

jSj s ¼

m P

st =m qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi Pn 2 st ¼ i¼1 ðxti – xt Þ =n – 1 t¼1

s0 T tnew,a

tr S Tt2

tTi ¼ fti1 ;ti2 ;:::;tiA g

¼ u

m P t¼1

ut

v

Ct =nm ¼

m P t¼1

ut =m

sample autocorrelation function (K K) sample variance– covariance matrix sample generalized variance average standard deviation sample standard deviation at stage t pooled residual standard deviation (m A) score matrix score of a new observation at the ath principal component trace of the covariance matrix Hotelling T2 control statistic at stage t A-dimensional score vector (ith row of matrix T) average sample number of nonconformities per unit sample average number of nonconformities per unit at stage t sample variance of the SPE sample


vt ¼ v(zt)

wk

wt Wt

x X X

x Nð; SÞ

x¯

x¯ t

Xt ¼ fx1t ;x2t ;:::;xKt g

x¯ ¼

m P

xt =m

control statistic to be plotted in the control chart square root of the explained sum of squares for the kth variable MEWMA control statistics at stage t multivariate control statistic for monitoring process variability (a direct extension of the univariate s2 control chart) key characteristic to be measured preprocessed Z (m K) data matrix (m K) prediction matrix x follows a multivariate normal distribution with mean vector m and covariance matrix S (K1) overall sample mean vector. (K 1) vector of sample means at stage t (n K) observations matrix of K characteristics measured on n units sampled at stage t grand average

t¼1

xTkt ¼ fxk1t ; xk2t ;:::; xknt g

xTnt ¼ fxn1t ; xn2t ;:::; xnKt g

x new;k

n-dimensional vector of sample values of kth variable (kth column of matrix X) K-dimensional observation vector of nth unit (nth row of matrix X) prediction of the kth variable in a new observation

99

xnew;k from the PCA model xTt ¼ fxt1 ; xt2 ;:::; xtn g

xˆ tþ1=t

Z

zt ¼ – ln½fðxt ; 0 Þ=fðxt ; 1 Þ zt ¼ MRð2Þt ¼ jxt – xt – 1 j

zt ¼ Rt ¼ maxðxt Þ – minðxt Þ P zt ¼ s2t ¼ ni¼1 ðxti – xt Þ2 =n – 1 Pn zt ¼ xt ¼ i¼1 xti =n zt ¼ z(xTt ;0 ) Y jS j

0 ˆt

observations vector of the key characteristic on n units sampled randomly from the production or measurement process at stage t one-step ahead forecast of x made at stage t (m K) data matrix likelihood ratio statistic moving range of two consecutive observations at stage t sample range sample variance sample mean suitable statistic (or seed) to build the chart on. (A A) covariance matrix of T generalized variance Type I risk number of standard deviations of the shift in the process parameter process parameter to be monitored in-control (target or reference) value of minimum variance unbiased estimator of based on the sample values xTt ¼

fxt1 ;xt2 ;:::;xtn g

smoothing constant of the EWMA control statistic (definition 1) population average number of

100

a


nonconformities in samples of size n (definition 2) smoothing constant of the MEWMA control statistic (definition 3) eigenvalue of the ath principal component

2K;

process mean autoregressive coefficient of AR(1) model 100(1–)% percentile of the

2 distribution with K degrees of freedom

1.04.1 Introduction: Basics of Process Monitoring Processes, whether chemical, biochemical, or analytical in nature, are subject to certain variability. You will not always get the same result each time no matter how tightly you operate a process. This is exactly what any customer perceives from the output (i.e., product, service, or information) of any process. If this output is to meet or exceed customer expectations, it should be produced by a process that is stable or repeatable.1 Variation remaining in a stable process reflects ‘common causes’ that are an inherent part of the process and which cannot be removed easily without fundamental changes in the process itself: for example, the design and operating settings of a polymerization reactor contribute to the common-cause variation of the viscosity of a polymer. As long as you keep the process the same, common-cause variation is the same from day to day, yielding a process consistent over time. A process affected by only common (random) causes of variation is said to be ‘in statistical control’. This means that the process is predictable, that is, you can predict what the process will make in the near future, for example, how many batches will meet the customer specifications for a particular key property. In addition to the common-cause system, other types of variability may occasionally affect the stability of a process: for example, operator errors, defective raw material, improperly adjusted or controlled equipment, sudden change in environmental conditions, and so on. These sources of variability are not part of the commoncause system and, thus, they are usually referred to as ‘assignable’ or ‘special causes’. They are sporadic in nature and cannot be predicted. A process also affected by this type of special causes of variation is said to be ‘out of control’ in the sense that it is unstable and, thus, unpredictable. It is crucial to distinguish between the two sources of variation because sharing the responsibility for process improvement depends on what type of variation is present: Frontline personnel are responsible for finding and taking decisions on assignable causes; common-cause variation, on the contrary, is management’s responsibility. Statistical process control (SPC) is a powerful collection of problem-solving tools useful in understanding process variability, achieving process stability, and improving process performance through the reduction of avoidable variability (i.e., special causes). The goal of any SPC scheme is to monitor the performance of a process along time to check whether the process behaves as it is expected to do (i.e., the predicted behavior from the common-cause system), and to detect any unusual (special) event that may occur. By finding assignable causes for them, significant improvements in the process performance can be achieved by eliminating these causes (or implementing them if they are beneficial). Control charts are the essential tools for pursuing this goal. A control chart is a picture of a process over time that helps to identify the magnitude and the type of variation present. To implement a control chart, one must register data from a process over time. Decisions are required about the variables to measure, the sample statistics to be monitorized, the sample size, the time between samples, the type of control chart, the control limits, and the decision rules. All these choices


101

determine the cost of the monitoring scheme and its ability to detect certain out-of-control conditions. The design of a control chart involves the following steps: 1. Identifying the key characteristic x to be measured (e.g., a quality property, a key process variable that in some way affects the quality of a product, or an analytical measurement of a control or reference sample), the process parameter to be monitored (e.g., process mean, variance, proportion of defectives or nonconforming items, rate of defects or nonconformities), and the probability distribution of x, f (x; ) (e.g., normal, binomial, or Poisson). In this context, a process is said to be in statistical control (i.e., in control) if the underlying statistical model representing the key characteristic is stable over time. In this case, the process is said to be operated under normal operating conditions (NOCs). If there is some change over time in this statistical model, the process is said to be out of control.2 2. Selecting a suitable statistic (or seed) zt ¼ zðxT t ; 0 Þ to build the chart, where 0 is the in-control (target or reference) value of the monitorized process parameter and xT t ¼ fxt 1 ;xt 2 ; . . .; xtn g is the observations vector of the key characteristic on n units sampled randomly from the production or measurement process at stage t (t ¼ 1, 2, . . .). In general, seeds of the form zt ¼ ˆt , where ˆt is the minimum variance unbiased estimator of based on the sample values xT t ¼ fxt1 ; xt 2 ; . . .; xtn g, are often considered. Another choice is the likelihood ratio statistic, zt ¼ – ln½f ðxt ; 0 Þ=f ðxt ; 1 Þ, motivated by the Neyman–Pearson theory of testing statistical hypotheses (H0: ¼ 0 versus H1: ¼ 1).3 The observations in xT t can be measurements (e.g., filling volumes, weights, viscosities, particle sizes, and analyte sample concentrations) or counts (e.g., defective/nondefective item or number of defects) yielding the so-called control charts for variables or attributes, respectively. When observations are measurements, sample size n is small with values ranging from 1 to 5; when dealing with counts, sample size n is usually large (e.g., 50 or even more). Standard control charts for variables assume that the key characteristic x at stage t is a normally distributed random variable that can be expressed as xt ¼ þ at , where is the process mean and at follows a stochastic ‘white noise’ process, that is, a series of independent and identically normally distributed (iind) random in xt disturbances with zero mean and variance 2 (at N(0, 2)). Therefore, in control, the observations Pn are assumed to be iind with mean 0 and variance 20 . The sample mean zt ¼ xt ¼ i¼1 xti =n is the seed usually chosen to monitor the process mean ¼ . To monitor the process variance ¼ 2, several seeds have been proposed: for example, the sample range zt ¼ Rt ¼ maxðxt Þ – minðxt Þ or the sample variance P zt ¼ st2 ¼ ni¼1 ðxti – xt Þ2 =n – 1 Control charts for attributes are based on counts (attributes or discrete data). There are two kinds of attributes data: (i) yes/no and (ii) number of defects. In the yes/no case, each unit (item or sample) inspected is either defective (i.e., it does not meet some preset criterion) or not defective (i.e., it meets the criterion). The key characteristic x is a dichotomous random variable with only two possible outcomes: It either passes or fails the preset criterion. Examples of this kind of yes/no data include the following: catalyst has been correctly activated or not; polymer reaction has suffered from fouling problems or not; batch meets the requirements on one or more product properties or not; and so on. The statistical principles underlying this attributes control chart are based on the binomial distribution. Suppose the production process is operating in a stable manner (i.e., in control), such that the probability that any unit will not meet the preset criterion (i.e., the unit is defective or nonconforming) is p, and that successive units produced are independent, then each unit produced is a realization of a Bernouilli random variable with parameter p. If at stage t a random sample of n units is selected and if Dt is the number of nonconforming units, then Dt has a binomial distribution with parameters n and p (Dt binomial (n; p)), with mean Dt ¼ np and variance 2Dt ¼ npð1 – pÞ. In this case, ¼ p is the only parameter to be monitorized and the seed usually chosen is the sample fraction nonconforming zt ¼ pˆt ¼ Dt =n. A nonconforming unit will contain at least one defect or nonconformity. If it is feasible to count these defects, the key characteristic x is the number of defects or nonconformities in each inspected unit. The statistical assumption in this case is that at stage t the occurrence of nonconformities in samples of size n, Ct , is well modeled by the Poisson distribution with parameter (Ct Poisson ( )), with mean and variance

102


Ct ¼ 2Ct ¼ . This requires that the number of opportunities for defects be infinitely large, the probability of occurrence of a nonconformity be small and constant, and that successive units produced are independent. This process depends only on parameter ¼ and usually the seed used is the sample average number of nonconformities per unit zt ¼ ut ¼ Ct =n. 3. Selecting the control statistic vt ¼ v(zt) most appropriate to detect certain out-of-control situations (e.g., changes in the parameter or other kind of abnormalities), the control limits of the chart, and the decision rules. The role of a control chart is to provide a procedure for a timely detection of a change in the value of . Depending on the particular goal, to detect a pulse, exponential decrease or increase, rectangular bump or sustained change in the parameter , a Shewhart, EWMA (exponentially weighted moving average), MA (moving average), or CUSUM (cumulative sum) type of control chart may be constructed on the selected seed zt, respectively.4 These various types of charts differ in the way present and past seeds zt are weighted, leading to different control statistics vt to be plotted in a time sequence on the control chart.

1.04.2 Phases in Process Monitoring Standard control chart usage involves two distinct phases, with two different goals: Phase I (model building) and Phase II (model exploitation). In Phase I, a set of process data is gathered and analyzed all at once in a retrospective way, estimating the parameters of the control statistics used and calculating trial control limits to determine if the process has been in control over the period of time where the data were collected. This is an iterative process where at each step the out-of-control observations are detected, carefully analyzed, and root causes are investigated. Those out-of-control observations considered as real anomalies are omitted from the data set and new trial control limits are recalculated from the resulting data. This phase ends when there is enough evidence that the data gathered come from an in-control process and hence the control limits can be used for Phase II. This means that the performance of the process has been understood and modeled, and the assumptions of its behavior and process stability are checked. No matter whether the collected data come from a historical database or they have been recently sampled, a wise use of Phase I control charts can increase process understanding and process improvement, assisting operating personnel and process managers in bringing the process into a state of statistical control. In Phase II, real online process monitoring is carried out by plotting the control statistics from new samples on the control charts defined in Phase I. As in Phase I, once an out-of-control sample is detected, this is carefully studied trying to find out the nature of the assignable causes affecting the process and implementing countermeasures to eliminate them if they deteriorate the process performance (or incorporate them in case they improve the process performance). Process knowledge and experience from operators and process managers are critical inputs for a successful diagnostic process. In simplex processes (e.g., univariate), the out-of-control action plans (OCAPs) as discussed by Montgomery1 can also be very useful in expediting this diagnostic process. The OCAPs are based on the analysis of known failure modes and provide the manual for identification and elimination (or incorporation if beneficial) of the assignable causes in an expert system-like fashion. In complex processes (e.g., multivariate), other diagnostic tools are needed as explained later on. As a result of this diagnostic process, process understanding and process improvement can also be obtained.

1.04.3 Shewhart Control Charts This type of monitoring schemes refers to any control chart developed according to the statistical principles proposed in the 1920s by Shewhart.5 In this case, the control statistic to be charted along stages t is the selected seed: vt ¼ zt ¼ zðxT t ; 0 Þ

ð1Þ


103

The chart contains a centerline that represents the in-control average value of the selected seed zt and two horizontal lines, called upper control limit (UCL) and lower control limit (LCL). These control limits are chosen so that if the process is in control, the expected fraction of seeds beyond the control limits takes a prespecified value . Choosing the control limits is equivalent to setting up the critical region for testing the hypothesis H0: ¼ 0 versus H1: 6¼ 0, being the Type I risk of the test (i.e., the chance of wrongly rejecting the null hypothesis when it is true; for more information on hypothesis testing, see Chapter 1.02). Under the assumption that zt ðiindÞ N ðz ;2z Þ, the control limits are set at z Lz , where L is the distance of the control limits from the centerline, expressed in standard deviation units, and this depends on . Usually, ¼ 0.0027, leading to the classical ‘three-sigma’ control limits (L ¼ 3). One of the most common Shewhart charts Pn for variables is the xR control chart. This is made of two charts: The x chart uses the sample mean ðzt ¼ xt ¼ i¼1 xti =nÞ to monitor the process mean ¼ , whereas the R chart uses the sample range ðzt ¼ Rt ¼ maxðxt Þ – minðxt ÞÞ to monitor the process variance ¼ 2. Control limits on both charts are estimated from preliminary samples taken when the process is thought to be in control. To obtain good where estimates, at least m ¼ 20–25 samples are required. The control limits on the x chart are set at x A2 R, Pm x ¼ t ¼2 xt =m is the grand average. The control limits on the R chart are plotted at D3 R (LCL) and D4 R (UCL), P where the average range R ¼ mt ¼1Rt =m is the centerline. Assuming the process is in control, the process standard 2 . The values for A2, D3, D4, and d2 are tabulated for various sample sizes n.1 deviation can be estimated by R=d The sample range R, although easy to calculate by hand and easy to understand for most people, is not the best pP ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi n t Þ2 =n – 1 give better estimates way to monitor process variability. Sample standard deviations st ¼ i¼1 ðxti – x of the process standard deviation, particularly for moderate to large sample sizes. The current availability of handheld calculators and personal computers for online implementation of control charts has eliminated any computational difficulty. If st replaces Rt as the seed, this leads to the x – s control chart. The control limits Pmon the x The centerline of the s chart is plotted at the average standard deviation s ¼ t¼1 chart are set at x A3 R. st =m, and the control limits at B3 s (LCL) and B4 s (UCL). Assuming the process is in control, the process standard deviation can be estimated by s =c 4 . The values for A3, B3, B4, and c4 are tabulated for various sample sizes n.1 There are many situations in which the sample size used for process monitoring is 1, that is, n ¼ 1. Some examples are (i) every unit manufactured is analyzed (e.g., 100% inspection in highly automated parts industry or batch processes), (ii) the process generates data on a very limited frequency (e.g., the test method used to analyze the process is very expensive to run or takes a long time, or when dealing with accumulated quantities such as daily energy consumption), and (iii) repeated measurements on the process differ only because of laboratory or analysis error, as in many chemical continuous processes. In such situations, a control chart for individual measurements is useful to monitor the process. The most common individuals control chart is the x – MRð2Þ individuals control chart (moving range (MR)). This actually includes two charts: The x chart displays the value of the key characteristic at stage t ðzt ¼ xt Þ to monitor the process mean ¼ , whereas the MR(2) chart uses the moving range of two consecutive observations ðzt ¼ MRð2Þt ¼ jxt – xt –1 jÞ to monitorPthe process variance ¼ 2. The MR(2) chart has centerline at the m average of the moving ranges MR ¼ t ¼2 MRð2Þt =ðm – 1Þ and control limits Pm at D3 MR (LCL) and D4 MR (UCL). The control limits on the x chart are set at x 3MR=d2 , where x ¼ t ¼1 xt =m is the sample average of all the observations. Assuming the process is in control, the process standard deviation can be estimated by MR=d2 . The values for D3, D4, and d2 are tabulated for various sample sizes n.1 Caution should be taken when normality assumption cannot be assumed. In this case, the control limits may be based on resampling methods.6 Regarding control charts for attributes, the most frequently used are the p chart and the u chart. The p chart displays the sample fraction nonconforming zt ¼ pˆt ¼ Dt =n to monitor the population fraction nonconforming p, where Dt is the number of nonconforming units in the sample. Assuming that the binomial probability model pffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi is adequate and can be approximated by a normal distribution, the controlPlimits are set at p 3 p ð1 – pÞ=n, P m m where the centerline is the average sample fraction nonconforming p ¼ t ¼1 Dt =nm ¼ t ¼1 pˆ t =m. In those cases where the normality assumption is not appropriate (i.e., when n pð1 – pÞ < 9), the control limits have to be set based on /2 and 1/2 percentiles from the binomial probability model.7 On the contrary, the u chart displays the sample average number of nonconformities per unit zt ¼ ut ¼ Ct =n to monitor the population average number of nonconformities per unit /n, where Ct is the occurrence of nonconformities in samples of size n. Assuming Ct is well modeled by the Poisson distribution with parameter and can be approximated by a

104


Table 1 Formulas for Shewhart control charts Chart

Centerline

Control limits

x (using R) R x (using s) s x (using MR) MR p u

x R

x A2 R UCL ¼ D4 R LCL ¼ D3 R; x A3 R LCL ¼ B3 s ; UCL ¼ B4 s x 3MR=d2 LCL ¼ D3 MR; UCL ¼ D4 MR pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi p 3 pð1 – pÞ=n pffiffiffiffiffiffiffi u 3 u=n

x s x MR p u

pffiffiffiffiffiffiffi normal distribution, the control limits 3 u=n, where the centerline is the average sample number Pm are set at u P m of nonconformities per unit u ¼ t¼1 Ct =nm ¼ t ¼1 ut =m. In those cases where the normality assumption is not appropriate (i.e., when < 9), the control limits have to be set based on /2 and 1/2 percentiles from the Poisson probability model.7 Table 1 summarizes the formulas to calculate the centerline and control limits for the different Shewhart control charts introduced. To determine whether a process is out of control, several rules have to be defined. These rules refer to extremely low probability events when the process is in control. So, when they occur, you are fairly certain that something has changed in the process and it is worth looking for an assignable cause. The main rule used on Shewhart control charts is one point exceeding the control limits. Using three-sigma limits for the control charts, this rule has an in-control probability of 0.0027. This is called the false alarm probability for this out-of-control rule. Other rules, called supplementary rules, have been suggested for detecting any nonrandom pattern of behavior. Some examples are as follows: a run of eight (or seven) consecutive points on one side of the centerline; eight (or seven) points in a row trending up (or down); and two out of three consecutive points beyond the twosigma warning limits ðz 2z Þ. Although the application of these supplementary rules increases the sensitivity of the Shewhart control charts to the detection of small process shifts, care should be exercised when using several decision rules simultaneously because of the increase in the overall false alarm probability.8 Shewhart control charts are extremely useful in Phase I implementation of SPC (model building), where the process is likely to be out of control and experiencing assignable causes resulting in large shifts in the monitored parameters. They are also very useful in the diagnostic aspects of bringing a ‘wild’ process into statistical control, because the patterns on these charts often provide guidance regarding the nature of the assignable cause.1 This advantage comes from the fact that they are plots of the actual data, providing a picture of what the process is actually doing, which makes the interpretation easy. Because they are not specialized, these charts act as global radars, being potentially capable of drawing attention to unusual kinds of behavior and hence to possible signals and causes previously unsuspected.4 The major disadvantage of Shewhart control charts is that they are relatively insensitive to small process shifts, quite often in Phase II monitoring (model exploitation). This drawback is due to the fact that at each sampling time t it uses only the contemporary information, ignoring any potential information contained in past samples (t1, t2, . . .). Time weighted control charts can be designed for this purpose (i.e., detecting small shifts). Out of the different proposals, the CUSUM control charts and the EWMA control charts are the best choices.

1.04.4 CUSUM Control Charts The CUSUM control chart was first designed by Page during the 1950s.9,10 In this case, the control statistic to be charted at stage t is the CUSUM of past and present deviations of the selected seed zt ¼ zðxT t ;0 Þ from a target (in-control) value 0: vt ¼ Ct ¼

t X k¼1

ðzk – 0 Þ

ð2Þ


105

This feature makes this scheme very powerful for detecting persistent changes in the process parameter , particularly a shift from the in-control value 0 to another (out-of-control) value 1. CUSUMs can be represented in two ways: the tabular (or algorithmic) CUSUM and the V-mask CUSUM. The tabular CUSUM, which is the preferred one, involves two statistics, Ctþ and Ct– . Ctþ is the sum of deviations above the target and is referred to as the one-sided upper CUSUM. Ct– is the sum of deviations below the target and is referred to as the one-sided lower CUSUM:

Ctþ ¼ max 0;zt – ð0 þ K Þ þ Ctþ– 1

Ct– ¼ max 0;ð0 – K Þ – zt þ Ct–– 1

ð3Þ ð4Þ

where the starting values are Ctþ ¼ Ct– ¼ 0. The constant K is referred to as the reference value, and it is often chosen as K ¼ j1 – 0 j=2, that is, halfway between the target value 0 and the out-of-control value 1 we are aiming to detect quickly. If the shift is expressed in standard deviation units as 1 ¼ 0 þ , then K ¼ ð=2Þ, that is, one-half of the magnitude of the shift. However, other values of K can be used. The two-sided CUSUM control chart plots the values of Ctþ and Ct– for each sample. If either statistics lies beyond a stated decision value (or threshold) H, the process is considered to be out of control. The choice of H is not arbitrary and should be chosen after careful consideration. A reasonable value is H ¼ 5. Actually, the proper selection of both parameters K and H is crucial for the good performance of the CUSUM control chart.3 CUSUMs have been developed for a large variety of sample statistics (monitoring seeds) such as individual observations, sample averages, ranges, and standard deviations of rational subgroups, sample fractions nonconforming, and average numbers of nonconformities per inspection unit.

1.04.5 EWMA Control Charts The EWMA control chart is also a good alternative to the Shewhart control chart for detecting small shifts. The EWMA control chart was first introduced by Roberts.11 In this case, the control statistic to be charted is an EWMA of present and past values of the selected seed zt : vt ¼ Et ¼ zt þ ð1 – ÞEt – 1

ð5Þ

where is a smoothing constant (0 < 1). The EWMA can also be expressed as Et ¼ ð1 – Þt E0 þ

t X

ð1 – Þt – k zk

ð6Þ

k¼1

where the starting value E0 is the process target 0. From this expression, it is clear how the weights ð1 – Þt –k decrease geometrically with the age of the sample statistic zt. The parameter is a tuning parameter. If is small, the EWMA performs like a CUSUM. On the contrary, if is close to one, the EWMA is quite similar to the Shewhart chart. Several EWMA charts have been developed for different goals using a large variety of sample statistics (monitoring seeds). The most used are the EWMA with individual observations ðzt ¼ xt Þ or sample averages qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi ðzt ¼ xt Þ for monitoring the process mean. The control limits are set at 0 Lz ð =ð2 – ÞÞ 1 – ð1 – Þ2t , where L is the width of the control limits and z is the standard deviation of the sample statistic z. Once the chart is started, after several time periods these control limits will approach steady-state values given by pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 0 Lz =ð2 – Þ. The process is considered out of control if one or more zt values exceed the control limits. For a given smoothing constant , the value L is chosen to give the desired performance monitoring of the EWMA chart. Usual values for both parameters are 0.05 0.25 and 2.6 L 3 (see Hunter12 for details). In practice, the EWMA chart is used more than the CUSUM chart because it is easier to set up and operate. For monitoring the variability of normally distributed processes, several EWMA-based statistics have been proposed. One example is the exponentially weighted mean square error (EWMS) chart. This chart uses the sample statistic zt ¼ ðxt – 0 Þ2 . To avoid the sensitivity of this chart to shifts in process mean, the exponentially

106


weighted moving variance (EWMV) chart has been proposed by replacing 0 in the sample statistic of EWMS with an estimate of the process mean obtained from the classical EWMA statistic for the process mean. EWMA charts can also be used for monitoring the average numbers of nonconformities per inspection unit. This is called the Poisson EWMA, and uses the sample average number of nonconformities per unit zt ¼ ut ¼ Ct =n as the sample statistic to be weighted. For technical details on control limits of these charts see Montgomery.1

1.04.6 Performance Measures of Control Charts The performance of a control chart is usually evaluated based on two criteria: how often the control chart signals for an in-control process (false alarm rate) and how slow the control chart is in detecting a specific change in the process (excessive delay in detection). Unfortunately, both criteria cannot be optimized at the same time. For example, to reduce the false alarm rate, the control limits can be moved farther from the centerline. However, widening the control limits will increase the risk of a point falling between the control limits when the process is really out of control. Therefore, it will take more time to detect a change in the process, hence causing excessive delay in detection. This concept is closely related to Type I error (rejecting the null hypothesis when it is true) and Type II error (accepting the null hypothesis when it is false) in hypothesis testing. Two different performance measures, in-control average run length (ARL0) and out-of-control average run length (ARLD), are defined for evaluating the false alarm rate and the excessive delay in detection, respectively. ARLD depends on the magnitude of the shift in the process parameter to be monitored, D ¼ j1 – 0 j ¼ . The average run length (ARL) is defined as the expected (average) number of samples that must be plotted before an out-of-control signal is observed. For an in-control process ARL0 should be as high as possible. For example, if the process observations are uncorrelated, then for any Shewhart control chart with no supplementary out-of-control rules ARL0 ¼ 1/, where is the probability that any point exceeds the control limits when the process is in control. Under normality and three-sigma control limits, ARL0 ¼ 1/0.0027 ¼ 370. This means that, even if the process remains in control, a point exceeding the control limits will appear every 370 samples, on the average. For a control chart with given control limits, ARL0 is unique. On the contrary, if the process is out of control owing to a shift of magnitude D, ARLD is a function of D and it should be as small as possible, meaning that the control chart signals the change as soon as possible right after the shift occurs. The performance of a particular control chart can be described by plotting the ARLD values against different suspected shifts D (note that ARLD¼0 ¼ ARL0). This is also a useful tool to compare the performance of different control charts. When comparing the performance of different control charts, ARL0 should be equal for all the schemes. This can be done by adjusting the control limits of the control charts: For example, in a CUSUM control chart, for a given particular K value (chosen to efficiently detect a particular shift in the process parameter D ¼ j1 – 0 j), the value of threshold H is selected to give the desired ARL0. On the contrary, in an EWMA control chart, for a given smoothing constant , the value L is chosen to give the desired ARL0. 1.04.6.1

Example 1

Let us consider the following example elaborated from a data set available in Montgomery,1 page 387: the first 20 observations were drawn at random from a normal distribution z0 N ð0 ¼ 10; 20 ¼ 1Þ; the last 10 observations were drawn at random from a normal distribution z1 N ð1 ¼ 11; 20 ¼ 1Þ. This data set could be considered, for example, as reaction run times for 30 batches; the last 10 as corresponding to batches produced with a different catalyst. Figures 1–3 display the x individuals Shewhart, CUSUM, and EWMA control charts, respectively. All the charts have been adjusted to yield the same in-control ARL, ARL0 ¼ 370, by setting L ¼ 3 in the Shewhart chart, H ¼ 4.773 and K ¼ 0.5 in the CUSUM chart, and L ¼ 2.8 and ¼ 0.15 in the EWMA chart. The control limits of the charts have been worked out based on the parameters of the normal distribution z0. The CUSUM and EWMA charts signal an out-of-control point: Sample 29 is exceeding the UCL. See the cross points in Figures 2 and 3. Both charts also show a stable upward trend in the last 10 observations, indicating that a shift in the process mean likely occurred around samples 21 and 22. This matches the time


107

14 13 12 x

11 10 9 8 7 6 0

5

10

15 20 Observation

25

30

Figure 1 x individuals Shewhart control chart for the reaction run time data (L ¼ 3). Cross points indicate out-of-control observations. Dotted lines indicate control limits.

7

CUSUM

5 3 1 –1 –3 –5 0

5

10

15 20 Observation

25

30

Figure 2 CUSUM individuals control chart for the simulated reaction run time data (H ¼ 4.773; K ¼ 0.5). Cross points indicate out-of-control observations. Dotted lines indicate control limits.

11

EWMA

10.5 10 9.5 9 0

5

10

15 20 Observation

25

30

Figure 3 EWMA individuals control chart for the simulated reaction run time data (L ¼ 2.8; ¼ 0.15). Cross points indicate out-of-control observations. Dotted lines indicate control limits.

when a new catalyst was used. The slope of the trend estimates the magnitude of the shift, D ¼ j1 – 0 j. In this case, D is approximately 1, corresponding to one standard deviation of the run time, D ¼ ¼ 1. Figure 1 shows that the Shewhart chart apparently looks in control (there is no point exceeding the control limits). Only if the supplementary rules are considered, a run of seven consecutive points above the centerline occurs at sample 29, suggesting the process is out of control. This example illustrates the benefits of using supplementary rules: increasing the sensitivity of the Shewhart chart to detect small shifts in the process. However, the shift in the process is much more evident from CUSUM and EWMA control charts.

108


1.04.7 Control Charts for Autocorrelated Processes As commented in Section 1.04.1, the standard model assumed in most SPC applications is that when the process is in control the key characteristic x to be measured at stage t can be expressed as xt ¼ þ at

ð7Þ

where at is normally and independently distributed with mean zero and standard deviation (i.e., at follows a white noise stochastic process). In many cases, the monitoring schemes are robust to slight deviations from normality, and then the statistical properties of the control charts are not compromised. The independence assumption, however, is much more critical and has a profound impact on the performance of the control charts if indeed violated. For example, if the data are positively correlated (i.e., values above the mean tend to be followed by other values above the mean and vice versa), control charts will give misleading results in the form of too many false alarms and they will not signal real out-of-control situations. Lack of independence between successive observations (i.e., data correlation over time) shows up whenever the interval between samples becomes small relative to the process dynamics (i.e., inertial elements as raw materials flow, storage tanks, reactors’ residence times, or environmental conditions, defining the settling time of the process). This being more the rule than the exception in modern process environments because of the information technologies that allow registering data from every part produced (e.g., in computer integrated manufacturing (CIM) environments) or at high sampling rates (e.g., in continuous and batch process industries). Most of the present-day analytical systems contain mechanical, electrical, and/or optical parts, in which the correlation over time can be of great importance. Some examples are lamp aging of the photometric detector, a decrease in the separation capacity of the chromatographic column, changes in the detector response with use, the influence of points of contamination on subsequent responses in continuous flow analytical systems, sensor deterioration, or changes in the environmental conditions (temperature, humidity, etc.), causing drifts in the results.13

1.04.7.1

Example 2

As an example, consider the data in Figure 4. This graph plots the x–MR(2) individuals control chart from the results of 56 consecutive sulfate determinations in a control sample using a sequential injection analysis (SIA) system with UV/visible detection for determining sulfates in water.14 Note that mainly the x individuals control chart signals many out-of-control measurements (cross points in Figure 4): for example, measurement 8 completes a run of seven consecutive points below the centerline; or measurements 11–18 are above the UCL. The sulfate determinations of the reference sample are drifting or wandering over time, indicating that the analytical process is apparently out of control. Before taking any decision, it would be of great interest to know whether the out-of-control signals are indicating real problems or whether they are false alarms due to autocorrelation (correlation over time). 240

24

230

20

x

MR(2)

220 210 200

16 12 8 4

190

0 0

10

20 30 40 Observation

50

60

0

10


50

Figure 4 x–MR(2) individuals control chart for sulfate determinations in a control sample. Cross points indicate out-of-control measurements. Dotted lines indicate control limits.

60


109

The correlation over a series of n time-oriented observations can be estimated by the sample autocorrelation function (ACF): rk ¼

Pn – k ðx – xÞðx t þk – xÞ t ¼1 Pn t Þ2 t ¼1 ðxt – x

ð8Þ

This assumes that the mean and the variance of the process are constant and that correlation between successive observations depends only on the lag k (number of time periods between observations). The sample ACF for the sulfate determinations is shown in Figure 5. Note that there is a statistically significant (p-value < 0.05) strong positive correlation till lag k ¼ 3. This means that measurements that are one, two, and three periods apart are positively correlated. This level of autocorrelation is sufficient to distort the performance of a Shewhart control chart and the interpretation rules. For example, supplementary run rules should never be used because they can be generated from autocorrelation (i.e., they are part of the common-cause system) and not from assignable causes of variability. On the contrary, estimating the variability of a positive autocorrelated process from moving ranges (as done in Figure 4) may lead to a considerably high underestimation of the real process variability, yielding control limits too much narrower than should be and, hence, increasing the frequency of measurements exceeding them. It is important to highlight that autocorrelation must be considered as part of the common-cause system affecting the process, and any attempt to get rid of it (e.g., by reducing the sampling rate) will lead to a very inefficient use of available data, degrading the performance of the resulting monitoring scheme. Therefore, we should model the autocorrelation. For this purpose, autoregressive integrated moving average (ARIMA) models15 can be used. In addition to the sample ACF, the so-called partial ACF15 is a useful tool to give more information about the correlation structure of the data. The partial ACF measures the correlation between measurements that are shifted k lags without the influence of the intermediate values. Figure 6 shows the partial ACF for the sulfate determinations. In this case, only the partial ACF at lag 1 is statistically significant (p-value < 0.05). Figures 5 and 6 suggest a first-order autoregressive model, AR(1), to describe this analytical process xt – ¼ ðxt –1 – Þ þ at

ð9Þ

where is the process mean, is the autoregressive coefficient ðj j < 1Þ, and at follows a white noise stochastic process, at N(0, ). The standard deviation of xt is given by x ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð1 – 2 Þ

ð10Þ

1

r(k)

0.6 0.2 –0.2 –0.6 –1 0

3

6

9

12

15

Lag Figure 5 Sample ACF for the sulfate determinations in a control sample. Dotted lines indicate limits for statistical significance ( ¼ 0.05).

110


1

r(k)

0.6 0.2 –0.2 –0.6 –1 0

3

6

9

12

15

Lag Figure 6 Partial ACF for the sulfate determinations in a control sample. Dotted lines indicate limits for statistical significance ( ¼ 0.05).

The estimated AR(1) model for the sulfate measurements is xt – 215:70 ¼ 0:85ðxt –1 – 215:70Þ þ et

ð11Þ

where the residuals et ¼ xt – xˆt are approximately normally and independently distributed with mean zero and constant variance (et N(0, s ¼ 4.85)). This is validated by computing the simple and partial ACF of the residuals (not shown) where no coefficient are statistically significant (p-value > 0.05). There are several approaches to monitor this analytical process. One is to estimate the process variability qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ˆ 2 . No supplementary from Equation (10) and set the control limits on the x individuals chart at ˆ 3s= ð1 – Þ rules should be used when interpreting the chart. Note that in the sulfate data the process standard deviation estimated from the average moving range ðMR=d2 ¼ 2:99Þ is approximately one-third of the one estimated (i.e., 9.21) from Equation (10). This explains the large number of measurements exceeding the control limits in Figure 4. Another approach is to apply classical control charts to the residuals. Points out of control or unusual patterns on such charts would indicate that the parameters , , or had changed, implying that the analytical process was out of control. Figure 7 is an x–MR(2) individuals control chart of the residuals from the fitted AR(1) model. Most of the out-of-control points in the x–MR(2) individuals control chart of the original data (Figure 4) have disappeared. This confirms the hypothesis that they were real false alarms due to autocorrelation. As shown in Figure 7, there is only one out-of-control point (measurement 36) in the x chart that generates two out-of-control points (measurements 36 and 37) in the MR(2) chart. By looking at the original data (Figure 4) at measurement 35, there is a drastic jump to measurement 36 that is signaled by the MR(2) chart. This momentary dysfunction of the analytical systems can be caused by bubbles that temporarily produce some anomalous data but, once they have disappeared, the systems begin to function correctly again.16 27

24 20 MR(2)

x

17 7 –3

16 12 8 4 0

–13 0

10


50

60

0

10


50

60

Figure 7 x–MR(2) individuals control chart for the residuals of AR(1) model from sulfate determinations in a control sample. Cross points indicate out-of-control observations. Dotted lines indicate control limits.


111

Sulfate (mg l–1)

240 230 220 210 200 190 0

20

40

60

Observation Figure 8 One-step ahead forecast sulfate determinations from AR(1) model.

To exploit the inertial properties of the process, in addition to the residual control charts, one-step ahead forecasts, xˆt þ1=t (i.e., prediction for the measurement at stage t þ 1 made at stage t), can also be plotted in a run chart: xˆt þ1=t ¼ þ ðx t – Þ

ð12Þ

Figure 8 displays the run chart of the one-step ahead forecasts, xˆtþ1=t , for the sulfate measurements. Note the big gap (error) between the real measurement 36 and its prediction when measurement 35 was obtained, xˆ36=35 . If this monitoring scheme was implemented online, this kind of anomalies could be detected in real time. Note that for estimating this model, outliers should be eliminated. Several methods can be used for this purpose. See for example Box et al.15 Finally, it is important to note that in correlated processes, if there is any easily manipulated variable related to the key characteristic of interest, process dynamics can be exploited to design control rules to force the key characteristic to track a desired trajectory. By integrating control theory with SPC ideas, process improvement can be achieved as commented in the following section.

1.04.8 Integration of SPC and Engineering Process Control The traditional control charting discipline of ‘monitor, then adjust when out of control’ is not the best strategy for process improvement in processes with some kind of dynamics. This is especially true in processes having a tendency to drift or wander away from the target. In these contexts, another approach called stochastic control, also known as engineering process control (EPC), is needed. This approach is based on process compensation and regulation (feedback and/or feedforward control), in which some easily manipulable process variables are adjusted following some control rules with the goal of keeping the key characteristic (controlled variable) close to the desired target. This requires good knowledge of the relationship between the controlled variable and the manipulated variables, as well as an understanding of process dynamics. If the cost of making adjustments (control actions) in the manipulable process variables is negligible, then the variability of the controlled variable is minimized by taking control actions in every sample. This is in sharp contrast to SPC.1 SPC and EPC originated in different industries – the parts industry and the process industry – and have been developed independently with some controversy in the past because of the different meaning of ‘control’ in both approaches: process monitoring (SPC) versus process regulation (EPC). It is true that a dynamic process should not be adjusted using SPC, but it is also true that assignable causes cannot be detected with EPC. In fact, regulation schemes react to process upsets; they do not make any effort to remove the root causes. For example, let us consider a control loop that regulates the inside reactor temperature through a thermostat (feedback controller) by making adjustments in the heating flow. If there appears a leaking problem in the heating jacket, the reactor temperature decreases and the thermostat tries to compensate for the deviation by increasing the heating flow. By continuous adjustments (provided that the heating system can keep up with the temperature

112


differential), the reactor temperature is kept around the target but the assignable cause (leaking) goes undetected. The EPC scheme masks the presence of the assignable cause keeping the process at the desired target at all costs. In the above example, this will result in high energy and repair costs (at the very least the heater will eventually breakdown). By monitoring, for example, the adjustments in the heating flow, the leaking problem would have been detected, saving energy and avoiding the breakdown of the heating system. Therefore, EPC and SPC should be considered as two complementary (not alternative) strategies for quality improvement. SPC monitoring procedures seek to reduce output variability by detecting and eliminating assignable causes of variation. On the contrary, EPC tries to minimize output variability by making regular adjustments exploiting the process dynamics (common-cause system). Hence, ideas from both fields can be used together in an integrated EPC/SPC system called to secure both optimization and improvement. The notion of superimposing SPC on a closed-loop system has opened new lines of research in the area of quality improvement. This new strategy called engineering statistical process control (ESPC), also known as algorithmic statistical process control (ASPC),17 is a proactive approach to quality improvement that reduces predictable variations in quality characteristics using feedback and feedforward techniques and then monitors the entire system (by plotting deviations from the target, prediction errors, adjustments, etc.) to detect and help remove unpredictable process upsets. ESPC integrates two complementary approaches: process adjustment and process monitoring. As such, ESPC is a marriage of control theory and SPC that aims to reduce both short- and long-term variability by replacing the traditional control charting discipline of ‘monitor, then adjust when out of control’ with ‘adjust optimally and monitor’.18 Successful applications and thorough research of this methodology can be found in the literature.4,19–24

1.04.9 Multivariate Control Charts Most SPC schemes currently in practice are based on charting one or a small number of product quality variables in a univariate way. These approaches, although effective in the past where data were scarce, are totally inadequate for modern continuous and batch processes and manufacturing industries where massive amounts of highly correlated variables are being collected. As commented by Professor John F. MacGregor ‘‘the presence of variable interactions in experimental designs leads to the same difficulties in interpreting the results of one factor at a time experimentation as does the presence of correlation among variables in interpreting univariate SPC charts’’.25 Applying univariate SPC charts to each individual variable separately will force the operator to inspect a large number of control charts. When special events occur in a process, they affect not only the magnitude of the variables but also their relationship to each other. These events are often difficult (or even impossible) to detect by charting one variable at a time in the same way as the human eye cannot perceive perspective by looking at an object with one eye at a time.26 Multivariate statistical process control (MSPC) schemes that treat all the variables simultaneously are required in these new data-rich environments. Several approaches have been proposed. In this chapter, they are going to be classified as original variables-based MSPC and latent variables-based MSPC.

1.04.9.1

Original Variables-Based MSPC Schemes

Let us consider Xt ¼ fx1t ; x2t ; . . .; xKt g, the n K observations matrix of K characteristics measured on n units sampled randomly from the production or measurement process at stage t (t ¼ 1, 2, . . .), where xT kt ¼ fxk1t ; xk2t ; . . .; xknt g is the n-dimensional vector of sample values of the kth variable (the kth column of matrix X) and xT nt ¼ fxn1t ; xn2t ; . . .; xnKt g is the K-dimensional observation vector of the nth unit (the nth row of matrix X). Original variables-based MSPC schemes are based on multivariate statistics built from measured variables Xk (k ¼ 1, . . ., K). They can be thought of as the multivariate counterpart to the Shewhart, CUSUM, and EWMA control charts for monitoring the process mean and variability. The Hotelling T 2 and multivariate EWMA (MEWMA) control charts are the most used in practice.


1.04.9.1.1

113

Hotelling T 2 control charts for monitoring process mean

The Hotelling T 2 control chart is the multivariate extension of the univariate Shewhart x control chart for monitoring the process mean. This approach assumes that the (K 1) vector of measured variables x is not time dependent and follows a K-dimensional normal distribution, x NK ðm; SÞ. The chart checks if the (K 1) mean vector of the process m remains stable, assuming a constant (K K) covariance matrix S. This can be used for either individual (n ¼ 1) or subgroup data (n > 1).

1.04.9.1.1(i) Subgroup data Assume that there are m samples of K measured variables each of size n > 1 available from the process. The Hotelling T 2 control chart is based on monitoring at each time t the Hotelling T 2 statistic:1,27–30 Tt2 ¼ nðx t – xÞ S – 1 ðx t – xÞ T

ð13Þ

where x t is the (K 1) vector of sample means at stage t, x the (K 1) overall sample mean vector (estimate of the in-control (K 1) mean vector m), and S is the (K K) pooled sample variance–covariance matrix (estimate of the in-control (K K) covariance matrix S) from the m samples. The Tt2 statistic represents the estimated weighted distance (Mahalanobis distance) of any observation from the target m. Therefore, only UCLs are defined for this chart. The Phase I and II control limits at significance level (Type I risk) are given by UCLðT 2 ÞPhase I; ¼

K ðm – 1Þðn – 1Þ FðK ;ðmn – m – K þ1ÞÞ; mn – m – K þ 1

ð14Þ

UCLðT 2 ÞPhase II; ¼

K ðm þ 1Þðn – 1Þ FðK ;ðmn – m – K þ1ÞÞ; mn – m – K þ 1

ð15Þ

where FðK ;ðmn – m – K þ1ÞÞ; is the 100(1)% percentile of the corresponding F distribution. Thus, if the value of the Tt2 statistic plots above the UCL, the chart signals a potential out-of-control process. The difference in both control limits comes from the fact that in Phase II, when the chart is used for monitoring new observations, x t is independent of x and S sample estimators. This is not the case in Phase I. However, when the number of samples m is large (m > 50), both limits match and the Tt2 statistic can be approximated by a 2 distribution with K degrees of freedom. Therefore, UCLðT 2 Þ ¼ 2K ; , where 2K ; is the 100(1)% percentile of the corresponding 2 distribution.

1.04.9.1.1(ii) Individual observations As commented in Section 1.04.3, there are situations where the subgroup size is naturally n ¼ 1. This occurs frequently in chemical and process industries and in measurement processes. In this case, the Hotelling T 2 statistic at stage t is given by Tt2 ¼ ðxt – x ÞT S – 1 ðxt – x Þ

ð16Þ

where xt is the (K 1) vector of measured variables at stage t, x T ¼ fx1 ; xP xK g is the sample mean vector 2 ; . . .; (estimate of the in-control (K 1) mean vector m), and S ¼ ðm – 1Þ – 1 mi¼1 ðxi – x Þðxi – x ÞT is the (K K) sample variance–covariance matrix (estimate of the in-control (K K) covariance matrix S) from the m samples. Following Tracy et al.,31 in Phase I the Tt2 =ðm – 1Þ2 m – 1 statistic follows a beta distribution with K/2 and (mK1)/2 degrees of freedom. Therefore, UCLs at significance level (Type I risk) can be obtained for Phase I as UCLðT 2 ÞPhase I; ¼

ðm – 1Þ2 BðK =2;ðm – K – 1Þ=2Þ; m

ð17Þ

where BðK =2;ðm – K – 1Þ=2Þ; is the 100(1)% percentile of the corresponding beta distribution. This can be computed from the 100(1)% percentile of the corresponding F distribution by using the relationship BðK =2;ðm – K – 1Þ=2Þ; ¼

ðK =ðm – K – 1ÞÞFðK ;m – K – 1Þ; 1 þ ðK =ðm – K – 1ÞÞFðK ;m – K – 1Þ;

ð18Þ

114


In Phase II, the Tt2 =K ðm – 1Þ2 ½mðm – K Þ – 1 statistic follows an F distribution with K and (mK) degrees of freedom. Thus, the corresponding UCL at significance level (Type I risk) is given by UCLðT 2 ÞPhase II; ¼

K ðm2 – 1Þ FðK ;ðm – K ÞÞ; mðm – K Þ

ð19Þ

If the multivariate normality assumption is not reasonable, control limits may be obtained from resampling methods such as bootstrap.6

1.04.9.1.2

Multivariate Shewhart-type control charts for monitoring process variability The Hotelling T 2 control charts assume that process variance–covariance structure, summarized in the (K K) covariance matrix S, remains constant. This assumption is not necessarily true and must be validated in practice. There are two indices for measuring the overall variability of a set of multivariate data: (i) the generalized variance jSj, that is, the determinant of the covariance matrix, and (ii) the trace of the covariance matrix trS, that is, the sum of the variances of the K variables. F. B. Alt32 presents two control charts for monitoring process variability based on the two previous indices. Assume that the vector x follows an NK ðm; SÞ and that there are m samples of size n > 1 available. The first procedure is a direct extension of the univariate s2 control chart. The procedure is equivalent to repeated test of significance of the null hypothesis that the process covariance matrix is S. The control statistic to be charted at stage t is Wt ¼ – Kn þ Kn lnðnÞ – n lnðjAt j=jSjÞ þ trðS – 1 At Þ

ð20Þ

where At ¼ (n1)St, St is the sample covariance matrix of the tth subgroup, and tr is the trace operator. The Wt statistic follows an asymptotic 2 distribution with K(K þ 1)/2 degrees of freedom. Hence, the UCL at significance level (Type I risk) is UCLðW Þ ¼ 2ðK ðK þ1Þ=2Þ; . The second approach is based on the sample generalized variance jSj (the square root of this quantity is proportional to the area or volume generated by a set of data). By using the mean and variance of jSj, that is, pffiffiffiffiffiffiffiffiffiffiffiffi ffi E(jSj) and V(jSj) respectively, the control limits are set at EðjSjÞ 3 VðjSjÞ. Although jSj is a widely used measure of multivariate dispersion, it is a relative simplistic scalar representation of a complex multivariate structure. Therefore, its use can be misleading in the sense that different correlation structures can yield identical generalized variance. To face this problem, the use of univariate dispersion charts in addition to the control chart for jSj is proposed.32 Other references to multivariate SPC methods for controlling the process variability can be found in Mason et al.33

1.04.9.1.3

MEWMA control charts The Hotelling T 2 control chart is a Shewhart-type control chart, that is, it uses information only from the current sample. As a result, it is relatively insensitive to small and moderate shift in the mean vector m. The MEWMA, introduced by Lowry et al.,34 is the multivariate extension of the univariate EWMA to provide more sensitivity to small shifts in the mean vector m. This is particularly useful in Phase II studies. Assuming individual observations (n ¼ 1), the MEWMA statistics is defined as wt ¼ xt þ ð1 – Þwt – 1

ð21Þ

where is a smoothing constant (0 < 1) and xt is the tth, K-dimensional observation sampled from a Kdimensional normal distribution, x NK ðm; SÞ. The control statistic to be charted at stage t is –1 MEWMA2t ¼ wT t wt wt

ð22Þ

where the covariance matrix is S wt ¼

1 – ð1 – Þ2t S 2–

ð23Þ


115

which is analogous to the variance of the univariate EWMA. The control limits and tuning parameters necessary for implementing the MEWMA control chart are dependent on the shift in the mean vector to detect the in-control ARL and the number of characteristics K to be monitored.35 MEWMA control charts, like their univariate counterparts, are robust to the assumption of normality, if properly designed. Bersimis et al.36 provide an extensive review of several types of MEWMA control charts for different purposes, for example, to account for multivariate time-dependent observations and to deal with monitoring changes in process variability. 1.04.9.1.4

Fault diagnosis Once the multivariate control chart signals an out-of-control alarm, it is required to diagnose an assignable cause for it. This involves two steps: first (diagnostic) find which measured variable(s) contributes to the out-ofcontrol signal and second (corrective) determine what happens in the process that upsets the behavior of these variables. For pursuing the first step (isolation of variables responsible for the out-of-control signal), several approaches have been reported in the literature. An extensive review and references of diagnostic procedures in original variables-based MSPC schemes can be found in Mason et al.33 Bersimis et al.36 and Kourti and MacGregor.37 Regarding the second step (identifying root causes of the problem), management and operator actions based on technical process knowledge will be required.

1.04.9.2

Latent Variables-Based MSPC Schemes

Although original variables-based MSPC is well sounded from a statistical point of view, it suffers from lack of applicability in data-rich environments, typical of modern processes where, as commented in Section 1.04.3, the subgroup size is naturally n ¼ 1. This serious drawback comes from the fact that the multivariate control statistics to be charted need the inversion of a covariance matrix (see Equations (13), (16), (20), and (22)). To avoid problems with this inversion, the number of multivariate observations or samples (m) has to be larger than the number of variables (K), and covariance matrix has to be well conditioned (slightly correlated variables). In addition, complete data (no missing values) are required to work out the Hotelling T 2 or MEWMA2 statistics for any particular sample (see Equations (13), (16), and (22)). Nevertheless, these requirements are not met in highly automated processes where not only a few product quality variables but hundreds (or even thousands) of process variables are measured at a higher frequency rate. For treating large and ill-conditioned data sets that are not full statistical rank (m < K), we advocate the use of latent variables-based MSPC in the way they were proposed by Kourti and MacGregor.37 Latent variable methodology exploits the correlation structure of the original variables by revealing the few independent underlying sources of variation (latent variables) that are driving the process at any time. Multivariate statistical projection methods such as principal component analysis (PCA)38 are used to reduce the dimensionality of the monitoring space by projecting the information in the original variables down onto low-dimensional subspaces defined by a few latent variables. The process is then monitored in these latent subspaces by using a few multivariate control charts built from multivariate statistics that can be thought of as process performance indices or process wellness indices.39 These charts retain all the simplicity of presentation and interpretation of conventional univariate SPC charts. However, by using the information contained in all the measured (process and quality) variables simultaneously, they are much more powerful for detecting out-of-control conditions. As commented by T. Kourti,40 by monitoring only the quality variables (as frequently done in original variables-based MSPC) we are in fact performing statistical quality control (SQC). For true SPC, one must look at all the process data as well. Monitoring the process variables is expected to provide much more information on the state of the process and to supply it more frequently. Furthermore, any abnormal events that occur will also have their fingerprints in the process data. Thus, once an abnormal situation is detected, it is easier and faster to diagnose the source of the problem, as we are dealing directly with the process variables. On the contrary, control charts on the quality variables will only signal when the product properties are no longer consistent with expected performance, but they will not point to the process variables responsible for the problem, making the fault diagnostic process more difficult. Another advantage of monitoring process data is that quality data may not be available at certain stages of the process. Sometimes, product quality is determined only by the performance of the product later, during another

116


process. For example, if a catalyst is conditioned in a batch process before being used for polymer production, the quality of the catalyst (success of conditioning) is assessed by its performance in the subsequent polymer production. It would be useful to know if the catalyst will perform well before using it; monitoring the batch process variables would detect abnormal situations and would provide an early indication of poor catalyst performance. In some cases, the scarce properties measured on a product are not enough to define entirely the product quality and performance for different applications. For example, if only the viscosity of a polymer is measured and kept within specifications, any variation in end-use application that arises owing to variation in chemical structure (branching, composition, and end-group concentration) will not be captured. In these cases, the process data may contain more information about events with special causes that may affect the product structure and thus its performance in different applications. Finally, by monitoring process variables, other abnormal operating conditions may be detected such as a pending equipment failure or a process failure.40 Finally, by using latent variables-based MSPC, missing and noisy data are easily handled, and predictive models based on projection to latent structures, such as partial least squares (PLS) or principal component regression (PCR), can also be used.41 Given the practical relevance of this approach, in the following an overview of implementation of MSPC schemes based on principal component analysis (PCA-based MSPC), describing both Phase I and II, is presented. 1.04.9.2.1 (Phase I)

PCA-based MSPC: exploratory data analysis and offline process monitoring

As commented in Section 1.04.2, the main goal in Phase I is to model the in-control process performance based on a set of historical in-control (reference) data. This data set is one in which the process has been operating consistently (stable over time) in an acceptable manner, and in which only good quality products have been obtained. Occasionally, this historical in-control data set is not directly available, but has to be extracted from historical databases in an iterative fashion as commented below. This explorative analysis of historical databases is a useful technique for improving process understanding and detecting past faults in the process (out-ofcontrol samples). By correctly diagnosing their root causes, some countermeasures can be implemented, optimizing the future performance of the process. Consider that the historical database consists of a set of m multivariate observations (objects or samples of size n ¼ 1) on K variables (online process measurements, dimensional variables, or product quality data) arranged in an (m K) data matrix Z. Variables in matrix Z are often preprocessed by mean centering and scaling to unit variance. With mean centering, the average value of each variable is calculated and then subtracted from the data. This usually improves the interpretability of the model because all preprocessed variables will have mean value zero. By scaling to unit variance, each original variable is divided by its standard deviation and will have unit variance. Given that projection methods are sensitive to scaling, this is particularly useful when the variables are measured in different units. After preprocessing, matrix Z is transformed into matrix X. PCA is used to reduce the dimensionality of the process by compressing the high-dimensional original data matrix X into a low-dimensional subspace of dimension A (A rank(X)), in which most of the data variability is explained by a fewer number of latent variables, which are orthogonal and linear combinations of the original ones. This is done by decomposing X into a set of A rank 1 matrices X¼

A X a¼1

t a pT a þ

rankðXÞ X

T ta pT a ¼ TP þ E ¼ X þ E

ð24Þ

a¼Aþ1

P (K A) is the loading matrix containing the loading vectors pa, which are the eigenvectors, corresponding to the A largest eigenvalues of the covariance matrix of the original pretreated data set X, and define the directions of highest variability of the new latent A-dimensional subspace. T (m A) is the score matrix containing the location of the orthogonal projection of the original observations onto the latent subspace. The columns ta of the score matrix T (ta ¼ Xpa) represent the new latent variables with variances given by their respective eigenvalues ( a). These new latent variables summarize the most important information of the original K variables, and thus can predict (reconstruct) X, by means X ¼ TPT, with minimum mean square error. Matrix E (m K) contains the residuals (statistical noise), that is, the information that is not explained by the PCA model.


117

The dimension of the latent variable subspace is often quite small compared with the dimension of the original variable space (i.e., A F,v1,v2} F distribution with v1 and v2 degrees of freedom alternative hypothesis

H0 N N(,2) NID(,2)

rxy s2p s2x s2yx t,v

128 129 129 131 133 134 134 134 135 136 137 138 140 140 141 142 144 145 147 148 151 153 154 159 163 164 165

null hypothesis number of observations (objects) normal distribution with mean and variance 2 independent random variables equally distributed as normal with mean and variance 2 sample correlation between x and y pooled estimated common variance variance (of x’s data) residual variance of the regression line upper -percentage point of a student’s t distribution with v degrees of freedom so that = pr{x > t,v}

127

128

Quality of Analytical Measurements: Univariate Regression

tv Var(X) x xc xd x xî y yî

student’s t distribution with v degrees of freedom variance of random variable X independent (predictor) variable decision limit (ISO 11843-1) capability of detection (ISO 11843-1) mean (of x’s data) predicted value of xi dependent (response) variable predicted value of yi significance level, probability of type I error, probability of false positive

j " xy 2 2v 2 ,v

probability of type II error, probability of false negative true coefficients random variable (experimental error) true mean correlation coefficient between x and y true variance (chi-squared) distribution with v degrees of freedom upper -percentage point of a chisquared distribution with v degrees of 2 } freedom so that ¼ pr{x > ,v

1.05.1 Introduction In Dux,1 a calibration model is defined as the verification of the response of an instrument to a material of known properties and, maybe, the correction by a factor to take the instrument to the corresponding mark, whereas standardization means to characterize the response of an instrument according to the known properties of the material. The standardization of an instrument is usually done by the ‘calibration curve’. These two terms are often mistakenly used interchangeably. Many of the present-day instruments consist of different systems and their complexity makes it difficult to calibrate them. In fact, what is done is standardizing the response by means of a series of samples of known concentration. These standards of concentration should be of enough purity, or a reference material with certified quality characteristics should be used. Many different meanings have been attributed to the term calibration: for example, the International Union of Pure and Applied Chemistry (IUPAC)2 recognizes two different versions: 1. Calibration function for species identification and qualitative analysis. That is, calibration of those analytical parameters that characterize types of chemical species by establishing a model for the purpose of identification and qualitative analysis (for instance, a model that identifies a compound on the basis of its chromatographic retention time). 2. Calibration function for quantitative analysis, whose main purpose is to obtain a function that allows one to calculate the concentration (or amount) of an analyte as a function of an instrumental signal. In most cases, the calibration function has to take into account the response relations of all relevant constituents and interferents. In the document from the International Organization for Standardization (ISO)3 related to the capability of detection, the practical relevance of the calibration is highlighted: ‘‘The quantitative calibration of a system of measurement is central in many chemical measurements. It is relevant for the trazability of the measurement, contributes to its uncertainty and many figures of merit of the analytical methods are directly related to the calibration.’’ The calibration curve is fit, usually, by the statistical technique of linear regression, which is a particular case of the more general framework of fitting models to experimental data. To fit empirical models involves specific peculiarities caused by the type of data used for the fitting; for this reason, a net chemometric approach has been adopted that emphasizes its utility for the calibration, the determination of the sensitivity of a method, the linear range, the detection of possible matrix and interferent effects, the comparison of two methods, the calculation of a detection limit, etc.


129

1.05.2 Linear Regression in Calibration: Elements and Procedure We will establish the notation and the basic aspects of regression in the simplest case of univariate linear regression. Let x denote the independent variable and y the response. When calibrating, x will be the concentration of the analyte in each standard sample and y denotes the recorded signal. The random error supposed for the response is denoted as ". The model for the observed experimental responses is supposed to be the following: y ¼ f ðx Þ þ "

ð1Þ

where f(x) is the calibration curve to be estimated. To do this, N observations are recorded, yi, that correspond to k calibration standards. If there are no replicate observations, then N ¼ k, otherwise N > k. As usual, for each yi, the corresponding concentration of the standard is written as xi, without distinguishing whether they are equal (replicates) or not. Therefore, yi ¼ f ðxi Þ þ "i ;

i ¼ 1; . . .; N

ð2Þ

In many calibration problems, the following hypotheses are considered whose compatibility with the experimental results will have to be carefully evaluated: 1. The true calibration curve f is linear in the studied range of concentrations. Hence yi ¼ 0 þ 1 xi þ "i ;

i ¼ 1; . . .; N

ð3Þ

2. The values of the predictor variable (xi, i ¼ 1,2 , . . ., N) are controlled values and their experimental uncertainty can be neglected compared with the uncertainty in the determination of the response. 3. The random errors in the observed response are uncorrelated, Covð"i ; "j Þ ¼ 0; if i 6¼ j , have zero mean, E ð"i Þ ¼ 0; i ¼ 1; . . .; N , and the same variance, Var ð"i Þ ¼ 2 ; i ¼ 1; . . .; N , independent of 0, 1, and xi. As a result of the hypotheses on the errors, each response yi is a random variable whose mean is a linear function of the value of the predictor E ðyi Þ ¼ 0 þ 1 xi , its variance is Varðyi Þ ¼ 2 , and the observed values (y1, . . ., yN) are uncorrelated, that is, Covðyi ; yj Þ ¼ 0; if i 6¼ j . Therefore, the problem of estimating the model of Equation (1), or calibration line, is technically equivalent to estimating the parameters 0, 1, and from the experimental results of the calibration set C ¼ fðxi ; yi Þ; i ¼ 1; . . .; N g. To propose a calibration model for experimental data, as the one in Equation (3), is the consequence of the theoretical knowledge of the problem, but its validity has to be verified on the basis of a critical examination of the experimental results. The model, f(x), is a working hypothesis and must be modified if the experimental data are against it. However, the model is not directly observable, and therefore the decision on its validity depends on the success when distributing the experimental response in the two addends of Equation (1): random part and functional part. In other words, the hypotheses on the random error, which are those described in previous hypothesis (3), should also be made explicit and validated. When we say that a model is linear or nonlinear, we are referring to linearity or nonlinearity in the parameters. The value of the highest power of a predictor variable in the model is called the order of the model. For example, y ¼ 0 þ 1 x þ 2 x 2 þ "i

ð4Þ

is a second-order (in x) linear (in the ’s) regression model. The calibration model of Equation (4) is quadratic; it is not a linear calibration but a linear regression.

1.05.2.1

Estimation by Least Squares

The least squares (LS) method does not require any hypotheses under the distribution of the errors, ", in Equation (1). To obtain the estimates b0 and b1 of 0 and 1, respectively, the straight line that best describes the experimental data is determined. The argument is as follows: for each couple of values b0 and b1, a different line

130


yˆ = b0 + b1x

eN

ei

e2 yˆ 1 e1

The linear calibration line fitted by least squares is the one that makes minimum the sum of squares of all these vertical distances

y1

x Figure 1 Illustration of least squares method to fit a calibration line.

is obtained, yˆ ¼ b0 þ b1 x , with the corresponding residuals ei ¼ yi – yî ; i ¼ 1; . . .; N . The LS estimators of 0 and 1 are the values that minimize the sum of squares (SSs) of residuals, Rðb0 ; b1 Þ ¼

N X

ei 2 ¼

i¼1

N X

ðyi – yî Þ2 ¼

i¼1

N X

ðyi – b0 – b1 xi Þ2

ð5Þ

i¼1

The line obtained when minimizing R as a function of b0 and b1 is the one that minimizes the sum of the distances ‘in vertical’ between the experimental data and the estimated ones with the fitted line (see Figure 1). It can be analytically proven that the function Rðb0 ; b1 Þ has a unique minimum (except for x1 ¼ x2 ¼ . . . ¼ xN , a clearly nonsense situation when calibrating) that can be explicitly determined. This solution provides the LS estimators as PN Þðxi – xÞ Covðx; y Þ i¼1 ðyi – y ¼ PN sx2 ð x Þ2 i¼1 i – x

b1 ¼

b0 ¼ y – b 1 x

PN

PN

ð6Þ ð7Þ

2 with y ¼ ¼ i¼1 yi =N the mean of all yi and x i¼1 xi =N the mean of all xi , and sx denotes the variance of the x’s. The procedure for obtaining the minimum shows that the LS line verifies N X

ei yî ¼ 0

ð8Þ

i¼1

and, if there is a 0 term in the model, as in our case, it is also true that N X

ðyi – yˆ i Þ ¼

i¼1

N X

ei ¼ 0

ð9Þ

i¼1

The LS method gives no estimator of 2 , but an estimator of 2 based on the LS estimators of 0 and 1 is the variance of the regression, defined as PN 2 syx ¼

– b0 – b1 xi Þ2 ¼ N –2

i¼1 ðyi

PN

PN 2 – yˆ i Þ2 e ¼ i¼1 i N –2 N –2

i¼1 ðyi

ð10Þ

Equations (6) and (7) imply that b0 and b1 are linear combinations of the random variables yi. Consequently, both estimates themselves are also random variables. As E ðyi Þ ¼ 0 þ 1 xi , from Equation (6), the following statement holds: E ðb1 Þ ¼

PN PN ÞE ðyi – yÞ Þð1 ðxi – xÞÞ i¼1 ðxi – x i¼1 ðxi – x ¼ ¼ 1 PN PN 2 Þ Þ2 i¼1 ðxi – x i¼1 ðxi – x

ð11Þ


131

and, analogously from Equation (7), E ðb0 Þ ¼ Eð y Þ – 1 x ¼ 0

ð12Þ

Equations (11) and (12) mean that both estimates are unbiased estimates. The variance of the estimates is given by PN

Þ2 Varðyi Þ 1 2 i¼1 ðxi – x ¼ PN PN 2 2 2 ð x – x Þ i ð Þ x – x i¼1 i¼1 i ! PN 2 x 1 x2 2 þ PN Varðb0 Þ ¼ PN i¼1 i 2 2 ¼ 2 N N i¼1 ðxi – xÞ ð x – x Þ i¼1 i Varðb1 Þ ¼

x 2 2 ð x – x Þ i i¼1

Covðb0 ; b1 Þ ¼ – PN

ð13Þ

ð14Þ ð15Þ

Equations (13) and (14) show that the precision attainable in the estimates line P P of the coefficients of a calibration is the product of two factors; one of them depends on the magnitude, Ni¼1 ðxi Þ2 , and dispersion, Ni¼1 ðxi – xÞ2 , of the concentrations and the other one, 2, depends on the uncertainty of the observed response. The second factor is estimated with Equation (10) after making the experimental determinations of the calibration, but the other factor is previously known and implies the introduction of the idea of designing the calibration, analogous to any other experiment. This subject, together with other aspects to be taken into account as a consequence of the validation of the model, will be dealt with in Section 1.05.5. When analyzing Equation (15) one should remember that, in a calibration, the values xi cannot be negatives; thus, x > 0 and the covariance (correlation) between b0 and b1 is always negative. We consider the class of the unbiased estimators of 0 and 1 that are linear functions of the random variables yi. In this class, the LS estimators b0 and b1 of the linear model of Equation (3) with the three hypotheses detailed in Section 1.05.2 have smaller variances than any other estimators. This important theorem is often referred to as the Gauss–Markoff theorem and guarantees the coherence between the linear calibrations by LS and the accuracy (trueness and precision) demanded by the methods of chemical analysis. 1.05.2.2

Estimation when the Distribution of " Follows a N(0, 2)

The evaluation and comparative analysis of the estimated parameters in a calibration (b0, b1, and syx) is a problem of statistical inference because all of them are random variables that depend on the distribution of the error ". The only supposition on the distribution of the errors in the LS method is that they have common variance 2 independent of the concentration xi. However, it is necessary to dispose of additional information on their distribution to statistically infer anything about the parameters. When the distribution of the random error, " in Equation (1), is known, the estimation of 0, 1, and can be done by the maximum likelihood method, which, in essence, looks for the values of b0, b1, and s that maximize the probability of obtaining the experimental results that were observed. The method is based on the computation of the likelihood function L of the N random variables ðy1 ; y2 ; . . .; yN Þ, that is, the joint density, g, of these random variables considered as a function of the parameters 0, 1, and . Therefore, Lð0 ; 1 ; 2 Þ ¼ g ðy1 ; y2 ; . . .; yN ; 0 ; 1 ; 2 Þ. The maximum likelihood estimator is the value of 0, 1, and that maximizes L. If, further to the hypotheses on the errors established in hypothesis in point 3 in Section 1.05.2, the data are assumed to be normally distributed, the consequence is that for each x, the experimental response is a normal random variable N ð0 þ 1 x; 2 Þ. Furthermore, as Covðyi ; yj Þ ¼ 0; if i 6¼ j , the random vector ðy1 ; y2 ; . . .; yN Þt follows a multivariate normal distribution N ðh; 2 IÞ with vector of means h ¼ ð0 þ 1 x1 ; 0 þ 1 x2 ; . . .; 0 þ 1 xN Þt and the variance–covariance matrix, 2I, which is a diagonal matrix with all the terms equal to 2 . Figure 2 shows the meaning of the hypotheses assumed; for each concentration xi and due to the random character of ", any value of the signal can be obtained but those values close to 0þ 1xi are more probable, because this number is the expected mean value when using a probability density function of a normal distribution, independent from one concentration to another and with common variance 2 .

132


y y = β0 + β1x

• •

•

•

•

•

•

•

XN

Xi

x

X1 Figure 2 Calibration model of Equation (3), assuming that the recorded signal comes from a normal distribution.

It can be proven that the likelihood function L is maximum when the sum of squares, R in Equation (5), is minimum. Consequently, the estimates b0 (Equation (7)) and b1 (Equation (6)) have the property of being maximum likelihood estimates. For 2 , the maximum likelihood estimate is PN s2 ¼

2 i¼1 ei

N

ð16Þ

which is biased, contrary to s2yx (Equation (10)). In any case, the relation ½N=ðN – 2Þs 2 ¼ syx2 holds. As a consequence of supposing normal distribution for the random errors, the following properties are obtained: 1. The estimates b0, b1, and syx2 (Equations (7), (6), and (10)) are of minimum variance in the class of all unbiased estimators, which is a class wider than the class of unbiased linear estimators. 2 =2 also does. 2. Ns 2 =2 follows a 2 distribution with N–2 degrees of freedom (d.f.), so ðN – 2Þsyx 3. b0 and b1 follow a bivariate normal distribution with covariance matrix whose elements are given by Equations (13)–(15). 2 . 4. b0 and b1 are independent of s 2 and, hence, of syx A last comment, if another distribution is supposed for the random term of the calibration line in Equation (3), the maximum likelihood estimate of 0, 1, and is obtained with other criteria. For example, instead of the normal distribution, let us suppose that the errors were independent and followed the Laplace distribution with null mean and variance 22, that is, its probability density function is f ð"i Þ ¼

1 – j"i j= e 2

ð – 1 "i 1Þ

ð17Þ

Then, application of the maximum likelihood principle for estimating 0 and 1 would involve the minimization of the sum of absolute errors N N X X j ei j ¼ jyi – yˆ i j i¼1

ð18Þ

i¼1

Of course, the LS solution is still available but it is not the maximum likelihood solution; thus, the statistical validity of the conclusions will be wrong. Consequently, the success in the selection of the distribution of the error is essential for a correct determination of the calibration line, including the criterion that should be followed to obtain the estimators. In the rest of the chapter, unless otherwise indicated, we will assume normal and independent residuals with zero mean and equal variance, that is, " is a NID(0,2) distribution.


1.05.2.3

133

The Correlation between Concentration, x, and Signal, y

Usually, the correlation coefficient is used as an index of the linearity of a regression model. The validity of its application to a calibration is a subject of controversy because it seems to be valid only when it is applied with two random variables and the concentration of the standard samples is not random. In the following, this subject is analyzed. We consider two random variables, v and w, which follow some continuous joint bivariate probability distribution. The correlation coefficient between v and w is defined by Covðv; wÞ vw ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffipffiffiffiffiffiffiffiffiffiffiffiffiffiffi VarðvÞ VarðwÞ

ð19Þ

For example, from Equations (13)–(15), the correlation coefficient between the two random variables b0 and b1 is sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi N b0 b1 ¼ – x PN 2 i¼1 xi

ð20Þ

which depends only on the values of the concentration of the calibration samples and which is completely independent of the recorded signals, yi. In general, – 1 vw 1. The quantity vw is a measure of the linear association between the random variables v and w. If vw ¼ 1, v and w are perfectly positively correlated and their values all lie on a straight line with a positive slope in the (v, w) plane. If vw ¼ 0, the variables are said to be uncorrelated, that is, linearly unassociated with each other; note that this does not mean that v and w are statistically independent. If vw ¼ – 1, v and w are perfectly negatively correlated and their values again lie on a straight line, but this time with a negative slope. If a sample of size Nfðvi ; wi Þ; i ¼ 1; :::; N g is available from the joint distribution, the quantity PN Þðwi – w Þ i¼1 ðvi – v ffiqffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi rvw ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi PN PN 2 Þ Þ2 i¼1 ðvi – v i¼1 ðwi – w

ð21Þ

is called the sample correlation coefficient between v and w. It is an estimate of vw and provides an empirical measure of the linear association between v and w. Like vw , – 1 rvw 1. When ðvi ; wi Þ; i ¼ 1; :::; N , are all nonrandom values, that is, they are not sample values from some distribution, rvw can still be used as a measure of linear association, because the values ðvi ; wi Þ; i ¼ 1; :::; N , can be considered as a finite discrete bivariate distribution with uniform marginals. In this case, rvw is a population rather than a sample estimation, that is, vw ¼ rvw . In our case, xi, i ¼ 1, . . ., N, are nonrandom values that represent the values of a finite discrete distribution for concentrations and the corresponding recorded signals yi, i ¼ 1, . . ., N, are observed random variables. As a consequence, it is possible to define xy by Equation (19). Equation (21), with x and y replacing v and w, can still be used to estimate yx by ryx. There is another permanent discussion about the usefulness or uselessness of the correlation coefficient to evaluate the linearity of a calibration.4–8 The paper by Asuero et al.9 is an updated review on the correlation coefficient and its use either as a way to infer correlation between variables or to test linearity with chemical applications. Here, we will only show that between the coefficient of the regression (the slope of the calibration, b1) and the correlation coefficient ryx, the following relation holds: qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi P pffiffiffiffiffiffiffiffiffiffiffiffiffi ðyi – yÞ2 =ðN – 1Þ Varðy Þ p ffiffiffiffiffiffiffiffiffiffiffiffiffiffi ryx b1 ¼ qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ r yx P Varðx Þ ðxi – xÞ2 =ðN – 1Þ

ð22Þ

Thus, b1 and ryx are closely related, although they provide different information about the calibration line. The unit-free and scale-free correlation coefficient ryx measures the degree of linear association between

134


concentration and signal, whereas b1 measures the size of the change in signal when a unit change is made in concentration. Scale changes in the data will affect b1 but not ryx. It is easy to see that ryx ¼ ryyˆ

ð23Þ

That is, the correlation between the recorded signals yi and the estimated ones yî is the same (in absolute value) as the correlation between the concentrations xi and the signals yi.

1.05.3 Statistical Validation of a Calibration Model 1.05.3.1

Validity of the Functional Model

1.05.3.1.1

F-test for significance of regression Obviously, the dispersion of the recorded signal has two sources: the one due to the regression and the residual one. If the dispersion attributable to the regression is higher, in statistical terms, than the residual one, the conclusion is that the model is significant, that is to say, it explains the recorded signals. The variation in the y’s about their mean can be written as N N N N X X X X ðyi – yÞ2 ¼ ððyi – yˆ i Þ þ ðyˆ i – yÞÞ2 ¼ ðyi – yˆ i Þ2 þ ðyˆ i – yÞ2 i¼1

i¼1

i¼1

ð24Þ

i¼1

because the crossproduct term is null. Therefore, the total variation of the y’s, SST, is decomposed into the P variation that can be ascribed to the regression line, SSR, and the residual variation, SSE ¼ Ni¼1 ðyi – yî Þ2 , due to the fact that the experimental observations do not all lie on the regression line. In a compact notation, Equation (24) is SST ¼ SS E þ SSR

ð25Þ

If SSR SSE (equivalently, if R ¼ SSR =SST is near one), it is clear that the regression explains the recorded signal as a function of the concentration. The degrees of freedom (d.f.) associated with any sum of squares are the number of independent pieces of information that we need for its calculation. To compute SST, there are only N1 independent data because it is necessary to calculate y. The SSE has N2 d.f. because we need to previously compute b0 and b1. By subtraction, SSR has 1 d.f. For each term in Equation (25), the mean of squares (MSs) is defined as the quotient between the sum of squares and its d.f., MS ¼ SS=d:f : The result of these computations is presented as an analysis of variance (ANOVA) table like Table 1. MSR and s2yx ¼ MSE, which are shown in Table 1, are random variables because they are obtained from the variables yi. It can be shown that their mean values are as follows: 2

E ðMSR Þ ¼ 2 þ 12

N X ðxi – xÞ2

ð26Þ

i¼1

2 ¼ 2 E ðMSE Þ ¼ E syx

ð27Þ

In addition, if "i are NID(0,2) and 1 ¼ 0, the random variable Fcalc ¼

MSR MSE

ð28Þ

Table 1 Analysis of variance for significance of a linear regression Source of variation

Sum of squares

Degrees of freedom

Mean of squares

Fcalc

Regression Error or residual Total

SSR SSE SST

1 N2 N1

MSR MSE

MSR/MSE


135

follows an F-distribution with 1 and N2 degree(s) of freedom in the numerator and denominator, respectively. If there is a linear relation between signal and concentration, the regression coefficient 1 must be different from zero. The appropriate hypotheses are H0 :1 ¼ 0

ð29Þ

Ha : 1 6¼ 0

and the null hypothesis would be rejected at significance level if Fcalc ¼ MSR =MSE exceeds the critical value at level , F,1,N2. In other words, the linear calibration is statistically significant at level . Instead of the critical value, it is usual to compute the probability pr{F1,N2 > Fcalc} called p-value. In this case, the null hypothesis is rejected (the model is significant) at level if the p-value is smaller than . In a calibration experiment, the null hypothesis of the test for significance of the regression is almost always rejected; hence the model is significant because the calibration is, by definition, based on the fact that the response of the instrument changes with the concentration of the standard solutions. Two additional relationships should be noted. In the explanation of Equation (25), we have defined R2 ¼ SSR =SST as an indicator of the variability of the signal that is explained by the variation of the concentration. R2 is called the determination coefficient and verifies for a fitted straight line that ryx ¼ ðsign of b1 ÞR

ð30Þ

where R is the positive square root of R2. Then, from Equation (23) we also have that, ryyˆ ¼ R

ð31Þ

1.05.3.1.2

Lack of fit test More interesting in practice is the lack of fit (LOF) test that allows us to decide if the supposed linear model in Equation (3) is compatible with the experimental data. The only information available on the discrepancy between the model and the experimental results is provided by the residuals of the fitting yi – yˆ i , which can be decomposed as yi – yˆ i ¼ ðyi – yˆ i Þ – ðE ðyi Þ – E ðyˆ i ÞÞ þ ðE ðyi Þ – E ðyˆ i ÞÞ ¼ ððyi – yˆ i Þ – E ðyi – yˆ i ÞÞ þ ðE ðyi Þ – E ðyˆ i ÞÞ ¼ qi þ Bi

ð32Þ

In Equation (32), Bi is the bias in the estimation when the concentration is xi. If the model is correct, then the bias is zero. Otherwise, Bi 6¼ 0 and its value depends on xi and on the postulated PN 2 model. The2 other term, the random variables qi, i ¼ 1, . . ., N, have null mean, are correlated, and E i¼1 qi ¼ ðN – 2Þ . Consequently, from Equation (32), it is deduced that MSE defined as MSE ¼

SSE ¼ N –2

PN

– yˆ i Þ2 N –2

i¼1 ðyi

ð33Þ

P is a random variable with mean 2 if the model is correct and 2 þ Ni¼1 Bi2 =ðN – 2Þ if the model is biased. Note that 2 ¼ MSE is an overestimation of 2 ¼ Varð"i Þ ¼ Varðyi Þ if the assumed model for the calibration is not correct, syx because it includes the bias of the model. The question is to have another independent estimate of 2 , for example, by previous knowledge of the system of measurement, and comparing it with MSE. If the latter is significantly greater, we say that there is LOF and we would reconsider the model for the calibration line. If no prior estimate of 2 is available, a usual procedure in calibration is to repeat measurements of the standard samples in such a way that we can use these replicates to obtain an estimate of 2 . It is said that this estimation represents the pure error because if xi is the same for several observations, then the variability among the observed responses is due to random fluctuations of measurements alone. It is important to understand that the replicated runs must be genuine replicates and not just repetitions of the same instrumental reading over the same analytical sample. The number and distribution of replicates in the concentration range is an important practical issue because it influences the variance of the estimates made from the calibration line and will be studied in Section 1.05.5.

136


Pk Suppose that in each calibration standard xi (i ¼ 1,P . . . ,k) mi signals

2 yij are recorded so that i¼1 mi ¼ N . The mi internal variability of these observations is given by j ¼1 yij – yi with mi1 d.f. Therefore, pooling over the k levels xi, we have the so-called sum of squares of pure error: SSPE ¼

mi k X X

2 yij – yi

ð34Þ

i¼1 j ¼1

P with ki¼1 mi – 1 ¼ N – k d.f. Dividing SSPE by its d.f. we have MSPE that verifies E(MSPE) ¼ 2. On the other side, we have SSE ¼

N X

mi k X

2 X

2 yj – yˆ j ¼ yij – yˆ i i¼1 j ¼1

j ¼1

mi k X k X

2 X ¼ yij – yi þ mi ðyi – yˆ i Þ2 ¼ SSPE þSSLOF i¼1 j ¼1

ð35Þ

i¼1

In Equation (35), we can see that the sum of squares for LOF, SSLOF, is a weighted sum of squared deviations between the mean response yi at each xi level and the corresponding fitted value yî . Table 2 shows the decomposition of the sum of squares of Equation (35) with the corresponding d.f. If Fcalc ¼ MSLOF =MSPE is greater than F,k-2,Nk, the null hypothesis is rejected at level , that is, we conclude that the model proposed for the calibration line has LOF (is biased). Alternatively, if Fcalc does not exceed F,k-2,Nk, then there is no experimental evidence for LOF, and MSPE and MSLOF are often combined to estimate 2.

1.05.3.2

Verification of the Hypotheses on the Residuals

The residuals are the differences ei ¼ yi – yî ; i ¼ 1; . . .; N , where yi are the experimental values and yî the corresponding fitted values. The random variables "i are not observable. The intuitive idea is that, if the calibration model is correct, the residuals will show tendencies that make us think that "i are effectively NID(0,2). The difficulty arises when expressing ei as a function of "i :10 ei ¼ ð1 – hii Þ"i –

XN

h " j ¼1; j 6¼i ij j

ð36Þ

with

1 ðxi – xÞ xj – x hij ¼ þ PN N Þ2 i¼1 ðxi – x

ð37Þ

and the NID assumption regarding "i implies that the ei must be normally distributed with Varðei Þ ¼ ð1 – hii Þ2

ð38Þ

Thus, unless the concentration values are only two, the variance of the residuals Varðei Þ will be changing and will differ from 2 . To avoid this effect, which can be important if the calibration range is large, it is better to use the studentized residuals defined as Table 2 Analysis of variance for significance and lack of fit of a linear regression Source of variation

Sum of squares

Degrees of freedom

Mean of squares

Fcalc

Regression Error or residual Lack of fit Pure error Total

SSR SSE SSLOF SSPE SST

1 N2 k2 Nk N1

MSR MSE MSLOF MSPE

MSR/MSE MSLOF/MSPE


ei 9 ¼

ei pffiffiffiffiffiffiffiffiffiffiffi syx 1 – hii

137

ð39Þ

As the studentized residuals come from a normal distribution with variance one, they have to vary between 3 and 3. 1.05.3.2.1

Graphical methods Usual graphics are the plots of residuals (or studentized residuals) versus the following: (1) the order of obtaining the results (possible bias due to the sequence of the experiments); (2) the responses yî computed with the model; and (3) the concentrations. In these plots, typical patterns like those of Figure 3 can be observed, which not only alert on the possible failure of the hypotheses but also aid in deciding on the actions to be taken to restore them. Figure 3(a) shows a suitable distribution of the residuals, whereas the rest display certain anomalies. For example, a tendency like the one observed in Figure 3(b), the variance of the residuals increases when increasing the variable represented in the abscissa, indicates a lack of homoscedasticity. Nevertheless, the tendency of Figure 3(c) shows a bias attributable to an inadequate model; a model of order 2 should be chosen as the calibration model. Figure 3(d) shows both anomalies, a failure of equality of variances and also a bias of the model. To evaluate normality, it is useful to draw the graph of normality of studentized residuals, which consists of plotting the cumulative relative frequency of each residual according to a normal distribution versus the residual itself. If the distribution is normal, the points have to appear aligned; additional details can be seen in Draper and Smith.11 When, instead of using the studentized residuals, their absolute values are used, the resulting graph is called a half-normal plot. The graphical possibilities of computers facilitate the drawing of these plots because the vertical scaling should be determined from the inverse function of a normal distribution. When " is not normally distributed, an additional problem can occur. Each hii is different for each estimated residual and, when it is large related to the residual itself, the corresponding ei will be approximately normally distributed because of the central limit theorem. When hii is small, the distribution of ei would be similar to that of "i, although closer to a normal distribution. Thus, we have the disturbing result that the residuals can have a distribution that is closer to normal than the distribution of the errors (even if they are not normally distributed). This property is known as supernormality by many authors. To rectify this problem, the use of (a) 2

(b) 2

1

1

0

0

–1

–1

–2

–2

(c) 2

(d) 2 1

1 0 0 –1 –1

–2

Figure 3 Schematic view of some possible tendencies on the residuals: (a) no tendency; (b) lack of homoscedasticity; (c) wrong model; and (d) both heteroscedasticity and wrong selection of model.

138


a ‘simulation envelope’ is proposed for a half-normal plot together with the use of the ‘externally studentized pffiffiffiffiffiffiffiffiffiffiffi residuals’ defined as ei ¼ ei =syxðiÞ 1 – hii , where syxðiÞ is the residual standard deviation when in the regression model the pair (xi, yi) is not considered (is suppressed). The envelope boundaries are made up by the estimators of the 5th and 95th percentiles of the distribution for each ei obtained by bootstrap. These simulated residuals are normally distributed and the boundaries constructed essentially provide a limit on how non-normal the residuals can appear when the errors have normal distribution. This technique has been deeply studied and there are variations. A review of the subject with examples of application can be seen in Ryan.12 1.05.3.2.2

Inferential methods

Serial correlation The Durbin–Watson test is usually used to verify if each residual is independent of the previous one and therefore if some relation between the results exists related to the concentration of the standards or the sequence in which they were measured. This is frequent in the laboratory because of indirect causes, for example, degradation of reagents and instrumental drift. From the Durbin–Watson test, some procedures have been developed to detect the lack of linearity in real calibration experiments. A comparative analysis, including a recent proposal, is made in the paper by Bru¨ggemann et al.13

1.05.3.2.2(i)

The usual tests are those of Kolmogoroff and 2. The Kolmogoroff test detects modifications in the form of the empirical histogram made with n data with respect to the theoretical one, in our case with respect to that of a normal distribution. The 2 test, similar to the Kolmogoroff test, detects whether the histogram of some experimental data adheres suitably to the one of a theoretical distribution (in our case a normal one). Unlike the previous one, the data are grouped into k intervals so that it demands greater sample sizes. 1.05.3.2.2(ii)

Normality

Homoscedasticity (equal variances) To be able to decide whether the variance of the responses stays or is nonequal for the different values of the independent variable, it is necessary to have several measured responses yi for the same value of the predicting variable xi, at least in two levels of concentration. The Bartlett test and the Cochran test are both adequate and, to a great extent, complementary. The Bartlett test detects the existence of groups of values of the independent variable with variances in the response similar to each other but different from one group to another. It is very sensitive to the failure in the hypothesis of normality. The test of Cochran detects the existence of one discrepant variance.

1.05.3.2.2(iii)

Example 1 To determine cadmium by anodic stripping voltammetry, the intensity at 630 mV has been recorded for five different samples. The results obtained for five replicates of each sample are shown in Table 3. The calibration line fitted by LS is y ¼ – 0:497 þ 0:265x

with the standard deviation of slope and the independent term equal to 0.004 and 0.186 nA, respectively. The residual standard deviation is syx ¼ 0.308, the correlation coefficient is 0.997, and the coefficient of variation is R2 ¼ 0.994. Table 3 Data for the calibration of cadmium Concentration (nmol l–1)

Current (nA)

20.18 30.27 40.36 50.24 60.08

5.00 7.92 9.89 12.49 15.27

4.85 7.05 9.77 12.67 15.49

5.13 7.48 10.00 12.62 15.90

5.02 7.81 10.72 12.83 15.26

4.50 7.49 10.34 12.51 15.98

Mean

Variance

4.90 7.55 10.14 12.62 15.58

0.060 0.116 0.149 0.019 0.117


139

Table 4 ANOVA for the determination of cadmium including the lack of fit test Source of variation

Sum of squares

Degrees of freedom

Mean square

Fcalc

p-value

Regression Error or residual Lack of fit Pure error Total

349.30 2.18 0.34 1.84 351.481

1 23 3 20 24

349.30 0.09 0.11 0.09

3686.37

t=2;N – 2 . Example 2 With the calibration data in Example 1, we want to decide at significance level ¼ 0.05 whether the independent term is equal to zero. Thus, the hypothesis test is now H0 : 0 ¼ 0 versus Ha : 0 6¼ 0 and the statistic, Equation (43), is tcalc ¼ ð – 0:497 – 0Þ=0:186 ¼ – 2:672. As jtcalc j > t0:025;23 ¼ 2:069, the null hypothesis is rejected and we conclude that the independent term is significantly non-null. Two tests have been shown for testing H0 :1 ¼ 0 versus Ha :1 6¼ 0, an F-test (Equations (28) and (29)) and the t-test (Equations (42) and (43)). The two tests are equivalent because a random variable that follows an F-distribution with one and d.f. is equal to the square of a t-distributed variable with d.f., and it is simple to 2 in Equation (43). prove that the statistic Fcalc in Equation (28) is equal to tcalc 2 The confidence interval, at significance level , for the residual variance syx : 2 2 syx ðN – 2Þ syx ð N – 2Þ ; 2=2;N – 2 21 – =2;N – 2

! ð44Þ

2 2 The null hypothesis that the residual variance is equal to a known value H0 :syx ¼ 20 against Ha :syx > 20 is rejected at level if

2calc ¼

2 syx ð N – 2Þ

20

is greater than the critical value of a 2 distribution with N2 d.f.

ð45Þ


141

In Draper and Smith11 and Asuero et al.,9 the procedure to compute a confidence interval for the correlation coefficient ryx can be found. Also, to test the null hypothesis H0 : ryx ¼ 0, where 0 is a specified value (e.g., zero), versus any of the alternative hypotheses Ha : ryx 6¼ 0, or ryx > 0, or ryx < 0. The relation between the correlation coefficient and the slope of Equation (22) makes the following relation holds: R2 ¼

ðb1 =sb1 Þ2 N – 2 þ ðb1 =sb1 Þ2

ð46Þ

Therefore, the significance of the slope of a calibration line, the significance of the regression (ANOVA), and the correlation and determination coefficients are related to each other in such a way that any of them can be obtained from the rest.

As a measure of the degree of linearity of a calibration, the use of the index ILC ¼ 1 – sb1 =b1 has been proposed.14,15 It is a hyperbolic transformation of the statistic of the t-test (for the significance of the slope) and the F-test (for the significance of regression) instead of evaluating the corresponding p-value. In fact, Equation (46) states that ILC is a transformation of the correlation coefficient that takes into account the size of the calibration set, N. Table 5 shows the values of the statistic b1 =sb1 between 1 and 100 together with the corresponding ILC, R2, ryx, and p-values supposing that the calibration size is N ¼ 10 or N ¼ 20. For example, stating that a calibration is good when ILC ¼ 0.95 is exactly the same as stating that the calibration has ryx ¼ 0.99015 (N ¼ 10) or ryx ¼ 0.97823 (N ¼ 20), which is equivalent, in terms of p-values, to 6.0 1010 and 2.7 1015, respectively. It is evident that to fix a threshold value for any of them is equivalent to fixing it in any other; the reader can verify that the practical meaning of all of them is the same.

1.05.4.2

Joint Confidence Region for 0 and 1

Equations (15) and (20) show that the estimates of b0 and b1 are correlated. Therefore, the joint confidence region at level 100(1–)% is not the Cartesian product of the corresponding individual confidence intervals, but is defined as the set of pairs (0, 1) that verify the following inequality: N ð0 – b0 Þ2 þ2

PN

i¼1 xi ð0

– b0 Þð1 – b1 Þ þ 2 2syx

PN

2 i¼1 xi ð1

– b1 Þ2

F;2;N – 2

ð47Þ

This region is useful to establish the accuracy of a method of analysis in a range of concentrations and to compare different analytical methods and/or calibrations. After the calibration model is obtained, it is used to estimate the concentrations xî corresponding to the calibration standards xi. Independent of the form of calibration (univariate, linear or nonlinear, multivariate, multiway, etc.), if the method is accurate, then the regression of values xî versus xi has to be a straight line with slope 1 and independent term zero. The regression 2 xˆ ¼ b 0 þ b 1 x and its corresponding syx are enough to calculate, Equation (47), the confidence region at significance level of ðb0 ; b1 Þ. If point (0, 1) belongs to that region, it can be concluded that the method is accurate. Table 5 Some values of x¼ b1 =sb1 and the corresponding index of linearity, ILC ¼ 1 – ð1=x Þ , coefficient of determination, R2, coefficient of correlation, ryx, and p-level for the t-distribution with the corresponding degrees of freedom (d.f.) R2

ryx

p-value

x

1(1/x)

N ¼ 10

N ¼ 20

N ¼ 10

N ¼ 20

t (8 d.f.)

t (18 d.f.)

1 5 10 20 50 100

0 0.8 0.9 0.95 0.98 0.99

0.11111 0.75758 0.92593 0.98039 0.99681 0.99920

0.05263 0.58140 0.84746 0.95694 0.99285 0.99820

0.33333 0.87039 0.96225 0.99015 0.99840 0.99960

0.22942 0.76249 0.92057 0.97823 0.99642 0.99910

0.08522 0.00013 3.9 107 6.0 1010 6.0 1014 5.5 1017

0.08231 0.00002 8.0 1010 2.7 1015 F;N1 – 2;N2 – 2

ð50Þ

the comparison is carried out by the statistic b11 – b12 ffi tcalc ¼ rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 2 syx1 syx2 PN1 P þ N2 2 2 i¼1

ðxi1 – x1 Þ

i¼1

ð51Þ

ðxi2 – x2 Þ

The distribution of tcalc in Equation (51) can be approximated by a t-distribution with d.f., where is the nearest integer to 1 c 2 =ðN1 – 2Þ þ ð1 – c Þ2 =ðN2 – 2Þ

ð52Þ

with c¼

PN1 ðxi1 – x1 Þ2 i¼1 ; P PN2 N1 2 2 2 = 2 = syx1 ð x – x Þ ð x – x Þ þ s i1 1 i2 2 yx2 i¼1 i¼1 2 syx1 =

if N1 N2

This value always lies between N12 and N1 þ N2 – 4. The adequate formulas for testing the equality of more than two regression lines can be found in Sachs.17 Another approximation to the same problem consists of considering a multivariate regression involving a dummy variable,11 z, which is zero for the values (xi ,yi) of the first calibration set C1 and one for the data of the second calibration set C2. The joint model is then y ¼ 0 þ 1 x þ 0 z þ 1 zx þ "

ð53Þ

The separate functional models for C1 and C2 lines are given by setting z ¼ 0 and z ¼ 1, respectively C1 calibration model: 0 þ 1 x

ð54Þ

C2 calibration model: ð0 þ 0 Þ þ ð1 þ 1 Þx ¼ 0 þ 1 x

ð55Þ

1. To test whether the two calibration lines are parallel (they have the same slope), we would fit model in Equation (53) and then test H0 : 1 ¼ 0 versus Ha : 1 6¼ 0. 2. To test whether the intercept is equal for both calibration lines, the null hypothesis used would be H0 : 0 ¼ 0 versus Ha : 0 6¼ 0. 3. To decide whether the two lines are equal, the null hypothesis is H0 : 0 ¼ 1 ¼ 0. In cases 1 and 2, the test is a t-test similar to those of Equations (42) and (43), that is, tcalc ¼ i =si but now si is the square root of the appropriate diagonal term of the (XtX)1s2 matrix, where s2 is the residual variance of the fitting of the model in Equation (53) and 0

1

B B 1 B B B B B B 1 X¼B B1 B B B. B .. B B @1

x1

0

x2

0

0

1

C 0 C C C C C C xN1 0 0 C C xN1 þ1 1 xN1 þ1 C C C C .. .. C . . C C xN1 þN2 1 xN1 þN2 A

ð56Þ

144


The jtcalc j is compared with the percentage point t=2;N1 þN2 – 4 for a two-sided test on the null hypothesis H0 : i ¼ 0 versus Ha : i 6¼ 0. The joint test of case 3 is similar to that in Equation (47). The details of this test can be consulted in Chapter 3.02 and also in Chapter 14 of Draper and Smith.11 In the model of Equation (53), it is assumed that the variance of the residuals is equal in all experimental data, and this should be carefully verified before applying the hypothesis test to decide about the calibrations C1 and C2. In chemical analysis, the decision on the equality of two calibration lines is important to decide if there is a matrix effect and, if so, what type. If this is the case, calibration C1 is made with samples spiked on a matrix (e.g., sulfonamide in pig kidney) and calibration C2 would be formed by samples of similar concentrations in the dissolvent. If there is no matrix effect, the null hypothesis of case 3 will not be rejected. It is also possible that the effect is a constant or proportional bias, in these cases the null hypothesis of case 2 or the null hypothesis of case 1 will be rejected respectively. When 0 and 1 in Equation (55) are significantly non-null, then both biases exist. Another important application is the verification that the calibration stays day to day; in this case, C1 and C2 would be the calibration made on the first and second day, respectively. 1.05.4.4

Confidence Interval for the Prediction

For a concentration x0 where no experiments were carried out, the response is estimated by yˆ0 ¼ b 0 þ b 1 x 0

ð57Þ

However, as b0 and b1 are random variables that follow a bivariate normal distribution, yˆ0 is also a random variable that follows a normal distribution whose mean and variance as computed with Equations (11)–(15) are E ðyˆ 0 Þ ¼ 0 þ 1 x0

ð58Þ !

Varðyˆ 0 Þ ¼

1 ðx0 – xÞ2 2 þ PN 2 N ð x – x Þ i i¼1

ð59Þ

Varðyˆ 0 Þ has a minimum when x0 ¼ x and increases as x0 separates from x in either direction. In other words, the most precise prediction is expected to be in the mean of the calibration range and precision is lost as we separate from x. The confidence interval at level (1)100% for the true mean value of the response for a given x0 is then computed as

yˆ 0 – t =2;N – 2 s y0 ; yˆ0 þ t =2;N – 2 s y0 sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 ðx0 – xÞ2 d sy0 ¼ Varðyˆ 0 Þ ¼ syx þ PN N Þ2 i¼1 ðxi – x

ð60Þ

If we join up all the lower endpoints and all the upper endpoints of intervals defined in Equation (60) as x0 changes, we would obtain the two dotted hyperbolas as shown in Figure 7 for the data in Example 1, Table 3. The individual values of the random variable yˆ0 are distributed around its mean with variance 2 independent of Varðyˆ 0 Þ. Therefore, the variance of the prediction of an individual observation would be

2

1 ðx0 – xÞ2 1þ þ PN N Þ2 i¼1 ðxi – x

! ð61Þ

2 and the corresponding estimated value is obtained by substituting 2 by syx in Equation (61). The confidence interval for a new observation is then

sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 ðx0 – xÞ2 yˆ0 t =2;N –2 s yx 1 þ þ PN N Þ2 i¼1 ðxi – x

ð62Þ


145

16

Current (nA)

14 12 10 8 6 4 20

30

40 50 Cadmium (nmol l –1)

60

70

Figure 7 Confidence intervals at 95% for the calibration data of Example 1, Table 3. Dotted hyperbolas are for the true mean. Continuous hyperbolas are for a new prediction with q ¼ 1 in Equation (63). The squares are the experimental points.

A confidence interval for the average of q new observations is obtained similarly as follows: sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 1 ðx0 – xÞ2 yˆ0 t =2;N – 2 s yx þ þ PN q N Þ2 i¼1 ðxi – x

ð63Þ

Again, by joining the corresponding endpoints of the confidence intervals for one new observation, we obtain two hyperbolas (those drawn with a continuous line in Figure 7). The rest of the hyperbolas that would be obtained for q > 1 would be located between the two ‘limiting’ hyperbolas (the dotted and the continuous lines in Figure 7). 1.05.4.5 Sample

Confidence Interval for the Determination of the Concentration of a Problem

The most important application of a calibration line is the determination of the concentration of an analyte in a problem sample. We have fitted the calibration line (y ¼ b0 þ b1 x with residual standard deviation syx) and now for a specified value of y, say y0, we wish to obtain an estimated value xˆ0, the value of sample concentration, as well as some sort of confidence interval for it. Initially, we suppose that y0 is the true mean value of the distribution of the signal for a sample of unknown concentration, then the solution will be adapted to the case of having one or several determinations on the same sample whose concentration we want to obtain. By using the calibration line, the estimated concentration is xˆ0 ¼

y0 – b0 b1

ð64Þ

A confidence interval for xˆ0 at level (1)100% will be made up by the values of the concentration x such that the confidence interval at level (1)100% for the corresponding value y ¼ b0 þ b1 x contains y0 . According to Equation (60), this is equivalent to considering all the x values that verify sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 ðx – xÞ2 jy0 – ðb0 þ b1 x Þj t=2;N – 2 syx þ PN N Þ2 i¼1 ðxi – x

ð65Þ

Besides being consequence of the distribution of the response, this interval is also a likelihood ratio confidence region as shown in Brown18 and also corresponds to the inversion of a similar test of the hypothesis x ¼ x0 (see Cox and Hinkley19). It is closely comparable to a Bayesian posterior distribution for x corresponding to vague prior knowledge.

146


After squaring and rationalizing Equation (65), we obtain a second degree equation. Its solutions are:11,18. 1 1 syx x þ ðxˆ 0 – xÞ t =2;N – 2 1–g 1 – g b1

sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 ðy0 – yÞ2 ð1 – g Þ þ 2 PN N b1 i¼1 ðxi – xÞ2

ð66Þ

with g¼

t2 –2 =2;N ¼ P 2 = b12 = syx ðxi – xÞ2

t=2;N – 2 2 b1 =sb1

ð67Þ

Note that the denominator of Equation (67) is the square of the statistic of the t-test to test H0 : 1 ¼ 0 versus Ha : 1 6¼ 0 (Equation (43)), which is equal to the F-test for significance of regression (Equation (28)). Thus, the more significant b1 is (e.g., smaller p-value in Table 1), the larger the denominator of g and the smaller g and ð1 – g Þ > 1 will be. This is a quite usual situation; so, instead of using Equation (66), the following approximation for the confidence interval20,21 is used: sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi xˆ0 t =2;N – 2

syx b1

1 ðy0 – yÞ2 þ 2 PN N b1 i¼1 ðxi – xÞ2

ð68Þ

The above calculations are true mean value calculations. Nonetheless, in calibration, y0 is regarded not as a true mean value but as the mean of q observations. Thus, Equation (63) will be used for Varðy0 Þ and the confidence interval at level (1)100% is syx xˆ0 t =2;N – 2 b1

sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 1 ðy0 – yÞ2 þ þ 2 PN q N b1 i¼1 ðxi – xÞ2

ð69Þ

Then q ¼ 1 provides the confidence interval for the concentration when we have an individual observation, y0 , of the signal. Equations (13), (14), and (61) prove that the variance of the slope, intercept, and yˆ0 are the multiplication of 2 2 (estimated by syx ) by a factor that depends on the following: (1) the proximity of x0 to x, (2) the number N of calibration samples, and (3) the value xi of the calibration standards. As regards the estimated variance of xˆ0 , from Equation (69), is obtains the estimated variance of xˆ0 as ðy0 – yÞ2 1 1 2 1 dðxˆ 0 Þ ¼ syx Var þ þ P b12 q N b12 Ni¼1 ðxi – xÞ2 ! 1 1 ðxˆ 0 – xÞ2 2 1 þ þ ¼ syx PN b12 q N Þ2 i¼1 ðxi – x

! ð70Þ

that depends on the calibration design (N and xi values), on the proximity between y0 and y , and also on the estimated slope. To obtain the most precise estimation of the concentration, the problem sample should provide a signal (or mean value of q signals), y0, close to the central value of those of the calibration concentrations in such a way that P the numerator of Equation (70) is small. Moreover, one obtains better precision if the sum of squares Ni¼1 ðxi – xÞ2 in the denominator is large, which is achieved if the concentration of the standards is far from the mean value. The remaining addends in Equation (70) also indicate that precision is better if the number of calibration standards N is larger and/or if the number of replicate determinations in the problem sample, q, is larger too. But we also obtain better precision in the estimated concentration if the slope b1 is larger, that is, as the sensitivity of the calibration increases, because b1 intervenes (squared) in the denominator of Equation (70). As in all confidence intervals, the value of N also influences through the value of t/2,N2, which can be important. For example, for ¼ 0.05, if N ¼ 5, t0.025,3 is 3.182, whereas for N ¼ 10, t0.025,8 is 2.306.


147

Example 4 With the data of the calibration of cadmium, Example 1, Table 6 shows the variation of the confidence interval at 95% for the concentration of a problem sample as a function of its position in the calibration range and the number of replicates used. To clearly see the effect, the values of the signals y0 supposedly obtained were 4.75, 10.20, and 15.50 nA, which are values near to the minimum, the mean and the maximum respectively of the recorded calibration signals. Furthermore, in each case, the signal corresponds to one, three, or five replicate(s), that is, q ¼ 1, q ¼ 3, or q ¼ 5 in Equation (69). In our case, t=2;23 ¼ 2:0687. From the results in Table 6, one can observe that for the signal of 10.20 nA to consider 3 or 5 replicates causes a reduction in the radius of the confidence interval of 40 and 52%, respectively. This reduction is much higher than that caused by the position of the sample in the calibration range, which represents a reduction of 4% when only one replicate is used and 16% when five replicates are used. Note the interaction between the number of replicates and the position of the calibration range. In general, it is advisable to determine at least three replicates to make the precision increase significantly. In any case, it is quite simple using Equation (70) to explore the combined effect of the position of the signal of the sample and the number of replicates, similar to Table 6, for any calibration. The expressions of the intervals and confidence region, Equations (40), (41), (44), (60), and (68), show that all of them become ‘wider’ if syx is larger, that is, the estimates are less precise. Furthermore, in the statistics of the t-test, Equations (43), (48), and (51), the standard deviation of the estimate, for example, appears in the denominator and increases when increasing syx. Thus, tcalc decreases and it becomes more difficult to reject the null hypothesis. In summary, an overestimation of the variance of the residuals, 2, has a negative effect on all the inferences, including those related to the calculated concentration. At this point, the reader must remember the analysis of the LOF in Section 1.05.3.1.2. Concretely, Equations (32) and (33) show that the bias of the model necessarily causes an increase in syx. Therefore, much attention has to be paid to the problem of assuring the adequacy of the model because in case the model is biased, all the inferential aspects are altered.

1.05.4.6

The Intersection of Two Linear Regression Lines

In some analytical techniques, it is necessary to determine the abscissa of the point where two straight line segments of an analytical signal intersect (e.g., conductometric and photometric titrations). In this case, we have two series of measurements, {(xi1,yi1), i ¼ 1, . . ., N1} and {(xi2, yi2), i ¼ 1, . . ., N2}, before and after the break 2 2 and syx2 as estimated residual point, and fit linear regression lines y ¼ b01 þ b11x and y ¼ b02 þ b12x with syx1 variance, respectively. The estimated abscissa in the intersection point is xˆ0 ¼

b01 – b02 b0 ¼ b12 – b11 b1

ð71Þ

Table 6 Confidence interval for the determination of the concentration of a problem sample with the calibration in Example 1 q¼1 y0 (nA) xˆ0 4.75 10.20 15.50

dðxˆ 0 Þ Var

19.80 1.518 40.37 1.405 60.37 1.515

q¼3

q¼5

h (nmol l1) Var dðxˆ 0 Þ

h (nmol l1)

dðxˆ 0 Þ Var

h (nmol l1)

2.549 2.452 2.546

1.626 1.469 1.622

0.437 0.324 0.434

1.368 1.178 1.363

0.618 0.504 0.614

y0, signal of the problem sample; q, number of replicates used in the determination; h, radius of confidence interval ( Xˆ0h, xˆ0þh ) at 95% (Equation (69))

148


The statistical problem of the intersection of two experimental lines has been tackled in depth.21,22–25 A (1)100% confidence interval on the intersection can be obtained as the roots of the following quadratic equation in xˆ0 :

2 2 xˆ0 2 b12 – t 2 sb – 2xˆ0 b0 b1 – t 2 Covðb0 ;b1 Þ þ b02 – t 2 sb ¼0 1 0

ð72Þ

where t denotes the critical value t=2;N1 þN2 – 4 . 2 2 Supposing that the variance of the error, syx1 and syx2 , is the same in both segments, the pooled estimated common variance of Equation (49) is used to obtain the variances and covariances of b0 and b1. They are computed by the propagation of errors method, which is exact in these cases because b0 and b1 are linear functions of random variables. 2 sb 1

2 sb 0

¼

sp2

¼

sp2

1 1 PN1 2 þ PN2 ð x – x Þ ð x 2 Þ2 i1 1 i¼1 i¼1 i2 – x

!

1 1 x1 2 x2 2 þ þ PN1 2 þ PN2 N1 N2 1 Þ 2 Þ2 i¼1 ðxi1 – x i¼1 ðxi2 – x

Covðb0 ; b1 Þ ¼

sp2

x1 2 x2 2 PN1 2 þ PN2 1 Þ 2 Þ2 i¼1 ðxi1 – x i¼1 ðxi2 – x

ð73Þ ! ð74Þ ! ð75Þ

The groups of symbols enclosed in brackets in Equations (73) and (74) have the form of a two-tailed t-test for significant differences of slopes and significant differences of intercepts, respectively. When the hypothesis of equal slopes is not rejected using the statistic in Equation (48), the coefficient of xˆ0 2 in Equation (72) becomes negative and no real roots are obtained; hence, the confidence interval embraces the entire x axis at the chosen level of confidence.

1.05.5 The Design of a Calibration In this section, the effect of the distribution of the calibration standards on the quality of the calibration line will be analyzed. In principle, it is necessary to answer a series of questions based on the prior knowledge of the analyst on the behavior of the system of measurement in terms of the model supposed for the calibration. In relation to the calibration set, which is the range of concentrations of interest? Is the total number, N, of measurements to be made limited? Is it possible to replicate in all the levels of concentration? Should the values of the concentration of the calibration standards be selected among some fixed values or can they take any value from the specified range? Other questions are relative to the calibration model such as is it probable that the relation signal concentration is not linear in the specified range? To decide this, one’s previous knowledge or knowledge of other experts should be used. Does one believe that the curvature, if it exists, can be modeled with a polynomial model of low order (for example, of order 2 or 3)? Because, if this is not so, a study with a model of higher order should be initiated. The answer to these questions will be reflected in the number of d.f. needed to estimate the LOF and even in the possibility of fitting a calibration model of higher order to the experimental data. Finally, questions about the variance of the error should be posed: Is there any a priori information about it? Should it be estimated together with the slope and the intercept? If the answer is yes, then replicates are mandatory, which are much more important if, in addition, the possibility of failure of the equality of variances in the chosen range of concentrations exists. Once the above questions are answered, the quality of a calibration model is related to the variances of b0 and b1 and the correlation between them; thus, Equations (13), (14), and (20) should be used. Equations (13) and (14) show that the variance of both parameters is equal to a factor, called variance inflation factor (VIF), that depends only on the values of xi and not on the recorded signals. The second factor, 2, is the variance of the


149

signal, which is either known or will be obtained from the experimental results. In both cases, we must accept that it is a characteristic of the measurement system and cannot be modified during the calibration stage. It is evident that by designing the distribution of the calibration standards, the value of VIF can be decreased and, therefore, the variance of the estimators decreases. The previous argument is the same as that under the methodology for response surfaces (see Chapter 1.12 of the book dedicated to design of experiments) but there is a difference; basically, a calibration model is not used to predict the instrumental signal, y0, but the concentration of a problem sample. The structure of Equation (59) of the variance of the estimated response is also the product of 2 by a variance function. This function is quadratic and its minimum is 1/N,P which is reached in the mean value of the concentrations. The form of this function depends on the factor 1= Ni¼1 ðxi – xÞ2 , which is also the VIF of b1 (Equation (13)). Consequently, the quality of the design of a linear calibration will be evaluated with the VIF and with the correlation between the slope and the intercept. Let us now continue our discussion in terms of a specific example. Let us suppose that the analyst has decided to do N ¼ 17 determinations in concentrations around 80 ppb (a hypothetical permitted limit (PL) for a residue); the calibration range considered is between 0.5 and 1.5 PL, that is, between 40 and 120 ppb (as usual in official norms). Furthermore, she or he wants the sample with 80 ppb to be always among the calibration samples. Also, the analyst thinks that the relationship is probably first order but is not absolutely sure. Finally, it can be considered that the variance 2 does not change in the calibration range, although it is unknown and should be estimated with the calibration data. Figure 8 shows some of the possibilities. Each dot represents one experimental determination at the concentration specified in ppb. For the same experiments and to avoid excessively small numbers, instead of the VIF, Table 7 shows the square root of the VIF, VIF1/2, for the independent term and the slope at each of the possibilities in Figure 8. VIF1/2 is in fact the factor that when multiplied with the residual standard deviation gives the standard deviation of the parameter. The number of different standards, k, and the d.f. of pure error and those for the LOF have also been shown in Table 7. As can be seen, all the designs include concentration 80 ppb and cover the proposed range. Design (1) is not acceptable because it does not allow considering the pure error. Designs (8) and (9) are not acceptable either because the estimation of the LOF is made with only 1 d.f. (in this case, if the LOF was significant, the calibration line of order 2 will pass obligatorily through the mean value of the signals obtained in the three concentrations 40, 80, and 120 ppb). Nevertheless, the factors of design (9) can be taken as the reference of the minimal VIF1/2, whereas those of design (1) as the maximal. When comparing these two designs, it is clear that factor VIF1/2 for the slope can be improved by about 36%, and thus also its standard deviation by the same percentage. The remaining designs would fulfill the conditions imposed, allowing, in case of curvature, the fitting of a polynomial of degree two or three. Designs (4) and (5) are especially adequate because they have 5 d.f. to estimate the LOF, and with design (5) a higher reduction (23%) of the standard deviation of b1 is obtained. It is interesting to note that when decreasing the number k of different standards, the value of VIF1/2 is not necessarily reduced – compare designs (3) and (4), 1/2 (5) and (6), or (7) and (8). The values of VIF P 2 for the independent term follow a similar tendency, although they are much higher because of the term xi that appears in the numerator. The strong linear dependency between the slope and the independent term must be observed, which stays between 0.96 and 0.90. To study the effect of the design on the variance of the computed concentration, Equation (70) will be used, 2 =b12 and VIFðxˆ 0 Þ where which is the product of syx VIFðxˆ 0 Þ ¼

1 1 ðxˆ 0 – xÞ2 þ þ PN q N Þ2 i¼1 ðxi – x

! ð76Þ

2 The value syx =b12 will be obtained after fitting the calibration line; both the residual variance and the slope are specific to the system and do not depend on the design. However, VIFðxˆ0 Þ of Equation (76) depends on the design and on the estimated value, xˆ0 , of the concentration of the problem sample. The minimum is reached when xˆ0 ¼ x (and is ð1=q Þ þ ð1=N Þ), and the maximum is in one of the extremes of the calibration range. Table 7 also contains

150


(1) 40 50 60 70 80 90 100 110 120 ppb (2) 40 50 60 70 80 90 100 110 120 ppb (3)

40 50 60 70 80 90 100 110 120 ppb (4) 40 50 60 70 80 90 100 110 120 ppb (5) 40 50 60 70 80 90 100 110 120 ppb (6) 40 50 60 70 80 90 100 110 120 ppb (7)

40 50 60 70 80 90 100 110 120 ppb (8)

40 50 60 70 80 90 100 110 120 ppb (9)

40 50 60 70 80 90 100 110 120 ppb Figure 8 Nine different possible designs for the calibration. Each dot represents an experiment at the concentration (ppb) in the horizontal axis.

the maximum values of VIFðxˆ0 Þ1=2 , which are the maximum inflaction factors of the standard deviation of xˆ0 for the nine designs analyzed. The minimum value when q ¼ 1, that is, when only one signal is recorded for the problem sample, is 1.0290 and the maximum varies from 1.1026 to 1.0589; nevertheless, when q ¼ 2 (two signals are recorded for the problem sample and the mean is taken), the minimum is 0.7475 and the maximum varies, depending on the design, from 0.8460 to 0.7882. In the case of q ¼ 5, the minimum is 0.5087 and the maximum varies from 0.6447 to 0.5669. It is clear that the effect due to the number of replicates in the problem sample is much greater than that due to the design. From Equation (76) it is clear that for q ¼ 1 is VIFðxˆ 0 Þ1=2 > 1, therefore it is advisable to make at least two replicates of the problem sample. In the example, this action diminishes the factor at least by 15%. Design (5) with q ¼ 2 is a suitable choice to obtain acceptable inflaction factors.


151

Table 7 Characteristics of the different designs in Figure 8 VIF(x)1/2 Design

VIF(b0)1/2

VIF(b1)1/2

Corr(b0,b1)

q¼1

q¼2

q¼5

k

d.f. for PE

d.f. for LOF

1 2 3 4 5 6 7 8 9

0.8284 0.7695 0.6849 0.7375 0.6514 0.6927 0.6054 0.6262 0.5557

0.0099 0.0091 0.0080 0.0087 0.0076 0.0081 0.0069 0.0072 0.0063

0.9562 0.9490 0.9352 0.9444 0.9281 0.9367 0.9162 0.9220 0.8997

1.1026 1.0919 1.0777 1.0863 1.0725 1.0789 1.0657 1.0687 1.0589

0.8460 0.8320 0.8133 0.8247 0.8063 0.8149 0.7973 0.8013 0.7882

0.6447 0.6262 0.6012 0.6165 0.5918 0.6034 0.5747 0.5849 0.5669

17 9 9 7 7 5 5 3 3

0 8 8 10 10 12 12 14 14

15 7 7 5 5 3 3 1 1

VIF(.) represents the maximum of the variance inflation factor of xˆ0, VIF(xˆ0); PE, pure error; LOF, lack of fit; q, the number of replicates; k, the number of calibration standards; d.f., the degrees of freedom.

Introducing the individual cost of each determination and by varying N and q, the cost can be optimized in addition to the quality of the design. In Section 2.2.8 of Meier and Zu¨nd,21 it is shown from Equation (70) that using the wrong combination of N and q enormously increases costs.

1.05.6 The Capability of Detection, the Decision Limit, and the Capability of Discrimination Computed from a Regression Model One of the main reasons for the development of new instrumental techniques is to increase their capability to determine much smaller quantities. It is therefore necessary to establish a rule to ensure whether the analyte is present or not in the sample. In general, a detection rule should be as follows: affirm that the analyte is present in a sample if the response obtained is above a threshold value. When applying this decision rule, two mistakes might happen: (1) affirming that the analyte is present in the sample when it is in fact not present, in other words, giving a ‘false positive’, and (2) affirming that the analyte is not present in the sample when it is in fact present, or giving a ‘false negative’. The classic definition for the detection limit26 only evaluates the probability of a false positive but does not explicitly quantify the probability of a false negative. A good revision of the evolution of the stance of the IUPAC can be found in the paper by Currie.27 Chapters 5 and 6 of Gibbons’s book28 contain an interesting discussion of the methods proposed for the calculation of the detection limit and include a good selection of bibliography on the subject. The capability of detection for a given probability of false positive, , as defined by the ISO,3 is ‘‘the true net concentration of the analyte in the material to be analysed which will lead, with probability 1, to the correct conclusion that the concentration in the analysed material is larger than that in the blank material’’. The need to evaluate the probability of false positive, , and of false negative, , has also been recognized by the IUPAC, and in the EU it is mandatory for the identification and quantification of toxic residues or residues that come from veterinary treatments in products for human consumption (European Decision 2002/657/EC). The concept of capability of detection is applicable to any level of concentration x0 and not only for the case of concentration zero. The following one-tail test30 is performed to estimate the capability of detection: H0 : x ¼ x0 (the concentration of the analyte in the sample is x0) versus Ha : x > x0 (the concentration of the analyte in the sample is greater than x0). Thus, the capability of detection, named CC in the above-mentioned European decision and xd in the ISO 11843-1 standard, can be estimated in the univariate case31–33 through a linear calibration model signal versus concentration using the following expression: xd ¼

ð; Þwx0 syx b1

ð77Þ

152


where (, ) is the value of noncentrality parameter of a noncentral t-distribution related to the probabilities and , and wx20 is the VIF(x0) already defined in Equation (76). When the analyte is banned, the minimum detectable net concentration is estimated with x0 ¼ 0 in Equations (76) and (77) and and are the probabilities of false positive and false negative, respectively. However, if the analyte is not banned and a PL has been fixed, the capability of detection must also be evaluated at those levels of concentration. In this case, the capability of detection is estimated31,32,34 as in Equations (76) and (77) with x0 ¼ PL and and are the probabilities of false noncompliance and false compliance, respectively. Regarding the decision limit (Equation (78)), the ISO 11843-2 standard has defined the critical value (at zero) of the net concentration as ‘‘the value of the net concentration the exceeding of which leads, for a given error probability , to the decision that the concentration of the analyte in the analysed material is larger than that in the blank material’’. xc ¼

t;N – 2 wx0 syx b1

ð78Þ

where wx0 is the same as in Equation (77). For substances with a PL, the decision limit also has to be estimated35,29,36 as in Equation (78) but substituting xˆ0 by PL in Equation (76) to compute wx0. As the decision of accepting or rejecting the presence of analyte in the sample is established based on the signal, for an unknown sample the null hypothesis has to be rejected if the recorded signal is above a threshold yd (signal of detection); hence, the critical region of the test is the set of signals beyond the signal of detection, and a calibration function that relates signal with concentration is needed. The program in Sarabia and Ortiz33 implements the previously described procedure when x0 ¼ 0 and calculates the characteristic curves associated with the hypothesis test. In this way, the statistical method establishes the capability of detection of a specific analytical procedure, given by the characteristic curves of the decision rule expressed as a hypothesis test in terms of the concentration taking into account the calibration line. This procedure (despite being published earlier than the year 2000) is the one proposed by ISO 11843 and Commission Decision 2002/657/EC, estimating the minimum detectable net concentration xd, taking into account the probabilities and . Example 5 In the determination of ascorbic acid in soft drinks by High Performance Liquid Chromatography – Diode Array Detection (HPLC-DAD) (see Garcıá et al.37 for more details), we want to compute the capability of detection for the experimental data in Table 8. The LS regression line computes an intercept, b0, equal to 221.5 with sb0 ¼ 41.6 and a slope, b1, 308124.0 with sb1 ¼ 2688.2. The residual standard deviation of the regression is 89.6 a.u. and the correlation coefficient is 0.9994. Furthermore, it is checked that the regression model is significant (p-value < 104) and that there is no significant LOF (p-value 0.24). There is neither failure in the equality of variances (the p-values for the Cochran and Bartlett tests are 0.63 and 0.14, respectively) nor failure of the normality of residuals. Using DETARCHI,33 Equations (76) and (77) with xˆ0 ¼ 0, with one replicate (q ¼ 1) it is possible to detect 0.0011 mg l1 with a probability of false positive, , equal to 0.05 and a probability of false negative equal to 0.05. As the computed capability of detection is less than the lowest concentration of the calibration standards, this value cannot be considered and the only fact we can assert is that the proposed procedure has a capability of Table 8 Calibration of ascorbic acid to determine the capability of detection Concentration (mg l1)

Peak area (a.u.)

0.004

0.006

0.008

0.012

0.016

0.020

0.026

1454 1420 1437

2063 2198 2078

2597 2586

3863 4042

5259 5241

6463 6279 6255

8280 8117 8305


153

detection equal to 0.004 mg l1 (the lowest concentration whose signal was recorded) with a probability of false positive equal to 0.05 and a probability of false negative less than 0.05. Garcıá et al.37 contains a discussion about the capability of discrimination at several non-null nominal concentrations, xˆ0 6¼ 0. The decision limit and the capability of detection defined in Equations (77) and (78) have the property of being invariant for linear transformations of the response36 (the signal). As a consequence, one obtains the same capability of detection and decision limit using the regression estimated concentration versus true concentration. In practice, this is useful because even when using nonlinear univariate calibration methods, or calibrations based on multiway and multivariate regression methods or based on neural networks, etc., it is always true that the relation between the estimated concentration, xî , and that of the calibration standards, xi , must be linear. This regression enables the evaluation of the accuracy of different calibration methods (see Section 1.05.4.2) and also the capability of detection and the decision limit of any of them. The capability of discrimination is another piece of information of interest provided by the calibration line. A method is said to be sensitive (IUPAC,29 Chapter 10) if a small change in concentration, x, causes a large change in the measurement, y; that is, when the derivative dy=dx is large. In the case of a linear calibration line, dy=dx ¼ b1 . As the slope is a random variable, it is not enough to consider only its mean but also its probability distribution or at least its variance. The capability of discrimination, that is, the smallest difference jx – x0 j such that the signal of x can be distinguished from that of x0, will depend on the distribution of both and also on the assumed and (risk of, respectively, concluding that there is a difference when there is none and concluding that there is no difference when in fact there is difference). Formally, it is about testing H0 :jx – x0 j ¼ 0 versus Ha :jx – x0 j > 0. The following equation has been proposed for the sensitivity (capability of discrimination):36 jx – x0 j ¼

ð=2; Þwx0 syx b1

ð79Þ

As for the capability of detection, the capability of discrimination can be computed using Equation (79) for the regression model xˆ ¼ b 0 þ b 1 x and, therefore, it is of use with any kind of calibration method (multivariate or multiway). In the same paper,36 several applications with partial least squares (PLS) calibration models are shown.

1.05.7 Standard Addition Method A calibration line cannot be used to determine an analyte in a sample when it is known that the matrix of the sample interferes with the determination and no blank is available for spiking at levels around the expected (‘nominal’) level. A possible solution to this problem is to apply the method of standard addition in which the sample is used as a kind of ‘blank’ to which further analyte is spiked. To do this, several quantities of the analyte to be determined are added to aliquots of the unknown sample. These spiked samples as well as the unknown sample are measured. Figure 9(a) is the usual representation of the data signal versus concentration added, that (b) Signal y

Difference y–y0

(a)

y0

y0

0 Estimated concentration

Amount added

0

Amount added

Estimated concentration

Figure 9 Procedure for the standard addition method: (a) extrapolation and (b) interpolation.

154


is, the pairs (xi, yi) i ¼ 1, . . ., N. If conditions about linearity and NID(0,2) errors are fulfilled, the model fitted by LS is y ¼ b0 þ b1 x and the quantity of analyte in the sample, xs, is obtained by extrapolating the line to the abscissa (y ¼ 0, xs ¼ – ðb0 =b1 Þ) if there is no systematic error. The estimated value for the concentration of the problem sample is thus xˆs ¼ b0 =b1 and its variance can be obtained from Equation (70) for value xˆs : 1 1 ð – ðb0 =b1 Þ – xÞ2 2 1 dðxˆ s Þ ¼ syx þ þ PN Var 2 b1 q N Þ2 i¼1 ðxi – x

! ð80Þ

This variance can be reduced if the regression line is made with the data (xi, yiy0), i ¼ 1, . . ., N, as indicated in Figure 9(b). It is clear that the slope of the new line of standard addition is the same as before. The concentration of the problem sample is the one corresponding to ordinate y0 in the graph, which is also the value b0 of the previous regression model, so its variance is 1 1 ððb0 =b1 Þ – xÞ2 2 1 dðxˆ s Þ ¼ syx þ þ PN Var b12 q N Þ2 i¼1 ðxi – x

! ð81Þ

less than the previous one, Equation (80), due to the positive sign of b0/b1. In a concrete problem, to guarantee the absence of bias in the determination and therefore that the standard addition method is valid, the following should be demonstrated: (1) that proportional error does not exist, verifying that the slope of the standard addition line is the same as the calibration line obtained for the pure analyte in the absence of the matrix, and (2) the absence of absolute systematic error that will be verified by means of reference spiked samples. The second point is the weakness of the method of standard addition because on many occasions it is not possible to have reference samples without an analyte to evaluate the recovery. The H-point standard addition method38 is a modification of the standard addition method that permits correcting for both absolute and proportional systematic errors. In the simplest case, an interferent whose spectrum is known, the method is based on obtaining a standard addition line at two wavelengths, 1 and 2, where the interferent shows the same absorbance. The two standard addition lines intersect at the so-called Hpoint (xH, yH) and the estimated concentration is xˆs ¼ – x H . This method has been generalized39,40 to obtain the total concentration or concentrations of different chemical forms of an analyte when the matrix of the sample is completely unknown. Another solution comes from the calibration methods based on the decomposition of three-way data that identify the multivariate signal of the analyte and separate it from the signal(s) of the remaining constituents of the sample. The procedure was initially developed41 for the ‘direct trilinear decomposition’ method known as the ‘second-order standard addition method’ (SOSAM). It has been applied with other decomposition methods such as multivariate curve resolution,42 or PARAFAC,43 or direct trilinear decomposition (TLD);44 even an adaptation of this method to handle the case in which the standard addition method causes a reduction of the rank has been proposed.45

1.05.8 Weighted Least Squares and Generalized Least Squares Sometimes the signals used in a calibration have different variance. This can be deduced from a priori knowledge, from graphics y versus x, or from the analysis of the residuals of an LS fit. With a transformation of the signal, y, and possibly of the concentration, x, it might happen that a linear relation between the new variables is valid and a constant-variance additive error is obtained. When this is not possible, the adequate estimation is obtained with a weighted least squares fitting. The presence of correlation in the errors leads to a generalized least squares fitting. To make the analysis easier, the matrix notation will be introduced for the model in Equation (3). ! 0 þ "i i ¼ 1, . . ., N, may be written in the matrix form as The set of N equations yi ¼ ð 1 xi Þ 1 y ¼ Xb þ e

ð82Þ


155

where y is the column vector comprising the values of the response, " is the column vector of residuals, b is the column vector with the slope and the independent term, and finally X is the matrix 0

1

x1

1

C B B. . C X ¼ B .. .. C A @ 1 xN

ð83Þ

and EðeÞ ¼ 0; CovðeÞ ¼ 2 W; and e follows a N ð0; 2 W Þ

ð84Þ

where W, the variance–covariance matrix of the errors, is an N N positive definite square symmetric matrix. Because of the properties of W, it can be shown that it is possible to find a nonsingular symmetric matrix U such that U – 1 U ¼ Ut U ¼ UU ¼ W

ð85Þ

1

By writing e ¼ U e, it is easy to prove that Eðe Þ ¼ 0 and Covðe Þ ¼ 2 I

ð86Þ

In addition, as e is a linear combination of normal distributions, it also holds that e follows a N(0, 2I). Now, if in Equation (82) both members are multiplied by U1, the following new model is obtained: U – 1 y ¼ U – 1 Xb þ U – 1 e

ð87Þ

ð88Þ

y ¼ X b þ e

2

whose residuals are NID(0, ). The LS solution of Equations (82) and (88) is written in the matrix form as b¼

b0 b1

!

–1 ¼ ðX t X Þ – 1 X t y ¼ Xt W – 1 X Xt W – 1 y

ð89Þ

and its variance–covariance matrix is given by

–1 CovðbÞ ¼ Xt W – 1 X 2

ð90Þ

ˆ where The residuals that must be checked are those corresponding to e that are obtained for e ¼ U1(Y Y), Yˆ ¼ Xb and b is obtained from Equation (89). These weighted residuals are used by the Analytical Methods Committee4 to evaluate the LOF. An estimate of is given by s¼

rffiffiffiffiffiffiffiffiffiffiffi et e N –2

ð91Þ

Often the error covariance matrix W is not completely known and the information provided by the residuals can suggest a parametric specification with just a few unknown parameters. The Gauss–Markov optimality of ˆ except for large sample sizes. Equation (89) is lost when W is substituted by an estimate W According to the hypotheses on the residuals, several procedures have been studied in detail. When matrix W is not a diagonal matrix, that is, when the residuals of Equation (82) are correlated, it is known as generalized least squares (GLS) and, when it is diagonal, it is known as weighted least squares (WLS). In the last case, the recorded signals for the calibration samples, and hence the residuals, are independent but with different variance from one level of concentration to another. This is the most frequent situation in chemical analysis. ˆ i , bi is estimated, Also, an iterative procedure to estimate the diagonal W has been proposed. By taking a trial W ˆ then re-estimating Wiþ1 and biþ1 until convergence, it is named iterative weighted least squares (IWLS). See, for example, Ryan’s book.12 The generalized linear model (GLIM) is about the case in which the distribution of the error is an exponential one (the normal distribution is one of them) and/or additionally, h(y), a nonlinear, monotone,

156


differentiable function of the vector of responses y, that verifies h(y) ¼ Xb þ e. The maximum likelihood solution for estimating the parameters, b of GLIM, involves the use of generalized (weighted) LS in an iterative way. Since the work by Nelder and Wedderburn,46 the GLIM procedure has become a technique of increasing use although it needs special computation programs that are more and more accessible. The complexity of GLIM prevents its in-depth discussion in the present chapter, and the interested reader can consult Chapter 16 of the book by Draper and Smith11 for an introduction and the references cited therein for further details. In many practical situations, it is difficult to initially have specific information on W, Equation (84). For this reason, usually it is necessary to make the hypothesis (mistaken) that W ¼ I and to try to discover something about matrix W examining the residuals of the usual LS regression. The LS method provides estimates of the coefficients as bLS ¼ ðXt XÞ – 1 Xt y and E ðbLS Þ ¼ ðXt XÞ – 1 Xt Xb ¼ b but CovðbLS Þ ¼ ðXt XÞ – 1 Xt WXðXt XÞ – 1 2 . If the correct analysis is done, according to Equation (90), –1 CovðbGLS Þ ¼ ðXt W – 1 XÞ 2 holds. The consequence of this wrong estimation is that the variances of the independent term and slope in bLS are greater than the correct ones in bGLS. However, as the estimate bLS is unbiased, the calculated values of the estimates are rarely affected by the fact that the correct regression has not been made. Therefore, the analysis of the residuals of the fitting by LS allows checking the lack of homogeneity of variances and even estimating the weights to be used, that is, matrix W. In an analytical calibration, it can be assumed that W is a diagonal matrix and, thus, the WLS approach proves to be adequate. Sometimes, the observation of the residuals allows fitting the variance as a function, f, of the predictor variable.11 Davidian and Haaland47 describe a procedure to estimate the GLS and the variance function, f, by means of an iterative procedure. It is also possible to use a GLIM to estimate the values of W. A common alternative consists of estimating the values wi of the main diagonal of matrix W by using replicates in each level of concentration, xi. In a calibration experiment, the estimate of the concentration of the problem sample, xu, is similar when using the LS or WLS regression, but its standard deviation will be different, and, further, depending on the position in the range of concentration, the incorrect estimation VarLS(xu) may be lesser or greater than VarWLS(xu).20,21,48 Tellinghuisen49 has analyzed recently this subject by using Monte Carlo simulation and showed that the use of replicates gives great improvement at small concentrations but can underperform LS regression in the mid-tolarge concentration region. Nevertheless, an estimation of the variance function approximates minimum variance even though the true variance functions are not well reproduced. The WLS regression for calibration has been recommended by several organizations;50,51 consequently, it is being used with increasing frequency in analytical chemistry. A survey of the analytical applications of the use of weighted linear regression is given in Table 10 of Asuero and Gonza´lez.52 Therefore, it is interesting to describe the basic inferential aspects of WLS regression. As much if the variance for each xi is estimated with replicates, si2, as if this is done by a function s2 ¼ f(x), it is usual to estimate the elements, wi, of the diagonal of W by 1=si2 (or 1=f ðxi Þ). The confidence intervals and the hypothesis tests for the slope and the independent term are computed as in the LS regression but taking the values of variances of Equation (90) and the residual standard deviation in Equation (91). Given a value of concentration, x0, the response is estimated by yˆ0 ¼ ð 1 x0 Þ

b0

! ð92Þ

b1

and the estimate of its variance is dðyˆ 0 Þ ¼ ð 1 Var

ˆ 1 X 1 x0 Þ Xt W

1 x0

! s2

ð93Þ

Thus, the confidence interval for the expected mean value at level (1)100% is yˆ 0 – t =2;N – 2

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi dðyˆ 0 Þ; yˆ0 þ t =2;N – 2 Var dðyˆ 0 Þ Var

ð94Þ


157

The estimated variance of the prediction for the average of q new observations is " dðyˆ 0 Þ ¼ Var

1 1 1 W1 X þ ð 1 x0 Þ Xt c w0 q x0

!# ð95Þ

s2

where w0 is the weight corresponding to x0. As a consequence, the confidence interval is vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi !ffi u 1 1 u 1 c1 X þ ð 1 x0 Þ Xt W yˆ0 t =2;N – 2 s t w0 q x0

ð96Þ

Finally, once the calibration is over, for a problem sample with value y0, the mean of q recorded signals in q replicates, the estimated value of the concentration is xˆ0 ¼

y0 – b0 b1

ð97Þ

An argument similar to that used to obtain Equation (69) provides the corresponding approximate confidence interval s xˆ0 t =2;N – 2 b1

sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 1 ðy0 – yw Þ2 þ PN þ 2 PN w0 q b1 i¼1 wi ðxi – xw Þ2 i¼1 wi

where again w0 is the weight corresponding to x0 and yw ¼ are the weighted means.

PN i¼1

wi yi =

PN i¼1

ð98Þ

wi and xw ¼

PN i¼1

wi xi =

PN i¼1

wi

Example 6 For the determination of benzaldehyde by differential pulse polarography,53 a calibration was made with nine levels of concentration and four replicates in each level. Table 9 shows the intensity recorded together with the means and variances in each level of concentration. The LS regression for these N ¼ 36 data (xi, yi) is y ¼ 0.0021 þ 1.9401x The estimated standard deviations are 0.004697 and 0.04147 for the independent term and slope, respectively. Figure 10(a) shows the standardized residuals and evidences that the dispersion of the residuals increases with concentration. The Cochran test to decide if one of the nine variances is significantly different from the rest has p-value equal to 0.18; thus, there is no evidence to reject the null hypothesis at level 0.05. This is a quite general situation in analytical calibrations because even when the variances vary much, usually they vary progressively in such a way that none of variances is significantly greater or smaller than the others. However, Table 9 Experimental data (with means and variances) of the determinations of benzaldehyde Concentration (mmol l–1)

Current (mA)

0.0199 0.0450 0.0593 0.0884 0.0980 0.1270 0.1370 0.1640 0.1740

0.0330 0.0823 0.1233 0.1769 0.2068 0.2553 0.2825 0.3427 0.3660

0.0371 0.0914 0.1149 0.1573 0.1874 0.2243 0.2417 0.3175 0.3311

0.0340 0.0905 0.1134 0.1606 0.1913 0.2340 0.2612 0.2961 0.3097

0.0330 0.0845 0.1155 0.1680 0.1777 0.2456 0.2689 0.3078 0.3447

Mean, xi

Variance, si2

0.0343 0.0872 0.1168 0.1657 0.1908 0.2398 0.2636 0.3160 0.3379

3.73 106 2.00 105 1.96 105 7.58 105 1.47 104 1.83 104 2.90 104 3.93 104 5.59 104

158


(b) 2.5

2.0

2.0

1.5

Standardized residuals

Standardized residuals

(a) 2.5

1.0 0.5 0.0 –0.5 –1.0 –1.5 –2.0 –2.5

1.5 1.0 0.5 0.0 –0.5 –1.0 –1.5

0

0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 Benzaldehyde (mmol l –1)

–2.0

0

0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 Benzaldehyde (mmol l –1)

Figure 10 Standardized residuals: (a) for the LS regression and (b) for the WLS regression.

the Bartlett test, which is adequate to detect groups of variances, has a p-value equal to 0.01, which means that, at a significance level of 0.05, the null hypothesis of homogeneity of variances should be rejected. It is interesting to note that the residuals in Figure 10(a) are compatible with the hypothesis of normal distribution. The tests of 2, Kolmogoroff, Saphiro–Wilks, skewness, and kurtosis have p-values of 0.43, 0.78, 0.97, 0.65, and 0.33, respectively; thus, clearly, at a significance level of 0.05, the hypothesis of normal residuals cannot be rejected. The plot of si2 versus xi suggests a quadratic relationship, which we estimate by LS as s 2 ¼ 0:00004 – 0:00195 x þ 0:02670 x2

which explains 96.27% of the variance of s2. Each individual xi value of concentration can be substituted in this equation, then estimate si2 (i ¼ 1, . . ., 36) and invert these values to give the estimated weights wî of the diagonal of matrix W. Then, from Equation (89), the WLS regression is y ¼ – 0:0021 þ 1:9565x

with estimated standard deviations 0.00196 and 0.03392 for the independent term and the slope, respectively. As the theory says, the estimated values of b0 and b1 are similar as both regressions provide unbiased estimates. However, the difference is in the standard deviation estimated for each one, which is reduced by 58% for the independent term and by 18% for the slope. Figure 10(b) shows the standardized residuals of WLS regression; unlike the LS case now they are practically equal for all the values of the concentration. Strictly speaking, they should have been represented pffiffiffiffiffi versus wi xi , but to make the comparison with the LS residuals easier, the unweighted values xi are maintained; this does not modify the clear pattern. Figure 11 shows the length of the confidence intervals (x–h, x þ h) at level 95%. For each abscissa value, x, the ordinate shows the two values þh and –h. The red ones are those obtained when using LS; the continuous line represents values that correspond to a mean signal (Equation (60)) and the dashed line those obtained in prediction of a future signal, q ¼ 1 in Equation (63). The length of the analogous intervals obtained with WLS is in blue; the continuous line represents the mean signal (Equation (94)) and the dashed line for the prediction (Equation (96)). The remarkable asymmetry in the length of the intervals introduced by the weighting is evident. Up to 0.1 mmol l1 of benzaldehyde, the intervals are shorter with WLS than with the incorrect LS, especially between 0.03 and 0.04 mmol l1. Much more interesting from the analytical point of view is the confidence interval (x0h, x0 þ h) for the concentration x0 determined in a problem sample, Equation (69) for LS and Equation (98) for WLS. In both cases, we use q ¼ 1 and 95% confidence level. Figure 12 shows along abscissas the values of x0 and along


159

Length of the confidence interval

0.06 0.04 0.02 0.00 –0.02 –0.04 –0.06 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 0.20 Benzaldehyde (mmol l –1) Figure 11 Length of confidence intervals for LS (in red) and for WLS (in blue). Continuous lines mark values corresponding to the mean signal and the dashed lines those obtained for prediction of a future sample.

Length of the confidence interval

0.03

0.02

0.01

0.00

–0.01

–0.02

–0.03 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16 0.18 0.20 Benzaldehyde (mmol l –1) Figure 12 Length of confidence intervals for LS (in red) and for WLS (in blue).

ordinates the values of h: red for LS and blue for WLS. Up to the concentration of 0.11 mmol l1, the interval determined by WLS is shorter than that determined when incorrectly using LS. The analysis of Figure 12 shows that the most important impact of WLS is in the decision limit and the capability of detection. When x0 ¼ 0, with the (incorrect) LS calibration model, the capability of detection is 0.0232 mmol l1 for ¼ ¼ 0.05. However, with the WLS calibration model, the capability of detection is 0.0118 mmol l1 for the same probabilities of false positive and false negative.

1.05.9 Robust Regression in Calibration In a linear univariate calibration, the visual inspection of the data can indicate the presence of outlier data, that is, data that are different from the majority and do not follow the general linear tendency. The first observation is that the outlier data are not necessarily incorrect or mistaken data. Sometimes, their presence indicates some

160


phenomenon unexpected when the experimentation began but that has chemical or experimental explanation. The problem raised by outlier data is regarding their detection, after which the analyst will be able to evaluate their significance and act accordingly. The outlier data give rise to highly asymmetric distributions or distributions with great accumulated probability far from the central values. In these conditions, the hypothesis of normality is not acceptable. In addition, it is necessary to consider that, in a calibration, the sample sizes are not so large so that the normality tests behave conservatively and do not detect ‘lack of normality’. The effect of outlier data on LS regression may be notable because it first affects the estimates of the intercept, b0, the slope, b1, and the concentration, xˆ0 , of a problem sample. It also affects the precision of these 2 estimates through syx . On examination of the equations of the confidence intervals of the coefficients of the model (Equations (40), (41), (44), and (47)), those of the predictions (Equations (60), (62), and (63)), and those for the concentration of the problem sample (Equations (68) and (69)), and also considering all the hypothesis tests used on these 2 of the random error, ", of parameters, it is clear that they depend on a good estimate of the residual variance syx the model. It suffices to observe how this estimate is made in Equation (10) to deduce that the presence of outlier data inflates this value and, as a consequence, all the intervals become larger, that is, the estimates will be 2 less precise. On the contrary, in the hypothesis tests about the slope and intercept, the syx appears in the denominators, therefore causing the statistic tcalc to take smaller values, leading to the more frequent acceptance of null hypotheses when they should not be accepted. Several robust regression methods insensitive (to a greater or lesser extent) to the presence of outlier data have been developed.11,54 This is why they are able to detect the true linear calibration curve and allow the analyst to decide, with objectivity, which data disagree and to re-establish the conditions under which the LS regression is optimal. Different robust regression methods have already been introduced to analytical chemistry (updated revisions of these applications can be found in Ortiz et al.55 and Daszykowski et al.56). Also, the ISO has recommended its use (ISO 5725-5). Given the relevance of the subject, the present book devotes a complete chapter to the introduction of robust methods (Chapter 3.07). The robust regression methods are usually evaluated according to two indices: (1) The ‘sampling breakdown’: the smallest fraction of contamination that can cause the estimator to take on values arbitrarily far from the correct ones. (2) The ‘influence function’ that describes the effect on the estimator of an infinitesimal contamination in the sample. In a robust procedure, this function must be bounded. For LS regression, one outlier suffices to carry LS over all bounds. Therefore, for a sample of size n, its breakdown is 1/n, which tends to 0 for increasing sample size. With regard to the influence function, the LS estimator is not bounded on both axes; hence, it is very sensitive to any deviation of data as much in the abscissas (‘leverage point’) as in the ordinates (‘outlier point’), and, of course, both deviations can come together (‘outlier-leverage point’). Rousseeuw57 developed the so-called least median squares (LMS) regression and Massart et al.58 showed the advantages of its use in chemical analysis. The theory and versatility of its application can be seen in Rousseeuw and Leroy54 and Rousseeuw.59 The least median squares estimators of 0 and 1 are the values that minimize, instead of the sum of squares, the median of squares of residuals, that is, those that minimize the following function R:

Rðb0 ; b1 Þ ¼ median ei2 ; i ¼ 1; 2; :::; N ¼ median ðyi – yî Þ2 ; i ¼ 1; 2; :::; N

¼ median ðyi – b0 – b1 xi Þ2 ; i ¼ 1; 2; :::; N

ð99Þ

The LMS method has a bounded influence function on both axes and 50% breakdown. The effect of outlier data in an LS regression depends on their relative position in the cloud of points, which is measured by indices as the ‘leverage effect’.10 In a calibration, there will not be any leverage points, that is, samples whose concentration greatly differs from the rest. But if the experimental data include concentration values beyond the linear range of the analytical method, it is clear that they will exert a similar influence on regression to that of a leverage point. Furthermore, the LMS estimator has the property of the ‘exact fit’, which guarantees that if at least 50% of the data (x, y) follow a linear model, the LMS regression finds it. This property is of great interest when one


161

needs to determine the range of linearity at the same time as the sensitivity of the analytical method, which is usually the case in calibration problems. The relation of LMS with the least trimmed squares (LTS) and a comparison with other 10 procedures (including fuzzy methods) can be seen in Ortiz et al.55 For more statistical background about LTS and other references, see Rousseeuw and Leroy54 and Stromberg;60 for computational advances, see Watson.61 The recent developments in PROGRESS62 and fast LTS63,64 have been included in a MATLAB library of robust statistical methods.65 The procedure for the calibration with LMS regression consists of the following: (1) obtaining with the calibration data the LMS regression line, (2) evaluating the standardized robust residuals to establish the data that can be considered outlier (if the residual in absolute value is greater than 2.5, the data are rejected), and (3) re-making the LS regression without the outlier data detected with the LMS regression. Traditionally, the detection of nonaligned data, decisive for the establishment of the range of linearity, is carried out by a detailed inspection of the residuals of the LS regression and many indices such as Cook distance, DFFITS, and DFBETAS.10,11 This procedure requires great experience in the analysis of regressions, which is perhaps not available in the routine working of a chemistry laboratory. The availability of a computerized method such as the LMS regression, with sufficient mathematical properties to successfully perform this task, is the key to the calibration proposed. Example 7 The linear range in the determination of nickel by square-wave adsortive-stripping voltammetry (SWAdSV) using dimethylglyoxime (DMG) was studied.66 The saturation of the electrode surface, caused by the voluminous complex of Ni–DMG, leads to an increasing curvature. Table 10 shows the experimental data corresponding to a calibration of this system. For more details, see Sanllorente et al.66 The LS regression estimates the slope as 193.12 and the independent term as 9.64 with standard deviations 14.05 and 3.37, respectively. The correlation coefficient is 0.96 and syx is 7.64 mA. The externally studentized residuals are not large; all of them are less than 2.5 in absolute value. Further, the p-values of the normality tests are 0.26, 0.77, 0.09, and 0.58 for the tests of 2, Kolmogoroff, Shapiro–Wilks, and skewness, respectively. However, the residuals show a systematic tendency, nonrandom, as can be seen in Table 10 Experimental data and residuals of the regression least squares (LS) and least median squares regression (LMS) for the calibration of nickel Nickel (mmol l–1)

Current (mA)

Externally studentized LS residuals

LMS standardized residuals

0 0.024 0.047 0.070 0.093 0.116 0.139 0.161 0.183 0.205 0.227 0.249 0.270 0.291 0.312 0.333 0.374 0.395 0.415

2.50 7.22 12.71 19.13 25.15 30.79 39.02 46.14 52.83 57.13 60.76 66.50 70.47 73.12 73.46 76.27 72.13 76.87 75.01

1.04 1.00 0.84 0.56 0.34 0.17 0.33 0.72 1.06 1.07 0.98 1.20 1.19 0.99 0.48 0.31 1.42 1.33 2.47

0.54 0.17 0.48 0.30 0.29 0.42 0.66 1.33 1.78 1.09 0.07 0.07 0.66 2.02 4.51 5.79 12.96 13.32 16.73

162


Current (μA)

80 60 40 20 0 0

0.1

0.2 0.3 Nickel (μmol l –1)

0.4

0.5

Figure 13 Regression line by least squares for data of Table 10.

Figure 13. At the beginning, they are negative, then positive, and finally negative again. The most important conclusion is that the calibration line does not reproduce the experimental data in the region of the concentrations in which the electrode has not still saturated and the response is theoretically linear. The LMS regression, with independent term 1.389 and slope 261.91, points as outlier data those corresponding to the greatest five concentrations (fourth column of Table 10) because their standardized robust residuals are greater than 2.5 in absolute value. In addition, they show a systematic tendency for those five data; the residuals ei ¼ yi – yî are negative and decreasing. That is, the experimental data of intensity do not lie on a straight line, and deviate more and more from it with increasing concentration, which is concordant with the effect of saturation of the electrode. Once the outlier data are removed and the LS regression is repeated with the remaining 14 data, the intercept is 2.05 and the slope is 257.66 with standard deviations of 1.05 and 6.04, respectively. The standard deviation of the regression syx is now 2.04 and the correlation coefficient is 0.997. When comparing these results with those of the first LS regression, the negative effect of the outlier data on the estimates and, thus, on the confidence intervals and the capability of detection is evident. Figure 14 shows the calibration line obtained with LS without the outlier data detected by LMS regression. Still a systematic tendency is observed in the residuals, a tendency that is assumable given the correlation coefficient and the residual standard deviation, and taking into account the fact that the LOF cannot be tested because there are no replicates. As a consequence, the linear range of this calibration can be established as the interval of concentrations up to 0.291 mM. The property of the exact fitting of LMS regression allows removing the influence of the curvature in the determination of the end point in titration analysis, as graphically shown in Figure 15. This approach has been used67 to determine the equation of the linear segments in amperometric, conductimetric, spectrophotometric, and potentiometric titrations.

Current (μA)

80 60 40 20 0 0

0.05

0.1 0.15 0.2 Nickel (μmol l –1)

Figure 14 Regression line by LS without the outlier data.

0.25

0.3

163

Corrected signal


V V′ Added volume Figure 15 V9 is the end volume calculated when the first linear section has been fitted by LS (dashed line). V is the end volume calculated when the first linear section has been fitted by LS without outlier data detected by means of LMS (solid line).

1.05.10 Errors in Both Variables In the previous sections, the predictor variable (e.g., the concentration of the standard samples) is supposed to be determined without random error, unlike the recorded signal that is affected by a random variation. In practice, there is no quantity experimentally determined without variability (random error) but, in many occasions, the error in the predictor variable is much less than that of the response; in this case, a model as the one in Equation (3) is adequate. However, consider for instance the case of comparing two analytical methods by linear regression; in this case, the abscissa contains the concentrations determined by one method and the ordinates those by the second method; therefore, the model supposed in Equation (3) cannot be adequate because both concentrations will have a similar experimental variability. The problem is posed by accepting that both the predictor variable, x, and the response, y, are yi ¼ i þ "i ;

i ¼ 1; :::; N

x i ¼ i þ i ;

i ¼ 1; :::; N

ð100Þ

and there is a linear relation between the true unobserved variables y ¼ 0 þ 1

ð101Þ

The solution to the problem of estimating the parameters 0 and 1 and the variance of the errors " and depends on the hypothesis under the distribution of the errors. Adcock68 suggested 130 years ago, minimizing the sum of squared perpendicular distances to the fitted line. Let us assume that "i are NID(0,2") and that i are NID(0,2) with " and uncorrelated. It can be proved11 that the solution by LS will provide bias in the following cases: (1) If 2 is small compared with the or no PN little 2 2 2 spread in the i values ( ¼ i¼1 i – =N ). (2) If the x’s are fixed by the experimenter. (3) If we want to fit yi ¼ i þ "i , where i ¼ 0 þ 1 xi (the observed xi) instead of Equation (101). The maximum likelihood estimates of 0 and 1 have an identifiability problem; the estimation cannot be carried out without some additional information being added, for example knowledge of the ratio ¼ 2" =2 . This situation can be assumed when analysis methods are compared because the repeatability of both methods can previously be known. If were known, maximum likelihood leads to estimates PN b1 ¼

P Þ2 – Ni¼1 ðxi – xÞ2 i¼1 ðyi – y þ P 2 Ni¼1 ðxi – xÞðyi – yÞ

b0 ¼ y – b 1 x

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi P N

2 PN

2ffi P Þ2 – Ni¼1 ðxi – xÞ2 þ 4 Þðyi – yÞ i¼1 ðyi – y i¼1 ðxi – x P 2 Ni¼1 ðxi – xÞðyi – yÞ

ð102Þ

164


qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi P P PN P Note that, if is estimated as Ni¼1 ðyi – yÞ2 = Ni¼1 ðxi – xÞ2 , then b1 ¼ Þ2 = Ni¼1 ðxi – xÞ2 , which is i¼1 ðyi – y the geometric mean functional relationship. In this case, b1 is the geometric mean of the slope in LS fit of y versus x and the inverse of slope of x versus y, so b1 is a compromising value lying in between the two equation slopes. In addition, the geometric mean functional relationship minimizes the sum of areas obtained by drawing horizontal and vertical lines from each data point to the fitted line. A more detailed discussion can be seen in Draper and Smith.11 An important particular case of Equation (102) corresponds to ¼ 1, that is, when both variances 2" and 2 are equal. The solution, called orthogonal regression, is the line that minimizes the sum of squares of perpendicular deviations from the line. This line is the same as the first principal component of the data (xi, yi), i ¼ 1, . . ., N, and is the solution proposed by Adcock. In Section 15.2 of Jackson’s book,69 the inferential procedures for the orthogonal regression line can be found. A very detailed and rigorous statistical analysis of diverse general models for the errors in Equations (100) and (101) can be found in the book by Fuller.70 Also, solutions based on LS have been proposed, that is, without supposing any distribution on the errors " and . In the approach by Ripley and Thompson,71 it is considered that " and are uncorrelated and heteroscedastic. The solution is then obtained by an IWLS procedure. By adequately modifying the weights, the previous iterative procedure allows considering correlated errors in both axes, the so-called bivariate LS procedure.72,73 Finally, it is proved that the total least squares (TLS)74 procedure estimates the true parameters of Equations (100) and (101) consistently, that is, the solution that provides TLS converges to the true values of 0 and 1 when N tends to infinity. This property of TLS estimates does not depend on any assumed distribution for the errors " and . TLS is a very useful method for the problem of errors in variable estimation.

1.05.11 Final Remark There are many regression procedures that can be of interest for calibration in analytical chemistry; however, there is no room for them in a chapter of introduction to the subject. Even so, the user of a regression for analytical calibration must draw her/his calibration data. The numerical indices that serve to evaluate the quality of a regression and therefore guarantee the analytical result and its precision are a limited vision of the calibration. The advantage of the univariate calibration is the possibility of the visual inspection of the data and it should not renounce to it, as well as to a careful reflection on the type of errors that are expected to obtain. The unthinking uses of a linear regression lead to seriously misleading conclusions. In this respect, the example of Anscombe75 is didactic; Table 11 shows the results of four linear regressions, N ¼ 11, with a single replicate. The equality of the estimates and their standard deviations is evident; thus, any other parameter deduced from them (e.g., the linearity coefficient, value and precision of the estimated response) will also be equal. Even the statistic for the significance of the regression indicates that they are ‘equally’ significant. It suffices to show the data graphically, Figure 16, to see the error made in the last three cases when using a linear regression because the data, except for case 1, were not compatible with the supposed model. Table 11 Analysis by means of a linear least squares regression of the Anscombe’s data

Intercept, b0 Slope, b1 Standard deviation of b0 Standard deviation of b1 Standard deviation of regression, syx Correlation coefficient, r Fcalc (Equation (28))a p-value of Durbin–Watson test a

p ¼ 0.002.

Case 1

Case 2

Case 3

Case 4

3.0001 0.5001 1.1248 0.1179 1.2366 0.82 17.99 0.010

3.0009 0.5000 1.1253 0.1180 1.2372 0.82 17.99 0.372

3.0024 0.4997 1.1245 0.1179 1.2363 0.82 17.97 0.402

3.0017 0.4999 1.1239 0.1178 1.2357 0.82 18.00 0.290


(b)

12.2

Response variable, y


(a)

10.2 8.2 6.2

4

6

8 10 12 Predictor variable, x

9.1 7.1 5.1

14

4

(d)

13.3



11.1

3.1

4.2

(c)

165

11.3 9.3 7.3

6


14


20

13.2 11.2 9.2 7.2 5.2

5.3 4

6


14

8

10

18

Figure 16 Graphical representation of the data from Anscombe: (a) case 1; (b) case 2; (c) case 3; and (d) case 4 of Table 11. The lines are those fitted by least squares.

References 1. Dux, J. P. Handbook of Quality Assurance for the Analytical Chemistry Laboratory, 2nd ed.; Van Nostrand Reinhold: New York, 1990. 2. IUPAC Recommendation, Guidelines for Calibration in Analytical Chemistry. Part 1 Fundamentals and Single Component Calibration. Pure Appl. Chem. 1998, 70, 993–1014. 3. International Standard ISO 11843. Capability of Detection. Part 1. Terms and Definitions, 1997 and Part 2. Methodology in the Linear Calibration Case, 2000; International Organisation for Standardisation: Geneva, Switzerland, 1997 and 2000. 4. Analytical Methods Committee. Is my Calibration Linear? Analyst 1994, 119, 2363–2366. 5. Mulholland, M.; Hibbert, D. B. Linearity and the Limitations of Least Squares Calibration. J. Chromatogr. A 1997, 762, 73–82. 6. Huber, W. On the Use of the Correlation Coefficient r for Testing the Linearity of Calibration Functions. Accred. Qual. Assur. 2004, 9, 726. 7. Hibbert, D. B. Further Comments on the (Miss-)Use of r for Testing the Linearity of Calibration Functions. Accred. Qual. Assur. 2005, 10, 300–301. 8. Ellison, S. L. R. In Defence of the Correlation Coefficient. Accred. Qual. Assur. 2006, 11, 146–152. 9. Asuero, A. G.; Sayago, A.; Gonza´lez, A. G. The Correlation Coefficient: An Overview. Crit. Rev. Anal. Chem. 2006, 36, 41–59. 10. Cook, R. D.; Weisberg, S. Residuals and Influence in Regression; Chapman and Hall: New York, 1982. 11. Draper, N. R.; Smith, H. Applied Regression Analysis, 3rd ed.; John Wiley and Sons: New York, 1998; p 43. 12. Ryan, T. P. Modern Regression Methods; John Wiley and Sons: New York 1997; p 64. 13. Bru¨ggemann, L.; Quapp, W.; Wennrich, R. Test for Non-Linearity Concerning Linear Calibrated Chemical Measurements. Accred. Qual. Assur. 2006, 11, 625–631. 14. Cuadros, L.; Garcıá, A. M.; Jimeńez, C.; Romań, M. Estimation of Performance Characteristics of an Analytical Method Using the Data Set of the Calibration Experiment. Anal. Lett. 1993, 26, 1243–1258. 15. de Beer, J. O.; de Beer, T. R.; Goeyens, L. Assessment of Quality Performance Parameters for Straight Line Calibration Curves Related to the Spread of the Abscissa Values Around their Mean. Anal. Chim. Acta 2007, 584, 57–65. 16. Arroyo, D.; Ortiz, M. C.; Sarabia, L. Multiresponse Optimization and Parallel Factor Analysis, Useful Tools in the Determination of Estrogens by Gas Chromatography–Mass Spectrometry. J. Chromatogr. A 2007, 1157, 358–368. 17. Sachs, L. Applied Statistics. A Handbook of Techniques; Springer-Verlag: New York, 1982; p 442. 18. Brown, P. J. Measurement, Regression and Calibration; Clarendon Press: Oxford, 1993; pp 26–29. 19. Cox, D. R.; Hinkley, D. V. Theoretical Statistics; Chapman and Hall: London, 1974. 20. Massart, D. L.; Vandeginste, B. G. M.; Buydens, L. M. C.; de Jong, S.; Lewi, P. J.; Smeyers-Verbeke, J., (eds.), Handbook of Chemometrics and Qualimetrics, Part A, Data Handling in Science and Technology 20A; Elsevier: Amsterdam, 1998. 21. Meier, P. C.; Zu¨nd, R. E. Statistical Methods in Analytical Chemistry, 2nd ed.; John Wiley and Sons: New York, 2000; pp 118–120. 22. Fisher, R. A. Statistical Methods for Research Workers, 13th ed.; Oliver and Boyd: Edinburg, 1963. 23. Lark, P. D.; Craven, B. R.; Bosworth, R. C. L. The Handling of Chemical Data; Pergamon: London, 1968. 24. Liteanu, C.; Rica, I. Statistical Theory and Methodology of Trace Analysis; Ellis Horwood: Chichester, 1980. 25. Carter, K. N. Jr.; Scott, D. M.; Salmon, J. K.; Zarcone, G. S. Confidence Limits for the Abscissa of Intersection of Two LeastSquares Lines such as Linear Segmented Titration Curves. Anal. Chem. 1991, 63, 1270–1278.

166


26. International Union of Pure and Applied Chemistry. Analytical Chemistry Division. Commission on Spectrochemical and Other Optical Procedures for Analysis. Nomenclature, Symbols, Units and Their Usage in Spectrochemical Analysis-II. Data Interpretation. Spectrochem. Acta B 1978, 33, 241–245. 27. Currie, L. A. Detection and Quantification Limits: Origins and Historical Overview. Anal. Chim. Acta, 1999, 391, 127–134. 28. Gibbons, R. D. Statistical Methods for Groundwater Monitoring. Wiley: New York, 1994. 29. Incze´dy, J.; Lengyel, T.; Ure, A. M.; Gelencse´r, A.; Hulanicki, A. Compendium of Analytical Nomenclature, 3rd ed.; International Union of Pure and Applied Chemistry, Port City Press, Inc.: Baltimore 2000; p 74. 30. Clayton, C. A.; Hines, J. W.; Elkins, P. D. Detection Limits with Specified Assurance Probabilities. Anal. Chem. 1987, 59, 2506–2514. 31. Garcıá, I.; Sarabia, L. A.; Ortiz, M. C. Detection Capability of Tetracyclines Analysed by a Fluorescence Technique: Comparison between Bilinear and Trilinear Partial Least Squares Models. Anal. Chim. Acta 2004, 501, 193–203. 32. Ortiz, M. C.; Sarabia, L. A.; Garcıá, I.; Gimeńez, D.; Meleńdez, E. Capability of Detection and Three-Way Data. Anal. Chim. Acta 2006, 559, 124–136. 33. Sarabia, L. A.; Ortiz, M. C. DETARCHI: A Program for Detection Limits with Specified Assurance Probabilities and Characteristic Curves of Detection. Trends Anal. Chem. 1994, 13, 1–6. 34. Verdon, E.; Hurtaud-Pessel, D.; Sanders, P. Evaluation of the Limit of Performance of an Analytical Method Based on a Statistical Calculation of its Critical Concentrations According to ISO Standard 11843: Application to Routine Control of Banned Veterinary Drug Residues in Food According to European Decision. Accred. Qual. Assur. 2006, 11, 58–62. 35. Ortiz, M. C.; Sarabia, L. A.; Herrero, A.; Sańchez, M. S.; Sanz, B.; Rueda, M. E.; Gimeńez, D.; Meleńdez, M. E. Capability of Detection of an Analytical Method Evaluating False Positive and False Negative (ISO 11843) with Partial Least Squares. Chemom. Intell. Lab. Syst. 2003, 69, 21–33. 36. Sanz, M. B.; Sarabia, L. A.; Herrero, A.; Ortiz, M. C. Capability of Discrimination: Application to Soft Calibration Methods. Anal. Chim. Acta 2001, 446, 297–311. 37. Garcıá, I.; Ortiz, M. C.; Sarabia, L.; Vilches, C.; Gredilla, E. Advances in Methodology for the Validation of Methods According to the International Organization for Standardization. Application to the Determination of Benzoic and Sorbic Acids in Soft Drinks by High-Performance Liquid Chromatography. J. Chromatogr. A 2003, 992, 11–27. 38. Bosch-Reig, F.; Campins Falco´, P. Campins Falco´ P., H-Point Standard Additions Method. Part 1 Fundamentals and Application to Analytical Spectroscopy. Analyst 1988, 113, 1011–1016. 39. Campıńs-Falco´, P.; Verdu´-Andre´s, J.; Bosch-Reig, F.; Molıńs-Legua, C. Generalized H-Point Standard Additions Method for Analyte Determinations in Unknown Samples. Anal. Chim. Acta 1995, 302, 323–333. 40. Tortajada-Genaro, L. A.; Campıńs-Falco´, P.; Blasco-Go´mez, F.; Bosch-Reig, F. The Generalized H-Point Standard-Additions Method to Determine Analytes Present in Two Different Chemical Forms in Unknown Matrix Samples. Part I. General Considerations. Analyst 2000, 125, 771–776. 41. Booksh, K.; Henshaw, J. M.; Burgess, L. W.; Kowalski, B. R. A Second-Order Standard Addition Method with Application to Calibration of a Kinetics-Spectroscopic Sensor for Quantitation of Trichloroethylene. J. Chemom. 1995, 9, 263–282. 42. Saurina, J.; Tauler, R. Strategies for Solving Matrix Effects in the Analysis of Triphenyltin in Sea-Water Samples by Three-Way Multivariate Curve Resolution. Analyst 2000, 125, 2038–2043. 43. Sena, M. M.; Trevisan, M. G.; Poppi, R. J. Combining Standard Addition Method and Second-Order Advantage for Direct Determination of Salicylate in Undiluted Human Plasma by Spectrofluorimetry. Talanta 2006, 68, 1707–1712. 44. Herrero, A.; Zamponi, S.; Marassi, R.; Conti, P.; Ortiz, M. C.; Sarabia, L. A. Determination of the Capability of Detection of a Hyphenated Method: Application to Spectroelectrochemistry. Chemom. Intell. Lab. Syst. 2002, 61, 63–74. 45. Bahram, M.; Bro, R. A Novel Strategy for Solving Matrix Effect in Three-Way Data Using Parallel Profiles with Linear Dependencies. Anal. Chim. Acta 2007, 584, 397–402. 46. Nelder, J. A.; Wedderburn, R. W. M. Generalized Linear Models. J. R. Stat. Soc. Ser. A 1972, 135, 370. 47. Davidian, M.; Haaland, P. D. Regression and Calibration with Nonconstant Error Variance. Chemom. Intell Lab. Syst. 1990, 9, 231–248. 48. Miller, J. N. Basic Statistical Methods for Analytical Chemistry. Part 2. Calibration and Regression Methods. Analyst 1991, 116, 3–14. 49. Tellinghuisen, J. Weighted Least-squares in Calibration: What Difference Does it Make? Analyst 2007, 6, 536–543. 50. Analytical Methods Committee. Uses (Proper and Improper) of Correlation Coefficients. Analyst 1988, 113, 1469–1471. 51. EPA, Document # 815 R 05 006. Statistical Protocol for the Determination of the Single-Laboratory Lowest Concentration Minimum Reporting Level and Validation of Laboratory Performance at 0 Below the Minimum Reporting Level, 2004. 52. Asuero, A. G.; Gonza´lez, G. Fitting Straight Lines with Replicated Observations by Linear Regression. III. Weighting Data. Crit. Rev. Anal. Chem. 2007, 37, 143–172. 53. Sanz, B. Metodologıá para determinar ıńdices de calidad de tećnicas analı´ticas que incorporan calibrados multivariantes. Aplicacioń al Electroana´lisis. Ph. D. thesis, University of Burgos, 2002. 54. Rousseeuw, P. J.; Leroy, A. M. Robust Regression and Outliers Detection. John Wiley and Sons Inc.: Hoboken, NJ, 2001. 55. Ortiz, M. C.; Sarabia, L.; Herrero, A. Robust Regression Techniques. A Useful Alternative for the Detection of Outlier Data in Chemical Analysis. Talanta 2006, 70, 499–512. 56. Daszykowski, M.; Kaczmarek, K.; Vander Heyden, Y.; Walczak, B. Robust Statistics in Data Analysis – A Review. Basic Concepts. Chemom. Intell. Lab. Syst. 2007, 85, 203–219. 57. Rousseeuw, P. Least Median of Squares Regression. J. Am. Stat. Assoc. 1984, 79, 871–880. 58. Massart, D. L.; Kaufman, L.; Rousseeuw, P. J.; Leroy, A. Least Median of Squares: A Robust Method for Outlier and Model Error Detection in Regression and Calibration. Anal. Chim. Acta 1986, 187, 171–179. 59. Rousseeuw, P. J. Tutorial to Robust Statistics. J. Chemom. 1991, 5, 1–20. 60. Stromberg, A. J. Computing the Exact Least Median of Squares Estimate and Stability Diagnostics in Multiple Linear Regression. SIAM J. Sci. Comput. 1993, 14, 1289–1299. 61. Watson, G. A. Computing the Least Quantile of Squares Estimate. SIAM J. Sci. Comput. 1998, 19, 1125–1138.


167

62. Rousseeuw, P. J.; Hubert, M. In L1-Statistical Procedures and Related Topics; Dodge, Y., Ed.; Institute of Mathematical Statistics: Hayward, CA, 1997; p 201. 63. Pison, G.; Van Aelst, S.; Willems, G. Small Sample Corrections for LTS and MCD. Metrika 2002, 55, 111–123. 64. Rousseeuw, P. J.; Van Driessen, K. Computing LTS Regression for Large Data Sets. Estadı´stica 2002, 54, 163–190. 65. Verboven, S.; Hubert, M. LIBRA: A MATLAB Library for Robust Analysis. Chemom. Intell. Lab. Syst. 2005, 75, 127–136. 66. Sanllorente, S.; Ortiz, M. C.; Arcos, M. J.; Lo´pez-Palacios, J. Application of an Optimization Procedure in Adsorptive Stripping Voltammetry to the Determination of Nickel with DMG. Electroanalysis 1996, 8, 285–292. 67. Ortiz, M. C.; Herrero, A. Regression by Least Median Squares, A Methodological Contribution to Titration Analysis. Chemom. Intell. Lab. Syst. 1995, 27, 231–243. 68. Adcock, R. J. A Problem in Least Squares. Analyst (des Moines, Iowa) 1878, 5, 53–54. 69. Jackson, J. E. A User’s Guide to Principal Components. John Wiley and Sons: New York, 1991. 70. Fuller, W. A. Measurement Error Models. John Wiley and Sons: New York, 1987. 71. Ripley, B. D.; Thompson, M. Regression Techniques for the Detection of Analytical Bias. Analyst 1987, 112, 377–383. 72. Lisy´, J. M.; Cholvadova´, A.; Kutej, J. Multiple Straight-Line Least-Squares Analysis with Uncertainties in All Variables. J. Comput. Chem. 1990, 14, 189–192. 73. Riu, J.; Rius, F. X. Univariate Regression Models with Errors in Both Axes. J. Chemom. 1995, 9, 343–362. 74. Van Huffel, S.; Lemmerling, P.; Eds. Total Least Squares and Errors-In Variables Modeling. Kluwer Academic Publishers: Dordrecht, 2002. 75. Anscombe, F. J. Graphs in Statistical Analysis. Am. Stat. 1973, 27, 17–21.

168



Marıá Cruz Ortiz received her Ph.D. in chemistry from the University of Valladolid (Spain) in 1988. She is a member of the staff of analytical chemistry in the University of Burgos (Faculty of Sciences) since 1989. She does research and teaching on analytical chemistry and chemometrics. Her research activity has focused on experimental design, optimization, pattern recognition, quality assurance, and validation of analytical methods according to official regulations, multivariate and/or multiway regression models. All of these applied to problems in food chemistry, typification, and so on, which produce about 100 papers. She is at present the head of an active research group, the chemometrics and qualimetrics group of the University of Burgos.

M Sagrario Sańchez obtained her Ph.D. in mathematics from the University of Valladolid in 1997. She is working at the University of Burgos since 1991 and as a member of the permanent staff since 2002. The teaching activities are mostly directed to graduate and postgraduate courses in chemistry and science and food technology. She is also a permanent member of the chemometrics and qualimetrics group since its foundation. Her main research activities are developed into the areas of interest of the group that include modeling and analysis of n-way data, modeling of categories, design of experiments, optimization, and so on, by using classical methods as well as computationally intensive methods (such as neural networks or evolutionary algorithms).


Luis A. Sarabia received his Ph.D. in Statistics from the University of Valladolid (Spain) in 1979. Since 1974, he has been teaching Statistics and Mathematics mostly to graduate and postgraduate students of Chemistry. At present, his research is centred on Chemometrics as a member of the Chemometrics and Qualimetrics group of the University of Burgos. His research activities include development of software and implementation of nonparametric and robust statistical methods, genetic algorithms, neural networks, etc. He is also involved in multivariate/multiway regression methods, methodology of the experimental design, quality assurance, and validation. He is the author of about a hundred papers on these matters.

169


1.06 Resampling and Testing in Regression Models with Environmetrical Applications J. Roca-Pardinãs, University of Vigo, Vigo, Spain C. Cadarso-Sua´rez and W. Gonza´lez-Manteiga, University of Santiago de Compostela, Santiago de Compostela, Spain ª 2009 Elsevier B.V. All rights reserved.

1.06.1 1.06.2 1.06.3 1.06.3.1 1.06.4 1.06.4.1 1.06.5 1.06.5.1 1.06.6 1.06.6.1 References

Introduction to Bootstrap Bootstrap Resampling Methods for Regression Generalized Additive Models Fitting Generalized Additive Models Constructing CIs in GAMs Application to Air Pollution Data Generalized Additive Model with Interactions Estimation Procedure Bootstrap-Based Methods for Testing Interactions Application to Environmetrical Data

171 173 174 175 176 176 179 180 180 181 184

1.06.1 Introduction to Bootstrap The software revolution that has taken place in recent years has led to the development of different statistical methodologies of data analysis that rely on computer-based calculation. Preeminent among these techniques are the so-called resampling methods, which consist of generating a large number of samples to study the behavior pattern of given statistics. There are various procedures for generating artificial samples on the basis of an initial sample. Possibly the best known of these is the bootstrap method introduced by Efron in 1979,1 a type of procedure that uses simulation to assess statistical accuracy. At present, bootstrap methods can be regarded as a general tool for statistical work, tending to be used in combination with other statistical techniques rather than in isolation. These types of methods are applied to different statistical areas, including the construction of confidence intervals (CIs), testing of hypotheses, and regression or principal components analysis, among others (see, for instance, the monograph by Efron and Tibshirani2). The interest that bootstrap methodology has aroused among the statistics community is reflected in the considerable number of textbooks devoted to justifying its theoretical bases or to discussing its applications in specific areas such as biology, environment, or medicine.2–6 In addition, various packages have been developed by implementing a variety of bootstrap methods applied to different statistical methods, and most of these packages are in languages such as Fortran, R, or S-plus. In many practical situations, the goal of research is to make inferences about a given characteristic (or parameter), , of a variable of interest, X. Estimation of this parameter is obtained on the basis of a statistic, a statistical inference ˆ ¼ ˆðX1 ; . . . ; Xn Þ, calculated as a function of a sample of size n ðX1 ; . . . ; Xn Þ, for example, P about the mean population ¼ is based on the sample mean statistic ˆ ¼ X ¼ ni¼1 Xi =n. A good part of conventional statistics is based on laws that enable the sample distribution of a statistic ˆ to be approximated for sufficiently large sample sizes, e.g., where n is large enough, then, in accordance withpthe ffiffiffi central limit theorem, the distribution of sample mean X approximately follows the distribution N ð; S= nÞ, qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi Pn 2 where S ¼ i¼1 ðXi – X Þ =n is the standard sample deviation. For instance, if one were seeking to construct pffiffiffi

a 95% CI for the population mean, this would be given by X 1:96S= n . 171

172

Resampling and Testing in Regression Models with Environmetrical Applications

Procedures of this type based on an asymptotic approach may display certain limitations in practice, as their performance will depend on the information furnished about the population by the sample. Hence, when not enough data are available, a good approximation of the distribution of ˆ will not be obtained, and so the asymptotic method will not produce good results. At other times, moreover, one has to work with statistics that have no asymptotic laws governing their sample distribution. This is the case, for example, of difference in sample medians, or of sample asymmetry the sampling distribution of which is unknown. In the face of these types of limitations, various alternative procedures that enable the sample distribution of the target statistic to be obtained have emerged, by simulating a large number of random samples directly constructed on the basis of initially observed data. Among these resampling techniques is the bootstrap method, which is described in detail below. As an alternative to the asymptotic method, the simplest version of the bootstrap method approximates the distribution of the statistic ˆ ¼ T ðX 1 ; . . . ; X n Þ, in accordance with the following procedure: 1. Based on sample X ¼X1 ; . . . ; Xn ; a random sample X b ¼ X1 b ; . . . ; Xn b is artificially simulated by resampling with replacement. In other words, after the extraction of an element, this is replaced in the original sample such that it can be chosen again. 2. For each sample obtained, the value of the statistic T b ¼ T ðX1 b ; . . . ; Xn b Þ is calculated. 3. Steps 1 and 2 are repeated a large number (B) of times so as to obtain bootstrap values T 1 ; . . . ; T B . At this stage in its development, computer software allows for the computational cost entailed in the generation of a large number of samples to be estimated. 4. Finally, the distribution of T and its corresponding quantiles is approximated by means of a histogram obtained on the basis of values T 1 ; . . . ; T B , that is, an empirical approach to the sample distribution of statistic T is obtained, without any assumptions having been made as to the theoretical distribution to which the latter conforms. The bootstrap resampling method outlined above is known as naive bootstrap. In step 1, the bootstrap samples are simulated by P means of resampling with replacement, that is, based on the empirical distribution Fˆn ðx Þ ¼ n – 1 ni¼1 I fXi x g of the sample. Other resampling approaches have also been considered: When the form of the population distribution is known, the use of parametric bootstrap allows for a better approximation of the sampling distribution of the test statistic. That is to say, if F is known to belong to a parametric family fF : PYg and ˆ is an estimator of (e.g., the estimator of maximum likelihood), then X can be taken as a random sample from Fˆ. For instance, assuming one knows that the observations came from a normal distribution X PN ð;Þ, then one would draw repeated bootstrap samples from a normal distribution N ðX; S Þ. Where the variable X is assumed to be continuous with the density function, f, the use of smoothed bootstrap might be more appropriate. Instead of resampling directly from the empirical distribution Fˆn ðx Þ, one first smoothes it out. RThis smoothed version is then x used to generate new samples from the smoothed distribution Fñ ðx Þ ¼ – 1 fˆn ðx Þ dx, where fˆn ðx Þ is an estimate of density function f (x). As previously pointed out, rather than being perceived as an isolated method, bootstrap is instead used in combination with other statistical techniques. As a review of all bootstrap applications published in recent years would prove extremely difficult, the main thrust of this chapter will focus on how bootstrap works in a regression context. In particular, the possibilities of bootstrap will be analyzed in the context of generalized additive models (GAMs). These types of models were proposed by Hastie and Tibshirani7 as a unifying family of flexible models covering a wide range of regression models with different types of responses (e.g., Gaussian, binomial, and Poisson). Rather than seeking to provide a comprehensive overview of all bootstrap applications in this regression context, this chapter will confine itself to discussing bootstrap resampling methods for achieving two different goals in a GAM context, namely (1) construction of CIs for covariate effects on response and (2) implementation of tests capable of detecting the statistical significance of interaction among the effects of the different covariates. This chapter is laid out as follows: Section 1.06.2 presents different bootstrap resampling methods for regression; Section 1.06.3 introduces GAMs and briefly discusses the process of estimating these models; and


173

Section 1.06.4 addresses the construction of bootstrap-based CIs for covariate effects in GAMs. The advantage of bootstrap methodology versus a classical method of constructing asymptotic intervals is illustrated in Section 1.06.4.1, by reference to air pollution time-series studies.8 The effect of a given covariate on response may often vary with the values taken by another covariate, something that in turn leads to the concept of interaction. Section 1.06.5 introduces GAMs that include interactions among covariate effects. Section 1.06.6 poses the problem of test hypotheses for the detection of significant interactions in GAMs. Finally, Section 1.06.6.1 contains an application to real SO2 binary pollution time-series data.9

1.06.2 Bootstrap Resampling Methods for Regression In many fields of research, it is important to establish the relationship between a response variable of interest and one or more explanatory covariates. Regression studies enable mathematical models to

be obtained that ; . . . ; X . This section sets link the conditional mean ðXÞ ¼ E ðY jXÞ of response Y to p covariates X ¼ X 1 p

out different bootstrap resampling procedures for regression data sets X1; Y1 ; . . . ; ðXn ; Yn Þ of (X,Y). These procedures will be used in Section 1.06.4 to construct CIs for covariate effects, and in Section 1.06.6 to test the significance of interaction terms. Let us assume that the relationship between covariate X and response variable Y can be expressed

as Y ¼ ðXÞ þ ", where " is an error variable having zero mean E ð"jXÞ ¼ 0. Bootstrap samples X 1 ; Y1 ; . . . ;

Xn ; Yn are generated by being drawn from some estimated distribution Rˆn of the true distribution of (X,Y). If ˆ Rn is chosen as the empirical distribution of ðX1 ; Y1 Þ; . . . ; ðXn ; Yn Þ, then the bootstrap samples are generated by being drawn with replacement from the original sample. This bootstrap procedure is called pairwise resampling.10,11 Another alternative is the use of smooth bootstrap described above, where the bootstrap samples are drawn from a smoothed distribution estimate of the joint density of X and Y. For a more detailed discussion of this bootstrap, the reader should refer to Cao-Abad and Gonza´lez-Manteiga.12 In bootstrap procedures proposed until now, both X and Y are assumed to be random variables. This sample scheme is known as random design. However, there are applications where the researcher is able to control the values of the covariates X, and Y is the only random variable. The scheme known as fixed design will now be discussed. The first point to be borne in mind here is that the error variables "i are independently

and identically distributed (i.i.d.) with zero mean. In this case, the bootstrap samples X1 ; Y1 ; . . . ; Xn ; Yn can be obtained with Yi ¼ ˜ ðXi Þ þ "î , where ˜ ðXi Þ is a pilot estimate of ðXi Þ and "ˆ1 ; . . . ; "ˆn is a random sample drawn from the empirical distribution of the centered residuals "î ¼ Y i – ˜ ðXi Þ. This bootstrap procedure is called residual resampling. Errors cannot be assumed to be i.i.d. in all cases, for example, this condition is not fulfilled when the response variance depends on covariate X. In a situation of this kind, possibly the most popular resampling method is the so-called wild bootstrap13–15 introduced by Wu.16 It allows for heterogeneous variance in the residuals. In wild bootstrap, each residual "î is drawn from a distribution that seeks to imitate the distribution of "î up to the first three moments, namely, E ½"ˆ i ¼ 0, E ½"ˆ i 2 ¼ "î 2 , and E ½"ˆ i 3 ¼ "î3 . These pffiffiffi

conditions are pffiffiifffi "i is drawn from the two-point distribution, pwith ffiffiffi probability mass at a ¼ 0:5 1 – 5 "î met and b ¼ 0:5 1 þ 5 "î occurring with probabilities q ¼ 0:1 5 þ 5 and 1q, respectively. A fuller review of resampling methods in regression can be found in Schimek.17 The resampling methods proposed thus far have basically been designed for continuous response regression models. Yet this type of resampling does not adapt to other types of distribution, for example, for binary response models, values other than zero or one, Yi , will be obtained, with the result that the model will perform in quantitative but not qualitative terms. Similarly, should there be a Poisson response, there is no guarantee that the values Yi will not be negative. A general way of approaching these types of situations is to consider the exponential family,18 which includes the majority of distributions in statistics, that is, Gaussian, binomial, or Poisson. In the exponential family, the conditional density function f ðY jXÞ is given by y – b ðÞ f ðY jXÞ ¼ exp þ c ðy; Þ a ð Þ

ð1Þ

174


where denotes the natural parameter, is the scale or dispersion parameter, and a, b, and c are known specific functions in the exponential family. The natural parameter depends on the conditional mean of the response ðX Þ ¼ E ðY jXÞ, and so one can write ¼ ððX ÞÞ. Moreover, the conditional variance ˆ ððXÞÞ, where V is of the response, VarðY jXÞ, depends on ðX Þ, via the relationship VarðY jXÞ ¼ V the so-called variance function, which is known and is determined by the exponential distribution chosen. Bootstrap resampling techniques for obtaining bootstrap samples are based on the conditional distribution of Y|X, determined by ðXÞ and , with the result that the density function f ðY jXÞ given in Equation (1) can therefore be written as f ð ðXÞ; Þ. The bootstrap technique consists of generating a large number (B) of

b b bootstrap replicates X1; Y1 ; . . . ; Xn ; Yn (b ¼ 1, . . . , B), with Yi b f ð˜ ðXi Þ Þ being based on the original sample ðX1 ; Y1 Þ; . . . ; ðXn ; Yn Þ, and ˜ ðXi Þ being the pilot estimates obtained on this basis.19,20 In special situations, the dispersion parameter is known and the conditional distribution of Y is specified by ðXÞ. For instance, for binary response Yi BernoulliððXi ÞÞ, the bootstrap replicates will be generated in line with Yi Bernoullið˜ ðXi ÞÞ. Likewise, for Poisson response Yi PoissonððXi ÞÞ, the bootstrap replicates will be generated in line with Yi Poissonð˜ ðXi ÞÞ. Otherwise, if is unknown, an estimate ˆ for ˆ ððXÞÞVarðY jXÞ ¼ V ˆ ððXÞÞ or equivalently parameter can be obtained, in line with VarðY jXÞ ¼ V

¼ VarðY jXÞ=V ððXÞÞ, from

ˆ ¼

n X ðYi – ˆ ðXi ÞÞ2 i¼1

nV ðˆ ðXi ÞÞ

where ˆ ðXi Þ are the estimates of ðXi Þ obtained on the basis of the original sample fðXi ; Yi Þgni¼1 . For instance, in the case of Gaussian response Yi – N ððXi Þ;Þ the parameter is unknown and coincides with ˆ where ˆ 2 ¼ n 2.P In such a case, the bootstrap responses will be generated in line with Yi – N ððX ˜ i Þ;Þ, n 2 –1 2 ˜ ðX i ÞÞ is the estimation of . It should be noted that in this approach, the variance 2 is a i¼1 ðYi – constant and the regression model is thus assumed to be homoscedastic. Nevertheless, our interest also lies in heteroscedastic regression models, where the variance 2 ¼ 2 ðXÞ depends on the vector of the covariates X. To estimate 2(X), a nonparametric regression model can be fitted to Ri ¼ ðYi – ˜ ðXi ÞÞ2 on X. Another alternative is to use the wild bootstrap method outlined above.

1.06.3 Generalized Additive Models The models that will be studied here can be viewed as a generalization of the well-known generalized 18 mean ðXÞ ¼ E ½Y jX of response Y depends on linear model (GLM). In these

GLMs, the conditional the covariates X ¼ X1 ; . . . ; Xp via X þ þ p Xp , where H is a known monotone ðXÞ ¼ H þ 1 1

link function and ; 1 ; . . . ; p is a vector of coefficients. In some instances, GLMs can be very restrictive, because they assume linearity in the covariates. This constraint can be avoided by replacing the linear index ¼ þ 1 X1 þ þ p Xp with a nonparametric structure. Accordingly, here we will concentrate on the GAM7, which is a generalization of the GLM, by introducing one-dimensional, nonparametric functions in lieu of linear components. Specifically, GAMs express the conditional mean (X) as

ðXÞ ¼ H þ f1 ðX1 Þ þ þ fp ðXp Þ

ð2Þ

where H is a known link function, is a constant, and f1 ; . . . ; fp are unspecified, unknown, zero-mean functions. In the case of an identity function, H, one speaks of an additive model. On assuming that effects are additive, GAMs maintain the interpretability of GLMs.18,21 Yet, at the same time, they incorporate the flexibility of nonparametric smoothing methods because, rather than following a fixed parametric form, the effect of each of the covariates, Xj, depends on a totally unknown function, fj, which is only required to possess a certain degree


175

of smoothness so that it can be estimated. The adaptability of data belonging to an exponential distribution, together with the GAM compromise between flexibility, dimensionality, and interpretability, ranks these types of models among the statistical tools with the greatest capacity for data analysis in different fields of research. To learn more about these models, the reader should consult Gu,22 Ruppert et al.,23 and Wood.24 Although this chapter was mainly focused on GAMs, alternative statistical methodologies, such as the alternating conditional expectation (ACE) algorithm developed by Breiman and Friedman,25 can also be used in this context. This algorithm estimates the transformations of a response and a set of covariates in multivariate regression that produce the maximum linear effect between the transformed independent variables and the transformed response variable. These types of transformed models can be fitted to data by means of R routine ACE.

1.06.3.1

Fitting Generalized Additive Models

This section describes an algorithm for fitting the effects f1 ; . . . ; fp in the GAM in Equation (2). The algorithm shown below is a combination of local scoring and backfitting techniques.26 In brief, local scoring is analogous to the use of iteratively reweighted least squares18 for solving likelihood and nonlinear regression equations. At each iteration of the estimation algorithm, the estimates fˆj were obtained by applying some smoother to the liberalized transformed responses. Although our focus is on local scoring, there are other types of procedures that allow for nonparametric estimation of GAMs. Sperlich et al.27 presented methods based on marginal integration. Wahba28 and Guo29 proposed the use of smoothing spline analysis of variance (ANOVA) methods. Coull et al.30 and Ruppert et al.23 investigated alternative methods based on penalized splines (P-splines), and Wood31 used thin plate regression splines. Other studies, such as the paper by Brezger and Lang,32 also used P-splines, and developed Bayesian versions of GAMs and extensions to generalized structured additive regression. the local Given a sample ðX1 ; Y1 Þ; . . . ; ðXn ; Yn Þ of (X,Y), the steps ofP

scoring algorithm are

as follows: Initialization. Compute the initial estimates ˆ ¼ H – 1 n – 1 ni¼1 Yi and î 0 ¼ 0, fˆj 0 Xij ¼ 0, for i ¼ 1; . . . ; n. Step 1. Construct the linearized response Y˜i ¼ ˆ 0i þ ðYi – H ðˆ i0 ÞÞ=H 9ðˆ i0 Þ and the weights Wi ¼ 2 H 9ðˆ i0 Þ =V ðˆ i0 Þ, where H 9ðÞ ¼ H = is the first derivative of H with respect to . Step 2. Fit a nonparametric additive regression model to Y˜i on Xi and compute the updates î , using the following backfitting procedure:

Step 2.1. For j ¼ 1; . . . ; p, obtain the updated nonparametric estimates fˆj Xij by using a linear smoother P P given by the following weighted (local) average of the partial residuals Rji ¼ Y˜i k<j fˆk ðXik Þ – k>j fˆ k0ðXik Þ (obtained by removing the estimated effects of the other covariates): fˆj ðx Þ ¼

n X

W i W j Xij Rij

ð3Þ

i¼1

where W j is a weight function depending on a smoothing parameter j .

Step 2.2. Repeat Step 2.1 with fˆ1 0 ðXi1 Þ; . . . ; fˆp 0 Xip replaced by fˆ1 ðXi1 Þ; . . . ; fˆp Xip for i ¼ 1; . . . ; n, until convergence.

Step 3. Repeat Steps 1 and 2 with î0 replaced by î ¼ ˆ þ fˆ1 ðXi1 Þ þ þ fˆp Xip until there is no change in the estimates. In expression (3), different types of estimates will be obtained,

on the

type of smoother used to P depending obtain weights W j . For instance, if weights W j ðx Þ ¼ n – 1 ni¼1 K Xij – x = j are used, with K being any given univariate density function, the Nadaraya–Watson estimator introduced by Nadaraya33 and Watson34 in 1964 is obtained. Other commonly used smoothers in nonparametric regression are local linear kernel smoothers,35 smoothing splines,7 or P-splines.36

176


At all events, it is well known that estimates depend heavily on the smoothing parameters j used in the estimation. The smoothing parameter j was selected automatically by minimizing the following weighted cross-validation error: CVj ¼

– iÞ

n X

2

Wi fˆ j – iÞ Xij – Ri j

i¼1

where fˆ j Xij indicates the fit at Xij, excluding the ith data point. Nowadays, the existence of standard software makes it easy to fit GAMs in practice. Thus, we may find a variety of freely available packages written in R language (http://cran.r-project.org/) using different estimation methods and smoothers. So far, the most widely used packages are gam,7 Semipar,23 VGAM,37 mgcv,24 and gamlss.38

1.06.4 Constructing CIs in GAMs To construct the CIs for partial effects, fj , in model (2), one has to know the distribution of estimations fˆj – f j , where fˆj is the nonparametric estimation of the true fj . As seen in Section 1.06.3.1, however, iterative algorithms based on nonparametric smoothers are used for estimation of the GAM in Equation (2), which makes obtaining asymptotic results for the distribution of fˆj – f j somewhat complicated. In Gu39,40 and Hastie and Tibshirani,7 smoothing splines are proposed as smoothers to construct the following (1) 100% limits for the CI for fj ðx Þ:

fˆj ðx Þz1 – =2 ˆ j ðx Þ

where ˆ j 2 ðx Þ is some estimate of the variance 2j ðx Þ of fˆj ðx Þ, and zp is the p-percentile of the standard Gaussian distribution. Yet, this procedure for constructing CIs does not always perform well. For instance, these types of approximations have been shown to suffer from an important limitation, in that the standard error j ðx Þ of the estimated effect of any covariate is underestimated and so the ensuing CIs are too narrow in cases where concurvity is present in the data.41 Briefly, concurvity is the nonparametric analog of multicollinearity, in which a partial function of one of the independent variables can be approximated by a linear combination of functions of the remaining variables. One way of eliminating the problem of underestimation of j ðx Þ in the presence of concurvity is to resort to bootstrap resampling techniques. The steps for construction of the bootstrap-based CIs for fj ðx Þ are as follows: Pp Step 1. Estimate the model in Equation (2) so as to obtain pilot estimates ˜ ðXi Þ ¼ H ˜ þ j ¼1 fˆj Xij for i ¼ 1, . . . , n, and then f˜j ðx Þ.

n Step 2. For b ¼ 1; . . . ; B generate the sample Xi ; Yi b i¼1 , where each Yi b is drawn from the estimated density function fˆð˜ ðXi ÞÞ (see Section 1.06.1 for a detailed description of this type of bootstrap resampling) and obtain bootstrap estimates fˆj b ðx Þ. Once this process has been completed, the (1) 100% limits of the CI for the true fj ðx Þ are given by fˆ j ðx Þ – fˆj ð1 – =2Þ ðx Þ; fˆj ðx Þ – fˆj ð=2Þ ðx Þ

where fˆj ðpÞ ðx Þ represents the p-percentile of the B differences fˆj 1 ðx Þ – f˜j ðx Þ; . . . ; fˆj B ðx Þ – f˜j ðx Þ. 1.06.4.1

Application to Air Pollution Data

Time-series studies with Poisson regression have been the reference method for analyzing the short-term effects of air pollution on health. To control for the possible confounding effect of unmeasurable variables that generate trend and seasonality in the series, different parametric functions (linear, quadratic, and sinusoidal of varying width) of the variable ‘time’ were initially used. However, these models could prove very ‘rigid’ and unadaptable in the case of confounding variables of seasonal behavior of varying width and frequency. Application of GAM-based nonparametric smoothing methods (smoothing splines or LOESS) has meant a great advance in that it has eradicated the need for researchers to specify the parametric forms of seasonality a


177

priori and enabled flexible adaptation to seasonal behavior. Consequently, most studies published in recent years on the short-term effects of pollution on health have applied this methodology.42–44 As has already been stated, however, in cases where concurvity is present in the data, asymptotic CIs prove too narrow.41,45–47 Alternatively, CIs can be obtained using bootstrap resampling techniques. Indeed, a comparative study was undertaken into the coverages obtained with asymptotic and bootstrap CIs,8 using the time series Yt, denoting the number of deaths per day in the city of Vigo (Spain) in 1996, 1997, and 1998 (t ¼ 1; . . . ; 1096). For each day t, black smoke (BS) (shown in Figure 1) BSt and temperature tempt were recorded. This study considered the Poisson model

Yt Poissonðt Þ with logðt Þ ¼ þ BSt þ ftrend ðt Þ þ ftemp tempt

ð4Þ

in which ftrend is a partial function that depends on time (seasonal and trend component) and ftemp represents the functional form of the effect of temperature on mortality. The main objective was to obtain valid CIs for the coefficient , which denoted the log relative rate of Y associated with an increase in BS, based on the sample

1996 t ; tempt ; BSt ; Yt t ¼1 . Data were analyzed using routine gam of S-Plus-based smoothing splines, with the estimated coefficients ˆ ¼ 3:09 and ˆ ¼ 0:001 being obtained. The estimated effects fˆtrend and fˆtemp are plotted in Figure 2. The concurvity measure proposed by Ramsay et al.41 was based on the correlation between the fitted values obtained from the GAM, with pollution, BSt , as the response and with time and temperature as the predictors. Specifically, to assess the degree of concurvity in our data, one should (1) fit the GAM BSt ¼ g1 ðt Þ þ g2 ðtempt Þ þ "t , where the partial functions g1 and g2 are estimated using smoothing splines, ct , and "t is a zero mean error variable, and (2) then compute the squared correlation (R2) between BSt and BS using the fitted values from model in Equation (4), namely

2 ct ¼ gˆ1 ðt Þ þ gˆ2 temp concurvity ¼ correlation BSt ; BS t

In the data, concurvity was 0.29. To vary the concurvity and thereby assess its influence on CI coverage and ˆ in model (4), the original BS was replaced by a weighted average of the bias in the parameter estimate, , ct , and the difference BSt – BS ct . Specifically, the new variable BSst was calculated original BSt , the estimated BS

Daily mean of BS

200

150

100

50

0 0

200

400

600 Day (t )

800

1000

Figure 1 Daily mean black smoke (BS) levels in Vigo for 1096 days, in the period 1996–98. Reproduced from Figueiras, A,; Roca-Pardinãs, J.; Cadarso-Sua´rez, C. A. Bootstrap Method to Avoid the Effect of Concurvity in Generalized Additive Models in Time-Series Studies of Air Pollution. J. Epidemiol. Community Health 2005, 59, 881–884, with permission from BMJ Publishing Group Ltd.

178


0.4 2.9

fˆtrend

fˆtemp

0.3

2.4 0.2 1.9 0.1 1.4 0.0

0.9

–0.1

0.4 0

200

400

600 Day (t)

800

1000

0

5

10

15 Temp

20

25

Figure 2 Estimated seasonal and trend component, and estimated smoothed temperature effect. Reproduced from Figueiras, A,; Roca-Pardinãs, J.; Cadarso-Sua´rez, C. A. Bootstrap Method to Avoid the Effect of Concurvity in Generalized Additive Models in Time-Series Studies of Air Pollution. J. Epidemiol. Community Health 2005, 59, 881–884, with permission from BMJ Publishing Group Ltd.

c t þ ð1 – w1 – w2 Þ"ˆt . Thus, by varying both weights w1 P½0; 1 and w2 P½0; 1, in line with BSst ¼ w1 BSt þ w2 BS concurvity may easily be modified from 0 (w1 ¼ w2 ¼ 0) to 1 (w1 ¼ 0, w2 ¼ 1). In the study in question, a number of scenarios were defined by using different values for w1 and w2, with the result that the degree of concurvity varied from 0.05 to 0.65. In each of the scenarios, 1000 samples

1096

t ; tempt ; BSst ; Yt t ¼1 were considered with Yt generated in accordance with Yt Poisson t , with

log t ¼ þ BSt þ ftred ðt Þ þ ftemp tempt

where ¼ 3:09, ¼ 0:001, mtrend ¼ fˆtrend and mtemp ¼ fˆtemp , and fˆtrend and fˆtemp were the estimated partial functions obtained with the original data. Lastly, the corresponding bootstrap and asymptotic 95% CIs were calculated for ¼ 0:001. CI coverages were calculated as the proportion of samples in which these included the true . In Figure 3, CI coverages are compared for different degrees of concurvity, using asymptotic GAM and GAM bootstrap. In this figure, the vertical line is that of the true concurvity, 0.29, obtained with w1 ¼ 1 and 1.00

Coverage

0.95

0.90 Asymptotic 0.85

Bootstrap

0.80 0.05

0.15

0.25

0.35 Concurvity

0.45

0.55

0.65

Figure 3 Effect of concurvity on 95% CI coverage. Reproduced from Figueiras, A,; Roca-Pardinãs, J.; Cadarso-Sua´rez, C. A. Bootstrap Method to Avoid the Effect of Concurvity in Generalized Additive Models in Time-Series Studies of Air Pollution. J. Epidemiol. Community Health 2005, 59, 881–884, with permission from BMJ Publishing Group Ltd.


179

w2 ¼ 0. It will be seen that, with the asymptotic approach, increases in concurvity were accompanied by a progressive decline in coverage, which dropped as low as 80% when concurvity was 0.6. With the bootstrap method, however, coverage remained above 94% throughout, irrespective of the degree of concurvity. Accordingly, the results of our study show that application of bootstrap techniques makes it possible for the CIs of contaminant-related effects to be calculated in time-series studies with optimum coverage, regardless of the degree of concurvity.

1.06.5 Generalized Additive Model with Interactions The results obtained with the pure GAM in Equation (2) are easily interpretable, as the effect of each of the covariates Xj is given by a single partial function fj via the known, monotone link function H. It is known, however, that in certain situations the effect of a given covariate on response may vary with the values taken by one or more of the remaining covariates, something that in turn leads to the concept of interaction. GAMs with interactions take the form p X X

ðXÞ ¼ H þ fj Xj þ fjk Xj ;Xk þ j ¼1

1j Fcalc} called the p-value. Therefore, the null hypothesis is rejected (the model is significant) at level if the p-value is smaller than . Table 5 shows the ANOVA for the model of first order fitted to the data in Example 1. It is evident that the model is not significant at the customary level of 0.05. 1.12.4.2

Lack of Fit Test

The estimate of 2 obtained in Equation (13) is dependent on the model. That is, it depends on the form of the fitted model, and therefore, if the proposed model is inadequate, s2 is no longer a valid estimate of 2. If replicate runs are available (i.e., more than one observation on y at the same x design point) a modelindependent estimate of 2 can be obtained. Let us suppose that some of the N experiments are replicates in one or several points of the experimental domain; it is clear that the variability in the response obtained at these points is only attributable to the random error ". SupposeP that at each point xi (i ¼ 1, . . ., n) of the n experimental design mi, observations y are obtained so that i¼1 mi ¼ N . The internal variability of Pmi ij 2 these observations is given by j ¼1 yij – yi with mi 1 degrees of freedom. Therefore, pooling over the levels xi we have the so-called sum of squares of pure error: SSPE ¼

mi n X X yij – yi Þ2

ð29Þ

i¼1 j ¼1

with

Pn

i¼1 ðmi

– 1Þ ¼ N – n degrees of freedom.

Table 4 Analysis of variance for significance of regression Source of variation

Sum of squares

Degrees of freedom

Mean square

Fcalc

Regression Error or residual Total

SSR SSE SST

p1 N–p N–1

MSR MSE

MSR =MSE

Table 5 ANOVA for significance of first-order model fitted to data of Table 2 Source of variation

Sum of squares

Degrees of freedom

Mean square

Fcalc

p-value

Regression Error of residual Total

1.2838 0.8303 2.1141

2 4 6

0.6419 0.2076

3.092

0.154

356

Response Surface Methodology

Table 6 ANOVA for significance of regression and lack of fit Source of variation

Sum of squares

Degrees of freedom

Mean square

Fcalc

Regression Error of residual Lack of fit Pure error Total

SSR SSE SSLOF SSPE SST

p–1 N–p n–p N–n N–1

MSR MSE MSLOF MSPE

MSR =MSE MSLOF =MSPE

In contrast, we have SSE ¼

mi mi n X n X n X

2 X

2 X yij – yî ¼ yij – yi þ mi ðyi – yî Þ2 ¼ SSPE þ SSLOF i¼1 j ¼1

i¼1 j ¼1

ð30Þ

i¼1

In Equation (30) we can see that the sum of squares for lack of fit, SSLOF, is a weighted sum of squared deviations between the mean response yi at each xi level and the corresponding fitted value yî . Table 6 summarizes the decomposition of the sum of squares of Equation (30) with the corresponding degrees of freedom. E(MSPE) ¼ 2 holds. To evaluate E(MSLOF) we suppose that the model to be fitted is y ¼ bX þ e

ð31Þ

but the true model is y ¼ bX þ b2 X2 þ e –1

Then the estimate b ¼ ðX9XÞ X9y (Equation (12)) of b is biased because the expected mean of the estimate is not b but E ½b ¼ b þ Ab2

ð32Þ

A ¼ ðX9XÞ – 1 X9X2

ð33Þ

E ½MSLOF ¼ 2 þ b92 ðX2 – XAÞ9ðX2 – XAÞb2

ð34Þ

where the bias matrix A is

Consequently, it can be shown that

Thus, provided that X2 – XA is non-null, E ½MSLOF will be inflated when b2 is nonzero. If " is N(0, 2) and b2 ¼ 0 it can be shown that Fcalc ¼

SSLOF =ðn – pÞ MSLOF ¼ SSPE =ðN – nÞ MSPE

ð35Þ

is distributed as a Fn p,N n. Therefore, we can test the null hypothesis of the lack of fit: H0 : b2 ¼ 0 Ha : b2 6¼ 0

ð36Þ

If Fcalc > F,n p,N n the null hypothesis is rejected at level , that is, we conclude that the model proposed has lack of fit (is biased). Alternatively, if Fcalc does not exceed F,n p,N n, there is no experimental evidence of lack of fit, and MSPE and MSLOF are often combined to estimate 2. For the data in Table 2, N ¼ 7 and n ¼ 5. For each experimental point, mi ¼ 1, i ¼ 1, . . ., 4 and m5 ¼ 3. The proposed model has p ¼ 3 coefficients. The lack of fit test is shown in Table 7. As we have an estimate of the variance based on the replicates, and thus independent of the model, we discover that the first-order model is


357

Table 7 ANOVA with test for lack of fit for first-order model fitted to data of Table 2 Source of variation

Sum of squares

Degrees of freedom

Mean square

Fcalc

p-value


1.2838 0.8303 0.8281 0.0022 2.1141

2 4 2 2 6

0.6419 0.2076 0.4140 0.0011

3.092

0.154

368.25

0.003

not adequate because the null hypothesis of the test for lack of fit should be rejected. Consequently, the F-test for significance of the first-order model should not be carried out. Obviously, the conclusion is clear: It is recommendable to include replicates at some points of the design to evaluate the lack of fit. In Example 1, the increase of experimental effort with respect to the D-optimal design is justified by the type of information that it has provided on the nonvalidity of the first-order model. In most cases, the lack of fit initiates a search for a more adequate model. The nature of the inadequacy would first be sought by analyzing the residuals and remedial measures, perhaps involving transformation of y or one or more of the X’s or possibly by use of a radically different model, would then be taken. Occasionally, investigations might show that the deficient model is, nevertheless, sufficient for the purpose at hand and therefore may be used with proper caution. As Box and Draper accurately say22 ‘‘Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful.’’ The lack of fit in Example 1 is due to a strong curvature that would remain undetected without experiments in the interior of the experimental design. To augment the design with new experiments needed to fit a secondorder model (which needs at least three levels per factor) a central composite design was selected with the additional experiments shown in Table 8. The characteristics of the design will be studied later. The proposed second-order model is y ¼ 0 þ 1 X1 þ 2 X2 þ 12 X1 X2 þ 11 X12 þ 22 X22 þ "

ð37Þ

The design is a standard central composite one (see Section 1.12.5.2.1) and it is optimal in relation to the criteria just described. The ANOVA (Table 9) shows that there is no experimental evidence, at the 5% significance level, to reject the null hypothesis of the test for lack of fit. Thus, we should accept that there is no bias in the model. Further, the model is significant, that is the null hypothesis (all the coefficients are zero) is rejected, and thus the model significantly explains the variance in the response. Ideally, as in Example 1, we find that the null hypothesis of the lack of fit test (unbiased model, Equation (36)) is not rejected and also the null hypothesis of the test for significance of regression (Equation (27)) is rejected. However, this does not guarantee that the model will be satisfactory as a prediction equation that is the first objective of RSM. This makes us to carefully examine the procedure followed, above all because it is advisable to follow a sequential procedure starting with a simple model and augmenting the degree of the polynomial, as was done in Table 8 Design and experimental response to augment the design in Table 2 Design matrix (codified variables)

Experimental matrix (natural variables)

Run

X1

X2

U1 ¼ Vhexone (ml)

U2 ¼ HCl (M)

Response

8 9 10 11

1.414 1.414 0 0

0 0 1.414 1.414

2.93 17.07 10.00 10.00

3.00 3.00 0.17 5.83

1.135 1.501 0.052 1.354

Source: Ortiz, M. C.; Herrero, A.; Sanllorente, S.; Reguera, C. The Quality of the Information Contained in Chemical Measures; Servicio de Publicaciones de la Universidad de Burgos, Burgos, 2005. http://www.ubu.es/investig/grupos/cien_biotec/QA-4/index_eng.htm.

358


Table 9 ANOVA with test for lack of fit for second-order model fitted to data of Table 2 augmented with that of Table 8 Source of variation

Sum of squares

Degrees of freedom

Mean square

Fcalc

p-value


3.4227 0.0624 0.0601 0.0022 3.4851

5 5 3 2 10

0.6845 0.0125 0.0200 0.0011

54.86

< 3 104

17.83

0.054

the example. The hypothesis tests are designed to reject the null hypothesis at significance level , which means they are built such that is the probability of rejecting H0 when it is true. In the case of the test for lack of fit, would be the probability of stating that the model is biased (b2 6¼ 0 in Equation (32)) when really it is not. However, what happens when there is no experimental evidence to reject the null hypothesis? Then it is necessary to know exactly what the alternative hypothesis is to evaluate the probability of taking the wrong decision. That is to say, to evaluate the probability of accepting the null hypothesis (the model is not biased) when it is false. Evidently, in this case, the accepted model will not have capacity of prediction. Another aspect to consider is that, if the number of degrees of freedom is small, statistical tests loses power (probability becomes large). In other words, much experimental evidence is needed to reject the null hypothesis. We will return to this question later. To specify the alternative hypothesis, it is necessary to define the family of models adequate for our purposes. The models to be used are defined by the corresponding model matrix and remember that the experimental design plays a part in the model matrix. To explain this question let us pose a hypothetical case. Example 2: We have three factors and one response and we look for a model to predict the response at any point of the experimental domain, which in codified units is the cube whose side is [1, 1]. It is accepted that the response could be well fitted with a polynomial of degree 2, but we start by proposing a first-order model. The experiments made and the responses obtained are in Table 10. The way the problem is posed and following the notation in Equations (31) and (32), we have X, which is the model matrix corresponding to the first-order model to be fitted to the experimental data and X2 that corresponds to the second-order model matrix without the terms of degree 1 (which can be possible extensions for the model). Table 11 shows the ANOVA for the lack of fit. In this case, the value of B2 is written as B 2 ¼ b92 ðX2 – XAÞ9 ðX2 – XAÞb2 ¼ b2 9F9Fb2 (Equation (34)), which is a quadratic form in b2 . In our Example 2, F9F ¼ diagð8; 8; 8; 0; 0; 0Þ and thus E ½MSLOF ¼ 2 þ

8 2 2 2 þ 23 12 þ 13 5

ð38Þ

This is to say that the design proposed in Table 10 is only sensitive to lack of fit produced by the cross-product terms in the second-order model, but is completely insensitive to lack of fit produced by the pure quadratic Table 10 Example 2: Design matrix (codified variables) and response Run

X1

X2

X3

Y

1 2 3 4 5 6 7 8 9 10 11 12

1 1 1 1 1 1 1 1 0 0 0 0

1 1 1 1 1 1 1 1 0 0 0 0

1 1 1 1 1 1 1 1 0 0 0 0

30.36 41.58 38.06 56.24 45.30 54.31 63.00 73.69 49.23 49.06 51.02 51.75


359

Table 11 Detailed structure of the analysis of variance shown in Table 6 Source of variation

Sum of squares

Degrees of freedom

Mean square

Regression Error of residual Lack of fit

SSR SSE SSLOF

p–1 N–p n–p

MSR MSE MSLOF

Pure error Total

SSPE SST

N–n N–1

MSPE

E[MS]

2 þ 2

B2 5

terms. Therefore, if the response depends on the square of one or several factors, the model is inadequate and the lack of fit test cannot detect it. Box and Draper23 consider a strategy of experimental design such that, in the second stage, the selected design is the one that gives a good detection of model inadequacy. Designs maximizing the criterion b2 9F9Fb2 belong to the class of T-optimal designs6 and are clearly attractive. But the T-criterion can only be evaluated provided that the parameter vector b2 is either a scalar or is known and this is not common. The criterion most widely used in building designs optimal for detecting departures from the fitted model is the determinant jF9Fj. This one has particular appeal in that is essentially a D-optimal or GSD criterion and all the advantages of such criteria immediately accrue. Other measures, such as the trace and the smallest eigenvalue of F9F, are also proposed. A detailed description of the variations can be seen in Haines.8 When introducing Example 2, we pointed out that the class of the matrices to augment the first-order model to a second-order one depends on the design used in the first stage. For example, suppose that instead of the design of Table 10, the design shown in Table 12 had been selected. Its points are at the same distance to the center point with those of Table 10 but arranged on the axes. This design is worse than the one in Table 10 according to the quality criteria but serves our purpose. By repeating the computation of the expected value for MSLOF we have E ½MSLOF ¼ 2 þ

6 ð11 – 22 Þ2 þð11 – 33 Þ2 þð22 – 33 Þ2 3

ð39Þ

Now, the test is insensitive to the lack of fit produced by the cross-product terms. Further, even if the ratio MSLOF =MSPE is small there is no guarantee that the coefficients ii are null but only that they are equal. To estimate the coefficients ij, we would need to augment the design in Table 12 with points in the vertices of the cube, that is, the first eight in Table 10. In summary, to state the concrete meaning of the alternative hypothesis for the lack of fit test it is necessary to (1) define the complete space of the model matrices, (2) evaluate the expected value of MSLOF, and (3) augment the design with points adequate according to the previous and some of the criteria based on matrix F9F. The ANOVA of the data in Table 10 forms Table 13; the conclusion is that there is no experimental evidence to reject the first-order model. However, it can be due to the adequacy of the first-order model or,

Table 12 Alternative design Run

X1

X2

X3

1 2 3 4 5 6 7 8 9 10

1.7321 1.7321 0 0 0 0 0 0 0 0

0 0 1.7321 1.7321 0 0 0 0 0 0

0 0 0 0 1.7321 1.7321 0 0 0 0

360


Table 13 ANOVA with test for lack of fit for the data of Table 10 first-order model Source of variation

Sum of squares

Degrees of freedom

Mean square

Fcalc

p-value


1356.54 56.97 51.67 5.30 1413.51

3 8 5 3 11

452.18 7.12 10.33 1.77

63.50

< 104

5.85

0.088

taking into account Equation (38), because some quadratic terms (corresponding to the cross products) are needed and the test cannot detect it. In this case, taking into account that we are using a sequential strategy, the design is augmented with the points that will serve to estimate the second-order model. To do this, we use the axial points (in fact, they complete a central composite design as we will see later). Table 14 summarizes the new experiments and the results obtained. Table 15 summarizes the ANOVA for the fitting of a second-order model to the data in Table 10 augmented with those in Table 14. There is no evidence to reject the null hypothesis of the lack of fit test at the 5% significance level. It has already been explained that in the lack of fit test (Equation (36)), the ideal situation is not to reject the null hypothesis as, for example, in Tables 9 and 15. In this case, the probability of accepting the null hypothesis when it is false is relevant and should be inspected. In general, the experimental designs in RSM consist of few experiments and, consequently, is large. For example, in Table 9 the distribution of Fcal has three and two degrees of freedom (in Table 15, five and three). If it is considered that the variance 2 þ ðB 2 =5Þ is different from 2 in Table 11 when it is double or more, then is 0.93 for the lack of fit test in Table 9 and 0.90 for the one in Table 15.

1.12.4.3

Using Test Points to Detect Lack of Fit

There is an alternative practical approximation to the problem of ensuring the possibility of detecting the lack of fit that is frequently used. We introduce it with the same Example 2. Once the first experiment has been done and analyzed (ANOVA of Table 13), we provisionally accept the fitted first-order model: y ¼ 50:30 þ 6:14X1 þ 7:43X2 þ 8:76X3

ð40Þ

Table 14 New experiments (codified variables) added to those of example 2 (Table 10) Run

X1

X2

X3

Y

13 14 15 16 17 18

1 1 0 0 0 0

0 0 1 1 0 0

0 0 0 0 1 1

56.02 69.12 31.68 50.31 36.95 51.28

Table 15 Analysis of variance with test for lack of fit for the data of Table 10 Source of variation

Sum of squares

Degrees of freedom

Mean square

Fcalc

p-value


2301.38 22.62 17.32 5.30 2324.00

9 8 5 3 17

255.71 2.83 3.46 1.77

90.43

< 104

1.96

0.31


361

The idea is to use some test points that, in general, are arranged in such a way that they are the farthest possible from those of the design carried out; or that they minimize some criterion like jF9Fj, or considering the adequate ones to fit a second-degree polynomial. In Example 2, we use the experiments in Table 14 as test points. The next step is to decide if the fitted first-order model (Equation (40)) describes adequately the experimental values obtained at the test points; the procedure to compare the experimental results with those predicted with the first-order model is described next. The variance of the prediction with the fitted model, yˆðxÞ, at any point of the experimental domain is varðyˆ ðxÞÞ ¼ ð1 þ d ðxÞÞ2 , where d ðxÞ is the value of the variance function at the point (cf. Equation (24)). From the data in the ANOVA of the fitted first-order model (Table 13), we have an estimate of the residual variance s2 ¼ 7.12 with eight degrees of freedom. Now, it is possible to decide if every experimental value obtained at the test points, yexp , is significantly different from the value estimated with the model of Equation (40), y.ˆ The hypothesis test is

H0 : yˆ ¼ y exp or yˆ – y exp ¼ 0 Ha : yˆ 6¼ y exp

ð41Þ

pffiffiffiffiffiffiffiffiffiffiffiffiffi and the statistic is tcalc ¼ yˆ – y exp = var ðyˆ Þ. The null hypothesis is rejected at the significance level if jtcalc j > t=2; , where are the degrees of freedom of the estimated variance. Table 16 shows the results of applying the hypothesis test to the test points in Table 14. At all the points, except for number 17, the null hypothesis should be rejected, that is, the predicted values are significantly different from the experimental values in the test points. Consequently, the first-order model of Equation (40) is invalid in the experimental domain. Then, a second-order model is fitted with all the data. In this case, we have already seen (Table 15) that this second-order model does not evidence lack of fit. Strictly, the experimentation should be continued by establishing the family of models respect to which E[MSLOF] is going to be evaluated, and to determine the most adequate additional points to maximize E[MSLOF] or jF9Fj. Alternatively, the test points could be chosen as the most distant of the 18 present ones, for example, the vertices of a tetrahedron inscribed in a sphere with radius 0.5 and centered at the origin. The ANOVA to detect possible third-order terms and the experimental design needed can be seen on p 234 of the book by Box and Draper.22 1.12.4.4

Discriminating Models

It is possible to define experimental designs to choose between two or more alternative models; none of them is a particular case of the others. The optimal design to discriminate model y2 ¼ b2X from model y1 ¼ b1X should give a large E[MSLOF] for b2. Like the case of the lack of fit, the T-optimal designs work and they can be constructed in a sequential way by adding the new experimental points one by one (Atkinson and Donev,6 Chapter 20). As we have already mentioned in the section on lack of fit, the T-optimal designs depend on which model is true and on the parameters of the true model. The sequential procedures overcome this defect, but if sequential optimization is not possible, we are left with a design which is only locally optimum. One possibility is to specify prior distributions and then to take the expectation of design criterion over this distribution. This Bayesian approach can be very useful. Table 16 Comparison of responses: yexp are the experimental values; yˆ those predicted by model of Equation (40) Run

yexp

yˆ

tcalc

p-value

13 14 15 16 17 18

56.02 69.12 31.68 50.31 36.95 51.28

44.16 56.44 42.87 57.73 41.54 59.06

4.042 4.324 3.815 2.530 1.566 2.651

0.0037 0.0025 0.0051 0.0353 0.1560 0.0292

362


1.12.4.5

Confidence Intervals and Significance of the Coefficients

With the hypothesis of the normal distribution of residuals, each coefficient j follows a Student’s t distribution with degrees of freedom (those of the estimation of s2) and variance cjjs2. If MSE is used to estimate s2, there are N p degrees of freedom. Therefore, the confidence interval on the estimated coefficients bj at the 100(1 )% confidence level is

pffiffiffiffiffiffiffiffi pffiffiffiffiffiffiffiffi bj – t=2;N – p c jj s 2 ; bj þ t=2;N – p c jj s 2 ; j ¼ 0; 1; 2; :::; p – 1

ð42Þ

where t=2; is the =2 critical value of a Student’s t-distribution with degrees of freedom. Once fit to a model, it can be interesting in some cases to decide if a coefficient is significantly different from zero. The hypothesis test for each one is H0 : j ¼ 0 Ha : j 6¼ 0

ð43Þ

The statistic for this hypothesis test is bj tcalc ¼ pffiffiffiffiffiffiffiffi c jj s 2

ð44Þ

The null hypothesis is rejected at the significance level if jtcalc j > t=2;N – p . It is also possible to decide with a hypothesis test if a group of the model coefficients are significantly different from zero. In RSM, the basic purpose is to adequately describe the experimental response and, in general, it is useless to eliminate the no significant regression coefficients. 1.12.4.6

Joint Confidence Region on the Regression Coefficients

Equation (18) shows the general form of the confidence region. At the 100(1–)% confidence level, the confidence region for all parameters in b is ðb – bÞ9X9Xðb – bÞ p MSE F;p;N – p

ð45Þ

where F,p,N p is the critical value of a Snedecor F distribution with p and N p degrees of freedom at the significance level . 1.12.4.7

Confidence Interval for the Response

According to the variance of the estimated mean response at point x (Equation (24)), a 100(1 )% confidence interval is qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi yˆðxÞ – t =2;N – p x9ðX9XÞ – 1 xs 2 ; yˆðxÞ þ t =2;N – p x9ðX9XÞ – 1 xs 2

1.12.4.8

ð46Þ

Prediction of New Response Observations

If the fitted model is used to predict future observations on the response y corresponding to variables x, then the estimated response is yˆðxÞ ¼ x9b, and the 100(1 )% confidence interval is qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi

yˆðxÞ – t =2;N – p 1 þ x9ðX9XÞ – 1 x s 2 ; yˆðxÞ þ t =2;N – p 1 þ x9ðX9XÞ – 1 x s 2

ð47Þ

The intervals defined in Equations (42), (46), and (47) and the confidence region in Equation (45) are useful in the phase of selection of the design but also, once s2 is estimated, for the maintenance control of the model.


1.12.4.9

363

Coefficients of Determination, R2 and R2adj

The coefficient of multiple determination is defined by R2 ¼

SSR SSE ¼ 1– SST SST

ð48Þ

It is a measure of the amount of reduction in the variability of the response obtained using the variables X1, X2, . . ., Xp 1, in the model. However, a large value of R2 does not necessarily imply that the regression model is a good one. Thus it is possible for models that have large values of R2 to yield poor prediction of new observations or poor estimates of the mean response. Because R2 always increases as we add terms to the model, some practitioners prefer to use an adjusted R2 statistic defined as R2adj ¼ 1 –

SSE =ðN – pÞ N –1 ¼ 1– 1 – R2 SST =ðN – 1Þ N –p

ð49Þ

For Example 1 with the second-order model, we have R2 ¼ 0:982 and R2adj ¼ 0:964, which are similar.

1.12.4.10

Residuals Analysis

The residuals contain within them information on why the model might not fit the experimental data. Therefore it is worthwhile to check the behavior of the residuals and allow them to tell us about any peculiarities of the regression fitted that might occur. An enormous amount has been written on the study of residuals and there are several excellent books.24–27 To verify the adequacy of the model to fit the experimental data implies also to check that the residuals are compatible with the hypotheses assumed for ", that is, to be NID with mean zero and variance 2. A check of the normality assumption can be done by means of a normal probability plot of the residuals as in Figure 2 for the absorbance of Example 1. If the residuals are aligned in the plot then the normality assumption is satisfied. Figure 2(a) reveals no apparent problems with the normality of the residuals. There are many inferential procedures to check normality. The usual ones are the 2-test, Shapiro–Wilks test, the z score for skewness, Kolmogorov’s, and Kolmogorov–Smirnof’s tests among others. When they are applied to the residuals of Figure 2(a), they have p-values of 0.73, 0.88, 0.99, 0.41, 0.95, and greater than 0.10, respectively. Since the smallest p-value among the test performed is greater than 0.05, we cannot reject the assumption that residuals come from a normal distribution at the 95% confidence level. Figure 3(a) shows the residuals versus the predicted response also for the absorbance. Visually, the residuals scatter randomly on the display suggesting that the variance of original observations is constant for all values of y. It is usual to work with scaled residuals instead of the ordinary least-squares residuals. One type of scaled residual is the standardized residual, ei ei di ¼ pffiffiffiffi ¼ pffiffiffiffiffiffiffiffiffiffi ; i ¼ 1; 2; . . .; N MSE s2

ð50Þ

These standardized residuals have mean zero and unit variance. It is more reasonable to standardize each residual by using its variance because it is different depending on the location of the corresponding point. If the estimated model (Equation (12)) is applied to all the points of the design, the vector of fitted responses is yˆ ¼ Xb ¼ XðX9XÞ – 1 X9y ¼ Hy

ð51Þ

The matrix H is called the ‘hat’ matrix because it maps the vector of observed values into a vector of fitted values. The residuals may be written in matrix notation as e ¼ y – yˆ ¼ ðI – HÞy and

364


(b) 0.95

0.95

0.9

0.9

0.75

0.75 Probability

Probability

(a)

0.5

0.25

0.5

0.25

0.1

0.1

0.05

0.05 –0.1

–0.05

0 Residuals

0.05

0.1

–3

–2

–1 0 Studentized residuals

1

2

Figure 2 Normal probability plot of residuals of the second-order model fitted with data of Table 2 augmented with those of Table 8: (a) residuals and (b) studentized residuals.

(a)

(b)

0.15

2 1.5

0.1

1 Studentized residuals

Residuals

0.05 0 –0.05 –0.1

0.5 0 –0.5 –1 –1.5

–0.15 –0.2 –0.2

–2 0

0.2

0.4 0.6 0.8 1 Predicted response

1.2

1.4

1.6

–2.5 –0.2

0

0.2

0.4 0.6 0.8 1 Predicted response

1.2

1.4

1.6

Figure 3 Plot of residuals vs. predicted response for absorbance data of Example 1 fitted with a second-order model: (a) residuals and (b) studentized residuals.

CovðeÞ ¼ CovððI – HÞyÞ ¼ ðI – HÞCovðyÞðI – HÞ9. As the ðI – HÞ matrix is symmetric and idempotent, it turns out that the covariance matrix of the residuals is CovðeÞ ¼ ðI – HÞs 2

ð52Þ

From Equation (52), each ei has a different variance given by the corresponding diagonal element of Cov(e), which depends on the model matrix. varðei Þ ¼ ð1 – hii Þs 2 ¼ ð1 – hii ÞMSE

ð53Þ

The studentized residuals, ri, are precisely these variance scaled residuals: ei ei ri ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ; ð1 – hii Þs 2 ð1 – hii ÞMSE

i ¼ 1; 2; . . .; N

ð54Þ


365

The studentized residuals have variance constant regardless of the location of xi when the model proposed is correct. Therefore most of them should lie in the interval [3, 3]. Any studentized residual outside this interval is potentially unusual. Violations of model assumptions are more likely at remote points, and these violations may be hard to detect from inspection of ei or di because their residuals will usually be smaller. Remember that when minimizing the sum of squares, the farthest points from the center have large values of hii; if, to the time, there is a large residual, the ratio that defines ri will detect this situation better. Figures 2(b) and 3(b) show the studentized residuals. It is advisable to analyze both types of residuals to detect possible influential data (large hii and ei). Figure 2(b) shows clearly that there are no problems with the normality of the studentized residuals either. Prediction error sum of squares (PRESS) provides a useful information about residuals. To calculate PRESS we select an experiment, for example the ith, fit the regression model to the remaining N1 experiments, and use this equation to predict the observation yi. Denoting this predicted value yˆðiÞ , we may find the so-called ‘prediction error’ for the point i as eðiÞ ¼ yi – yˆðiÞ . This procedure is repeated for each xi, i ¼ 1,2, . . ., N. Then the PRESS statistic is defined as PRESS ¼

N X

2 eðiÞ

ð55Þ

i¼1

The idea is that if a value eðiÞ is large, it means that the estimated model depends specifically on xi and therefore that point is very influential in the model, that is, an outlier. It can be proved that eðiÞ ¼

ei 1 – hii

ð56Þ

and consequently the prediction error is not independent of the fitting with all the data. It is easy to see that the prediction error eðiÞ is just the ordinary residual weighted according to the diagonal elements of the hat matrix. From this point of view, PRESS is affected by the fitting with all the data. Finally, we note that PRESS can be used to compute an approximate R2 for prediction analogous to Equation (48), which is: R2pred ¼ 1 –

PRESS SST

ð57Þ

PRESS is always greater than SSE as 0 < hii < 1 and thus 1–hii < 1. If the difference is very great, this is due to the existence of a large residual ei that is associated to a large value of hii, that is to say, a very influential point in the regression. The meaning of variance explained in prediction of R2pred as opposed to the one of variance explained in fitting of R2 must be used with precaution, given the relation between eðiÞ and ei. For the response of Example 1, PRESS ¼ 0.433 and R2pred ¼ 0:876. The detection of outlier points, that is to say influential points that modify the regression model, is a central question and several indices have been designed to try to identify them. Like both shown here (studentized residuals and residuals in prediction), all of them depend on the fitting already made. More concretely, they depend on the estimates of the residuals ei and on the residual variance weighted by diverse factors. Therefore, if the regression is affected by the presence of outliers, then the residuals and the variances that are estimated from the fitting are also affected. This produces a masking effect that makes one think that there are not outliers when in fact there are. An efficient alternative to treat this problem is to use a regression method that is little or not at all sensitive to the presence of outliers. Among these robust procedures, they are of special use in RSM, those that have the property of the exact fitting. That is to say, if at least half of the observed results yi in an experimental design follows a multiple linear model, the regression procedure finds this model independent of which other points move away from it. This way, the residuals identify outliers with respect to the proposed model. The least median of squares (LMS) regression has this property. In LMS, the coefficients, b, are estimated as the ones that make minimum the median of squares of the residuals,

Min Mediani ðyi – xi 9bÞ2 b

ð58Þ

366


Once the residuals eLMS of the fitting are computed, they are standardized with a robust estimate of the dispersion, so that we have the residuals dLMS that are the robust version of di. If the absolute value of a residual dLMS is greater than some threshold value (usually 2.5), the corresponding point is considered outlier. An analysis of the advantages of using a robust regression for the diagnosis of outliers, as well as the properties of LMS regression can be seen in the book by Rousseeuw and Leroy27 and in Ortiz et al.28 where its usefulness in chemical analysis is shown. Once the outlier data are detected, the usual least-squares regression model is built with the remaining data.

1.12.5 Experimental Design for Fitting Response Surfaces There is a large amount of experimental designs in the literature. Some of them come from theoretical studies and are consequences of the optimality criteria. Others have been generated to solve concrete problems. In addition, there are several computer packages that provide designs, based on an optimality criterion, just for the need of the user. The important issues associated with computer-generated designs of experiments will be discussed in Chapter 1.14 of this book. However, it is important to revise the properties that should be taken into account when the choice of a response surface design is made. Otherwise, it is possible to arrive at the absurd situation to modifying the problem to use the experimental strategy that the researcher knows. 1.12.5.1

Design of Experiments for First-Order Models

Suppose we wish to fit the first-order model with k variables and p coefficients (p ¼ k þ 1) of Equation (5). Also let us assume that each codified variable lies in the

interval [1,þ1]. For the first-order model and a fixed sample N, if xj P½ – 1; þ1 for j ¼ 1, 2, . . ., k, then var bj =2 for j ¼ 1, 2, . . ., k is minimized if the design is orthogonal and all levels of xj are 1 for j ¼ 1, 2, . . ., k. Consequently, the elements on the diagonal of ðX9XÞ – 1 are minimized (recall Equation (14)) by making the off-diagonal elements of X9X zero and by forcing the diagonal elements of X9X to be as large as possible. The two-level factorial designs and fractions in which the main effects are not aliased with each other (resolution 3) do, in fact, minimize the variances of all coefficients. 3 The design 2made up by the first eight experiments of Table 2 is a 2 factorial design and it is easy to check that var bj = ¼ 1=8, j ¼ 1, 2, 3, the least possible value for a design with N ¼ 8 experiments. The 2k designs do not afford an estimate of MSPE and MSLOF unless some runs are replicated. A common method for this is to augment the design with several replicates at the center. The addition of N0 center points does not alter the orthogonality of the design or influence var(bj), j ¼ 1, 2, . . ., k, but b0 is the grand average of all observations, thus varðb0 Þ=2 ¼ 1=ðN þ N0 Þ. Data of Table 2 illustrate the use of a 23 design augmented with three center points to fit a first-order model. Equation (16) already showed that varðb0 Þ=2 ¼ 1=ð8 þ 3Þ. It is possible the use of fractional designs from a 2k in such a way that they are saturated, that is with N ¼ p. For example, the Hadamard design for a first-order model with three variables (four coefficients) has the following model matrix: 0

1

B B1 B X¼B B1 @ 1

1

1

–1

1

1

–1

–1

–1

–1

1

C 1C C C 1C A –1

ð59Þ

The scaled variance var bj =2 ¼ 1=4, j ¼ 1, 2, 3, is minimal but it is impossible to estimate MSE even including points at the center. It is necessary to add to the given experimental design some points uniformly distributed in the experimental domain to be able to establish the significance of the regression. This observation is valid for any saturated design (N ¼ p). Another orthogonal first-order design is the simplex that is also a saturated design. For two variables it is formed by the vertices of an equilateral triangle, and for three variables that of a regular tetrahedron. Analogous designs may be constructed for any number of factors.


(a)

(b) X1

(c) X1

X2

367

(d) X1

X1

X2

X2

X2

Figure 4 Equiradial designs for two factors: (a) equilateral triangle, (b) square, (c) regular pentagon, and (d) regular hexagon.

Finally, we can mention the equiradial designs for the two variables, which are represented in Figure 4. All the designs described in this section are also rotatable.

1.12.5.2

Design of Experiments for Second-Order Models

1.12.5.2.1

Central composite designs The class of central composite designs was informally introduced in Example 1 (Tables 2 and 10) for fitting a second-order model. This is the design most commonly used for fitting these models and was proposed by Box and Wilson1 and it has been subjected to much attention as much in the theoretical development of its properties as in its practical use. Much of the motivation of the central composite designs (CCD) evolves from its use in sequential experimentation as shown in Example 1. A CCD consists of N experiments distributed as follows:

1. nF experiments of a two-level factorial design. These experiments are the only points that contribute to the estimation of the two-factor interactions. It is known that the complete 2k or fractional 2k r of resolution V are optimal for this task. 2. 2k experiments of a ‘star’ or ‘axial’ design with coordinates ð; 0; . . .; 0Þ; ð0; ; . . .; 0Þ; . . .; ð0; 0; . . .; Þ. The axial points do not contribute to the estimation of interaction terms. 3. nC center points at ð0; 0; . . .; 0Þ. These experiments provide an estimate of MSPE and contribute toward the estimation of quadratic terms. Whatever the number of factors, in a central composite design to fit a second-order model, pffiffiffi they are all at five different levels (–, 1, 0, 1, and ) except for ¼ 1. Figure 5 shows a CCD for ¼ 2 and two factors. The sequential character of the design is evident. The factorial points make up an optimal design for a firstdegree model or a model with two-factor interactions. Center runs clearly provide information about the existence of curvature in the system as shown in Tables 5 and 6 in our example. If curvature is found in the system, the addition of axial points allows for efficient estimation of the pure quadratic terms. The flexibility in the use of central composite designs resides in the selection of (the axial distance) and the nC (number of center points). The election of these two parameters can be very important and have an impact in the optimality of the resulting design. The value of is related with the shape and size of the experimental domain, the CCD is rotatable for values of such that 1=4

¼ nF

ð60Þ

The value of nC much influences the shape and magnitude of the variance function d(x) in the experimental domain, in particular, its uniformity. It is impossible to obtain a constant variance function, but it may vary little and regularly. A necessary condition to obtain uniform precision is that the design is rotatable. In this case, it can pffiffiffiffi ffi 2 be proved29that the number of center points nC needed to obtain the uniformity is nC ¼ nF þ 2 – nF – 2k, 1=2 =ð4ðk þ 2ÞÞ. where ¼ ðk þ 3Þ þ ð9k2 þ 14k – 7Þ A CCD cannot be orthogonal, but it is p possible ffiffiffiffi pffiffiffiffitoffi 2reduce

1=4 the correlation between the estimated coefficients holds. It is not possible to simultaneously achieve to the minimum if the relation ¼ nF N – nF =4 this near-orthogonality and the uniform precision. The reason is that each one depends on a different number of

368


1.5

1

0.5

X2

0

–0.5

–1

–1.5 –1.5

–1

–0.5

Figure 5 Central composite design for two factors with ¼

0 X1

p

0.5

1

1.5

2: Solid points are the experiments of the design.

center points and it is always larger to achieve the near orthogonality than for uniform precision. Therefore, the usual practice is to maintain the uniform precision. For more than four variables, nF is quite large when using a complete factorial design, that is why it is usual to consider fractional designs. In Table 17 for k ¼ 5 and 6, several CCD are shown, some of them fractional with the corresponding generator. This possibility is of great practical interest and other fractions of 2k can be used provided that they are of resolution V. For rotatable designs, that fulfil Equation (60), Table 17 shows the values of nC that guarantee the uniform precision as well as the minimum value to have an acceptable design. Given the present availability of software, it is reasonable to study the joint effect of both parameters in the context of the problem under study. Suppose that in a problem with two variables it is needed to augment a factorial design to fit a model of second order. Figure 6 shows the level curves for the six cases when is 1, 1.414, or 2, and 1 or 5 center points are planned. When ¼ 1 (Figures 6(a) and 6(b)), the experimental domain is a square whose sides are parallel to the axes; for ¼ 1.414 (Figures 6(c) and 6(d)), it is a circle; and if ¼ 2 (Figures 6(e) and 6(f)), it is a square whose vertices are in the axes. Also, they differ remarkably in its surface, which is 4, 6.28, and 8, respectively. Table 17 Summary of the most used CCD up to six factors – the design size is N ¼ nF þ 2k þ nC Number of factors k 2

3 2

4 3

5 4

Factorial design

2

2

2

nF nC,up nC,min p

4 1.414 5 2–3 6

8 1.682 6 3 10

16 2 7 3–4 15

nC,up, optimum number of center points for uniform precision. nC,min minimum number of center points for acceptable quality.

6 51

2 I X 12345 16 2 5 5 21

26–1 I X 123456 32 2.378 9 5 28

(a) 2

(b)

1.5

1.5 0.6 0.70.8

0.8 0.7 0.6

1

0

X2

0.5

0.5

0 0.2

0.4

–0.5

0.4

0.6 0.7 0.8

–1

0.6 0.7 0.8

0.5 0.6 0.7

–1

–2

–2 –2

–1.5

–1

–0.5

0

0.5

1

1.5

–2

2

–1.5

–1

–0.5

X1

(d)

1

1 0.8

0 –0.5

2

1 0.9 0.8 0.7

0.9

0.4

0.2

X2

0.8

0.7

0.7

0.9

0.9

0 0.3

–0.5

–1

0.5

–1 0.70.810.9

10.90.80.7 0.6

0.8 1

1 0.8

–1.5

–1.5

–2

–2 –2

–1.5

–1

–0.5

0

0.5

1

1.5

2

–2

–1.5

–1

–0.5

X1

(e)

1.5

1 0.9 0.8 0.7

0.5

0.7 0.5 0.6 0.6 0.5

1

2

1

1 0.8

0.5

0.5

1.5

0.7 0.9

0.9 0.7

0

X1

2 1.5

X2

0.5 0.6 0.7

0.3

–1.5

–1.5

(c)

0.50.6 0.7

0.4

0.5

0.4

–0.5

0.70.6 0.5

1

0.4

0.5

X2

2

(f)

2 0.7

1.5

0

0.5

1

2

2 1.5 0.8

0.8 0.4

1

1.5

X1

1 0.2

0.5

X2

0.5

X2

0 0.4

0.4 0.6 0.3 0.50.7

0

0.4

0.5

–0.5

–0.5 0.4

–1

–1 0.6 0.5 0.8

–1.5

0.8

0.8

–1.5 –2

–2 –2

–1.5

–1

–0.5

0

X1

0.5

1

1.5

2

–2

–1.5

–1

–0.5

0

0.5

1

1.5

2

X1

Figure 6 Variance function d(x) for a two-factor second-order central composite design for several values of axial distance, , and number of center points, nC: (a) ¼ 1, nC ¼ 1, (b) ¼ 1, nC ¼ 5, (c) ¼ 1.41, nC ¼ 1, (d) ¼ 1.41, nC ¼ 5, (e) ¼ 2, nC ¼ 1, and (f) ¼ 2, nC ¼ 5.

370


Figure 6 shows that in the experimental domain acceptable value for dmax(x) is obtained, namely 1, 1, and 0.8 for ¼ 1, 1.41, and 2, respectively. However, when nC ¼ 1, the surface d(x) presents a maximum near the center point whose value depends on : It is the global maximum and equals 1.0 when ¼ 1.414 (Figure 6(c)); in the other two cases it is a local maximum and equals 0.5. The same happens with the minimum, which is 0.4, 0.5, and 0.4, respectively. In addition, the minimum values are reached at points in different locations and the function d(x) greatly differs in the neighborhood of each one, as can be seen in Figures 6(a), 6(c), and 6(e), respectively. On the contrary, for nC ¼ 5, the minimum is always reached at the center point of the design and the variance function regularly increases without presenting local extrema when going to the boundary of the domain. Moreover, it is precisely in the boundary where the relationship between rotatability and uniformity is best seen, above all for the non-rotatable designs, note that, for ¼ 1, d(x) varies between 0.6 and 1 whereas for ¼ 2, it varies between 0.5 and 0.8. Table 18 collects the values of the optimality criteria for the CCD with two, three, and four factors and three, three, and four center points. 1.12.5.2.2

Uniform shell (Doehlert) designs The uniform shell (Doehlert) designs are generated30 from a k-dimensional simplex (regular solid with k þ 1 vertices), which for two factors, for example, is a regular triangle. By juxtaposing simplex, it is possible to cover the whole k-dimensional space, so that the vertices make up a grid of equally spaced points. The nearest points to the points in the grid are disposed as a hypersphere centered in it. In the space of codified variables, the Doehlert design is made up by the points in the grid situated in one of these hyperspheres. According to its orientation in the space, there are three different groups of matrices31,32 from one standard D1. Like for CCD, it is advisable to include additional center points. Although we will see that they may be inferior to the CCD according to some of the criteria already described, they are still of excellent quality and they have interesting additional properties about displacing and expanding by adding a new factor. Thus, they are designs useful for a sequential experimentation. We will show these properties for a Doehlert design of two factors (D1), which is made up by the vertices of a regular hexagon. Figure 7 shows the variance function with three center points. We can see that the design is rotatable, nearly orthogonal and has uniform variance of prediction unlike the designs for more than two factors. The minimum of d(x) is 0.30 at distance 0.5 from the center and d(0) ¼ 0.33. Table 19 summarizes the values of the optimality criteria for Doehlert designs. By comparing these data to those of Table 18, differences are observed that are to be attributable above all to the loss of orthogonality. In any case, it has enough quality to fit a second-order model. The uniform shell design gives us the possibility of extending or displacing the design quite easily. New hexagonal designs may be built in any of the six directions orthogonal to each side of the initial design. The new design overlaps with the old design and requires only three extra experiments plus nC 1 if nC center points are desired in the new design. A possible displacement is shown in Figure 8. The domain of a Doehlert design can be expanded by adding another factor that was held constant at a known level. The hexagonal design can be extended to three dimensions by adding six experiments because the two-factor design is a subset of the three-factor design and so on. This is shown in Table 20. The table evidences that the number of levels is not the same for all factors; for example, when k ¼ 2, there are five levels for X1 and three levels for X2. This is useful when some of the factors show experimental difficulties to be varied. Table 18 Properties of CCD p

k

nC

N

VIFa (bi, max{bij})

jMj1=p

trðX9XÞ – 1

dmax (X)

Geff

Khuri index

6 10 15

2 3 4

3 3 4

11 17 27

(1.00, 1.09) (1.00, 1.16) (1.00, 1.14)

0.62 0.68 0.75

1.19 1.19 0.96

0.62 0.67 0.58

87.27 87.81 91.84

100 100 100

a

The VIF common to the coefficients of the first-degree terms and the maximum of the remaining terms.


0.9

1

X2

0.9

0.4

0.8 0.6

0.5

371

0

0.9

0.9

0.7 0.5

–0.5

–1 –1

–0.5

0

0.5

1

X1

Figure 7 Variance function d(x) for a two-factor second-order Doehlert design with three center points. The diamonds represent the experimental points.

Table 19 Properties of Doehlert designs p

k

nC

N

VIFa (bi, max{bij})

jMj1=p

trðX9XÞ – 1

dmax (X )

Geff

Khuri index

6 10 15

2 3 4

3 3 4

9 15 24

(1.00, 1.04) (1.00, 1.20) (1.00, 1.28)

0.24 0.13 0.08

4.00 8.17 13.88

0.83 0.75 0.70

80.00 88.89 89.28

100 98.7 96.3

a


X1

X1

X1 X2

X2

X2

Figure 8 Several displacements of a two-factor second-order Doehlert design adding three new experimental points.

The possibility of augmenting the number of factors together with the displacement of the experimental domain by using experimental results already obtained has been used to plan sequential optimizations very efficient in terms of experimental effort.33 1.12.5.2.3

Box–Behnken designs Box and Behnken34 have proposed some designs for a spherical domain whose most specific property is that each factor takes only three levels. The class of designs is based on the construction of balanced incomplete block designs. The design for three variables is formed by three blocks; in each of them, two variables are

372


Table 20 Structure of nested Doehlert designs for two to five factors k

X1

X2

2

0 1 0.5 0.5 0.5 0.5 1

0 0 0.866 0.866 0.866 0.866 0

0.5 0.5 0 0.5 0.5 0

0.289 0.289 0.577 0.289 0.289 0.577

0.816 0.816 0.816 0.816 0.816 0.816

0.5 0.5 0.5 0 0 0.5 0 0

0.289 0.289 0.289 0.577 0 0.289 0.577 0

0.204 0.204 0.204 0.204 0.612 0.204 0.204 0.612

0.791 0.791 0.791 0.791 0.791 0.791 0.791 0.791

0.5 0.5 0.5 0 0 0 0.5 0 0 0

0.289 0.289 0.289 0.577 0 0 0.289 0.577 0 0

0.204 0.204 0.204 0.204 0.612 0 0.204 0.204 0.612 0

0.158 0.158 0.158 0.158 0.158 0.632 0.158 0.158 0.158 0.632

3

............................... 4

0 0 0 0 0 0 0 0 0 0 0 0 0

X5

...................................

.......................

0 0 0 0 0 0 0

X4

......................

...........

X3

........................................

5

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

0.775 0.775 0.775 0.775 0.775 0.775 0.775 0.775 0.775 0.775

combined following a 22 factorial design and the remaining third variable is maintained at level zero. Also several center points are added. The resulting design pffiffiffi is shown in Table 21. Notice that all points of the Box– Behnken design (BBD) lie on a sphere of radius 2. Also this design does not contain any points at the vertices of the cubic region created by the upper and lower limits for each variable. In general, BBDs of k factors are composed of k(k–1)/2 factorial designs 22, taking each pair of factors in turn, and keeping the other factors at the (codified) level zero. The BBDs are nearorthogonal and the VIF are usually low. Their properties are summarized in Table 22 for three, four, and five factors.

1.12.5.2.4

Hybrid and related designs Roquemore35 developed a set of saturated or near-saturated second-order designs, which may be of considerable value when it is important to reduce the number of experiments as much as possible. Hybrid designs exist for three, four, and six variables and the original ones were modified by Franquart36 and Peissik37 using the methods of Chapter 1.14 to obtain D-optimal designs by means of an algorithm of interchange. A list of designs for second-order models can be consulted in Appendix III of the book by Lewis et al.,33 in the one by Myers and Montgomery,16 and in Chapter 15 of the book by Box and Draper.22


373

Table 21 Box–Behnken design for a second-order model and three variables Run

X1

X2

X3

1 2 3 4 5 6 7 8 9 10 11 12 13

1 þ1 1 þ1 1 þ1 1 þ1 0 0 0 0 0

1 1 þ1 þ1 0 0 0 0 1 1 þ1 þ1 0

0 0 0 0 1 1 þ1 þ1 1 þ1 1 þ1 0

Table 22 Properties of Box–Behnken designs P

k

nc

N

VIFa (bi, max{bij})

jMj1=p

trðX9XÞ – 1

dmax (X)

Geff

Khuri index

6 10 15 15

3 4 5 5

3 3 2 6

15 27 42 46

(1.00, 1.01) (1.00, 2.33) (1.00, 1.96) (1.00, 1.20)

0.37 0.42 0.17 0.17

2.27 2.29 4.30 3.55

1.39 0.77 4.09 3.55

47.87 72.44 12.24 13.79

96.97 79.40 98.46 98.46

a


1.12.5.3

Replication, Randomization, and Blocking

Until now, one has assumed implicitly that all the experiments of a design are made under homogeneous conditions. The devices for dealing with unavoidable sources of variability are replication, randomization, and blocking. As we have already shown, by means of experiments replicated in the same conditions, the pure error MSPE may be estimated, which is essential to detect the lack of fit and to apply the statistical methodology. 1.12.5.3.1

Randomization It is the cornerstone underlying the use of statistical methods in experimental design. By randomization, we mean that the allocation of the experimental material and the order in which the individual experiments of the design are to be performed are randomly determined. As is stated in the definition of the model (Equations (1) and (2)) and in Section 1.12.4, the requirement is that the realizations of the random variable " are independent. But in many occasions, systematic variations in the homogeneity of runs linked to time take place, for example aging of reagents, better use of the equipment by the workers, and temperature. This drift causes a dependency among the experimental results. Usually randomization recovers the independence hypothesis. In extreme cases, the linear drift can be modeled making the experiments in a certain order but, in general, it will require a specific treatment.38 By properly randomizing the experimental design, we also assist in ‘averaging out’ the effects of the nonconsidered factors that may be present. 1.12.5.3.2

Blocking A block is a portion of the experimental design that should be more homogeneous than the entire set of experiments. In many RSM situations, the experiment is too big to allow all runs to be made under homogeneous conditions. Therefore, it is important to consider designs that make blocking easy, by assigning the experimental points to blocks to be made so as to minimize the impact on the model coefficients. The block is modeled by means of an additional variable that takes so many values as the number of blocks in which the entire design has been divided. The desirable thing is that this variable is orthogonal to the k of the model, in

374


this case the block effect only affects the independent term. That is, one assumes that the model that relates the variables is the same one in each block and that the block effect is limited to a constant displacement by block in the values of the response. The conditions for orthogonal blocking in second-order designs have been determined16 (p 330 and following sections). In the case of CCD, the design should be divided always maintaining the axial part in a unique block; by modifying the number of center points nC and the value, an orthogonal or near-orthogonal blocking can be obtained. Table 23 shows some CCD that block orthogonally; comparing with Table 17, it is observed that for some designs, the value of is a little different from the necessary one to maintain the rotatability. The user must value which one takes according to whether the rotatability or the orthogonal blocking are high priority. The BBD for k ¼ 4 and 5 are other second-order designs that block orthogonally. Also, orthogonal blocking is possible with equiradial designs. The details of these designs can be consulted in the books by Myers and Montgomery16 and by Box and Draper.22 In other occasions, the research hopes that a block effect exists due to the different behavior of the system. This is a case that frequently takes place when optimizing the instrumental conditions of some procedures of chemical analysis. Example 3:39 In the determination of estrone and 17--ethinylestradiol by gas chromatography–mass spectrometry, the optimization of the derivatization reaction has been carried out with a Doehlert design. Three factors will be explored: the volume of the silylation agent, time, and temperature of the reaction (X1, X2, and X3 in codified units). The complete design has been replicated twice with a delay of 6 h. The model put forward for each experimental response is y ¼ 0 þ B XB þ 1 X1 þ 2 X2 þ 3 X3 þ 11 X12 þ 22 X22 þ 33 X32 þ 12 X1 X2 þ 13 X1 X3 þ 23 X2 X3 þ "

ð61Þ

The variable XB in Equation (61) models the block effect. The model estimated for the internal standardized peak area of estrone is yˆ ¼ 59:36 – 2:50X B – 1:06X 1 – 4:26X 2 þ 1:19X 3 þ 6:71X 1 2 þ 4:92X 22 þ 1:76X 32 þ 3:23X 1 X 2 – 9:01X 1 X 3 þ 8:36X 2 X 3

and that for 17--ethinylestradiol is yˆ ¼ 32:63 – 1:74X B – 0:52X 1 – 2:02X 2 þ 1:26X 3 þ 0:69X 12 – 1:40X 22 – 0:91X 32 – 0:06X 1 X 2 – 0:39X 1 X 3 þ 0:15X 2 X 3

Both models are significant (p ¼ 0.0005 and 0.0019, respectively) and do not evidence lack of fit (p ¼ 0.85 and 0.76, respectively). The coefficient of determination, R2, is respectively 0.82 and 0.80. Therefore, both models adequately reproduce the experimental data. The block effect is significant (hypothesis test in Equation (43)) because the coefficient of XB is statistically different from zero in the two models (p ¼ 0.032 and 0.002, respectively). Consequently, there is a common model for the derivatization reaction as a function of the experimental variables, which can be expressed by the model of second

Table 23 Design parameters for some CCD blocked designs Number of factors k 2

3 2

4 3

Factorial block

2

2

Number of blocks in factorial portion Number of points in each block nF Number of added center points in each block

1 4 4 3

2 4 8 2

Axial block Number of added center points Total size of design, N Value of for orthogonal blocking

4 3 14 1.414

6 2 20 1.633

5 4

6

2 8 16 2

2 I X 12345 1 16 16 6

26–1 I X 123456 2 16 32 4

8 2 30 2.000

10 1 33 2.000

12 2 54 2.366

2

5–1


375

order, but the analytical signal has a significant change in each experimental session that is modeled by the corresponding coefficient of XB. Recently, Trinca and Gilmour40 have tackled the problem of blocking response surface designs when the block sizes are prespecified to control variations efficiently and the treatment set is chosen independently of the block structure. In their work, an interchange algorithm based on scores defined by Mead to improve a given blocked design is presented.

1.12.6 Analysis of a Quadratic Response Surface 1.12.6.1

Canonical Analysis

Canonical analysis is a method of rewriting a second-degree equation in a form in which it can be more readily understood. Assume that the estimated response is fitted by a second-order model as yˆ ¼ b 0 þ

k X j¼1

bj xj þ

XX

ð62Þ

b ij x i x j

ij

Given the matrices 0 0

1

x1 B C B x2 C B C C x¼B B .. C; B . C @ A xk

0

1

b1 B C B b2 C B C C b¼B B .. C; B . C @ A bk

b11

B B B1 B b B 2 12 B¼B B B .. B . B @1 b1k 2

1 b12 2 b22 .. . 1 b2k 2

1 1 b1k 2 C C 1 C b2k C 2 C C C .. .. C . . C C A bkk

ð63Þ

Equation (62) is written as yˆ ¼ b 0 þ x9b þ x9Bx

ð64Þ

Mathematically, a second-degree polynomial (Equation (64)) has necessarily a unique stationary point, S, that can be a maximum, a minimum, or a saddle point. By translation of the origin to the stationary point and rotation of axes, Equation (64) becomes yˆ ¼ yˆS þ 1 z1 2 þ 2 z2 2 þ þ k zk 2

ð65Þ

which is known as the canonical equation. In Equation (65), yˆS stands for the estimated value at the stationary point S and i, i ¼ 1, . . ., k, are the eigenvalues of the symmetric matrix B. The canonical coordinates, z, of a point x are obtained as z ¼ M9ðx – xS Þ, where M (by columns) is the matrix of normalized eigenvectors of B, and xS are the coordinates of S. Written as in Equation (65), the analysis of the quadratic equation is very simple. If all the coefficients i, i ¼ 1, . . ., k, are positive, the stationary point S is a minimum; if all the coefficients are negative, S is a maximum; and if some coefficients are positive and other negatives, S is a saddle point. We illustrate this analysis with Example 1. The second-order fitted model is as follows: yˆ ¼ 1:579 þ 0:094 x1 þ 0:512 x2 – 0:161x12 – 0:468x22 – 0:061x1 x2

ð66Þ

The contour lines of this surface are shown in Figure 9 and the canonical equation is yˆ ¼ 1:725 – 0:158z1 2 – 0:471z2 2

Therefore, the stationary point S is a maximum as we also observe in Figure 9.

ð67Þ

376


1.5

1.1 1.2

1.3

1.1 1.2

1.3 1.6

1

S

0.5

1.7 1.3 X2

0

1.5 1.4 1.2

−0.5

1

−1

1.1

0.8

0.9

0.6 0.4

0.7 0.5 0.3

0.1 −1.5 −1.5

−1

−0.5

0 X1

0.2

0.5

1

1.5

Figure 9 Level curves of the second-order model fitted for data of Example 1 (S: stationary point).

The transformation z ¼ M9ðx – xS Þ is z1 ¼ 0:995x1 – 0:097x2 – 0:139 z2 ¼ 0:097x1 þ 0:995x2 – 0:550

ð68Þ

and its inverse x1 ¼ 0:995z1 þ 0:097z2 þ 0:192 x2 ¼ – 0:097z1 þ 0:995z2 þ 0:534

ð69Þ

If z1 ¼ z2 ¼ 0 in the system of Equations (69), the coordinates of the stationary point, xS ¼ (0.192, 0.534), are obtained. As the experimental domain is a circle with radius 1.414 and the distance from xS to the center is qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð0:192 – 0Þ2 þð0:534 – 0Þ2 ¼ 0:567, that is less than the radius, the conclusion is that the stationary point is an interior point of the domain. Again, this information is also seen in Figure 9. The estimated response in S, yˆS , is 1.725 (independent term in Equation (67)). The eigenvectors associated to the eigenvalues are (0.995, 0.097)9 and (0.097, 0.995)9, which are the coefficients of x1 and x2, respectively, in Equations (68). 2 is the largest eigenvalue in absolute value; thus, the greatest curvature of the surface round the maximum is in the direction of its associated eigenvector (0.097, 0.995)9. This direction is almost parallel to the X2-axis and in Figure 9 one observes that the absorbance strongly decreases when diminishing (or increasing) the concentration of HCl from stationary point, S. The opposite happens in the direction of 1, which is the smallest in absolute value. Its corresponding eigenvector is (0.995, 0.097)9 that is practically parallel to the X1-axis, that is the volume of hexone. Around the maximum, the absorbance is less sensitive to changes in the volume of hexone than to changes in the molarity of HCl. In the example, the conclusions reached from the canonical analysis are also easily obtained from Figure 9. However, with three or more factors it is impossible to see the level curves in a global manner, whereas the canonical analysis is carried out in the same way and independently of the number of variables.


1.12.6.2

377

Ridge Analysis

The analysis of matrix B in Equation (64) provides information about the shape of the response surface, but further exploratory work may be needed to understand how the response surface changes in an optimal way as values of variables depart from the center of design space. For making this task, the ridge analysis was introduced by Hoerl41,42 and given a more formal analysis by Draper.43 In ridge analysis we maximize (or minimize) the response on hyperspheres centered at the center of the experimental domain. If yˆðxÞ is the fitted value, then a ridge analysis (involving maximization) solves the following optimization problem as function of the radius, r: yˆmax ðxr Þ ¼ max2 yˆ ðxÞ

ð70Þ

x9x¼r

where xr is the value of x that maximizes yˆðxÞ subject to the restriction x9x ¼ r 2 . Similarly, we have yˆmin ðxr Þ ¼ min2 yˆ ðxÞ

ð71Þ

x9x¼r

A typical output of ridge analysis might simply be a set of two-dimensional plots, despite the value of k, the number of variables. One plot is the ‘optimal response plot’, which is a plot of yˆmax ðxr Þ or yˆmin ðxr Þ as a function of r. The other plot, ‘optimal coordinate plot’, is an overlay plot of the coordinates xir ði ¼ 1; . . .; kÞ as a function of r. Example 4 (taken from Rueda et al.44 with permission): In this case there are three design variables, the carrier solution flow rate (U1), the conditioning cell potential Ea (U2), and the working electrode potential EL1 (U3) of the analytical cell in a flow injection system with electrochemical detection for the determination of hydroquinone. The purpose is to optimize the analytical signal by simultaneously maximizing its magnitude (y1, peak height in mA) and minimizing its variability (y2, coefficient of variation in percentage, CV%). Table 24 shows the experimental design and the responses. For each response, a second-order model has been fitted by using a CCD with ¼ 2 and nC ¼ 7. The fitted response surfaces are yˆ1 ¼ 1:51 þ 0:25X1 – 0:05X2 þ 0:06X3 – 0:03X12 þ 0:02X22 þ 0:02X32 þ 0:03X1 X2 þ 0:07X1 X3 – 0:02X2 X3

ð72Þ

yˆ2 ¼ 0:71 – 0:41X1 – 0:13X2 – 0:23X3 þ 0:44X12 – 0:07X22 þ 0:13X32 þ 0:24X1 X2 þ 0:08X1 X3 – 0:14X2 X3

ð73Þ

Table 24 Data of Example 4 Design matrix (codified variables)

Experimental matrix (natural variables) 1

Responses

Run

X1

X2

X3

U1( ) (ml min )

U2(Ea) (mV)

U3(EL1) (mV)

Peak height (A)

CV%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

1 1 1 1 1 1 1 1 2 2 0 0 0 0 0 0 0 0 0 0 0

1 1 1 1 1 1 1 1 0 0 2 2 0 0 0 0 0 0 0 0 0

1 1 1 1 1 1 1 1 0 0 0 0 2 2 0 0 0 0 0 0 0

0.5 1 0.5 1 0.5 1 0.5 1 0.25 1.25 0.75 0.75 0.75 0.75 0.75 0.75 0.75 0.75 0.75 0.75 0.75

150 150 50 50 150 150 50 50 100 100 200 0 100 100 100 100 100 100 100 100 –100

50 50 50 50 300 300 300 300 175 175 175 175 75 425 175 175 175 175 175 175 175

1.39 1.71 1.28 1.73 1.37 1.97 1.20 1.91 0.84 1.81 1.64 1.40 1.36 1.69 1.53 1.48 1.48 1.65 1.51 1.39 1.43

1.95 0.52 1.76 0.58 2.24 0.43 0.79 0.63 3.07 2.06 0.69 0.34 2.05 0.58 0.60 0.70 0.56 0.30 1.09 1.35 0.52

378

Response Surface Methodology yˆ1 (μA)

(a)

(b) 2 X1 X3

2.0 1.7

1

X2 X3

1.4

2

1

1.1

1

0

1

X1

2

yˆ2 (CV%)

(c)

r 2X 2

1

–1 r

2

0

–2

(d) 2

X2

3.2 1

X3

2.4

X1

1.6

2

0.8

1

1

r 2X 3 X2

–1 r

2

1

0

1

2

–2

X1

Figure 10 Ridge analysis. (a) Optimal response plot and (b) optimal coordinate plot for the response y1, peak height, Equation (72); (c) optimal response plot and (d) optimal coordinate plot for the response y2, CV%, Equation (73). Reproduced with permission from Rueda, M. E.; Sarabia, L.A.; Herrero, A.; Ortiz, M. C. Optimisation of a Flow Injection System with Electrochemical Detection using the Desirability Function. Application to the Determination of Hydroquinone in Cosmetics. Anal. Chim. Acta 2003, 479, 173–184.

The representation of the optimum path computed from the ridge analysis of the response surface fitted for the peak height is shown in Figure 10(a). The radius r is indicated in abscissas. In ordinates, the yˆmax ðxr Þ value is shown on the right side of the graph and the yˆmin ðxr Þ value is shown on the left side of the graph. Note that both sides of the abscissas axis have the same sign, as they represent positive distances from the center of the design. It is clear that the maximum peak height (2.05 mA) is reached at the boundary of the experimental domain, at distance 2. Figure 10(b) shows the coordinates for each factor, in codified variables, of the points xr of Figure 10(a). The coordinates for the maximum (distance 2) transformed into natural variables, correspond to a flow rate, a conditioning potential, and a working electrode potential of 1.11 ml min1, 110 mV, and 337 mV, respectively. In addition, Figure 10(b) shows that, close to the maximum or to distances close to 2 from the center of the experimental domain, the peak height is more sensitive to variations in flow rate (X1) and in working electrode potential (X3) than to changes in the conditioning cell potential (X2). To reach the maximum, X1 and X3 must have larger values while the conditioning cell potential must tend toward center values. For small values, close to the minimum peak height, only the flow rate is influential while the analytical potentials lose importance. The ridge analysis of the CV% is shown in Figure 10(c). The CV%, when it is about minimizing (left side of the graph), slightly decreases from the center of the domain to the boundary where it reaches the minimum. Its coordinates, in natural decodified variables, are 0.75 ml min1, 12.5 mV, and 275 mV for U1, U2, and U3, respectively, as can be seen also in Figure 10(d). The minimum is highly sensitive to the variation of the conditioning potential (X2), less so to the change in the analytical cell potential (X3) and practically insensitive to variations in the flow rate (X1).


379

The optimal response plot allows one to see how the mean response changes in an optimal way regardless of the number of factors. Hence, as the number of factors increases, ridge analysis becomes increasingly important and contours plots less interpretable. Variations on the subject have been proposed by considering the variance of the estimated response, or a weighted least-squares regression, or generalized models and also a confidence region for the optimal response plot.45 A updated revision of these contributions can be seen in Lin and Peterson.46

1.12.6.3

Analysis of Multiple Responses

It is frequent that the real problems are characterized by several responses of interest, as it is the case of the quality of the analytical signal in Example 4. Hill and Hunter47 were perhaps the first authors to make reference to multiresponse applications in chemistry and chemical engineering. Excellent reviews48–50 with different viewpoints can be found. A review of RSM techniques applicable to multiresponse experiments can be found in the literature.2,51 The optimization problem in a multiresponse setting is not as well defined as in the single-response case. In particular, when two or more functions are simultaneously considered, their values are multivariately distributed. In this case, the meaning of optimum is unclear, because there is no unique way to order such data. Obviously, the univariate approach of optimizing the fitted responses individually and independently of one another is not recommended. Solutions that are optimal for one function may be far from optimal for other functions. This is the case in Example 4, the vector of variables xr that maximizes y1 (Figure 10(b)) is completely different from vector xr that contains the coordinates of the minimum of y2 (Figure 10(d)); therefore, a compromising solution must be reached. Several methodologies have been developed, and some of them are described next. In the graphical approach to multiresponse optimization, contour plots are generated and then are superimposed to locate one or more regions in the factor space where all the fitted responses attain a certain degree of acceptability. There can be several candidate points from which the experimenter may choose. Note that these plots limit consideration of the factors to only two, although recently an improved graphical technique was deployed using computer-generated contour surfaces with three factors depicted in the same diagram.52 Derringer and Suich53 developed a procedure based on the so-called desirability function. For a vector of several fitted responses, yˆ ðxÞ ¼ ðyˆ 1 ðxÞ; . . .; yˆ s ðxÞÞ, a desirability function, Dðyˆ ðxÞÞ, is a (weighted) geometric mean of s individual desirability functions di ðyˆ i ðxÞÞ; i ¼ 1; . . .; s. Each di ðyˆ i Þ is scaled between 0 and 1, with 0 indicating the unacceptable value of yî and 1 indicating the completely acceptable value. If the response should be maximized, the one-sided desirability function will be used, defined as 8 0 if yî < a > > > < yˆ – as i di ðyî Þ ¼ > b–a > > : 1 if b < yî

if

a < yî < b

ð74Þ

In Equation (74), b is the target value and the response is unacceptable if it is smaller than a; for intermediate values of yî , a monotone increasing function is considered, which can be linear (s ¼ 1), concave (s > 1), or convex (0 < s b are considered unacceptable, the function is 8 1 if yî < a > > > < b – yî s di ðyî Þ ¼ if > b–a > > : 0 if b < yî

a < yî < b

ð75Þ

Figure 11(a) shows a schema of the shape of the family of functions defined in Equation (74) and Figure 11(b) for the one in Equation (75).

380


(a)

(b) 1.0

1

0.6

s=1 s=2

0.4

s=8

0.2

s = 0.2 s = 0.4

0.8

s = 0.2 s = 0.4

Desirability

Desirability

0.8

s=1

0.6 s=2 s=8

0.4 0.2

0.0

0 a

b

a

Response value

b Response value

Figure 11 Family of individual desirability functions corresponding to (a) Equation (74) and (b) Equation (75).

Finally, if the values of a certain response need to be held close to a particular value (target) or to remain between two threshold values (target interval [a1, b1]), the bilateral desirability function may be used. For a target interval, it is defined as follows: 8 0 if yî < a > > > > s > > yî – a 1 > > if a < yî < a 1 > > > > < a1 – a di ðyî Þ ¼ 1 if a1 < yî < b 1 > > > b – yˆ s2 > > i > if b1 < yî < b > > b–b > 1 > > > : 0 if yî > b

ð76Þ

where s1 and s2 can be different. When a target value vt is given, it suffices to take vt ¼ a1 ¼ b1 in Equation (76). Then, the overall desirability function Dðyˆ ðxÞÞ is defined as

1=w Dðyˆ ðxÞÞ ¼ d1w1 d2w2 dsws ; wi 0; i ¼ 1; :::; s and w ¼ w1 þ w2 þ þ ws

ð77Þ

In this way, the desirability functions collect the researcher own priorities (weights, wi) and desires on the fitted response functions (individual desirabilities, di). Because Dðyˆ ðxÞÞ is a geometric mean of the di ðyˆ i Þ’s, it can only be close to 1 if all of the di ðyˆ i Þ values are. Likewise, Dðyˆ ðxÞÞ will be small if any of the di ðyˆ i Þ values are sufficiently close to zero. The overall desirability function is not differentiable, so the optimization process must use a method that is free of derivatives, such as the (Nelder and Mead) simplex method, a genetic algorithm, or a simulated annealing algorithm. Recent contributions, including modifications of the desirability functions so that they are differentiable, can be consulted in Khuri and Valeroso51 and Lin and Peterson.46 In Example 4, linear desirability functions were chosen, defined as 1. In the case of peak height, yˆ1, we find a maximum so Equation (74) is used with a ¼ 1.5 mA and b ¼ 1.8 mA. Between these two values, the desirability function varies linearly (s ¼ 1). 2. For the CV, yˆ2, we find a minimum so Equation (75) is adequate with a ¼ 1.5% and b ¼ 2.5%. As for the peak height, the function varies linearly between these two values (s ¼ 1). For the overall desirability function Dðyˆ ðxÞÞ, equal weights were used, since both responses are considered equally important, that is w1 ¼ w2 ¼ 1 in Equation (77).


381

The overall desirability function, built from the individual functions, gives an optimum point of coordinates (1.25, –0.32, 0.73) in codified variables, that is, 1.06 ml min1, 116 mV, and 266 mV in natural variables. At this point, the peak height is equal to 1.9 mA and the CV% is equal to 0.8%. This solution fulfils all the restrictions imposed on the individual functions and reaches overall and individual desirability values of one. The graphic study is done using the plots in Figure 12, which correspond to the three-dimensional representations of the global function in the space of the variables versus Ea (Figure 12(a)), versus EL1 (Figure 12(b)), and Ea versus EL1 (Figure 12(c)). In each case, the variable not represented remains constant with the value corresponding to the optimum of the global function. Although these representations of the global desirability function are only partial since in each case one of the three factors must remain constant, they confirm that the optimum reached is robust, that is, it is not very sensitive to small accidental changes in the analyzed factors.

(a)

(b)

Desirability

Desirability

100 100

50 50 425 175

0 0

E L1 (mV) 0.25

Ea (mV) –100 –200

0.75

1.25

φ (ml min–1)

1.25

–75

0.75 0.25

φ (ml min–1) (c) Desirability 100

50

50

Ea (mV)

425

–100 –200 –75

175 E L1 (mV)

Figure 12 Three-dimensional representations of the global desirability function in the space of the variables and Ea, plot (a), and EL1, plot (b), and Ea and EL1, plot (c). In each case the variable not represented remains constant with the value corresponding to the optimum of the global function, that is, EL1 ¼ 266 mV in (a), Ea ¼ –116 mV in (b), and ¼ 1.06 ml min1 in (c). Reproduced with permission from Rueda, M. E.; Sarabia, L. A.; Herrero, A.; Ortiz, M. C. Optimisation of a Flow Injection System with Electrochemical Detection using the Desirability Function. Application to the Determination of Hydroquinone in Cosmetics. Anal. Chim. Acta 2003, 479, 173–184.

382


In contrast, the analysis of the sensitivity of this maximum to variations in the experimental conditions cannot be done by means of the ridge analysis or similar techniques (such as analysis of the curvature of Dðyˆ ðxÞÞ around the maximum) because the function Dðyˆ ðxÞÞ is not differentiable. This means that one has to vary each coordinate of the optimum in a radius of 0.1, in codified variables, and calculate the global desirability at the points of this complete factorial design centered on the optimum. In our case, we obtain Dðyˆ ðxÞÞ ¼ 1 in all of them, such that we can affirm that the maximum is insensitive to small variations in the studied experimental variables. The optimization based on the desirability functions is adequate when the responses should comply external specifications, which is frequent in industrial products. However, as the specifications are incorporated a priori it can happen that other possibilities are not detected. The approach based on desirability functions is essentially to convert a multiresponse problem into a singleresponse one. Therefore, there is a crucial difference between single- and multiresponse optimization, which is ignored when using the transformation method. If two (or more) experimental responses are conflicting, we have a set of trade-off solutions where a gain in one response calls for a loss in the other response. The central question is that none of these trade-off solutions is the best for both responses. Thus, in problems with more than one conflicting response, there is no single optimum but a number of solutions exists which are all optimal. Without any other information, no solution from the set of optimal solutions can be said to be better than any other. In a multiresponse optimization problem, once the models have been fitted to the experimental data for several responses, the procedure is a two-step one: (1) to find the multiple trade-off optimal solutions with a wide range of values for responses, and (2) to choose one of the obtained solutions using higher level information. This approach has been recently developed54 by using an evolutionary algorithm to find the Pareto-optimal front of solutions. The algorithm, which is an adaptation of NSGA-II posed by Deb et al.,48 is based on the concept of a nondominated solution. A solution x1 to the problem to optimize several responses is said to dominate another solution x2 if both conditions (1) and (2) hold: 1. The solution x1 is no worse than x2 in all responses yî ðxÞ; i ¼ 1; . . .; s 2. The solution x1 is strictly better than x2 in at least one fitted response function. The dominance relation is transitive, not reflexive, not symmetric, and not antisymmetric. Among a set of solutions, P, the nondominated set of solutions is made for those solutions that are not dominated by any member of the set P. The Pareto-optimal set is the nondominated set of solutions of the entire search space. With the fitted models for the peak height yˆ1 ðxÞ (Equation (72)) and for the CV, yˆ2 ðxÞ (Equation (73)), the procedure to determine the Pareto-optimal front has been applied to Example 4. Figure 13(a) shows the Paretooptimal front of the nondominated solutions. The meaning of this graph is the following: When fixing a value for the peak height, for instance, 2.3 mA, one gets that the least CV achievable is 0.98%. Analogously, if a value of the CV is selected, let us say 0.8%, the maximum expected value for the peak height is 2.2 mA. The Pareto-optimal front shows clearly the conflict between the responses: If an increase in the peak height is desired, it will be necessary to use experimental conditions that make the CV% worse (they make greater) and vice versa. In this case, the Pareto-optimal front is made up of 483 points of the experimental domain, whose coordinates are represented in Figure 13(b). In the Pareto-optimal front in Figure 13(a) we also observe two main tendencies: the pairs of values estimated until the ‘break’ point – observed around (2.1, 0.57) and marked as the discontinuous lines in Figures 13(a)-, and the rest. In this last case, there is also a different behavior: At the beginning, the peak height increases faster than the CV until the point around (2.45, 1.3) from which is the CV that increases faster than the peak height. These two main tendencies are related and clearly observed in the sudden change in the experimental conditions in Figure 13(b) before and after the continuous line. For the first (from left to right) 134 pairs of estimated responses until the ‘break’ point in the graph is X2 ¼ X3 ¼ 2, X1 increases almost linearly from 0.6 to 1.2; then, X2 ¼ 2 for the rest of 349 solutions, whereas X1 is slightly varying between 1.3 and 1.4 while X3 is fast increasing from 1 to 2 levels. Finally, after the solution approximately 400, with X2 still constant, X3 becomes constant again but in the high level (2) and X1 increases. The additional criteria to select optimal experimental conditions are yˆ1 ðxÞ > 1:8 mA for peak height and yˆ2 ðxÞ < 1:5% for the CV. The estimated peak height is above 1.89 mA in all the solutions that make


(a) 1.8

(b)

2 1.5

1.4

1

1.2

Level of factor

Coefficient of variation (%)

1.6

383

1 0.8

0.5 0 –0.5

0.6 –1 0.4 –1.5

0.2 0 1.8

–2 1.9

2

2.1

2.2

2.3

2.4

2.5

Peak height (μA)

2.6

0

100

200

300

400

500

Solution number

Figure 13 Example 4. (a) Pareto-optimal front and (b) the corresponding codified variables values: þ, factor X1(); o, factor X2(Ea); , factor X3(EL1). Reproduced with permission from Ortiz, M. C.; Sarabia, L.; Herrero, A.; Sańchez, M. S. Vectorial Optimization as a Methodogical Alternative to Desirability function. Chemom. Intell. Lab. Syst. 2006, 83, 157–168.

up the Pareto-optimal front and we should suppress the solutions that provide a CV above 1.5%, which are the last 23 ones; and looking at Figure 13(b) we can say that the increase in the flow rate of the carrier solution when the cell potentials are at their respective low and high levels is responsible for increasing the CV. In any case, we have still many experimental conditions that provide desirable values for the responses. By using the desirability function, a solution was found with yˆ1 ðxÞ ¼ 1:9 A and yˆ2 ðxÞ ¼ 0:8%. Figure 13(a) shows that this solution is not optimal: If the peak height is maintained in 1.9, a CV < 0.05% can be obtained; and if CV ¼ 0.8%, it is possible to reach peak height of 2.25 mA. It is clear that the analysis a posteriori obtained with the Pareto-optimal front is of interest and, in some way, complementary to the one corresponding to optimize the desirability function. There are other less spread approaches to the problem of the multiresponse optimization that can be used in concrete problems. An ample revision of them can be consulted in the references already cited by Lin and Peterson,46 Khuri and Valeroso,51 and Myers et al.2

1.12.6.4

Transformation of the Responses

The equality of variances (homoscedasticity) of the residuals is an important hypothesis to give validity to the statistical analysis of the estimates. Usually, the metric used to record the experimental response is considered to be intrinsic of the experimental measure. The effect of simple transformations, like the powers, depends on whether the values of the response are greater or less than 1 or whether they introduce asymmetries. However, the largest effect is on the structure of the residuals because the transformations are applied to the experimental results and, thus, to the corresponding residual. The most important consequence is that the residuals that should satisfy the normality, independence, and homoscedasticity are the transformed ones. A detailed analysis of this question is outside the scope of this chapter and can be consulted in Chapter 8 of Box and Draper.22 However, the main conclusion is that for (positive) data the selection of a transformation for the experimental response Y becomes important when the ratio max Y =min Y is large, say, greater or equal to 10 (Draper and Smith24) or greater than 3 (Myers and Montogomery16). In this case, one should consider the possibility of transforming Y. There are several kinds of transformations. A useful idea is to use powers, Y , and to try to look 0 for the best value of . There is an apparent problem when ¼ 0,

Y ¼ 1 and the transformed response is constant. However, when using transformations W ð Þ ¼ Y – 1 = ; 6¼ 0, the limit W ð Þ !0 ! lnðY Þ

384


holds. Another question is that the transformation from Y to W( ) makes the size of W vary; thus, it would be better to use the alternative form 8 > < Y –1 Y_ – 1 V ¼ > : _ Y lnðY Þ

if 6¼ 0

ð78Þ

if ¼ 0

where the quantity Y_ ¼ ðy1 ; y2 ; . . .; yN Þ1=N is the geometric mean of response values, yi, i ¼ 1, . . ., N. When formula (78) is applied we obtain a vector v ¼ (v1, v2, . . ., vN) and use it to the multilinear model fit by least squares: v ¼ Xb þ e

ð79Þ

instead of the model of Equation (11). More generally, we have to estimate as well as b. We do this by invoking the principle of maximum likelihood under the assumption that " is normally distributed, independent, and homoscedastic. This method is discussed by Box and Cox55 and is applied in the following way: 1. For each , compute V using Equation (78), fit the model of Equation (79) and record lnðS ð ; vÞÞ (or S( , v)), the natural logarithm of the residual sum of squares of the regression. 2. Determine the minimum of the (natural) logarithm of S, 0 ¼ min lnðS ð ; vÞÞ

ð80Þ

If the function S changes a little, it is preferable to minimize S ð ; vÞ above all if graphical representations are used. Obviously, the value of 0 is the same in both cases. 3. Determine a one-sided confidence interval for the minimum value lnðS ð 0 ; vÞÞ ðor S ð 0 ; vÞÞ. For computing the confidence interval, a Student’s t, a normal, or a 2 distribution with different degrees of freedom can be used. A proposal of Draper and Smith,24 (p 262) is at the 100(1 )% confidence level, " lnðS ð 0 ; vÞÞ; lnðS ð 0 ; vÞÞ þ

21;ð1 – Þ

#

N

ð81Þ

4. Let [ 1, 2] denote the interval in the abscissa axis that corresponds to the ordinates in Equation (81), see Figure 14(b). Use any value in this

interval ½ 1 ; 2 as power and transform the experimental response ðy1 ; y2 ; . . .; yN Þ to be y1 ; y2 ; . . .; yN (or to ðlnðy1 Þ; lnðy2 Þ; . . .; lnðyN ÞÞ if ¼ 0). With this new response, fit the surface, validate, and analyze it. One should remember that the power transformations are monotonic (the values of yi are supposed to be positive) so a stationary point of the fitted surface for the transformed response is the same as for the surface corresponding to the original response. With didactic purposes, to show the way the procedure is applied, let us look for the transformation of the response with the data of Example 1 (Tables 2 and 8) although no evidence exists about lack of fit neither about lack of normality of the residuals. Figure 14(a) shows lnðS ð ; vÞÞ for – 2 < < þ2 and in Figure 14(b) the region around the minimum (–4.64 for 0 ¼ 0.46) is enlarged. To apply Equation (81), 21;0:95 ¼ 3:84 and thus 1 ¼ 0.36 and 2 ¼ 0.56. Any power in this interval is appropriate. In our case, we use ¼ 0.5. Once the response is transformed by taking the square root, the corresponding ANOVA shows that the model is significant and that there is no evidence of lack of fit, as can be seen in Table 25. Furthermore, R2 becomes 0.998, R2adj ¼ 0.995, and R2pred ¼ 0.985. The second-order fitted model is as follows: 1=2 ¼ 1:257 þ 0:055X þ 0:326X – 0:065X 2 – 0:289X 2 – 0:053X X yd 1 2 1 2 1 2

The contour lines of this surface are shown in Figure 15, which is similar to Figure 9.

ð82Þ


(a)

385

(b) 10 –2.5 –3

ln(S(λ ,v))

ln(S(λ ,v))

5

–3.5 –4

0

ln(S(1,v)) = –4.29

Approximate

–4.5 95% confidence ln(S(1,v)) = –4.64

interval

–5 –2

–5 –1.5

–1

–0.5

0 λ

0.5

1

1.5

0

2

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 λ

1

Figure 14 Box–Cox transformation for data of Example 1.

Table 25 ANOVA with lack of fit test for second-order model fitted to data of Table 2 augmented with that of Table 8 with square root of response Source of variation

Sum of squares

Degrees of freedom

Mean square

Fcalc

p-value


1.3635 0.0032 0.0029 0.0004 1.3667

5 5 3 2 10

0.2727 0.0006 0.0010 0.0002

423.36

4 the simplex is a regular hyper tetrahedron. In the case of a mixture involving three components, the simplex space, representing all the feasible experimental region, will be depicted as a ternary diagram as shown in Figure 2. In general, for xi, i ¼ 1, 2, . . ., q, the simplex coordinate system is the most commonly employed coordinate system for mixture simplexes. For the three-component mixture, the simplex coordinate system is as displayed in Figure 3; the pure or one-component mixture (100% of component Xi) is located at each vertex. Moving from each vertex of the simplex, the horizontal dotted lines show increasing amounts

of component Xi. Along the edges, is the binary or two-component mixture xi > 0; xj > 0 and xi þ xj ¼ 1 , whereas the component not present in the mixture is at the opposite vertex. Interior points correspond to a three-component mixture (in a three-component system is also called complete or total mixture), where all components are included (xi > 0, i ¼ 1, 2, 3). At the center of the simplex the overall centroid is found, where the three components are present in equal proportion ð1=3:1=3:1=3Þ.Thanks to the literature on the mixture experiment, the reader will soon become familiar with the two-dimensional simplex displayed in Figure 2. This is because even though the mixture system under study involves more than three ingredients, many times, it may be possible to reduce or simplify the mixture system to three primary components.6 Moreover, as for systems with more components it

=1

X

3

1

+X

+X

2

X1

=1

X1(1,0,0)

X2(0,1,0)

X2 + X3 = 1

X3(0,0,1)

Figure 2 The triangular region for mixture with three components.

X1 0.1 0.2

0.9 0.8

0.3 0.4 0.5 0.6 0.7 0.8

0.7 0.6 0.5 0.4 0.3 0.2

0.9 X2

0.1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Figure 3 Simplex coordinate system for a three-component simplex.

X3

Experimental Design for Mixture Studies 395

X1(1,0,0,0)

X1(0.5,0,0,0.5)

50% of X4 X3(0,0,1,0)

X2(0,1,0,0) X4(0,0,0,1)

X2(0,0.5,0,0.5)

X3(0,0,0.5,0.5)

Figure 4 Response surfaces on ternary diagram for a four-component space.

is not possible to represent the mixture space graphically; the only way to analyze mixture experimental data (Figure 4) is via ternary diagrams that represent slices of the experimental region.

1.13.2.2

Mixture with Constraints

1.13.2.2.1

The L- and U-simplexes for constrained mixtures As mentioned earlier, in the study of the functional relationship between a response and a number of mixture variables, specific situations owing to physical, theoretical, and economic considerations may impose additional restrictions besides the classical mixture constraints listed in Equation (1). In the component proportion approach, this situation will be translated by placing some limitations on the component proportions as we are not interested in studying the pure components. Since the experimental domain of interest is not the whole simplex space, it is necessary to specify some constraint on the component proportions to define it well. In fact, as we need some information about this experimental domain of interest, we choose a design consisting of experiences placed just in this subregion. In practice, we can introduce some lower limits (or bound), some upper limits, or some lower and upper limits on the proportions of components of interest. However, we may have two possible types of restriction:

1. The individual (or single-component) constraints imposed on one or more mixture components in the form 0 < Li xi 1 0 xi Ui < 1

ð3Þ

0 < Li xi Ui < 1

2. The multicomponent or the ratio constraints of the form xi þ xj k 1 xi r1 r2 xj

ð4Þ

Independently of the type of the constraints placed on the component proportions, as mentioned before, their presence will lead to a reduction in the mixture space both by a simple homothety and a deformation reduction. This will correspond to one or several ways of generating experimental design according to the assumed model. Here, we will introduce only the case of mixtures with some or all the components that must fulfil the constraints. Once some restrictions are imposed on one or all components, further steps are devoted to verify whether the limits given are consistent or not, and thus the real existence and feasibility of the experimental domain of

396

Experimental Design for Mixture Studies

interest.7 When the experimenter places some explicit upper and/or lower constraints, she/he does not have a global idea about the experimental region that is implicitly determined. The additional constraints on the proportion of mixture components might be incompatible with one another and incoherent concerning the sum-to-one restriction. The limits placed on component proportions are coherent whenever each and every lower and upper bounds are possible to attain.8 To be consistent, a set of constraints must always be as follows: 0 < Li xi Ui < 1;

i ¼ 1; 2; . . .; q

and

q X

xi ¼ 1

ð5Þ

i¼1

Generally, it is assumed that the restrictions to be considered are internally consistent, that is, they do not introduce any impossible constraints. For example, let the following lower constraints be placed on a threecomponent system: 0:45 x1 1 ð6Þ

0:40 x2 1 0:30 x3 1

Though each constraint is attainable, from Figure 5 it is evident that inevitably the constrained experimental region does not exist. The sum of these lower limits (L) exceeds 1, where instead it should always be L¼

q X

Li < 1

ð7Þ

i¼1

The experimental subspace does not exist also if the following set of upper constraints is introduced in mixture experimentation: 0:00 x1 0:20 0:00 x2 0:30

ð8Þ

0:00 x3 0:25

since, in this case, it is the summation constraint of Equation (2) to be violated.

X1

45

%

%

30

X2

X3 40%

Figure 5 A three-component system with inconsistent lower bounds 0.45 x1, 0.40 x2, and 0.30 x3.


As observed above, verifying the existence of the experimental domain when some constraints are placed on a limited number of components (q ¼ 3 or 4) in the mixture is not at all complicated, it can be a very difficult task for q > 4. For example, in the production of glass, a set of constraints on the mixture components can be as follows: 0:392 SiO2 0:460

0:136 B2 O3 0:217

0:088 Al2 O3 0:132

0:171 Na2 O 0:193

0:000 P2 O5 0:021

0:027 MgO 0:066

0:000 F

0:000 others 0:089

0:0011

ð9Þ

It is evident that it is not so easy to check whether the constrained region given by the attainment of these bounds exists. To know the experimental region defined by the explicit bounds and, in particular, to verify its existence, there are some rules that can be followed and will be introduced hereafter. First, we will analyze the cases including one explicit (lower or upper) constraint and then the case where both are present. Case (i). Lower bounds for mixture components: 0 < L i x i 1 A useful formula to verify the existence of the design region when only lower bound is present on component proportions is 1.13.2.2.2

RL ¼ 1 –

q X

Li

ð10Þ

i¼1

In fact, when RL < 0, the design region does not exist and the lower constraints are inconsistent; RL ¼ 0, the design region corresponds to a point; RL > 0, the design region exists and the value of RL gives us a measure of the size of this region. The design region determined by consistent lower-bound restriction is called L-simplex and RL is its linear size. However, it must be noted that even when only lower bounds, Li, are considered in mixture experiments some implicit upper limits,Ui , are also determined: 0 Li xi Ui 1

ð11Þ

They are defined from the values of the explicit lower bounds as Ui ¼ Li þ RL

ð12Þ

The L-simplex, if existing, is a real simplex but reduced in size as it is located inside the original simplex. As for an unconstrained simplex the linear size is 1, the size of L-simplex will be indicated relatively by RL. Even in the L-simplex the number of the vertices will correspond to the q component, as occurring for the original X-simplex. However, unlike the latter, each vertex corresponds to a mixture of q pseudocomponents, namely the L-pseudocomponents z9i , and not to a pure component. In the constrained simplex each point inside and on the boundaries is in fact obtained from a mixture of q-pseudocomponents. In general, the pseudocomponents are combinations of the original components that make both the construction of the design to be used and the fitting of models to describe the constrained experimental region easier. The pseudocomponent transformation has, in fact, the same purpose as standardization for independent variables. L-pseudocomponent is given by the following linear transformation (the L-pseudocomponent transformation): z9i ¼

xi – Li 1–L

ð13Þ

398


where Li is the lower limit for the proportion of the component xi, L the sum of all lower bounds Li, and 1 L the experimental range of values for each xi. In fact, xi can range from Li to Li þ (1 L) and z9i from 0 to 1. Example 1: If, for instance, the lower explicit constraints placed on the formulation ingredients are 0:10 x1 1 0:30 x2 1

ð14Þ

0:30 x3 1

then the calculated linear size of the simplex is RL ¼ 1 – ð0:10 þ 0:30 þ 0:30Þ ¼ 0:30

ð15Þ

As RL > 0, the constrained subregion will be an L-simplex determined also by the implied upper constraints calculated as follows: U1 ¼ 0:10 þ 0:30 ¼ 0:40 U2 ¼ 0:30 þ 0:30 ¼ 0:60

ð16Þ

U3 ¼ 0:30 þ 0:30 ¼ 0:60

The complete set of proportion bounds in the formulation will be then the following: 0:10 x1 0:40 ð17Þ

0:30 x2 0:60 0:30 x3 0:60

which define the constrained mixture region depicted in Figure 6. Case (ii). Upper bounds for mixture components: 0 x i U i < 1 To check on the existence of an experimental region constrained only by upper bounds, the following measure is defined as 1.13.2.2.3

RU ¼

q X

Ui – 1

ð18Þ

i¼1

X1 X-simplex

X2

X3 L-simplex

Figure 6 The L-simplex defined by the lower constraints: 0.10 x1, 0.30 x2 , and 0.30 x3.


In fact, when RU < 0, the design region does not exist and the upper constraints are inconsistent; RU ¼ 0, the design region corresponds to a point; RU > 0, the design region exists and RU is the linear size of the so-called U-simplex. The U-simplex, also known as the region of the U-pseudocomponents, is defined using the linear transformation (the U-pseudocomponent transformation) introduced by Croiser8 as zi ¼

Ui – xi U –1

ð19Þ

where Ui is the lower limit for the proportion of the component xi, U the sum of all upper bounds Ui, and U 1 is the experimental range of values for each xi. It must be noted, however, that in the literature on the mixture the conventional transformation is the L-transformation, as there are few authors taking into consideration the orientation of the final simplex. As in case (i), however, even when only upper constraints are placed on component proportions, the design space will be delimited by implied lower constraints, L i , as well. These lower constraints are easily calculated from the explicit upper constraints: L i ¼ Ui – RU

ð20Þ

In the case where only upper bounds are present, the constrained subregion, if existing, has often an asymmetrical shape. However, the U-simplex may represent the experimental region whenever it is completely inside the original simplex (the X-simplex). Example 2: Given the following set of upper constraints, 0:00 x1 0:30 0:00 x2 0:40

ð21Þ

0:00 x3 0:60

using Equation (18), we may calculate the linear size of the U-simplex: RU ¼ ð0:30 þ 0:40 þ 0:60Þ – 1 ¼ 0:30

Then, from Equation (21), we can compute the implied lower constraints L 1 ¼ 0:30 0:30 ¼ 0:00 L 2 ¼ 0:40 0:30 ¼ 0:10 L 3 ¼ 0:60 0:30 ¼ 0:30

defining the L-pseudocomponent simplex. Thus, the experimental domain of interest will be determined by the following complete set of constraints: 0:00 x1 0:30 0:10 x2 0:40

ð22Þ

0:30 x3 0:60

and will correspond to a constrained region given by the U-simplex in Figure 7. Case (iii). Lower and upper bounds for mixture components: 0 < L i x i U i < 1 Furthermore, when placing both lower and upper constraints on component proportions, the constraint consistency must be checked before starting with mixture experimentation. The multiple constraints introduced explicitly by the experimenter will be consistent (and therefore the experimental domain will be coherent) only when each and every component proportion attains the established minimum and maximum values. 1.13.2.2.4

400


X1 X-simplex

U-simplex

X2

X3

Figure 7 The L- and U-simplexes defined by the upper constraints: x1 0.30, x2 0.40, and x3 0.60.

For checking consistency of multiple constraints, we need to compute the linear sizes of L- and U-simplexes (respectively, RL and RU) and, Ri, the range of each component Xi as follows: RL ¼ 1 –

q P

Li

and

RU ¼

i¼1

q P

Ui – 1

i¼1

ð23Þ

Ri ¼ Ui – Li ; i ¼ 1; 2; . . .; q

We can say that there is consistency for

the lower constraints if Ri < RU for each i ¼ 1, 2, . . ., q the upper constraints if Ri < RL for each i ¼ 1, 2, . . ., q.

In the case of inconsistency, however, the unattainable constraints (Li or Ui) could always, in theory, be adjusted by implicit constraint, L i or Ui , defined as L i ¼ Ui – RU and Ui ¼ Li þ RL

ð24Þ

Example 3: Suppose we want to place the following restrictions on mixture component proportions: 0:30 x1 0:80 0:20 x2 0:60

ð25Þ

0:20 x3 0:60

Once both lower and upper constraints are fixed explicitly, as seen above, we have to verify their consistency by first computing all component ranges, R1 ¼ 0:50;

R2 ¼ 0:40;

R3 ¼ 0:40

and then the sizes of L and U-simplexes, respectively, RL ¼ 1 – ð0:30 þ 0:20 þ 0:20Þ ¼ 0:30 RU ¼ ð0:80 þ 0:60 þ 0:60Þ – 1 ¼ 1:00

to point out some possible incompatibilities among the constraints and, in that case, to improve. Here, since R1 > R L ;

R2 > RL ;

R3 > R L


to obtain the consistency of the constraints, we can substitute the unattainable upper bounds by the following implied upper constraints: U1 ¼ L1 þ RL ¼ 0:30 þ 0:30 ¼ 0:60 U2 ¼ L2 þ RL ¼ 0:20 þ 0:30 ¼ 0:50 U3 ¼ L3 þ RL ¼ 0:20 þ 0:30 ¼ 0:50

so that the constraints on component proportions can be rewritten as 0:30 x1 0:60 ð26Þ

0:20 x2 0:50 0:20 x3 0:50

The constrained region of interest defined by corrected bounds is shown in Figure 8. As seen from the examples so far, when placing consistent lower or/and upper constraints on some or all component proportions, the experimental region resulting from the restrictions can still correspond to a regular simplex. But this is not always the case, and the experimenter has to pay attention to that. The shape of the region is a critical aspect since it will influence the design strategy to adopt for the experimentation. Therefore after having identified the consistent (explicit or implied) constraints, we need to know the shape of the design region obtained from the intersection of the L-simplex (defined by the lower bounds), with the U-simplex (defined by the upper constraints) and the original X-simplex (defined by the basic constraints). As it is not so easy to view that when q > 3, we may refer to the computed linear sizes of the L- and U-simplexes, RL and RU, and the component ranges Ri for i ¼ 1, 2, . . ., q, to derive a very helpful rule to know something on this matter. From their comparison, in fact, the following possibilities can be encountered: 1. RL ¼ RU, we can be sure that the experimental domain of interest will never correspond to a simplex as the Land U-simplexes have the same size in this specific case, but as always different orientation; 2. RL < RU, from which we know that the L-simplex is smaller than the U-simplex, but not if the first is inside the latter; 3. RL > RU, from which we know that the U-simplex is smaller than the L-simplex, but not if the first, as in the previous case, is inside the latter.

X1 X-simplex

U-simplex

X2

X3 L-simplex

Figure 8 The constrained region defined by 0.30 x1 0.60 = U 1, 0.20 x2 0.50 = U 2, and 0.20 x3 0.50 = U 3.

402


Knowing whether a simplex is inside another or not is very important since this means that the constrained region is a regular simplex too, as seen before in Examples 1–3. On the contrary, the experimental domain will be more complicated in the shape. Therefore, we need to employ another measure, Rp, defined as Rp ¼ minðRL ; RU Þ

ð27Þ

and to introduce the following rule: If all component ranges, Ri, are equal to Rp, then the constrained region is an L-simplex when RL < RU and a U-simplex when RL > RU. 1.13.2.3

The Polyhedral Mixture Region

As mentioned before a nonsimplex space can identify the feasible experimental region in the presence of only upper limits or when both lower and upper limits are placed on component proportions. Once the constraint consistency is verified, and after having adjusted the bounds found unattainable, from the intersection between U-simplex and the original X-simplex, or between the L- and U-simplexes, it may be possible to derive a constrained region of interest with an irregular shape. For mixture problems involving three components, the shape of the experimental region will be an irregular polygon; with four components, it will be an irregular polyhedron; and with five components, an irregular hyperpolyhedron. The irregular shape of the experimental subregion implies some aspects that undoubtedly may make more onerous the investigation. First, it implies the increase of the number of mixtures due to the fact that there is no longer a direct correspondence between the number of the vertices of the irregularly shaped space and the number of mixture components considered. Vertex number increases very rapidly with augmentation of the number of mixture components involved, and this inevitably enhances the number of candidate mixtures.9 Thus, the need to limit the number of experiences brought to the development of several algorithms for the construction of designs for the exploration of the irregular constrained mixture space. In this section, we present some examples of mixture experiments to analyze some typical situations that the experimenter may face when assigning constraints on component proportions lead to the exploration of an irregular mixture region. 1.13.2.3.1

Case (i). Upper bounds for mixture components: 0 x i U i < 1

Example 4: If the restrictions expressed in terms of the component proportions are 0:00 x1 0:20 0:00 x2 0:60

ð28Þ

0:00 x3 0:70

as previously seen, first we calculate the linear size of U-simplex: RU ¼ ð0:20 þ 0:60 þ 0:70Þ – 1 ¼ 0:50

Then, we identify the implied lower bounds: L 1 ¼ 0:20 – 0:50 ¼ – 0:30 L 2 ¼ 0:60 – 0:50 ¼ 0:10 L 3 ¼ 0:70 – 0:50 ¼ 0:20

Given the existence of the nonnegative constraint for mixture experiments, it is evident that L 1 ¼ 0:30 is not a valid bound and therefore it will be set at 0.00. The complete set of constraints will, therefore, be 0:00 x1 0:20 0:10 x2 0:60 0:20 x3 0:70

ð29Þ


X1 X-simplex

L-simplex

U-simplex X2

X3

Figure 9 The constrained region defined by 0 x1 0.20, L 2 = 0.10 x2 0.60, and L 3 = 0.20 x3 0.70.

In this case, the experimental domain of interest will be obtained by the intersection among the L-simplex (defined by the implied lower bounds), the U-simplex (defined by the explicit upper constraints), and the original X-simplex. The shape of the constrained region, as shown in Figure 9, corresponds to a convex polyhedron with four vertices. 1.13.2.3.2

Case (ii). Lower and upper bounds for mixture components: 0 < L i x i U i < 1

Example 5: If the lower and upper restrictions expressed in terms of the component proportions are 0:10 x1 0:20 0:30 x2 0:40

ð30Þ

0:20 x3 0:70

for checking constraint consistency we calculate the linear size of L- and U-simplexes: RL ¼ 1 – ð0:10 þ 0:30 þ 0:20Þ ¼ 0:40 RU ¼ ð0:20 þ 0:40 þ 0:70Þ – 1 ¼ 0:30

Then the ranges for each component included in the mixture study, R1 ¼ 0:10;

R2 ¼ 0:10;

R3 ¼ 0:50

to be able to compare these values to the linear sizes of the simplexes and to adjust the unattainable constraints as pointed out for component X3. In fact, as R3 > RL the upper constraint U3 (¼0.70) is inconsistent and we need to substitute it with the corresponding implied bound, that is, U3 ¼ L3 þ RL ¼ 0:20 þ 0:40 ¼ 0:60

Therefore, the correct restrictions on mixture components should be as 0:10 x1 0:20 0:30 x2 0:40 0:20 x3 0:60

ð31Þ

404


However, a new check on the bounds R1 ¼ 0:10;

R2 ¼ 0:10;

R3 ¼ 0:40

RL ¼ 1 – ð0:10 þ 0:30 þ 0:20Þ ¼ 0:40 RU ¼ ð0:20 þ 0:40 þ 0:60Þ – 1 ¼ 0:20

points out again a problem of consistency for the same component as this time R3 > RU is the lower constraint L3 (¼0.20) to be inconsistent and adjusted with the implied lower bound: L 3 ¼ 0:60 – 0:20 ¼ 0:40

The constrained experimental region should be defined by the corrected set of constraints: 0:10 x1 0:20 0:30 x2 0:40

ð32Þ

0:40 x3 0:60

Now as R1 ¼ 0:10;

R2 ¼ 0:10;

R3 ¼ 0:20

RL ¼ 1 – ð0:10 þ 0:30 þ 0:40Þ ¼ 0:20 RU ¼ ð0:20 þ 0:40 þ 0:60Þ – 1 ¼ 0:20

all component ranges Ri are equal or inferior to RL and RU. Also in this case the experimental region is a convex polyhedron with four vertices (Figure 10), although completely different in shape from that presented in Figure 9. Example 6: Let us consider the following lower and upper constraints on component proportions: 0:20 x1 0:60 0:20 x2 0:40

ð33Þ

0:30 x3 0:80

X1 X-simplex

U-simplex

L-simplex X2

X3

Figure 10 The constrained region defined by 0.10 x1 0.20, 0.30 x2 0.40, 0.40 x3 0.60.


The component ranges and the linear sizes of the L- and U-simplexes are computed as R1 ¼ 0:40;

R2 ¼ 0:20;

R3 ¼ 0:50

RL ¼ 1 – ð0:20 þ 0:20 þ 0:30Þ ¼ 0:30 RU ¼ ð0:60 þ 0:40 þ 0:80Þ – 1 ¼ 0:80

to check, as always, whether the constraint is consistent or not. Since R1 and R3 are greater than linear size of the L-simplex, we need to replace the inconsistent constraints (here the upper bounds) with their corresponding implied upper constraints, that is, U1 ¼ L1 þ RL ¼ 0:20 þ 0:30 ¼ 0:50 U3 ¼ L3 þ RL ¼ 0:30 þ 0:30 ¼ 0:60

so that the constraints on component proportions can be rewritten as 0:20 x1 0:50 ð34Þ

0:20 x2 0:40 0:30 x3 0:60

The constrained region of interest defined by constraint bounds reported in Equation (30) is shown in Figure 11.

1.13.2.3.3

Determination of the core points of the constrained mixture region Once the set of consistent restrictions on component proportions is defined, it is possible to determine the number of vertices of the constrained experimental region by applying a formula given by Croiser.9 After having computed RL, RU, Rp, and the component ranges Ri for i ¼ 1, 2, . . ., q, we can determine the number of the vertices (N ) of the convex polyhedron representing the experimental domain of interest according to the following equation: N ¼qþ

q q X X ðq – 2r Þ Lðq; r Þ þ ð1 – r Þ Eðq; r Þ r¼1

r¼1

X1 X-simplex

U-simplex

X2

X3 L-simplex

Figure 11 The constrained region defined by 0.20 x 1 0.50, 0.20 x2 0.40, 0.30 x3 0.60.

ð35Þ

406


where L(q, r) is the number of combinations of component ranges, Ri, whose sum is lower than Rp taken r at a time, for r ¼ 1, 2, 3, . . ., q and E(q, r) is the number of combinations of component ranges, Ri, whose sum is equal to Rp taken r at a time, for r ¼ 1, 2, 3, . . ., q. The following four-component example is presented to introduce the step-by-step application of the procedure used for determining the number of extreme vertices, defined as the intersections Pq of the constraint’s planes Xi ¼ Li, Xi ¼ Ui, i ¼ 1, 2, . . ., q, subject to the basic mixture constraint i¼1 xi ¼ 1: Example 7: Suppose that in a four-component mixture experiment, the following bounds on the component proportions are specified: 0:02 x1 0:15 0:05 x2 0:20

ð36Þ

0:10 x3 0:70 0:10 x4 0:70

The component ranges are R1 ¼ 0.13, R2 ¼ 0.15, R3 ¼ 0.60, and R4 ¼ 0.60, and since RL ¼ 0:73; RU ¼ 0:75;

and

Rp ¼ 0:73

the constraint consistency is verified, even if the resulting constrained experimental region is not a simplex as none of the component ranges is equal to Rp. To identify the polyhedron defined by the given constraints, we need to compute the number of vertices according to the Croiser’s formula. Therefore, first we have to determine the functions L(q, r) and E(q, r), respectively. For r ¼ 1; R1 < Rp ð0:13 < 0:73Þ

Lðq; 1Þ ¼ 1

Eðq; 1Þ ¼ 0

R2 < Rp ð0:15 < 0:73Þ

Lðq; 1Þ ¼ 2

Eðq; 1Þ ¼ 0

R3 < Rp ð0:60 < 0:73Þ

Lðq; 1Þ ¼ 3

Eðq; 1Þ ¼ 0

R4 < Rp ð0:60 < 0:73Þ

Lðq; 1Þ ¼ 4

Eðq; 1Þ ¼ 0

For r ¼ 2; R1 þ R2 ¼ 0:13 þ 0:15 < Rp

Lðq; 2Þ ¼ 1

Eðq; 2Þ ¼ 0

R1 þ R3 ¼ 0:13 þ 0:60 ¼ Rp

Lðq; 2Þ ¼ 1

Eðq; 2Þ ¼ 1

R1 þ R4 ¼ 0:13 þ 0:60 ¼ Rp

Lðq; 2Þ ¼ 1

Eðq; 2Þ ¼ 2

R2 þ R3 ¼ 0:15 þ 0:60 > Rp

Lðq; 2Þ ¼ 1

Eðq; 2Þ ¼ 2

R2 þ R4 ¼ 0:15 þ 0:60 > Rp

Lðq; 2Þ ¼ 1

Eðq; 2Þ ¼ 2

R3 þ R4 ¼ 0:60 þ 0:60 > Rp

Lðq; 2Þ ¼ 1

Eðq; 2Þ ¼ 2

For r ¼ 3; R1 þ R2 þ R3 ¼ 0:13 þ 0:15 þ 0:60 > Rp

Lðq; 3Þ ¼ 0

Eðq; 3Þ ¼ 0

R1 þ R2 þ R4 ¼ 0:13 þ 0:15 þ 0:60 > Rp

Lðq; 3Þ ¼ 0

Eðq; 3Þ ¼ 0

R1 þ R3 þ R4 ¼ 0:13 þ 0:60 þ 0:60 > Rp

Lðq; 3Þ ¼ 0

Eðq; 3Þ ¼ 0

R2 þ R3 þ R4 ¼ 0:15 þ 0:60 þ 0:60 > Rp

Lðq; 3Þ ¼ 0

Eðq; 3Þ ¼ 0

For r ¼ 4; R1 þ R2 þ R3 þ R4 ¼ 0:13 þ 0:15 þ 0:60 þ 0:60 > Rp

Lðq; 4Þ ¼ 0

Eðq; 4Þ ¼ 0


We can resume all the counts relating to the combination of component ranges as follows:

R

L(q, r)

(q 2r)

E(q, r)

(1 r)

1 2 3 4

4 1 0 0

2 0 2 4

0 2 0 0

0 1 2 3

According to Equation (35) the number of vertices of the constrained experimental region is N ¼ 4 þ 4ð2Þ þ 1ð0Þ þ 0ð2Þ þ 0ð4Þ þ 0ð0Þ þ 2ð1Þ þ 0ð2Þ þ 0ð3Þ ¼ 4 þ 8 þ 0 – 0 – 0 þ 0 – 2 – 0 – 0 ¼ 10

As the number of component increases, a little change in the bound constraints may lead to an important increment of vertices and, hence, not a negligible modification in the shape and size of the constrained experimental region. When we need to introduce some restrictions on component proportions, it is therefore useful to determine the acceptable tolerance interval for the constraints. In this way, it is possible to modify the constraints, also taking into account how the possible changes affect the constrained design region. Even if finding the minimum number of vertices is not the main objective, by virtue of the flexibility given to the constraints it becomes possible to define a constrained region of interest as simpler as possible.9

1.13.2.4

The Response Surface (or Polynomial) Models for Mixture

The basis for the application of the RSM to the analysis of mixtures has been published by Scheffe´ in 1958.1 His lattice design method for investigating properties of multicomponent systems as a function of composition still represents the fundamentals in this field and till date it is the origin of several applications. For experimentation on mixture, Scheffe´ introduced appropriate polynomial models having a special correspondence to some particular experimental designs, known as the simplex-lattice designs, with a number of points exactly the same as the number of terms in the polynomial. Similar to the RSM, even in mixture design context, the basic assumption is that, in the experimental domain defined by a number of components q, the response of interest can be approximated sufficiently well over the experimental range by a polynomial of the form ¼þ

X i

i xi þ

XX i

ij xi xj þ

XXX

j

i

j

ijk xi xj xk

ð37Þ

k

where indicates the expected value of the measured response and the 9s are unknown parameters to be estimated. However, as in mixture problem the component proportions, xi, always have to respect the fundamental restriction xi 0 ði ¼ 1; 2; . . .; qÞ and

X

xi ¼ 1

i

To simplify the system Scheffe´ proposed to carry out a simple transformation, so as to obtain an unique formula. In the Scheffe´ polynomial model, known as canonical polynomial, terms such as the constant 0, the pure quadratic xi2 , and the pure cubic xi3 are therefore omitted. Expressed in canonical form, the polynomials are characterized by a relatively simple form and are therefore easy to use for predicting the response over the factor space. As Scheffe´ called q the number of components and m the polynomial degree, his mixture models are also known simply as the {q,m} polynomial.

408


According to the degree (or order) m of the polynomial, the Scheffe´ canonical forms for mixture models that express a response, , as a function of composition are: X Linear: ¼ i xi X X Quadratic: ¼ i xi þ ij xi xj i < j X X X X ðFullÞ Cubic: ¼ i xi þ ij xi xj þ ij xi xj ðxi – xj Þ þ ijk xi xj xk i < j < k X X X X ij xi xj þ ij xi xj ðxi – xj Þ þ ij xi xj ðxi – xj Þ2 ðFullÞ Quartic: ¼ i xi þ X X X X þ iijk xi2 xj xk þ ijjk xi xj2 xk þ ijkk xi xi xk2 þ ijkl xi xj xk xl i < j < k < l

ð38Þ ð39Þ ð40Þ ð41Þ

The interpretation of the terms on the right-hand side of the model equations, the polynomial coefficients, is the following: i xi response due to the pure components, where i is the linear coefficient; ij xi xj response due to synergism of the binary mixtures, where ij is the quadratic coefficient of the binary synergism; ij xi xj ðxi xj Þ response due to synergism of the binary mixtures, where ij is the cubic coefficient of the binary synergism; iik xi xj xk response due to synergism of the ternary mixtures for components i, j, and k, where iik is the cubic coefficient of the ternary synergism; ij xi xj ðxi – xj Þ2 response due to synergism of the binary mixtures, where ij is the quartic coefficient of Pthe binary synergism; ijkl xi xj xk xl response due to synergism of the quaternary mixtures for components i, j, k, and l, where iikl is the quartic coefficient of the quaternary synergism. In general, if the terms in a first-order model adequately describe the response, we can say that the mixture components blend linearly and the response surface is a plane. When a second-order model is required as the first-degree model is not adequate, the quadratic coefficients are the quadratic deviation of the response surface from a plane. Cubic and higher order terms can be considered as response deviations from the first- and secondorder approximation. To determine the most important model coefficients, we have to compare them; however, we have to consider that their importance depends in part on the order of the term it is associated with as shown below: Term

Maximum value of the component

Relation between coefficient and maximum effect

ixi ijxixj iikxixjxk

xj ¼ 1 xixj ¼ 1/2 1/2 ¼ 1/4 xixjxk ¼ 1/3 1/3 1/3 ¼ 1/27

ixi ¼ 1 I ijxixj ¼ (1/4)ij iikxixjxk ¼ (1/27)iik

Higher values for quadratic and cubic coefficients do not necessarily mean large effects. A size of a cubic coefficient must be approximately 27/4 ¼ 6.75 times the size of the quadratic coefficient to represent effects of equal magnitude. Moreover, remember that because of the constraint that the sum of all components must be 1.0, quadratic and cubic terms cannot be interpreted as interactions but as nonlinear blending relationships (the so-called synergisms or antagonisms). It should be mentioned that even if the Scheffe´ polynomials have been extended to include the quartic by Gorman and Hinman, in practice the linear and, in particular, the quadratic models are the most used to describe the response over the region of interest. Generally, if the experimental domain of interest is small, the linear model may approximate the response under study, whereas if it is large a quadratic or a special cubic may be needed. Higher-order models are very rarely employed because of the large number of coefficients.10–14

Experimental Design for Mixture Studies 409 Table 1 Number of points in the {q, m} simplex-lattice design according to the mixture component number and the polynomial order for 3 q 8 and 1 m 4 m q

1

2

3

3 4 5 6 7 8

3 4 5 6 7 8

6 10 15 21 28 36

10 20 35 56 84 120

4 15 35 70 126 210 330

The number of coefficients in the canonical polynomial models as a function of the number of mixture components for 3 q 8 and 3 m 4, where the number of levels for each component is m þ 1, is shown in Table 1. The canonical polynomials for some {q, m} values are as follows: f3; 2g polynomial :

ð42Þ ¼ 1 x1 þ 2 x2 þ 3 x3 þ 12 x1 x2 þ 13 x1 x3 þ 23 x2 x3

f3; 3g polynomial :

ð43Þ

¼ 1 x1 þ 2 x2 þ 3 x3 þ 12 x1 x2 þ 13 x1 x3 þ 23 x2 x3 þ 12 x1 x2 ðx1 – x2 Þ þ 13 x1 x3 ðx1 – x3 Þ þ 23 x2 x3 ðx2 – x3 Þ þ 123 x1 x2 x3 f4; 2g polynomial :

ð44Þ

¼ 1 x1 þ 2 x2 þ 3 x3 þ 4 x4 þ 12 x1 x2 þ 13 x1 x3 þ 23 x2 x3 þ 14 x1 x4 þ 24 x2 x4 þ 34 x3 x4 f4; 3g polynomial :

ð45Þ

¼ 1 x1 þ 2 x2 þ 3 x3 þ 4 x4 þ 12 x1 x2 þ 13 x1 x3 þ 23 x2 x3 þ 14 x1 x4 þ 24 x2 x4 þ 34 x3 x4 þ 12 x1 x2 ðx1 – x2 Þ þ 13 x1 x3 ðx1 – x3 Þ þ 23 x2 x3 ðx2 – x3 Þ þ 14 x1 x4 ðx1 – x4 Þ þ 24 x2 x4 ðx2 – x4 Þ þ 34 x3 x4 ðx3 – x4 Þ

1.13.3 Mixture Design for Simplex (-Shaped) Regions 1.13.3.1

The Scheffe´ Simplex-Lattice Designs

The designs introduced by Scheffe´ for mixture experiments, in his already-mentioned famous pioneering paper of 1958, are referred to as simplex lattices since a uniformly spaced distribution of points over all possible mixtures of the components is given. The lattice designs are characterized by their simplicity of construction as the coordinate for each mixture component is determined by the formula 1 2 m Xi ¼ 0; ; ; . . .; m m m

ð46Þ

where i ¼ 1, 2, . . ., q are the components and m the degree of the polynomial. The proportions for each of the q component take the m þ 1 equally spaced values ranging from 0 to 1 over the simplex. This represents a favorable property of such design, since the symmetrical arrangement of points provides an appropriate configuration to perform the polynomial regression analysis. Therefore, the form of the simplex-lattice design mainly depends on the form of the model selected for approximating the blending problem in the region of interest. The total number of points of a simplex lattice is N¼

ðm þ q – 1Þ! m! þ ðq – 1Þ!

ð47Þ

410


(a)

(b)

X1

X1

1

1

4

7

4 8

5

9

2

3

5

X2

13

8

3

10

X2

X3

{q, m} = {3, 2}

(c)

9

7

12

2

X3

6

11

6

{q, m} = {3, 3}

(d)

X1

X1

1

1

5

12

13

6

14

6

12 7

8

11

5

8

X2 2

4 X4 13

7

10

X2 2

14

19

9

10

18 15

9

16 17

11

3

4 X4 20

3

X3

X3

{q, m} = {4, 2}

{q, m} = {4, 3}

Figure 12 Space configuration of some simplex-lattice designs ((d) lattice design and (O) check points).

As the lattice designs can have several combinations of q and m, they can be represented, in braces, by the corresponding couple of values, {q, m}, as seen for the canonical polynomials. They are called as a quadratic, cubic, or quartic lattice according to the number of points included. Figures 12(a), 12(b), 12(c), and 12(d) show the {3, 2}, {3, 3}, {4, 2}, and {4, 3} simplex lattices, respectively, which would consist of observations taken at appropriate points. They are seen to lie on the plane x1 þ x2 þ x3 ¼ 1, whose coordinates are listed in Tables 2–5.

Table 2 Design coordinates for the {3, 2} simplex-lattice design, along with checkpoints Design points

X1

X2

X3

1 2 3 4 5 6

1.000 0.000 0.000 0.500 0.500 0.000

0.000 1.000 0.000 0.500 0.000 0.500

0.000 0.000 1.000 0.000 0.500 0.500

Checkpoints 7 8 9

0.6667 0.1667 0.1667

0.1667 0.6667 0.1667

0.1667 0.1667 0.6667

Experimental Design for Mixture Studies 411 Table 3 Design coordinates for the {3, 3} simplex-lattice design, along with checkpoints Design points

X1

X2

X3

1 2 3 4 5 6 7 8 9 10

1.0000 0.0000 0.0000 0.6667 0.3333 0.6667 0.3333 0.0000 0.3333 0.0000

0.0000 1.0000 0.0000 0.3333 0.6667 0.0000 0.3333 0.6667 0.0000 0.3333

0.0000 0.0000 1.0000 0.0000 0.0000 0.3333 0.3333 0.3333 0.6667 0.6667

Checkpoints 11 12 13

0.6667 0.1667 0.1667

0.1667 0.6667 0.1667

0.1667 0.1667 0.6667

Table 4 Design coordinates for the {4, 2} simplex-lattice design, along with checkpoints Design points

X1

X2

X3

X4

1 2 3 4 5 6 7 8 9 10

1.000 0.000 0.000 0.000 0.500 0.500 0.000 0.500 0.000 0.000

0.000 1.000 0.000 0.000 0.500 0.000 0.500 0.000 0.500 0.000

0.000 0.000 1.000 0.000 0.000 0.500 0.500 0.000 0.000 0.500

0.000 0.000 0.000 1.000 0.000 0.000 0.000 0.500 0.500 0.500

Checkpoints 11 12 13 14

0.625 0.125 0.125 0.125

0.125 0.625 0.125 0.125

0.125 0.125 0.625 0.125

0.125 0.125 0.125 0.625

The mixture approach proposed by Scheffe´ includes some pros and cons, which are considered here. On the one hand, the Scheffe´ simplex-lattice design presents the following advantages:

Designing mixture experimentation is quite simple and the number of experiments is minimal. The number of experimental points is exactly the same as the number of the terms in the associated polynomial model. Therefore, the coefficients may be calculated easily without using a computer program for multiple linear regression. A sequential approach is possible, however, at least till the third-degree model. The validity or lack of fit of the model can be tested by including additional checkpoints.15,16 Such points are not design points and generally their coordinates are the ones presented in Table 6. It must be noted, however, that any mixture potentially interesting can be selected by the experimenter as a checkpoint independently of its composition (e.g., a commercial product can be chosen as a checkpoint). Some examples of locations of checkpoints in the design space are shown in Figure 12. The checkpoints can be included into the design using the same model or possibly a more complex model (a higher order polynomial) (Table 6). The model allows for a response prediction with a minimal variance (only when the model is valid).

412


Table 5 Design coordinates for the {4, 3} simplex-lattice design Design points

X1

X2

X3

X4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1.000 0.000 0.000 0.000 0.667 0.333 0.667 0.333 0.000 0.333 0.000 0.667 0.333 0.000 0.333 0.000 0.000 0.333 0.000 0.000

0.000 1.000 0.000 0.000 0.333 0.667 0.000 0.333 0.667 0.000 0.333 0.000 0.333 0.667 0.000 0.333 0.000 0.000 0.333 0.000

0.000 0.000 1.000 0.000 0.000 0.000 0.333 0.333 0.333 0.667 0.667 0.000 0.000 0.000 0.333 0.333 0.667 0.000 0.000 0.333

0.000 0.000 0.000 1.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.333 0.333 0.333 0.333 0.333 0.333 0.667 0.667 0.667

Table 6 Location of the internal checkpoints for the simplexlattice designs X1

X1

...

Xq

(q þ 1)/2q 1/2q ... ... 1/2q 1/q

1/2q (q þ 1)/2q ... ... 1/2q 1/q

... ... ... ... ... ...

1/2q 1/2q ... ... (q þ 1)/2q 1/q

On the other hand, the disadvantages presented are:

The simplex lattices are very expensive in terms of number of points when the model degree increases. The model coefficients are calculated on the basis of experimental data where only simple mixtures with q9 components (q9 q) are involved (e.g., pure elements and binary mixtures). This fact is particularly evident for the first-degree model, where the performed experiments concern only pure mixtures.

1.13.3.2

The Scheffe´ Simplex-Centroid Designs

To overcome some shortcomings in the simplex-lattice design, in 1963 Scheffe´ proposed a new design for experiment with mixtures, namely the simplex-centroid design.17 This new design enabled to reduce the problem of the sequentiality as well as to fall in number the model coefficients (and hence the number of observations on mixtures). Scheffe´ also proposed an appropriate polynomial function, called the special cubic model, which was formulated to contain only the cross-product terms that could be progressively added. The special equation, which involves the same number of terms as there are points in the associated simplexcentroid design, is ¼

X i

i x i þ

XX i

ij xi xj þ ijk xi xj xk

i<j 2. In the simplex-centroid design, in fact, only mixtures considered are the ones in which the components present are in equal proportions. Moreover, for a given number of q components there is only one simplex-centroid design for m ¼ 1, 2, 3, . . . . In the simplex-centroid design the 2q 1 observations are taken on

q points obtained by making the q possible permutations of (1, 0, 0, . . ., 0), all the pure blends; C2q points corresponding to C2q permutations of (1/2,1/2, 0, . . ., 0), all the binary blends; C3q points corresponding to C3q permutations of (1/3,1/3,1/3, 0, . . ., 0), all the ternary blends; and the point (1/q,1/q, . . ., 1/q) corresponding to the isobarycenter of the vertices of the (q 1)-dimensional hyper polyhedron representing the possible experimental domain, the overall centroid.

The formula for determining the number of points required for the simplex-centroid lattice, for m = 3, is N¼

qðq þ 1Þ q ðq – 1Þðq – 2Þ þ 2 6

ð49Þ

In the simplex-centroid design there are just enough points, in a suitable configuration, to enable the fitting of the special cubic polynomial. Although the number of terms in the special model is fewer than the full cubic model, the number of points of the simplex-centroid design increases rapidly with q, even if saturated. The Scheffe´ special polynomial for q ¼ 3 takes the form ¼ 1 x1 þ 2 x2 þ 3 x3 þ 12 x1 x2 þ 13 x1 x3 þ 23 x2 x3 þ 123 x1 x2 x3

ð50Þ

¼1 x1 þ 2 x2 þ 3 x3 þ 4 x4 þ 12 x1 x2 þ 13 x1 x3 þ 23 x2 x3 þ 14 x1 x4 þ 24 x2 x4 þ 34 x3 x4 þ 123 x1 x2 x3 þ 124 x1 x2 x4 þ 134 x1 x3 x4 þ 234 x2 x3 x4 þ 1234 x1 x2 x3 x4

ð51Þ

and for q ¼ 4,

Unfortunately, the simplex-centroid design still presents some of the disadvantages of the simplex lattice, such as not being a good spacefiller. Most of the observations are collected for mixtures composed of q9 components (q9 q), in correspondence to design points mainly located on the boundary of the domain. However, for a large majority of three-component problems a simplex-centroid design augmented with three interior points, (x1, x2, x3) ¼ (2/3, 1/6, 1/6), (1/6, 2/3, 1/6), and (1/6, 1/6, 2/3), for checking model lack of fit up to the special quartic polynomial ¼

X

i x i þ

X

ij xi xj þ

X

iijk xi2 xj xk þ

X

ijjk xi xj2 xk þ

X

ijkk xi xi xk2

i<j 3:0 11:4 < Disintegration time in min ð3 Þ < 18:9 Mean diameter in mm ð4 Þ > 1260 % of drug release ð5 Þ > 84

Step 2. Here, the assumed model considered for the approximation of the measured responses is the special cubic model: ¼ 1 x1 þ 2 x2 þ 3 x3 þ 4 x4 þ 12 x1 x2 þ 13 x1 x3 þ 23 x2 x3 þ 123 x1 x2 x3

ð60Þ

432


Step 3. The experimental design selected is therefore the simplex-centroid design with checkpoints. Replicates at every design point of the simplex centroid have been included. The checkpoints have been chosen so that they were in the interior of the region but located as far as possible from the original seven points of the design. The 10-point augmented simplex-centroid design is displayed in Figure 21. In Table 13, the design settings in the component proportions used for this mixture experiment involving a total of 17 blends are presented. The experiments were carried out in a random order as shown in the second column of the table labeled ‘rand’. The experimental design chosen, if the model describes the studied responses well, will allow us to predict the value of each response at any point of the experimental domain of interest with an acceptable quality. In Figure 22 the evolution of the prediction variance over the experimental region is shown. It is of interest to note that the maximum prediction variance is always 3:0 0:95 Prob½11:4 < 3 < 18:9 0:95

Being inside the optimal zone for the responses 1, 2 and 3 the experimenter wants to obtain values respecting the constraints with a probability (1). The wideness of the optimal zone is reduced (see Figure 30). Lactose X1

Desirability 0.40

0.20

X1 Lactose

X2 Avicel PH 101 0.00 0

X3 HPMC

X2 Avicel PH 101 Figure 29 Contour plot of desirability function.

X3 HPMC


Desirability

Lactose X1

0.20

X1 Lactose

0.10 X2 Avicel PH 101 0.00

X3 HPMC X2

X3

Avicel PH 101

HPMC

Figure 30 Reliability study. Variation of the desirability function.

1.13.5.4

Sensitivity Study

The optimal composition is known

Component

Proportion

Lactose Avicel PH 101 HPMC

0.186 0.384 0.430

However, actually, to provide these conditions to the experimenter, the acceptable tolerance of utilization of every proportion must also be provided. We want to know, at a point of the experimental domain of interest, the variation of the responses as a consequence of the imposition of little variations around this point in every proportion. Let us imagine that, in our example, the aim is to know if the component proportions could accept a 0.05 variation (it must be remembered that the sum of the proportions is always equal to 1). Lactose ¼ 0:186 0:05 Avicel PH 101 ¼ 0:384 0:05

ð66Þ

HPMC ¼ 0:430 0:05

Starting from the optimal composition, the evolution of the response 1 can be showed graphically (Figure 31), by progressively varying the component 1 proportions following the ax X1, the component 2 proportions following ax X2, and the component 3 proportions following ax X3. For response 1, the tolerance limits for the three components can be considered acceptable, whereas for response 2, the tolerance limit of 0.05 on the component A is not acceptable and hence the variation of response 2 is superior than the statistical limits. At the optimal point, the confidence interval of the calculated response is t;

p

ð1 þ dopt Þs

ð67Þ

442


Figure 31 Graphical evolution of the five studied responses starting from the optimal composition and varying the proportions of the three components.

where is the significance level (here ¼ 0.05 and 0.025%), dopt is the variance function at the optimal point, t, is the Student–Fischer’s t value, is the number of the degrees of freedom (here ¼ 8), and s is the estimate standard deviation of the response

S p t0.05,7 (1 þ dopt) s p t0.025,7 (1 þ dopt) s

Crushing strength

Porosity

Disintegration time

Mean diameter

Drug release

0.298 0.112 0.140

1.110 1.316 1.643

1.784 2.116 2.641

163.9 194.3 242.6

2.69 3.186 3.976


Therefore, more strict limits must be considered for lactose proportions. In particular, the following constraints can be imposed: Lactose ¼ 0:186 0:02 Avicel PH 101 ¼ 0:384 0:05 HPMC ¼ 0:430 0:05

As appears from the graphical representations in Figure 32, these tolerance limits are acceptable.

Figure 32 Graphical evolution of the five studied responses starting from the optimal composition and varying the proportions of the three components, imposing more strict tolerance limits for lactose proportions.

ð68Þ

444


Example 9(a) and (b): To illustrate the design strategy for defining an ellipsoidal region of interest within the simplex, let us consider two practical experiences relating a four-component system in the first case and a five-component system in the second case.29 In the first example, the aim of the investigation was to examine the effects of introducing slight variations in the formulation upon theophylline solubility and hence to test the robustness of the formulation with regard to this parameter. Starting from the overall centroid, corresponding to the maximal value of solubility, an experimental region was defined and explored to get important information on the solubility behavior. The experimental domain was the maximal ellipsoid that can be inscribed in the polyhedral. As a consequence, boundaries were defined implying strict constraints on mixture components, as reported in Table 15. This region can be represented in the mixture space by an ellipsoid having the following equation:

x1 – 0:05 2 x2 – 0:20 2 x3 – 0:30 2 x4 – 0:20 2 þ þ þ 1 0:05 0:10 0:15 0:10

ð69Þ

As explained earlier, to simplify the system, the ellipsoidal can be transformed to a spherical region. The coordinates of the experimental design points for both ellipsoidal and spherical regions – in terms of X1, X2, X3, X4, and Z1, Z2, and Z3, respectively – are shown in Table 16, along with the observed values of theophylline solubility (Yi). The transformation steps of the ellipsoid to the spherical region and the experimental designs are presented in Figure 33. A classical second-degree model and a Scheffe´ {4, 2} polynomial model for mixture were fitted to the experimental data collected at the Doehlert design points. Both resulting response surfaces revealed that the composition variations did not have a statistically significant influence on the solubility. This outcome led to

Table 15 Boundaries of the optimal region within the polyhedron and coordinates of point X0 Component

ai

X0i

bi

Polyethylene glycol (X1) Water (X2) Propylene glycol (X3) Ethanol (X4)

0 0.20 0.10 0.30

0.05 0.30 0.25 0.40

0.10 0.40 0.40 0.50

Table 16 Doehlert design point coordinates, zi with the corresponding mixture component proportions, Xi, along with the measured response value, Y1 Design point

Z1

Z2

Z3

X1

X2

X3

X4

Y1 (mg ml–1)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

1 1 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0 0.5 0 0 0 0 0

0 0 0.866 0.866 0.866 0.866 0.289 0.289 0.289 0.577 0.289 0.577 0 0 0 0

0 0 0 0 0 0 0.816 0.816 0.816 0.816 0.816 0.816 0 0 0 0

0.023 0.077 0.027 0.073 0.046 0.054 0.031 0.069 0.043 0.047 0.057 0.053 0.050 0.050 0.050 0.050

0.327 0.273 0.276 0.324 0.350 0.250 0.289 0.311 0.338 0.288 0.262 0.312 0.300 0.300 0.300 0.300

0.250 0.250 0.296 0.204 0.204 0.296 0.238 0.262 0.262 0.308 0.238 0.192 0.250 0.250 0.250 0.250

0.400 0.400 0.400 0.400 0.400 0.400 0.443 0.357 0.357 0.357 0.443 0.443 0.400 0.400 0.400 0.400

28.26 26.18 28.66 28.35 25.62 29.24 29.20 28.15 27.15 32.16 31.69 27.64 30.56 29.15 27.99 29.06


X1

X2 X2

Ellipsoid

X4

Sphere

z3 11

12

7

4 8

2 4

5 9

13

10

2

12

13

6

5 3

1

z1

11

6 1

8

7

10

3

9

z2

Mixture design

Doehlert design

Figure 33 The transformation steps of the ellipsoidal region to the unit spherical region, along with the corresponding experimental design. Reproduced from Campisi, B.; Chicco, D.; Voinovich, D.; Phan-Tan-Luu, R. Experimental Design for a Pharmaceutical Formulation: Optimisation and Robustness. J. Pharm. Biomed. Anal. 1998, 18, 57–65, with permission from Elsevier.

the conclusion that the solubility fluctuations observed in the region around the maximum were not due to variation of mixture composition and hence the formulation could be considered robust from this point of view. Let us consider another research experience which was conducted with the aim of preparing self-emulsifying pellets with precise granulometric characteristics (a relatively small particle size, Y1, and a high percentage of modal fraction, Y2) in a high shear mixer.30 These pellets comprised a complex mixture, based on five components (q ¼ 5), and the experimenter wanted to evaluate the effects of the components on the abovementioned granulometric characteristics (Yi). From preliminary trials, the quantitative limitations for each component of the mixture were found, thus obtaining an irregular experimental region representing only a small restricted subregion inside the simplex (Table 17).

Table 17 Lower and upper limits of the formulation components Components

Lower constraints (%)

Upper constraints (%)

Lactose (X1) Microcrystalline cellulose (X2) Water (X3) Polysorbate 80 (X4) Mono- and di-glycerides (X5)

16 27 37.5 2 1

24 37 42.5 8 5

446


In order to explore this region, an experimental strategy for mixture that allowed to achieve the optimization of the system with a reduced number of analysis was followed. Also in this case, the experimental domain of interest in the mixture space was represented by an ellipsoid defined by the following equation: x1 – 0:2 2 x2 – 0:32 2 x3 – 0:40 2 x4 – 0:05 2 x5 – 0:03 2 þ þ þ þ 1 0:04 0:05 0:025 0:03 0:02

ð70Þ

To determine the experimental runs to carry out inside this spherical region, a hybrid design, requiring a total of 16 observations, was applied. Five additional experimental checkpoints were added for testing lack of fit. In Table 18 the coordinates of the 16 experimental design points and the 5 experimental checkpoints are listed. The assumed polynomial model was ¼1 X1 þ 2 X2 þ 3 X3 þ 4 X4 þ 5 X5 þ 12 X1 X2 þ 13 X1 X3 þ 23 X2 X3 þ 14 X1 X4 þ 24 X2 X4 þ 34 X3 X4 þ 15 X1 X5 þ 25 X2 X5 þ 35 X3 X5 þ 45 X4 X5

ð71Þ

and the estimates of model coefficients and predicted values for the two response variables are reported in Tables 19 and 20. Response surfaces were drawn based on the fitted equations for the two response variables (Y1 and Y2), using as reference mixture the overall centroid of the experimental region (X1 ¼ 20%, X2 ¼ 32%, X3 ¼ 40%, X4 ¼ 5%, X5 ¼ 3%). In order to obtain a two-dimensional representation of the response surface, X4 and X5 were chosen as constant variables and fixed at 5 and 3%, respectively (Figures 34(a) and 35(a)). The effects were plotted using the model in Equation (71), to analyze the effects of each component of the mixture about the two selected experimental responses (Figures 34(b) and 35(b)).

Table 18 Hybrid design point coordinates Zi with the corresponding mixture component proportions Xi (%), and measured response values Yi Design points

Z1

Z2

Z3

Z4

X1

X2

X3

X4

X5

Y1

Y2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

0.354 0.354 0.354 0.354 0.75 0.75 0.75 0.75 0.707 0.707 0 0 0 0 0 0

0.612 0.612 0.612 0.612 0.433 0.433 0.433 0.433 0 0 0.866 0.866 0 0 0 0

0.612 0.612 0.612 0.612 0.433 0.433 0.433 0.433 0 0 0 0 0.866 0.866 0 0

0.354 0.354 0.354 0.354 0.250 0.250 0.250 0.250 0.707 0.707 0.5 0.5 0.5 0.5 1 0

19.45 22.51 19.12 21.52 18.14 21.50 18.12 20.52 17.13 22.88 17.74 21.06 19.57 21.08 19.55 20.00

31.14 31.58 34.97 30.04 33.43 30.01 33.39 28.47 31.85 29.92 35.06 33.66 31.32 33.69 31.29 32.00

39.79 38.86 38.86 41.39 41.39 38.47 38.47 40.99 40.99 40.12 40.12 38.21 42.04 40.42 39.82 40.00

4.69 3.41 3.41 3.41 3.41 6.39 6.39 6.39 6.39 5.18 5.18 5.18 5.18 2.91 7.44 5.00

4.93 3.63 3.63 3.63 3.63 3.63 3.63 3.63 3.63 1.90 1.90 1.90 1.90 1.90 1.90 3.00

688 967 918 742 842 893 649 946 816 869 672 646 760 512 835 982

90.0 69.5 85.5 60.1 78.3 79.3 93.3 66.3 75.5 60.7 78.9 71.2 59.7 50.8 65.4 86.3

Test pointsa 17 18 19 20 21

0.395 0.395 0 0 0

0.228 0.228 0.456 0 0

0.161 0.161 0.161 0.484 0

0.125 0.125 0.125 0.125 0.500

21.66 19.19 19.80 19.62 19.73

31.43 33.90 31.69 31.40 31.57

39.55 39.55 41.15 39.85 39.89

4.59 4.59 4.59 6.37 4.85

2.76 2.76 2.76 2.76 3.97

966 892 900 956 933

78.2 87.0 80.4 80.0 92.2

a

Checkpoints used for validity of the postulated model (Equation (71)).

Experimental Design for Mixture Studies 447 Table 19 Estimates and statistical significance of the model coefficients (Equation (71)) for the two measured response variables Coefficients

Y1

Significance (%)

Y2

Significance (%)

b1 b2 b3 b4 b5 b12 b13 b23 b14 b24 b34 b15 b25 b35 b45

812 820 6 880 3 681 20 275 2 055 7 326 10 727 10 739 4 828 23 054 32 161 33 464 34 626 22 826

18.0 12.1 6.2 7.7 3.26 16.3 9.9 6.1 5.1 10.0 3.73 3.08 2.75 3.72 4.91

159 24 498 487 407 251 926 798 1 193 975 1 316 980 989 1 201 1 261

18.2 58.0 16.6 11.5 29.8 25.4 15.3 15.9 9.1 9.8 12.7 19.3 17.8 20.5 17.1

( < 0.05).

Table 20 Predicted and experimental values for the two considered experimental responses in the checkpoints Experimental points

Yexp

Ycalca

Yexp Ycalc

du

s

(Yexp Ycal)/s

Significance (%)

Median diameter(Y1) 17 18 19 20 21

966.0 892.0 900.0 956.0 933.0

971.3 900.4 911.8 969.9 928.4

5.3 8.4 11.8 13.9 4.6

0.684 0.701 0.689 0.757 0.658

14.471 14.545 14.491 14.781 14.361

0.364 0.580 0.812 0.941 0.320

72.1 57.1 43.0 36.2 75.3

0.684 0.701 0.689 0.757 0.658

6.328 6.360 6.336 6.463 6.280

0.112 0.172 0.664 0.430 0.253

91.2 86.6 51.7 67.3 80.4

% in modal fraction(Y2) 17 18 19 20 21

78.20 87.00 80.40 80.00 92.20

77.49 88.09 76.19 82.78 90.61

0.709 1.093 4.206 2.778 1.589

a Values calculated by Equation (71). For the validation of the model, experimental points 17–21 have been omitted and considered for estimating the experimental variance.

(a)

(b) 0.610X 1 + 0.090X 2 + 0.003X 3

1 Lactose

Median diameter

A

2 Microcrystalline cellulos 3 Water

1000

4 Polysorbate 80 910

810

900 800 700 600 500

700

5 Mono- and di-glycerides

1 2

3

710

610

4

600 500

510 5

B

800

0.040X 1 + 0.660X 2 + 0.003X 3

C 0.040X 1 + 0.090X 2 + 0.573X 3

–0.04

–0.02

0.00

0.02

0.04

Component

Figure 34 (a) Contour plot and (b) response trace plot for median diameter in mm, Y1, in the restricted region for the mixture component system when X4 ¼ 5% and X5 ¼ 3%.

448


0.630X 1 + 0.080X 2 + –0.007X 3

(a)

(b)

A

1 Lactose

% in modal fraction

2 Microcrystalline cellulos 3 Water

50.0 90.0

58.0

4 Polysorbate 80 5 Mono- and di-glycerides 3

80.0 58.0 66.0 50.0 74.0

70.0

82.0

1

90.0

60.0 5 98.0

50.0 2

B

C

0.010X 1 + 0.700X 2 + –0.007X 3

–0.04

0.010X 1 + 0.080X 2 + 0.613X 3

4 –0.02

0.00

0.02

0.04

Component

Figure 35 (a) Contour plot and (b) response trace plot for the % of modal fraction, Y2, in the restricted region for the mixture component system when X4 ¼ 5% and X5 ¼ 3%.

The response trace plots show the effects of each component when moving away from the reference blend. The parabolic nature of the curves indicates that the estimated value is quite sensitive to the changes of the amount of each component. Particular results are pointed out for lactose: the response Y1 increases when a higher amount of lactose is included in the formulation. This may be explained by the fact that the water dissolves the lactose contained in the formulation, by increasing the agglomeration of the powder and giving a decrease of Y2 with a concentration superior to 20%. On the contrary, when the amounts of microcrystalline cellulose and of mono- and di-glycerides increase in the formulation, Y1 decreases, whereas Y2 increases. Moreover, as in this case, the interest was aimed to the optimization of all the responses at the same time, the use of the desirability function was considered. Here, the target was to carry out a simultaneous optimization of the two properties under study, to obtain pellets having a median diameter >800 mm (Y1) and a modal fraction >80% (Y2). The resulting desirability function is depicted in Figure 36.

(a)

(b) 0.560X 1 + 0.150X 2 + –0.007X 3

Desirability

A

1.10

Pseudo-X1 0.55 Pseudo-X2 0.6 0.80

–0.00

Pseudo-X3

B 0.010X 1 + 0.700X 2 + –0.007X 3

C 0.010X 1 + 0.150X 2 + 0.543X 3

Figure 36 Response surface of global desirability as function of five components when X4 ¼ 5% and X5 ¼ 3%.


1.13.6 Concluding Remarks This article presents some approaches that the experimenter may follow in the design and analysis of experiments with mixtures. In formulation problems, the choice of the experimental strategy to be used should consider the main objective(s) to attain, but without forgetting the available resources and, in particular, the contingent (technical or economic) restraints. Such aspects require careful consideration by the researcher in the plan phase before starting experimentation in order to make the best choice just from the beginning. Besides the classical mixture designs introduced by Scheffe´, the extreme vertices design and the computergenerated design of experiments are also discussed, as in product optimization studies it may frequently occur that the region of feasible blends, restricted by single- or/and multicomponent constraints, does not correspond to the whole simplex region. Moreover, as the interest is focused quite often on a particular blend, the designs around a referential mixture are also considered. Various numerical examples to favor the step-by-step comprehension of the theoretical aspects of the procedures presented as well as some real blending problems taken from the pharmaceutical field in order to illustrate the practicality and the usefulness of some techniques and tools for the data analysis and the optimization of one or more properties of interest are included. The methodological approaches presented are general and can be used easily in the solution of most blending problems. Papers on mixture literature give an evidence of the use of statistical design in various experimental fields, and hence of the broad range of applicability of mixture design. However, it must be kept in mind that some designs and models might be optimal for a given experimental situation and not so particularly appropriate for others as they are inevitably conditioned by previous experience or, simply, by the field of application. Just in virtue of that, experimental design for mixture studies may be still considered a developing research field since the many problems encountered in practice by experimenters had still left unanswered questions, so to recommend topics for future research.

References 1. Scheffe´, H. Experiments with Mixtures. J. R. Stat. Soc. Ser. B 1958, 20 (2), 344–360. 2. Piepel, G. F.; Cornell, J. A. Mixture Experiment Approaches: Examples, Discussion, and Recommendations. J. Qual. Technol. 1994, 26 (3), 177–196. 3. Cornell, J. A. Experiment with Mixture. Designs, Models, and the Analysis of Mixture Data, 2nd ed.; Wiley: New York, 1990. 4. Cornell, J. A. Experiments with Mixtures: A Review. Technometrics 1973, 15 (3), 437–455. 5. Cornell, J. A. Experiments with Mixtures: A Update and Bibliography. Technometrics 1979, 21 (1), 95–106. 6. Cornell, J. A. A Comparison Between Two Ten-point Designs for Studying Three-component Mixture Systems. J. Qual. Technol. 1986, 18 (1), 1–15. 7. Piepel, G. F. Defining Consistent Constraint Regions in Mixture Experiments. Technometrics 1983, 25 (1), 97–101. 8. Croiser, R. B. Mixture Experiments: Geometry and Pseudocomponents. Technometrics 1984, 26 (3), 209–216. 9. Croiser, R. B. Geometry of Constrained Mixtures. Technometrics 1986, 28 (2), 95–102. 10. Gorman, J. W.; Hinman, J. E. Simple Lattice Designs for Multicomponent Systems. Technometrics 1962, 4 (4), 463–487. 11. Murty, J. S.; Das, M. N. Design and Analysis of Experiments with Mixtures. Ann. Math. Stat. 1968, 39 (5), 1517–1539. 12. Snee, R. D. Design and Analysis of Mixture Experiments. J. Qual. Technol. 1971, 3 (4), 159–169. 13. Snee, R. D. Some Aspects of Nonorthogonal Data Analysis. Part I. Developing Prediction Equations. J. Qual. Technol. 1973, 5 (2), 67–79. 14. Snee, R. D. Experimental Designs for Quadratic Models in Constrained Mixture Spaces. Technometrics 1975, 17 (2), 149–159. 15. Snee, R. D. Developing Blending Models for Gasoline and Other Mixtures. Technometrics 1981, 23 (2), 119–130. 16. Shelton, J. T.; Khuri, A. I.; Cornell, J. A. Selecting Check Points for Testing Lack of Fit in Response Surface Models. Technometrics 1983, 25 (4), 357–365. 17. Scheffe´, H. The Simplex-Centroid Design for Experiments with Mixtures. J. R. Stat. Soc. Ser. B 1963, 25 (2), 235–263. 18. McLean, R. A.; Anderson, V. L. Extreme Vertices Design of Mixture Experiments. Technometrics 1966, 8 (3), 447–454. 19. Snee, R. D. Experimental Designs for Mixture Systems with Multicomponent Constraints. Commun. Stat. Theor. Meth. 1979, A8 (4), 303–326. 20. Snee, R. D. Computer-Aided Design of Experiments Some Practical Experiences. J. Qual. Technol. 1985, 17 (4), 222–236. 21. Cook, R. D.; Nachtsheim, C. J. A. Comparison of Algorithms for Constructing Exact D-Optimal Designs. Technometrics 1980, 22 (3), 315–324. 22. Snee, R. D.; Marquardt, D. W. Extreme Vertices Designs for Linear mixture Models. Technometrics 1974, 16 (3), 399–408. 23. Lewis, G. A.; Mathieu, D.; Phan-Tan-Luu, R. Pharmaceutical Experimental Design; Marcel Dekker Inc.: New York, 1999. 24. Box, G. E. P.; Draper, N. R. Empirical Model-Binding and Response Surface; John Wiley and Sons: New York, 1987. 25. Doehlert, D. H. Uniform Shell Designs. Appl. Stat. 1970, 19, 231–239.

450


26. Voinovich, D.; Rupena, P.; Moneghini, M.; Rubessa, F.; Coslovich, S.; Phan-Tan-Luu, R. Experimental Research Methodology Applied to Wet Pelletization in a High-Shear Mixer. STP Pharma Sci. 1993, 3, 130–135. 27. Vining, G. G.; Cornell, J. A.; Myers, R. H. A. Graphical Approach for Evaluating Mixture Designs. Appl. Stat. 1993, 42 (1), 127–138. 28. Derringer, G.; Suich, R. Simultaneous Optimization of Several Response Variables. J. Qual. Technol. 1980, 12, 214–219. 29. Campisi, B.; Chicco, D.; Voinovich, D.; Phan-Tan-Luu, R. Experimental Design for a Pharmaceutical Formulation: Optimisation and Robustness. J. Pharm. Biomed. Anal. 1998, 18, 57–65. 30. Franceschinis, E.; Voinovich, D.; Grassi, M.; Perissutti, B.; Filipovic-Grcic, J.; Martinac, A.; Meriani-Merlo, F. Self-Emulsifying Pellets Prepared by Wet Granulation in High-Shear Mixer: Influence of Formulation Variables and Preliminary Study on the In Vitro Absorption. Int. J. Pharm. 2005, 291, 87–97.



Dario Voinovich is professor of pharmaceutical technology at the University of Trieste, Italy, since 2001. He earned his doctorate in commodity sciences in 1992 from the University of Trieste, becoming researcher at the faculty of pharmacy in the same University in 1990. He attended several chemometrics and methodology of the experimental research schools. He is coauthor of several scientific articles where the experimental design is applied to the preparation and characterization of pharmaceutical, phytochemical, and food formulations.

Barbara Campisi is tenured professor assistant of commodity science at the University of Trieste, Italy, in the Department of Materials and Natural Resources since 2000. In 1996, she earned her doctorate in commodity science from the University of Trieste, Italy, and a master’s degree in quality management at the University Consortium of Quality Engineering of Pisa, Italy, in 2000. Her teaching and research interests focus on the application of techniques and methods for quality management and improvement.

452


Roger Phan-Tan-Luu was Professor at the University Paul Cezanne of Marseille, France. In 1970, he founded the Laboratory of Methodology of Experimental Research in which he developed both the methodological approach and the algorithms that enable construction of efficient designs. He was at the start of the establishment of trainings about the Methodology of Experimental Research in several universities and industries in France and abroad. He is author and co-author of numerous scientific articles dealing with experimental designs. He belongs to the scientific committees of several international congresses. In 1992, Roger Phan-Tan-Luu was awarded Doctor Honoris Causa of the University of Umea˚ (Sweden).

1.14

Nonclassical Experimental Designs

R. Phan-Tan-Luu and M. Sergent, University Paul Cezanne, Marseille, France ª 2009 Elsevier B.V. All rights reserved.

1.14.1 1.14.2 1.14.2.1 1.14.2.2 1.14.3 1.14.3.1 1.14.3.2 1.14.3.3 1.14.3.4 1.14.3.5 1.14.3.5.1 1.14.3.5.2 1.14.3.5.3 1.14.3.5.4 1.14.4 1.14.4.1 1.14.4.1.1 1.14.4.1.2 1.14.4.1.3 1.14.4.2 1.14.4.2.1 1.14.4.2.2 1.14.4.2.3 1.14.5 References

Introduction Methodological Approach The Most Economical Set Information of Sufficient Quality Criteria Criteria of Type I Criteria of Type II Criteria of Type III Criteria of Type IV Examples Study of the effects of solvents Screening of eight factors with nonclassical matrix Study in an experimental domain with constraints Box–Behnken design in a cubic domain Combined Designs Continuous Factors and Discrete Factors Construction of the experimental design Experimentation and exploitation of the experimental results Multicriteria optimization Continuous Factors and Mixture Factors Construction of the experimental design Experimentation and exploitation of the experimental results Multicriteria optimization Final Remark

453 455 455 455 457 457 457 459 459 460 460 466 469 470 471 471 472 476 480 484 485 489 491 494 497

1.14.1 Introduction In science, engineering, and production, processes and/or products can be improved by use of critical information about the operational factors and properties to optimize performance and reliability. However, there are a certain number of circumstances for which these experimental strategies do not apply, in particular:

Nonsymmetrical experimental domain In this case, the use of the classical experimental strategies is very limited, as they can be applied only to symmetrical experimental domains. In some cases, the experimental domain is limited by technological or economic constraints, which give it a nonsymmetrical shape. As certain combinations of factor levels may lead to experiments that are costly, hazardous, or impossible to perform, discontinuities may occur. Some possible experimental domains may be discrete and composed of a finite, sometimes reduced, set of possible experiments. This is the case when ‘qualitative’ factors, which can take a finite number of states (levels), are quantified by several natural variables (for example Quantitative Structure-Activity Relationship (QSAR)). Fixed number of experiments There may be cases where, for technical or economic reasons, the number of experiments is limited or fixed a priori. Very often, this number does not correspond to the number imposed by a standard strategy. 453

454


Linear or nonlinear mathematical model The classical experimental strategies are designed to allow the study of well-defined linear models and do not economically allow the study of a particular linear or nonlinear model that would have been postulated according to existing information. Exploration of a relatively large experimental domain In this case, it is hazardous to postulate a linear model a priori. We will look for the interesting region or regions in this domain in which it will be possible to perform a more detailed study. This is often applied to complex simulation problems that require prohibitive computation times. Complement to an experimental design Unless it has been anticipated, it is extremely unusual that experiments that have already been performed will be re-used. Often it happens that the experimenter wants to modify or complete the model in the light of the information provided by an experimentation, or maybe include new factors in the study. These modifications are not possible with the classical experimental strategies, which are by nature rigid and pre-established, and this all the more when these modifications have not been anticipated. Repairing an experimental design Even if it turns out to be possible to apply a classical experimental strategy in the above-mentioned circumstances, other difficulties may arise during experimentation. We can imagine a number of situations in which the set of experiments initially selected cannot be run. If for example, during experimentation, one or several experiments turn out to be impossible, the experimental results will not be complete and will not allow the objectives to be reached. The missing information will have to be obtained by running one or several additional experiments. Economic aspect The rigidity of the pre-established classical experimental strategies does not make it possible to take into account the following cases:

when all the experiments do not have the same cost, and when some experiments are hazardous or difficult to perform, when the constraints on the factors or on the factor combinations generate a nonsymmetrical experimental domain.

As the classical experimental strategies do not apply in these circumstances, the solution adopted by a great number of researchers consists in modifying the problem so as to solve it by using the classical methods. They adapt the problem to the method they have chosen. To avoid this, we have today at our disposal new powerful and very versatile tools that make it possible to build up, according to the problem, the most economical and informative experimental strategies taking into account the reality of the problem studied. However, and whatever the situation, we must be able to answer the questions that are asked in order to reach the objectives. Experiments must be run, but which ones? Those that will provide the required information.

When we have a set of possible experiments, and provided that the required information is within this set (how can this be verified? ), we will try to find (using which tools? ) a subset containing fewer (or more economical) experiments that will provide information of sufficient quality (how will this quality be judged? ) to reach the objectives. How can we find the answers to these questions? What do we mean by fewer experiments, more economical experiments, sufficient quality? We have at our disposal many quantitative indicators that make it possible to characterize the quality of the information provided by an experimental design for a given objective. These quantitative indicators are called criteria. We all know the criteria A, D, E, V, I, G, U, and so on. (called alphabetical criteria by Box et al.).1 There are a vast number of criteria, and this raises several questions: Are they all comparable? and in this case: Why several criteria? What are the domains and limits of application of these criteria? Are they universal? Mathematical or practical criteria?


455

It seems that the use of these criteria is the preserve of the search for nonstandard experimental strategies, but the following question must be asked: Are the standard experimental strategies always of good quality?

1.14.2 Methodological Approach We have now at our disposal new powerful and easy-to-use tools that enable us to set up, depending on the problem studied, the most informative experimental strategies, taking into account the reality of the problem. We want to know whether, within the set of possible experiments, there is one or several economical subsets that will provide us with sufficient quality to reach our objectives. Let us define what ‘the most economical set’ and ‘information of sufficient quality’ mean.

1.14.2.1

The Most Economical Set

There are several situations.

All the experiments have the same cost. In this case, we try to find, within the set of possible experimental designs providing information of sufficient quality, the one that contains the minimum number of experiments. All the experiments do not have the same cost. In this case, we try to find, within the set of possible experimental designs providing information of sufficient quality, the one whose overall cost is minimum. Some experiments are more difficult to perform or more hazardous than others. In this case, we try to find, within the set of possible experimental designs providing information of sufficient quality, the one which is the least costly or the least hazardous.

1.14.2.2

Information of Sufficient Quality

We run experiments to obtain certain well-defined pieces of information, and from this information we make decisions that contain a certain amount of risk. It is in our interest to keep this risk at a reasonable level and thus we need to obtain information of reasonable quality. How can this term ‘of reasonable quality’ be defined? We will say that we want the quality of the information we are looking for to be sufficient so that all the decisions we will make using this information will be made with a reasonable risk. We will deal with this point in the following paragraphs. Let us go back to the first part of the diagram of the methodological approach:2 Step A: Definition of the objective Step B: Choice of the experimental responses Step C: Choice of the factors and of the experimental domain of interest Depending on the objective, we will choose the experimental strategy that should provide the information that will enable us to reach our objective. It is obvious that the choice of this experimental strategy will depend on the type of objective. The objectives are countless, but nevertheless they can be classified into a few groups: screening, quantitative studies of the factors, response surface methodology, and mixtures. In general, two situations exist:

Experimental designs already built up exist, respecting the constraints, for the objective studied. There is no experimental design already built up, respecting the constraints, for the objective studied.

456


Let us imagine we are in the second situation: We thus need to build up a special experimental design that will take into account all the constraints. Among the well-known methods for setting up nonstandard experimental designs, the most used is based on the exchange algorithm proposed by Mitchell and improved by Fedorov.(3,4) (We mention only the basic improvements; other authors have proposed other improvements that are less fundamental.) We must take the following approach:

A set of NC experiments (candidate points) is proposed. This set contains all the possible experiments, which allows us to take into account the nonfeasible experiments, the constraints linked to the domain, even already existing experiments, and so on. We need to verify the coherence between the information required and the information contained within the candidate points. It is obvious that if with NC candidate points it is not possible to obtain the required information with sufficient quality, it is pointless to try to set up, from these NC points, an experimental design containing fewer experiments of sufficient quality. How can we know whether there is enough information within the NC candidate points? We have to use one or several criteria that will enable us to answer this question. But what criterion or criteria?

When the set of NC candidate points contains the information of sufficient quality, we will try to find whether there is a set of N < NC experimental points that will provide the information of sufficient quality. Let us recall the exchange algorithm approach: * * *

Set the minimum value of N ! NI Set the maximum value of N ! NF (NF NC) In the set of the experimental designs N, we will try to find the experimental design N that is optimal according to a chosen optimization criterion. The set contains a number of different experimental designs given by the relation NC ! ½ðNC – N Þ!N !

*

Repeat this algorithm for different values of N to obtain the set of optimal solutions N (according to the chosen criterion).

We will call criteria of type I the criteria that allow us, for a given value of N, to choose the optimal experimental design N (according to the type I criterion chosen). For each value of N, we have an experimental design N that is optimal according to the type I criterion chosen. We have a total of (NF NI þ 1) optimal experimental matrices (according to the chosen criterion). Which one(s) to choose? In order to determine the experimental designs of acceptable quality, we compare the N optimal experimental designs, taking into account the quality of the required information according to the objective. For this, we need criteria, but these criteria cannot be those of type I, which helped us to find, for N number of experiments, the optimal experimental design N . The reason is that the type I criteria do not take into account the quality of the required information according to the objective. We must use criteria that enable us to choose, within a set of N optimal experimental designs (according to the type I criterion chosen), the experimental design or designs of acceptable quality according to the objective. We will call these criteria criteria of type II.

We have a set of optimal experimental designs N (according to the type I criterion chosen) providing the required information of acceptable quality (according to the type II criterion chosen). We have to choose the final experimental design. This choice will take into account the different properties of experimental designs: number of experiments, cost of each experiment, easiness or dangerousness of experimentation, etc. We will call these criteria criteria of type III.


457

1.14.3 Criteria There are many criteria, but they are all not used for the same purposes. Let us examine them in detail. 1.14.3.1

Criteria of Type I

These are criteria that allow us to obtain, for a value of N, among a number of possible experimental designs, the optimal experimental design N . They are related to the structure of the dispersion design (X9X)1 (Chapter 1.12). Let us mention

the determinant of the dispersion matrix (X9X)1 (criterion D), the trace of the dispersion matrix (X9X)1 (criterion A), the spectrum of the dispersion matrix (X9X)1 (criterion E).

In the case of linear models, the most used criterion is D, because it is the easiest to use. 1.14.3.2

Criteria of Type II

These are criteria that enable us to verify that the chosen experimental design will provide the required information of acceptable quality. The choice of these criteria depends on the type of information required. Let us consider the following situation: We have M ¼ (NF NI þ 1) optimal experimental designs (according to the type I criterion chosen), where NI is the initial number of experiments and NF is the final number of experiments. Which design or designs to choose? According to which criterion or criteria?

We must use the criteria that will allow us to choose, within the set M of optimal experimental designs (according to the type I criterion chosen), the experimental design or designs of acceptable quality. These criteria must take into account the quality of the required information.

Quality of the model coefficients We want to know the estimates of the model coefficients with an acceptable quality. This is what happens during the screening and the quantitative study of the factors. We want to know the estimates of the model coefficients that will allow us to know the estimates of the weights of the factors or the estimates of the main effects or of the interaction effects, etc., with an acceptable quality. We will choose as criteria the following: *

The variance inflation factor (VIF)5 of the coefficient bj. This is the diagonal element of the inverted matrix of the correlation matrix, which can be readily calculated from the elements of the dispersion matrix (X9X)1: fðbj Þ ¼ c jj

X

2 xij – xj ; mean

ð1Þ

i

with c jj being the diagonal element of the dispersion matrix. When all the factors have two levels, then f (bj) 1. A value that is different from 1 indicates that the experimental design is no longer optimal (orthogonality). It is admitted that when the value of a VIF is superior to a limit value (3–6 depending on the authors), the quality of the information provided by the experimental design is not sufficient for the postulated model. When at least one of the factors has a number of states higher than 2, the minimum value of the VIF is not 1 but the above-mentioned rule can be accepted. This indicator characterizes the dependence of the information provided.

458 *


The coefficient of variance,5 cjj (diagonal element of the dispersion matrix). Let us recall that the variance of the estimator bj ¼ cjj2. This characterizes the precision of the information provided.

Qualities of the prediction provided by a validated model We want to know, in any point of the experimental domain of interest, the value of one or several experimental responses. This will allow us to find, if it exists, the domain within which all the experimental responses studied respect the constraints imposed by the specifications, what we call the region of acceptable compromise. The approach known under the name of ‘response surface methodology’ (Chapter 1.12) consists in postulating that the variation of the experimental response studied, within the experimental domain of interest, can be represented by an empirical model. What must the qualities of this model be? *

*

The model must represent quite well the experimental response studied within the experimental domain of interest. If this is verified, this should allow us to foresee, in any point of this experimental domain of interest, the value of the experimental response studied with the same quality as the one we would have obtained if we had performed the experiment in this very point.

Before this postulated empirical model could be used, it has to be validated. Even if the experimental design is chosen according to the type II criterion (allowing us to anticipate, in any point of this experimental domain of interest, the value of the experimental response studied with the same quality as the one we would have obtained if we had run the experiment in this very point), it is not possible to know whether the postulated empirical model actually represents the experimental response studied within all the experimental domain of interest. Knowing the predicted value of the response studied with a good precision is useless if this value does not represent at all the true value we should have. Very often, statistical criteria are used, which we will call criteria of type IV, that allow us to verify that the postulated empirical model actually represents the experimental response studied only at the experimental points and not at all at any point of the experimental domain of interest. We must consider two type II criteria: *

A criterion allowing us, when the empirical model is validated, to foresee, in any point of the experimental domain of interest, the value of the experimental response studied with the same quality as the one we would have obtained if we had run the experiment at this very point (Figure 1). Let us clarify this. If we run the experiment at point A, we will obtain a value of the studied response at point A, yexp,A, with a variance equal to the variance of the experimental error: var(yexp,A) ¼ 2. If, with the postulated model, we calculate the predicted value of the experimental response at this same point A, we will obtain a value ycalc,A with a variance equal to var(ycalc,A) ¼ dA2. It is demonstrated that the multiplicative coefficient, dA, that is called the variance function at point A depends, for a U2

X2

360 sec

Var (ycalc,A) = dAσ 2

A X1

270 sec

180 sec

2500 g

2750 g

Var (y exp,A) = σ 2 Figure 1 Variance of the experimental and calculated response at point A.

3000 g

U1


*

459

given model, only on the elements of the dispersion matrix (X9X)1 and of the model matrix at point A. The value of the variance function dA is independent of the value of the model coefficients. This is actually an a priori criterion. We will choose among the N optimal designs (according to the type I criterion chosen) those whose maximum variance function, dmax, within the experimental domain of interest does have a value inferior or close to 1. A maximal value of 1.3–1.4 seems to be a limit. A criterion allowing us to validate the postulated empirical domain at any point of this experimental domain of interest. The experimental design or designs that we select using the criterion dmax generally contain a number of points only slightly larger than the number of coefficients of the postulated empirical model. It is obvious that in most cases the model represents quite well the results obtained at the experimental points. To use this postulated model, we must verify that it actually represents the experimental response at any point of the experimental domain and not only at the experimental points (E) that are used to calculate the estimates of the model coefficients. We propose that the postulated model should be validated at points of the experimental domain of interest located the farthest possible from the points that are used to calculate the estimates of the model coefficients, and we call those points check points (T). For this, we use a criterion that takes into account the distances between the experimental points (E) and the check points ( T).

1.14.3.3

Criteria of Type III

These are criteria that allow us to choose, among a number of optimal experimental designs N (according to the type I criterion chosen) providing the required information of acceptable quality (according to the type II criterion chosen), either the most economical or the most convenient final experimental design (number of experiments, cost of experimentation, user-friendliness, etc).

1.14.3.4

Criteria of Type IV

The criteria of type I, II, and III are a priori criteria, independent of the experimental results, and they take into account, for a given model, only the distribution of the experimental points. They are the only ones that allow us to construct experimental strategies (experimental designs) of good quality. Once the experiments are run, and provided the model has been validated, we can use different criteria that depend mainly on the experimental results. These criteria are a posteriori criteria. They characterize the extent to which the experimental results are represented by the postulated model; they validate the model at the experimental points and not over the whole of the experimental domain of interest. Let us mention the study of residuals, the analysis of variance,6 etc. We call residual error, or residual, the difference between the experimental value and the value calculated at a given point. These residuals have a mean of 0 and an approximate variance: P MSE ¼

yexp;i – ycalc;i N –p

2

where MSE is the residual mean square. Let us recall however that these residuals are not independent (the sum of the residuals is 0) and that the statistical analyses performed from these residuals are valid only if the number of different experiments (N) is sufficiently higher than the number of model coefficients (p) or, in other words, (N p) is high enough. In this case, we can consider that, provided the postulated model actually represents the experimental response, MCE is an estimate of the response variance with (N p) degrees of freedom. s 2 MCE

Several other statistics are calculated from these residuals (standardized, studentized, etc.) Let us illustrate our methodological approach with some examples.

ð2Þ

460


1.14.3.5

Examples

1.14.3.5.1

Study of the effects of solvents After several theoretical approaches that did not quite succeed in explaining the effects of solvents, many authors have considered some empirical models that assume that the solvent effect can be considered as a ‘sum’ of different independent elementary effects linked to different properties that are characteristic of these solvents. Among all the empirical models proposed, we will use the one proposed by Koppel and Palm.7 They assume that a complete description of all the solute–solvent interactions must include both specific and nonspecific effects of the solvents. They propose a four-parameter model, linking a given property A, two nonspecific characteristics (Y, P), and two specific characteristics (E, B) of the solvent. A ¼ A+ þ yY þ pP þ bB þ Ee

ð3Þ

A value of the property dependent on the solvent A value of this property in a reference solvent Y parameter characterizing the polarization effect in the solvent–solute interactions P parameter characterizing the polarizability effect B parameter characterizing a nucleophilic solvation by the solvent (solvent ¼ Lewis base) E parameter characterizing an electrophilic solvation (solvent ¼ Lewis acid) y, p, b, e corresponding regression coefficients, indicating the sensitivity of property A to the four parameters characterizing the solvents Here is the description of the approach we use for this example:8,9 Step 0: We have a set of 66 solvents for which the four properties have been measured (Table 1).

Table 1 Properties of 66 solvents No.

Solvent

B

E

Y

P

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Benzene Toluene o-Xylene m-Xylene p-Xylene Mesitylene Carbon tetrachloride Chloroform Dichloromethane 1,2-Dichloroethane Fluorobenzene Chlorobenzene Bromobenzene Iodobenzene o-Dichlorobenzene m-Dichlorobenzene Nitrobenzene Acetonitrile Benzonitrile Di-iso-propyl ether Di-n-butyl ether Tetrahydrofurane Dioxane Cineole Diphenyl ether

52 54 59 59 58 62 3.1 39 43 49 34 45 49 53 43 42 63 103 97 134 129 145 128 155 57

2.10 1.30 1.30 0.70 1.20 0.80 0.00 3.28 2.70 3.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 5.20 0.00 0.00 0.00 0.00 4.20 2.60 0.00

0.231 0.239 0.255 0.239 0.229 0.230 0.226 0.359 0.422 0.432 0.374 0.378 0.373 0.353 0.428 0.365 0.479 0.480 0.471 0.329 0.289 0.405 0.226 0.352 0.321

0.228 0.226 0.229 0.226 0.226 0.227 0.215 0.210 0.204 0.210 0.217 0.234 0.244 0.260 0.242 0.240 0.242 0.174 0.235 0.184 0.195 0.198 0.203 0.215 0.250 (Continued )


Table 1

461

(Continued)

No.

Solvent

B

E

Y

P

26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

Anisole Phenetole Acetone Ethylmethylketone Diethylketone Methyl-n-propyl ketone Methy-iso-propyl ketone Methyl-n-butyl ketone Methyl-tert-butyl ketone Di-iso-propyl ketone Cyclopentanone Cyclohexanone Acetophenone Methyl acetate Ethyl acetate Diethyl carbonate Dimethyl formamide Dimethyl acetamide Dimethyl sulfoxide Methanol Ethanol n-Propanol iso-Propanol n-Butanol 2-Butanol iso-Butanol tert-Butanol n-Pentanol Diethylcarbinol tert-Pentanol 1-Hexanol 1-Heptanol 1-octanol Cyclohexanol Benzyl alcohol 1-Phenyl ethanol 2-Phenyl ethanol 3-Phenyl propanol 2-Methoxy ethanol Nitromethane Nitroethane

75 76 123 109 100 111 102 111 97 92 124 132 108 88 89 85 166 178 192 114 117 119 122 119 122 119 125 120 124 127 120 120 121 124 112 115 117 117 119 58 60

1.40 0.80 2.10 2.00 0.20 1.50 1.10 0.50 0.30 1.30 0.20 0.50 0.70 2.10 1.60 4.10 2.60 2.40 3.20 14.90 11.60 10.60 8.70 10.30 7.40 7.40 5.20 9.70 6.20 5.10 9.60 9.60 10.10 7.40 10.50 8.00 9.70 8.90 12.50 5.15 2.85

0.347 0.341 0.466 0.460 0.457 0.453 0.456 0.450 0.443 0.455 0.446 0.455 0.461 0.395 0.385 0.274 0.480 0.480 0.484 0.477 0.470 0.464 0.462 0.458 0.456 0.459 0.442 0.448 0.442 0.381 0.446 0.434 0.427 0.452 0.444 0.418 0.442 0.437 0.457 0.481 0.474

0.232 0.229 0.180 0.189 0.197 0.190 0.190 0.195 0.193 0.195 0.208 0.212 0.237 0.181 0.185 0.190 0.204 0.208 0.221 0.169 0.181 0.190 0.187 0.195 0.194 0.194 0.191 0.199 0.199 0.197 0.201 0.204 0.205 0.216 0.239 0.235 0.237 0.235 0.196 0.189 0.192

As we always work with dimensionless variables, we apply the following transformations to factors Y, P, B, and E (Table 2): X1 ¼ ðB – B0 Þ=B

B0 ¼ ðBmax þ Bmin Þ=2

B ¼ ðBmax – Bmin Þ=2

X2 ¼ ðE – E0 Þ=E

E0 ¼ ðEmax þ Emin Þ=2

E ¼ ðEmax – Emin Þ=2

X3 ¼ ðY – Y0 Þ=Y

Y0 ¼ ðYmax þ Ymin Þ=2

Y ¼ ðYmax – Ymin Þ=2

X4 ¼ ðP – P0 Þ=P

P0 ¼ ðPmax þ Pmin Þ=2

P ¼ ðPmax – Pmin Þ=2

Koppel and Palm had proposed the following model: ¼ 0 þ 1 x1 þ 2 x2 þ 3 x3 þ 4 x4

ð4Þ

462


Table 2 Center and step of variation Factor

Center

B E Y P

98 6.15 0.355 0.215

94 8.75 0.129 0.045

We have to verify the coherence between the postulated model, the objective to reach, and the set of 66 candidate points. Our objective must be well specified. Actually, we could have two distinct objectives: Objective 1: The experimental domain of interest is composed of 66 solvents only. In this case, it is obvious that if we ran the 66 experiments, at these 66 points, we would know the value of the property studied. Objective 2: The experimental domain of interest is described by the variation range of the factors studied. In this case, if we ran the 66 experiments, we would not necessarily know the value of the property studied with an acceptable quality at every point of the experimental domain of interest by using the postulated model. From the experimental design (Table 3) composed of the 66 candidate points, we calculate the model matrix (X), the information matrix (X9X), and the dispersion matrix (X9X)1. From the dispersion matrix, we can calculate, at each point of the experimental domain of interest, the variance function. We will keep the maximum value of this variance function: dmax. We obtain dmax ¼ 0.69. Thus, for a given experimental domain of interest described by the variation range of the studied factors, within the set of 66 points, there is a piece of information of sufficient quality.8,9 We can then proceed to Step 1. Step 1: We try to find out whether there is a set of N (1.2) In order to estimate by linear regression the coefficients of the postulated model, we considered the 26 points of the experimental design and the 9 experiment runs under the routine conditions (in italics characters in the Table 31). To test the validity of the model (Chapter 1.13), we used the four check points that have not been considered for the calculation of the coefficients and we performed a test of comparison between the experimental values and the calculated values. The difference between the experimental value yexp,j and the calculated value ycalc,j is denoted by j ¼ y exp;j – y calc;j

The deviation between the experimental value and the calculated value at point j can be tested statistically taking into account its standard deviation. Indeed, yexp;j – ycalc;j Sj

ð22Þ

follows a Student’s distribution. To carry out this statistical test, we use the estimate of the variance of the experimental error, calculated from the nine experiments performed in the same conditions. Besides, we know that

2 S ¼ 1 þ dj s 2 j

ð23Þ

with dj ¼ variance function of the response calculated at point j s2 ¼ estimate of the experimental variance calculated with eight degrees of freedom. At each of the check points, it is possible to calculate the ratio

yexp;j – ycalc;j Sj

This ratio has a Student-Fisher’s distribution with eight degrees of freedom (Table 32).

ð24Þ

490


Table 31 Experimental results Exp

Resolution IsoCl

Retention time

RN1

RX

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

1.15 1.18 1.16 1.36 0.76 1.47 1.48 1.66 1.10 1.36 1.29 1.50 0.94 1.23 1.36 1.25 1.18 0.94 1.45 1.35 1.73 1.45 1.10 1.56 1.05 1.21 1.10 1.30 1.05 1.21 1.33 1.30 1.34 1.24 1.31 1.26 1.33 1.21 1.15

10.90 11.82 12.02 12.63 7.65 18.31 20.02 20.88 15.06 11.74 10.14 13.99 8.48 16.23 17.34 14.29 15.76 10.42 14.27 13.86 20.11 15.50 11.75 17.92 10.84 11.98 11.75 15.50 11.70 11.98 13.35 13.40 13.26 13.33 13.47 13.40 13.30 13.45 13.19

0.88 0.84 0.76 0.65 0.82 0.57 0.47 0.69 0.72 0.30 0.21 0.31 0.67 0.95 0.87 0.84 1.10 0.89 0.73 0.44 0.31 0.56 0.71 0.64 0.82 0.65 0.68 0.69 0.85 0.65 0.73 0.85 0.84 0.78 0.68 0.66 0.80 0.78 0.68

1.27 1.09 1.17 1.29 0.83 1.27 1.35 1.67 1.02 1.10 1.00 1.49 0.99 1.34 1.19 1.08 1.37 1.10 1.17 1.07 1.45 1.40 1.01 1.39 0.99 1.02 1.11 1.09 0.99 1.02 1.19 1.23 1.13 1.25 1.24 1.29 1.29 1.22 1.14

Table 32 Table of residuals

Yexp;j – Ycalc;j =S

dj

Standard deviation

1.286 1.328 1.340 1.089

0.026 0.002 0.130 0.061

0.124 0.141 0.123 0.110

0.079 0.079 0.079 0.078

0.326 0.020 1.657 0.778

75.3 98.4 13.6 45.9

Response Y2: retention time 36 13.400 13.427 37 13.300 13.309 38 13.450 14.661 39 13.190 11.586

0.027 0.009 1.211 1.604

0.124 0.141 0.123 0.110

0.845 0.851 0.844 0.839

0.031 0.010 1.435 1.911

97.6 99.2 18.9 9.2

Yexp

Response Y1: RIsoCl 36 1.260 37 1.330 38 1.210 39 1.150

Ycalc

Difference

Exp

p value (%)

(Continued )


Table 32 Exp

491

(Continued) Yexp

Ycalc

Difference

dj

Standard deviation

Yexp;j – Ycalc;j S

p value (%)

Response Y3: RN1 36 0.660 37 0.800 38 0.780 39 0.680

0.745 0.652 0.715 0.787

0.085 0.148 0.065 0.107

0.124 0.141 0.123 0.110

0.080 0.081 0.080 0.080

1.057 1.841 0.815 1.348

32.1 10.3 43.9 21.5

Response Y4: RX 36 1.290 37 1.290 38 1.220 39 1.140

1.207 1.118 1.197 1.036

0.083 0.172 0.023 0.104

0.124 0.141 0.123 0.110

0.085 0.086 0.085 0.085

0.981 2.006 0.268 1.226

35.5 8.0 79.5 25.5

The mathematical model being validated for all the responses, all the results for the 39 experiments are considered to calculate the mathematical model again more precisely. The estimates of the coefficients for the six responses are reported in Table 33. From these models, we can draw the isoresponse curves that can be represented within the space of the continuous variables or within the space of the mixture variables. The interpretation of the experimental results will be detailed only for the experimental response Y3: ‘RN1’ in Figures 23–25. These graphs show the influence of the pH and of the ionic strength of the pH. Besides, it is to be noted that the composition of the formula, and particularly the proportion of methanol, has a very strong influence: The response decreases markedly when the methanol percentage increases.

1.14.4.2.3

Multicriteria optimization For the main experimental responses studied, the ‘acceptation’ limits were specified, in order to determine – provided it exists – the zone of the experimental domain where all the characteristics measured are satisfactory. The objectives for each of the responses are represented in Figure 26. The values of di are combined together using the geometrical mean:14 D ¼

p ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 4 d1 d2 d3 d4

ð25Þ

Table 33 Estimates of the coefficients for all the responses, with 39 experiments Coefficient

Y1

Y2

Y3

Y4

a1 a2 a3 a12 a13 a23 b1 b2 b3 b11 b22 b33 b12 b13 b23 g11 g12

1.093 0.827 1.469 0.666 0.878 0.484 0.049 0.037 0.319 0.101 0.055 0.035 0.052 0.047 0.048 0.000 0.057

26.617 11.643 13.430 20.910 4.011 12.126 0.150 1.361 3.273 0.546 0.104 0.676 0.247 1.054 1.113 0.508 0.860

1.678 1.432 0.048 1.744 1.239 0.156 0.115 0.168 0.089 0.159 0.185 0.101 0.183 0.134 0.024 0.109 0.407

2.111 1.577 1.724 2.257 1.846 1.501 0.077 0.088 0.244 0.028 0.178 0.000 0.034 0.205 0.070 0.008 0.005

492


Figure 23 2D isoresponse curves for the response RN1, in the mixture domain, for temperature ¼ 25 C.

The global desirability D is calculated at each point of the experimental domain and the function is maximized at the point represented in Table 34. The evolution of D, calculated from the previously defined objectives, is represented graphically in Figures 27 and 28. Study of fiability The algorithm makes it possible to determine the coordinates of an extremum and the region where the desirability function is maximum. However, the experimenter may also be interested in the evolution of the desirability function when an interval of confidence is imposed upon the experimental responses studied. Indeed, the experimental conditions proposed are obtained from estimated models and the experimenter may ask for – as a complement to the definition of the region in which the

1.14.4.2.3(i)


493

Figure 24 3D response surfaces for the response RN1, in the mixture domain, for temperature ¼ 25 C.

estimates of all the responses are calculated with a precision higher than or equal to (1 ¼ 0.95) – respect of the constraints imposed upon each of the responses. Let us suppose that the experimenter wants to impose, for four responses: Prob½1 > 1:20 0:95 Prob½12 < 2 < 18 0:95 Prob½3 > 0:70 0:95

ð26Þ

Prob½4 > 1:20 0:95

The experimenter does not want, if he runs experiments in the optimum zone, to obtain values for these responses that will not respect the constraints with a probability higher than 0.05%. The coordinates of the optimum will thus be modified (Table 35, Figures 29 and 30). The curves show that the surface of the optimum zone is smaller.

494


RN–1

Temperature 30.0

X3 1.00 0.7

0.8

1.1 0.6 0.5 0.4

0.50

25.0

0.6 X1

–1.00

–0.50

0.50

0.00

1.00 0.1 Temperature

X3

–0.50 0.5 0.5 20.0

0.4

X1

–1.00 pH

2.6

2.8

3.0 pH

Figure 25 Variation of the response RN1 in the plane: pH, temperature. Fixed factors: ionic strength ¼ 0.02 mol l–1. Fixed components: buffer ¼ 0.53, ACN ¼ 0.24, MeOH ¼ 0.23.

d1

d2

100%

100%

R IsoCl

0% 1.20

0%

1.28

12

d3

d4

100%

100%

R N–1

0% 0.70

0.72

15

18

Retention time

RX

0% 1.20

1.27

Figure 26 Partial desirabilities.

1.14.5 Final Remark Classic designs of experiments are now standardized and more often used. But those designs can be used only in an ideal environment (e.g., symmetrical experimental domain). That is how, during the past, scientific investigators sometimes adapted their problems to be able to use those designs. Other potential users could not use them because those designs were not able to consider different experimental restrictions. Since the 1970s, some statistical algorithms have allowed building designs considering all restrictions (experimental, economical, and more) present in an experimentation. But only the progress made in computing during the past


495

Table 34 Desirability function: coordinates and characteristics of the maximum Variable

Value

Factor

Z1 Z2 Z3 X1 X2 X3

0.300 0.690 0.010 0.607 0.063 0.774

Buffer ACN MeOH pH Ionic strength Temperature

Response

Value

Difference (%)

RIsoCl Retention time RN1 RX Desirability

1.34 15.19 0.93 1.57

100.0 93.6 100.0 70.9 93.4

Y1 Y2 Y3 Y4

lonic strength 0.0250

Value 0.527 0.262 0.211 2.7 0.0197 21.1

Desirability

X2

1.00

1.00 0.20 0.40 0.60

0.50

0.80

0.50 X1

0.0200 –1.00

–0.50

0.00

0.50

1.00

lonic strength

0.00 –0.50 X1 –1.00

0.0150

pH 2.6

2.8

3.0 pH

Figure 27 Variation of the desirability in the plane: pH, ionic strength. Fixed factors: temperature ¼ 21.1 C. Fixed components: buffer ¼ 0.527, ACN ¼ 0.262, MeOH ¼ 0.211.

few years (processing speed, storage capacity, etc.) allowed the standard use of those statistical tools. The resulting designs must then provide the needed information to reach the wanted objective. But the quality of the information we obtain should be good enough to make risk-free decisions. We therefore selected mathematical criteria to help us evaluate the quality of the wanted information. We demonstrate that those criteria, even well known, have to be classified into different categories, depending on whether they are used as an optimization in the different algorithm types (D-optimality, A-optimality, etc.) or as quality criteria of the wanted information (variance function, VIF, coefficient of variance, etc.). Such criteria are not universal and the choice of quality criteria essentially depends on the objective we want to reach. We showed that when we want to qualify the principal and interaction effects, then the VIF and the coefficient of variance are good quality indicators. If we want to know the value of all the concerned experimental responses in any point of the experimental domain of interest, then the higher value of the variance function is the only acceptable criterion. We used examples to

496


Desirability Pseudo-X 1

1.10

0.55 Pseudo-X 1 Pseudo-X 2 –0.00 0.80

0.60 Pseudo-X 3

Pseudo-X 2

Pseudo-X 3

0.00

Figure 28 Variation of the desirability in the plane: buffer, ACN, MeOH. Fixed factors: pH ¼ 2.7, ionic strength ¼ 0.0197 mol l–1, temperature ¼ 21.1 C.

Table 35 Desirability function: coordinates and characteristics of the maximum Variable

Value

Factor

Value

Z1 Z2 Z3 X1 X2 X3

0.399 0.524 0.077 0.446 0.262 0.856

Buffer ACN MeOH pH Ionic strength Temperature

0.533 0.252 0.215 2.7 0.0187 20.7

Response

Value

Difference (%)

RIsoCl Retention time RN1 RX Desirability

1.27 17.72 0.65 1.32

73.4 9.3 81.5 0.1 16.3

Y1 Y2 Y3 Y4

show how to choose and use those different criteria and focused on the risk involved if they are not used properly. We demonstrate that the choice of those factors is also essential when used with classic designs such as Box–Behnken design that are available only in the spherical domain. We considered the case where we have to study together categorical factors or continuous factors and mixture composition. The experimental domain is composed of several different domains. A specific design may correspond to each domain, and the experimental domain is composed of several nonoverlapping fields. Several examples complete the presentation of the experimental designs used for the optimization studies. They show that it is always possible to construct an adequate design, taking into account the experimental constraints or a particular mathematical model.


lonic strength

497

Desirability

X2

1.00 1.00

0.0250

0.50

0.50 X1

0.0200

–1.00

–0.50 0.10

0.00

0.50

1.00

lonic strength 0.00

X2

–0.50 X1 –1.00

0.0150

pH 2.6

2.8

3.0 pH

Figure 29 Variation of the desirability in the plane: pH, ionic strength. Fixed factors: temperature ¼ 20.7 C. Fixed components: buffer ¼ 0.533, ACN ¼ 0.252, MeOH ¼ 0.215.

Desirability Pseudo-X1

1.00

0.50

Pseudo-X1

Pseudo-X2 0.00

Pseudo-X3

Pseudo-X2

Pseudo-X3

Figure 30 Variation of the desirability in the plane: buffer, ACN, MeOH. Fixed factors: pH ¼ 2.7, ionic strength ¼ 0.0187 mol l–1, temperature ¼ 20.7 C.

References 1. Box, G. E. P.; Hunter, W. G.; Hunter, J. S. Statistics for Experimenters; John Wiley & Sons: New York, 1978. 2. Mathieu, D.; Phan-Tan-Luu, R. Approche Me´thodologique des Surfaces de Re´ponse. In Plans d‘expe´riences (Applications a` l’entreprise); Droesbeke, J. F., Fine, J., Saporta, G., Eds.; Technip Editions: Paris: France, 1998. 3. Mathieu, D. Contribution of Experimental Design Methodology to the Study of Structure-Activity Relations. PhD Thesis, University of Aix-Marseille: Marseille, 1981.

498


4. Fedorov, V. V.; Malyutov, M. B. Optimal Design in Regression Experiments. Math. Operationsforsch. Statist. 1972, 14, 237–324. 5. Lewis, G. A.; Mathieu, D.; Phan-Tan-Luu, R. Phamaceutical Experimental Design; Marcel Dekker: New York, Basel, Hong Kong, 1999. 6. Montgomery, D. C. Design and Analysis of Experiments, 5th ed.; John Wiley & Sons: New York, 2001. 7. Koppel, A.; Palm, V. A. In The Influence of Solvent on Organic Reactivity in Advances in Linear Free Energy Relationships. ; Chapman, N. B., Shorter, J., Eds.; S. J., Ed.; Chapter 5; Plenum Press: London,, 1972; pp 221. 8. Sergent, M.; Phan-Tan-Luu, R.; Elguero, J. Statistical Analysis of Solvent Scales. Part 1. Anal. Quim. Int. Ed. 1997, 93, 71–75. 9. Sergent, M. Contribution de la Me´thodologie de la Recherche Expe´rimentale a` l9e´laboration de Matrices Uniformes: Application Aux Effects de Solvants et de Substituants. PhD Thesis, University of Aix-Marseille: Marseille, France, 1989. 10. Plackett, R. L.; Burman, J. P. Design of Optimum Multifactorial Experiments. Biometrika 1946, 33, 305–325. 11. Box, G. E. P.; Behnken, D. W. Some New Three Level Designs for the Study of Quantitative Variables. Technometrics 1960, 2, 455–475. 12. Box, G. E. P.; Draper, N. R. Emperical Model-Building and Response Surfaces; John Wiley & Sons: New York, 1987. 13. Doehlert, D. H. Uniform Shell Designs. Appl. Stat. 1970, 19, 231–239. 14. Derringer, G.; Suich, R. Simultaneous Optimization of Several Response Variables. J. Qual. Technol. 1980, 12 (4), 214–219. 15. Draper, N. R.; Smith, H. Applied Regression Analysis, 3rd ed.; John Wiley & Sons: New York, 1998. 16. Cornell, J. A. Experiments with Mixtures; John Wiley & Sons: New York, 1990.




Michelle Sergent is Professor at the University Paul Cezanne of Marseille, France. She is Doctor of Science with a specialty in Designs of Experiments. She heads the Laboratory of Methodology of Experimental Research. She participated in the setting up of courses in the Methodology of Experimental Research at the University of Aix-Marseille and she collaborated in the creation of trainings in several universities and industries in France and abroad. She is author and co-author of numerous scientific articles and currently she is president of the ‘Groupe Français de Chimiome´trie’.

499


1.15 Experimental Designs: Conclusions, Terminology, and Symbols R. Phan-Tan-Luu, University Paul Cezanne, Marseille, France R. Cela, Universidad de Santiago de Compostela, Santiago, Spain ª 2009 Elsevier B.V. All rights reserved.

1.15.1 1.15.1.1 1.15.1.2 1.15.1.3 1.15.1.4 1.15.1.5 1.15.1.6 1.15.1.7 1.15.2

Terminology Responses Factors Experiments Experimental Design Experimental Plan Experimental Domain Experimental Domain of Interest Symbols

502 502 502 502 502 503 503 503 503

We have presented the state of the art while trying to avoid repetition of what appears in the numerous books dealing with experimental designs. In the various chapters, we have treated not only the numerous applications that we have already made or helped to make but also the recent progress achieved in the domain of design. Our objective is to provide the experimenters with a tool that will help them not only to use the designs but also to understand why and how to use them. These tools are only tools; what is essential is to know in which context to use or not to use them. We should not forget that if, when using a hammer, we hit our finger, it is not the hammer’s fault. The use of experimental design has become commonplace, and we can see that experimenters have started to change their habits, especially in the domains of application. Not long ago, a total of 20 or less factors was considered reasonable for a screening study and 10 or less was considered reasonable for a response surface study. Several reasons accounted for these arbitrary limits:

For a long time, the experimenters were studying one factor at a time, and it is only with difficulty that they accepted to study all the factors at the same time. Switching from 1 factor to k factors together was not so easy. Most of the classical designs had been set up before 1980, at a time when the capabilities of computers and especially of microcomputers could not be compared with those of the present-day computers, let alone those of future computers. Let us consider, for example, the catalog of Hadamard designs established by Plackett and Burman, which makes it possible to build Hadamard (Plackett–Burman) designs for a total of 100 factors or less. We could think that it should always be possible to find a suitable Hadamard design in case of screening of k factors at two levels. But what to do when dealing with several hundred factors? There are now new domains of applications. Sometimes, the experiments are decided through simulation (from complex simulation algorithm) and not experimentation. In this case, the number of factors can be very high (from several tens to several hundreds) and, even though computers are extremely fast, the calculation times are too long, and hence the number of factors has to be reduced. The experimental designs used in these domains of simulation are called numerical designs. They were set up taking into account the complexity of the simulation models: metamodel polynomials, kriging, splines, etc. Very often, these designs are specific: Latin hypercube sampling, space-filling design, Morris method, etc.

Nowadays, the world of mathematical simulation and that of experimentation are finding more and more common working spaces; therefore, the distinction between numerical designs and experimental designs is fading. 501

502

Experimental Designs: Conclusions, Terminology, and Symbols

We can mention, for example, the application of a Hadamard matrix for the screening of 718 factors in a simulation problem. The ever-changing face of science and technology developments will surely surprise us as new challenges appear. However, a constant challenge for the experimentalist is to produce the desired information with the required quality and with the highest efficacy and efficiency. We have learned through experimental design that there are better ways to accomplish this. Do not forget lessons learned.

1.15.1 Terminology Most of the specific terms we use in experimental design are classical terms, but they may gain slightly different meanings when used in certain domains of statistics. Let us recall the meanings of some important terms to better understand our presentation.

1.15.1.1

Responses

An experimental response (sometimes called ‘dependent variable’) is a measurable manifestation that can be observed when the factors studied are made to vary. What characterizes a response is that it is not possible to directly impose a value on it. This has to be done indirectly, by means of factors. We may have to study several characteristics or experimental responses. We can imagine any type of response, for example, yield, purity, selectivity, and organoleptic property. The last response, which can take only discrete values, often creates problems during interpretation of the results. It is also possible to study the function of a measured response, for example, the logarithm of an equilibrium constant. The term ‘experimental response’ is considered in a broad sense, as it can be either the result of an experiment or the result of a numerical simulation.

1.15.1.2

Factors

The factors are the supposed or certain causes that may have an influence on the phenomenon studied. They must all be taken into account; otherwise, the results may be distorted and thus useless. They may have different characteristics: A factor can be qualitative or quantitative, continuous or discontinous, or controllable or uncontrollable. A factor is characterized by one or several natural variables: Some qualitative factors can be described by several quantitative parameters (e.g., a molecule can be characterized by its steric effect and its hydrophobicity).

1.15.1.3

Experiments

An experiment (or experimental point, experimental unit, run) is defined by the specified settings of the experimental factors. The natural variables take precise values, and the factors that are not taken into account in the study must be kept constant throughout the whole of the experimentation.

1.15.1.4

Experimental Design

An experimental design is a mathematical object that represents, under a coded form, the set of experiments to be run. It is a table composed of N rows, corresponding to N experiments, and k columns, corresponding to k variables studied. The element xij of this design corresponds to the level that the jth (coded) variable takes during the ith experiment. It should be possible to use an experimental design, for a type of required information, whatever the domain of application (chemistry, physics, mechanics, biology, agronomy, etc.).


1.15.1.5

503

Experimental Plan

An experimental plan is the ‘translation’ of an experimental design into natural variables. It is therefore a table containing data expressed in terms of their units and it can be used directly by the experimenter. Like the experimental design, the experimental plan contains N rows and k columns. Each of its elements Uij corresponds to the level that the jth variable takes during the ith experiment. The experimenter must perform a critical analysis of the experimental plan taking into account all the information he/she has on the problem studied. If he/she realizes that one or several experiments cannot be performed, either because they are impossible to perform or because they may be hazardous (explosion, etc.), he/she should not run the experimental plan. In that case, the methodological approach must be restarted, including the information that had been neglected. If all the experiments are possible, the experimenter must decide on the planning of the experiments to run, and perform them in a random order so that the information contained in the experimental results is not biased.

1.15.1.6

Experimental Domain

The influence of each natural variable can be determined by imposing on the variable variations that will modify the behavior of the phenomenon studied. A variation range is defined for each variable, within which the variable can take different values or levels. We call factor space the set of the levels that each variable can take. Each element in this domain represents a set of experimental conditions.

1.15.1.7

Experimental Domain of Interest

The possible experimental domain (or domain of operability) is a subset of the factor space containing the feasible experiments. In practice, this domain is often reduced to experimental domain of interest within which we can find the information we are looking for. The interpretations that we will make from the experimental results will be valid only within this experimental domain of interest. Before any experimentation, we must define its size, its shape, the possibilities of extension, the suspected discontinuities of the system, etc.

1.15.2 Symbols j

bj

A yA s Xj xi ei Y X X9 (X9X) (X9X)1

coefficient of a model equation for response surface or mixture design: the true (unknown) value weight for screening designs: the true (unknown) value effect for studies of factors : the true (unknown) value estimate coefficients of a model equation for response surface or mixture designs estimate weight for screening designs estimate effect for studies of factors true value of a response y at point A measured value of a response y at point A number of degrees of freedom true (unknown) standard deviation estimate of standard deviation coded variable for the factor j level of Xi in the model for an experiment experimental error at experiment i vector response whose elements are experimental responses Y9: {y1, ..., yN} model matrix transpose of the model matrix information matrix dispersion matrix

504

cij E[ ] si Uj r k p q G N Zi Zi zi


an element of the matrix (X9X)1, row i and column j mathematical expectation number of levels of the ith variable of a (symmetric or asymmetric) factorial or screening design natural variable for the factor j fraction of a fractional factorial design 2kr number of independent factors or independent variables in the problem number of parameters (coefficients) in the model number of components in a mixture a generator number of experiments in the design transformed set of axes by canonical analysis of the second-order model a mixture component in a mixed process – mixture model level of the component Zi in a mixed process – mixture model




Rafael Cela Torrijos is a professor of analytical chemistry in the University of Santiago de Compostela, Spain, and head of the laboratory for analytical chemistry in the Research Institute of Food Analysis in the same university. Previously, he was at the University of Madrid (Complutense) and University of Cadiz, belonging to the group of analytical chemists that started the development of chemometrics in Spain in the 1980s. His research has focused on the analytical applications of separation science, particularly, the development and optimization of sample preparation techniques in chromatographic analysis, including experimental designs and computer-assisted chromatographic methods development. Currently, he is author or coauthor of more than 200 scientific papers and several textbooks.

505


1.16

Constrained and Unconstrained Optimization

B. W. Bader, Sandia National Laboratories, Albuquerque, NM, USA Published by Elsevier B.V.

1.16.1 1.16.2 1.16.3 1.16.4 1.16.4.1 1.16.4.2 1.16.4.3 1.16.4.3.1 1.16.4.3.2 1.16.5 1.16.5.1 1.16.5.2 1.16.5.3 1.16.6 1.16.6.1 1.16.6.1.1 1.16.6.1.2 1.16.6.1.3 1.16.6.2 1.16.6.2.1 1.16.6.2.2 1.16.6.2.3 1.16.7 References

Introduction Numerical Optimization Optimization in Chemometrics Unconstrained Optimization Methods Nonlinear Equations in One Variable Minimization in One Variable Minimization in Multiple Variables Newton’s method Quasi-Newton methods Globalization Strategies Steepest Descent Line Search Methods Trust Region Methods Constrained Optimization Methods Penalty, Barrier, and Augmented Lagrangian Methods Quadratic penalty function Augmented Lagrangian method Logarithmic barrier method Active Set Methods Linear programming and the simplex method Quadratic programming Sequential quadratic programming Discussion

Symbols A A(x) A (x) Ax ¼ b ai ared ak B B

B Bk c

general matrix Jacobian matrix of the constraints active set of constraints system of linear constraints in a linear program linear constraint from row of matrix A actual reduction in f, which equals

f ðxk Þ – f ðx – k þ 1Þ

approximate derivative f9(xk ) block diagonal matrix with blocks of size 1 1 or 2 2 submatrix consisting of the m columns of A with column indices iPB basic set of m indices from {1, 2, . . .n} inverse of the Hessian matrix approximation, Hk vector in QP that equals Ax0 b

c cB, cN

ci(x) c(x) F i

df f ðiÞ ðxÞ ¼ dx i jx

f(x) df f9ðxÞ ¼ dx jx 2

d f f0ðxÞ ¼ dx 2 jx

G g h, b

Hk PRnn

508 509 511 512 512 514 518 520 522 524 525 526 528 531 532 532 533 534 534 534 537 539 542 543

vector of costs in a linear program vector c partitioned according to the index sets B and N , respectively constraint function set of equality constraints subset of directions along set of constraints ith derivative of f with respect to x evaluated at x objective function first derivative of f with respect to x evaluated at x second derivative of f with respect to x evaluated at x symmetric n n matrix in QP vector in QP that equals Gx0 þ h n-vectors in QP approximate Hessian matrix,

r2 fðxk Þ

507

508


h(x) I i,j,k,p,q J J L L A(x, ; ) L (x, ) m,n,p,c m(xk þ s) min f(x) N

N P pred P(x ; ) Q(x ; ) q(x) q() R s U u W k W(x, ) x x xB, xN

xCP k

set of inequality constraints identity matrix scalar indices set of inequality constraints set of unique indices for inequality constraints lower triangular matrix augmented Lagrangian function Lagrangian P function,

f ðx Þ –

m i¼1

i ci ðxÞ

scalar constants local approximation to f at xk minimum value of f(X) submatrix consisting of the n – m columns of A with column indices iPN nonbasic set of n – m indices and is the complement of B permutation matrix predicted reduction in f by the local model logarithmic barrier function where > 0 is the barrier parameter quadratic penalty function where is the penalty parameter quadratic objective function of QP quadratic function in set of real numbers vector corresponding to the step upper triangular matrix vector of unknowns working set of constraints at the kth iteration Hessian of the Lagrangian,

r2xx L

ðx; Þ

vector of reals of length n minimizer of f(X) vector x partitioned according to the index sets B and N , respectively Cauchy point, xk – rf ðxk Þ

xN k {xk} yk

Newton point, xk þ Hk– 1 – rf ðxk Þ sequence of iterates {x0, x1, x2,. . .} vector representing the difference in function values,

rf ðxk Þ – rf ðxk – 1 Þ Z zi " " low, high B, N

b e ,

low ;high

i

1 ; 2

O (?)

ðÞ r r2

rf ðxÞ r2 f ðxÞ rx

r2xx

basis for the null space of A slack variable step length/line search constant trust region radius set of equality constraints set of unique indices for equality constraints trust region control parameters vector of Lagrange multipliers, i vector b partitioned according to the index sets B and N , respectively Lagrange multipliers for bound constraints Lagrange multipliers for equality constraints scalar constant fraction to reduce in a line search line search safeguard parameters ith eigenvalue of a matrix line search parameters asymptotic complexity of an algorithm function defined as f (xk þ sk) multidimensional partial differential operator multidimensional second partial differential operator is the gradient of f at x Hessian matrix of second partial derivatives of f with respect to x multidimensional partial differential operator just (with respect to x) multidimensional second partial differential operator (with respect to x)

1.16.1 Introduction This chapter covers the basic concepts, theory, and algorithms of numerical optimization as well as its application to problems in chemometrics. Optimization is a very useful tool for data analysis. Indeed, quite frequently the methods used in standard chemometric data-fitting problems are, at the heart, optimization techniques, through they may be disguised or described as something else. Hence, while optimization is at the


509

core of many standard chemometric analysis techniques, its presence may not be immediately recognized. For instance, many data-fitting techniques use algorithms for solving an optimization problem of minimizing the residual squared error. The goal of this chapter is to explain optimization theory and methods so that the practitioner has enough knowledge to understand the field, to know how to apply various optimization techniques, and to know where to go for further information on very difficult problems. This chapter starts with a general survey of the field of optimization and describes some motivating examples from chemometric data analysis. Then we delve into methods for solving unconstrained and constrained optimization problems, along with two popular strategies for global convergence. Finally, we end with a brief discussion of global optimization techniques and so-called direct search methods that do not require derivatives.

1.16.2 Numerical Optimization The field of numerical optimization deals with improving some aspect of a problem until it can be improved no more. Words like optimize, minimize, and maximize all signal that an optimization problem is at hand. For example, a chemical company may wish to maximize profit in their production of a certain chemical, or they may wish to minimize waste in their operations. In fact, with today’s sophisticated and competitive markets, these corporate objectives are routinely set up as optimization problems and solved repeatedly in real time. Sometimes, there may be constraints that must be modeled, such as physical limitations inherent to the equipment or, perhaps, operational constraints imposed by laws governing discharge of toxic by-products. Occasionally, these problems are so large that they take the largest computers days to solve, such as in the design of an optimal shape, like an airplane wing. All of these problems boil down to the same mathematical description and use the same methods to solve the problem. Let us consider the particular notation used. Generally speaking, there is some ‘objective’ that we desire, and we manipulate some knobs and dials to achieve this. Mathematically, we write the problem as min f ðxÞ x

ð1Þ

where f (x) is the objective function and x is the degree(s) of freedom, which can be either a single variable or a vector of unknowns, xPRn , as is frequently the case. While problem (1) specifically refers to the minimization of f (x), it should be noted that a maximization problem can be recast as a minimization problem: max gðxÞ ¼ min – gðxÞ x

x

ð2Þ

Thus, any maximization problem involving g(x) can be treated as a minimization problem by setting f (x) ¼ g(x). It should be noted that some disciplines tend to prefer the term maximization. For example, the computer science field of artificial intelligence talks of methods for ‘hill climbing’, which is a metaphor for maximization. In this chapter, we will only consider minimization, and we will treat ‘optimization’ as being synonymous with ‘minimization’. Technically speaking, the solution to problem (1) is the minimum value of f (x). However, we are usually interested in the particular value(s) of the variable x that produces the minimum, which we formally write as x ¼ argmin f ðxÞ

We call x the minimizer of f (x).

ð3Þ

510


f (x)

x∗

xa

xb

x

Figure 1 A bumpy function with lots of local minima that would be difficult to find the global minimizer.

If the point x is the minimum over all possible values of x, then we have found a global minimizer. This is challenging to do in practice because we usually have knowledge only in a limited region of this function. That is, local information cannot say anything with confidence about the function far from our area of exploration; how far away depends upon the function. If the function is as in Figure 1, then we may not have much confidence that our algorithm returned the global minimizer. In such cases, the point x is a local minimizer of f (x) for some neighborhood around x . In Figure 1, x is a local minimizer of f (x), for x P [xa, xb]. If constraints are involved in the optimization problem, then the minimization problem changes: min x

f ðxÞ ð4Þ

subject to cðxÞ ¼ 0 hðxÞ 0 m

where cðxÞPR is a set of m equality constraints and hðxÞPRp is a set of p inequality constraints. We may write this problem more compactly as min x

f ðxÞ;

subject to cðxÞ ¼ 0; hðxÞ 0

While constraints encountered in practice are not usually written such that they are greater than or equal to zero, it is always the case that with a little algebra the constraints can be rearranged to fit this form. Our convention is to write inequality constraint as greater than zero. The alternate form of less than or equal to zero may be used with a small change in algebra. While not all constrained optimization problems have both equality and inequality constraints, the distinction is an important one because, as we shall see, it affects the solution technique. Within the general optimization problem framework described by problems (1) and (4), there are numerous cases to consider. Table 1 categorizes all of these cases. In general, each case requires its own solution technique, which we describe in the sections that follow.

Table 1 Optimization problem types Name

f (x)

c (x)

Unconstrained optimization Linear programming Quadratic programming Linear constrained optimization Constrained optimization or nonlinear programming

Nonlinear Linear Quadratic Nonlinear Nonlinear

None Linear Linear Linear Nonlinear


511

1.16.3 Optimization in Chemometrics This section motivates optimization in chemometrics by providing some examples from chemometric data analysis. Data-fitting problems may be formulated as an optimization problem because, at their core, data-fitting problems seek to find a set of unknowns in a model that maximizes some goodness-of-fit metric, usually in terms of minimizing the sum of the squared error. In addition, with data analysis problems, for example, constraints may be added to impose certain restrictions upon the solution, such as orthogonal factors, XTX ¼ I. For example, principal component analysis (PCA) may be formulated as an optimization problem. The most straightforward problem statement for finding the first principal component is as a constrained optimization problem, x T Ax

max x

ð5Þ

subject to jjx jj2 ¼ 1

but this may be also written as an unconstrained optimization problem, max x

x T Ax xT x

ð6Þ

Here, A is a symmetric n n matrix. The global maximizer x is the eigenvector of A corresponding to the largest eigenvalue. For the more general PCA problem, the formal problem may be stated as follows min W ;P

A – AWP T

2

ð7Þ

subject to W T W ¼ I

where W is an orthogonal matrix. Variations in the data-fitting problem that resemble a PCA problem may be considered as well. For example, perhaps the data are nonnegative and the factor matrices are desired to be nonnegative to preserve the nonnegative quality of the original data and facilitate better interpretability. Given a nonnegative matrix A, the formal problem may be stated as min U ;V

A – UV T

2

subject to U 0; V 0

ð8Þ

This problem is called the nonnegative matrix factorization. The parallel factors (PARAFAC) model1, which is used, for example, in multiway analysis of excitation– emission fluorescence spectra of different samples, may be formulated as an optimization problem, min

A;B;C

P i;j ;k

ðxijk –

P

air bjr ckr Þ2

r

While PARAFAC is a multilinear model (in this case, trilinear), fitting it to data is still a nonlinear optimization problem due to the cross terms, air bjr ckr . Consequently, the objective function cannot be described by a simple quadratic function. In fact, fitting PARAFAC is a very difficult optimization problem because, in general, there are multiple minima. Missing data is another area where optimization techniques may arise in chemometric data analysis. Traditional data-fitting techniques, such as least squares or PCA, operate on arrays with complete data. With these techniques, it would be a mistake to enter missing data as zeros because then these zero values would bias

512


the model. Rather, the terms corresponding to the missing data should be removed from the objective function so that the model is fit according to only the data that are available. This may be accomplished by defining an objective function for the appropriate metric but only for the data that are present.

1.16.4 Unconstrained Optimization Methods To motivate optimization theory, we will start with simple one-dimensional examples. However, before we even discuss optimization, it is especially instructive to consider the closely related problem of root finding. That is, we wish to solve a nonlinear equation with one unknown.

1.16.4.1

Nonlinear Equations in One Variable

Consider this nonlinear function in one variable, f ðxÞ ¼ x 2 – e – x

ð9Þ

We wish to solve the nonlinear equation for x such that f (x) ¼ 0. The function in Equation (9) has a root at x 0.70346742. We start with an initial guess close to where we think a root might be, so we choose x ¼ 0 as our initial guess. Because we will iterate to find the solution, we denote this point x0, and each subsequent iteration, denoted with increasing subscripts, x1, x2, and so on, hopefully will be closer to the actual solution. The function value at x0 is f (0) ¼ 1, which indicates that x0 is not a root. The slope of the tangent line is f 9(0) ¼ 1, and it indicates which direction we must travel to improve our initial guess. With a negative function value and positive slope, we see that moving to the right would increase f (x) and improve our current estimate of the root. The question is how far do we move? One reasonable approach, which is supported by theory, is to move where the tangent line crosses the x-axis. The step to reach this intersection is given by the formula x ¼

– f ðxÞ – ð – 1Þ ¼ ¼1 f 9ðxÞ 1

which indicates that our next point is one unit to the right of x0. Hence, x1 ¼ 1. We check the function value once again, f (x1) ¼ 0.63212056, and see that we are closer to zero but still not at a root. Repeatedly applying this procedure, moves us ever closer to the actual root. Indeed, repeated application of this algorithm provides x2 ¼ 0.733043605, x3 ¼ 0.703807786. Collectively, these steps produce a famous algorithm called the Newton–Raphson method, or just Newton’s method: xkþ1 ¼ xk þ sk

ð10Þ

where sk ¼

– f ðxk Þ f 9ðxk Þ

ð11Þ

Here, we have isolated the step direction and called it sk. We will see later why it is helpful to compute the step direction and store it separately. It is useful for our understanding to describe Newton’s method in more abstract terms. At each iteration k, we are constructing a simplified, surrogate model of our function f (x) at the point xk. This simplified model is a


513

linear problem, and linear problems have a closed-form solution. Newton’s method is just finding the root of this linear model and using that solution as the step to move from xk to xkþ1. In fact, one way to derive Newton’s method is to truncate the Taylor series expansion of f (x) at xk and solve the resultant linear model for the step. Specifically, the Taylor series approximation of the function f (x) at a current point xk is f ðxÞ ¼ f ðxk Þ þ f 9ðxk Þðx – xk Þ þ ¼

1 X 1 i¼0

i!

1 f 0ðxk Þðx – xk Þ2 þ 2!

f ðiÞ ðxk Þðx – xk Þi

ð12Þ ð13Þ

This implies that given enough information about the local derivatives of f (x) at xk, then one can obtain the function value at some distance away from xk. This is a powerful notion. Truncating the Taylor series to just the linear terms (i.e., i ¼ 0 and 1 in the summation), we arrive at a linear model of f (x) at xk, and the root of this model is exactly the step (11). The Newton step (11) uses the exact derivative f 9(xk). In some cases, however, we may not have the exact derivative. For example, this is the situation when the functional form of f (x) is not known, and we only have a procedure for computing the function value. These are so-called ‘black boxes’ where only limited information is known, and which frequently may not include derivative information. The question then is how can we solve the nonlinear equation without exact derivatives? Fortunately, Newton’s method can be modified to use approximate or inexact derivative information. Suppose we have two points y and z (assume y < z), and we know their function values f (y) and f (z). Then a basic law of calculus states that the derivative of some other point in their interval, xP[y, z], must be equal to f ðyÞ – f ðzÞ y–z

This fact provides a means to compute approximate derivatives. Indeed, this is the concept behind finite differences, where if we choose an interval [y, z] sufficiently small around xk, then the approximation to the first derivative of f at xk can be very good. There is a large theory behind finite-difference derivatives that provides formulas for determining derivatives of various accuracies. Here, we use the most basic kind, which is first-order finite differences. If we use the current iterate xk as one of the points y or z, then that saves an extra computation, which is important if f (x) is expensive to compute. If the difference between xk and the second point is some step length , then the firstorder finite-difference formula is f 9ðxk Þ ak ¼

f ðxk þ Þ – f ðxk Þ

ð14Þ

The ‘first-order’ refers to the method’s accuracy, which is proportional to . Typically, we choose to be very small, like 107, but its optimal value depends on machine precision. We mention one last point for terminology. If we choose > 0 such that the second point is larger than xk, then the technique is further defined as forward finite differences. If < 0, then it is called backward finite differences. With an approximate value of f 9(xk) available, we change the calculation of the Newton step (11) to sk ¼

– f ðxk Þ ak

ð15Þ

This is called finite-difference Newton’s method, and this approximation usually does not slow the method down by too much. It turns out that the accuracy of the numerator f (xk) is more critical than the accuracy of the derivative in the denominator. We point out that finite-difference Newton’s method calculates two function values per iteration, f (xk) and f (xk þ ). If f (x) is expensive to compute, then it would be helpful if we could avoid any extra calculations. The

514


astute reader will note that Newton’s method already calculates extra points in its iterative process, so why not use them? Indeed, this is the basis for a class of methods called secant methods, or more generally quasi-Newton methods. Instead of computing f (xk þ ) for some arbitrary value of , we use the previous iterate xk1 and its function value f (xk1) in the calculation of the approximate derivative, f ðxk Þ – f ðxk – 1 Þ xk – x k – 1

ak ¼

In the context of the finite-difference Newton’s method, secant methods choose a step k ¼ sk at each iteration such that we point back to our previous iteration xk1. Because the step is changing size at each iteration, we may have a very poor approximation if is large. Typically, Newton’s method makes large steps in the early iterations, and then the steps get very small as it approaches the solution. This characteristic is good for us because as k gets smaller, ak becomes a better approximation to f 9(xk) and we get better directional information. For this method to work, we need just one extra function evaluation at the start to calculate a0. Usually, we compute a0 with a standard finite-difference calculation and small , like it is done in the finite-difference Newton’s method.

1.16.4.2

Minimization in One Variable

With a suitable background in solving nonlinear equations, we now return to optimization theory. Optimization is so closely related to solving nonlinear equations that we almost already know how to solve optimization problems. To introduce optimization theory, we continue with some simple one-dimensional examples. Consider this nonlinear function in one variable, f ðxÞ ¼ x 4 –

20 3 x þ 16x 2 – 16x þ 6 3

ð16Þ

Figure 2 shows this function over the range xP[0, 3]. By visual inspection, we see that the minimum of this function is at x ¼ 1. The key observation that allowed us to identify the minimum is that the slope of f (x) at x is zero, that is, the tangent line at x is flat. In fact, this characteristic holds for all minimization problems and actually forms a necessary condition. Points on f (x) that have a flat tangent line are called critical points or stationary points.

f (x ) 6 5 4 3 2 1 0 Figure 2 The function fðxÞ ¼ x that is not a minimizer. 4

20 3 3 x

x∗ 0

0.5

1

1.5

2

2.5

3

x

þ 16x 16x þ 6, which has a minimum at x ¼ 1 and another critical point at x ¼ 2 2


515

Explicitly stated, the minimum occurs at points such that f 9(x) ¼ 0. From Section 1.16.4.1, we already know how to solve a problem like this. Instead of finding the root of f (x), we wish to find the root of the function’s first derivative. That is, in the context of root finding, we wish to solve g(x) ¼ 0 where g(x) ¼ f 9(x). However, there are some subtle issues that are unique to optimization that do not arise when solving nonlinear equations. In particular, not every critical point of f (x) is a minimizer; it could be a maximum or not even an extreme point at all. To determine whether the critical point is a minimum or maximum, we need to examine the function’s curvature at x , which is the second derivative of f (x). This second characteristic is formally called a sufficient condition, which guarantees that x is a local minimizer. If d2 f =dx 2 jx > 0, then the point is a minimum, and if d2 f =dx 2 jx < 0, then the point is a maximum. If d2 f =dx 2 jx ¼ 0, then x is an inflection point and is neither a minimum nor a maximum. See Figure 3 for a graphical depiction of these conditions. The reader can verify that function (16) in Figure 2 has an inflection point at x ¼ 2. As there is no closed-form solution to solve a general nonlinear function, there is no closed-form solution to find the minimizer of an arbitrary function. Hence, we start with the iterative approach of Newton’s method for solving a nonlinear equation and adapt it for minimization. The modification of Newton’s method for nonlinear equations (10) and (11), when applied to optimization problems, is written here: xkþ1 ¼ xk þ sk

ð17Þ

– f 9ðxk Þ f 0ðxk Þ

ð18Þ

where sk ¼

Similarly, this algorithm is called Newton’s method for optimization. Once again, we compute the step direction sk and store it separately. The utility of this decision will become clear later. To illustrate some key points of optimization theory, we proceed with the example function in Equation (16), and solve the optimization problem min f ðxÞ x

We start with an initial guess of where we think the minimum might be. If the function is plotted as in Figure 2, then it is easy to make that determination and, in fact, may allow us to pick the minimizer very easily. If, however, we are just given the mathematical form of f (x) or we have even less information, such as a compiled function, call a black box, then the task is not as straightforward. We will choose x ¼ 0 as our initial guess. Without additional knowledge of the function, this is usually a reasonable starting point. Because we will iterate to find the solution, we denote this initial point as x0, and each subsequent iteration, denoted with increasing subscripts, hopefully will be an improvement over this initial point. f (x)

2

df dx

2

x∗

0 x

Figure 3 Second-order conditions of various critical points encountered in univariate optimization.

516


The function value at x0 is f (0) ¼ 6, and the slope of the tangent line is f 9(0) ¼ 16. Because x0 is not a critical point (f 9(x0) 6¼ 0), x0 cannot be at a minimum either. The slope indicates which direction we must travel to decrease f (x) and improve our initial guess. With a negative slope, we see that moving to the right (increasing x) would decrease f (x). The question is how far do we move in this direction? Newton’s method applied to the nonlinear equation f 9(x) ¼ 0 tells us the step should be sk ¼

– f 9ðxk Þ – ð – 16Þ ¼ ¼ 0:5 f 0ðxk Þ 32

which indicates that our next estimate of the minimum is 0.5 units to the right of x0. Hence, x1 ¼ 0.5. We check the first derivative there, f 9(x1) ¼ 4.5, and see that it is better but still not a critical point. Applying Newton’s method several more iterations, one arrives at x2 ¼ 0.8, x3 ¼ 0.95, and so on. Table 2 shows that the iterates are approaching a solution. To determine whether our final answer is a minimizer or not, we must check the first- and second-order conditions. For the final iterate x7, these values are f 9ðx7 Þ ¼ 0 f 0ðx7 Þ ¼ 4

This point is a critical point and has positive curvature. Therefore, x7 is a local minimizer and we label it x . In some cases, the difference between successive points grows so small that the function value hardly changes, or the slope is sufficiently close to zero for our needs. We may stop the algorithm at this point and say that the algorithm has converged. For this example, maybe we are satisfied with x5 0.99996 as our final solution because f 9(x) is nearly zero and is close enough to a critical point that we do not want to iterate any further. The decision of when to stop is left to the user in the form of convergence criteria. Stricter convergence criteria may dictate that we iterate a few more times with Newton’s method until we are satisfied with the solution. A very important quality of iterative methods deals with convergence rates (i.e., the rate at which the algorithm reaches the solution). For a sequence of iterates {x0, x1, x2, . . .}, the convergence rate is defined in terms of the error of successive iterates with respect to the actual solution x . For constants p > 1, c 0, and kˆ 0, if the sequence {xk} for all k kˆ obeys jjxkþ1 – x jj c jjxk – x jjp

then the sequence is said to converge to x with order p and constant c. For p ¼ 1 and p ¼ 2 the convergence is said to be linear and quadratic, respectively. In practice, linear convergence can be very slow if the constant c is large, whereas, quadratic convergence is eventually fast. Thus, a method that offers a quadratic convergence rate is more desirable than one that has linear convergence. Newton’s method has quadratic convergence (p ¼ 2), and this quality is evident in Table 2. After some point (around k ¼ 3), the error of xk from one iteration to the next is squared. In practice, this means that the number of correct digits in xk is doubling. Often, this same behavior is seen in the first derivative f 9(x), where in Table 2 the exponent of f 9(xk) is doubling at each iteration as xk approaches x . This quality is characteristic of Newton’s method. Table 2 Newton’s method for minimization of Equation (16) using doubleprecision floating point arithmetic (roughly 16 digits of accuracy) k

xk

f 9(xk)

0 1 2 3 4 5 6 7

0.00000000000000000 0.50000000000000000 0.80000000000000004 0.95000000000000018 0.99565217391304317 0.99996267960440366 0.99999999721468846 1.00000000000000000

1.6000000000000000eþ01 4.5000000000000000eþ00 1.1520000000000010eþ00 2.2049999999999770e01 1.7542861839402235e02 1.4929272509078828e04 1.1141246147872153e08 0.0000000000000000eþ00


517

f(x) 6 m (x 1 + s)

5 4 3 2

x1 x2

1 0

0

0.5

x∗ 1

1.5

2

2.5

3

x

Figure 4 A nonlinear function in one variable (solid line), and the local quadratic model at xk (dashed line).

At this stage, it is useful to step back and describe Newton’s method for optimization in more abstract terms and relate it back to solving nonlinear equations. At each iteration of Newton’s method, we are constructing a simplified local model of our function f (x) at point xk. In the nonlinear equation case, we construct a linear model of f (x) and solve it to find the step. In the optimization case, we are constructing a local quadratic model and moving to the minimum of it. This is represented graphically in Figure 4 for the local model at x1. In both cases, the local model may be derived by truncating the Taylor series to low-order terms. For nonlinear equations, we keep only up to linear terms in x (first two). Whereas for optimization, we keep up to quadratic terms (first three): 1 f ðxÞ f ðxk Þ þ f 9ðxk Þðx – xk Þ þ f 0ðxk Þðx – xk Þ2 2

The minimization of a quadratic function has a closed-form solution and is given by the same formula as in Equation (18). As was discussed previously for nonlinear equations, what if exact derivatives are not known? For nonlinear equations, only the first derivative is needed to compute the Newton step, but for optimization, both the first and second derivatives are needed in Equation (18). For the first derivative, we may compute an approximation by the first-order finite-difference formula, Equation (14). Because the first derivative of f (x), which appears in the numerator of Equation (18), is very important in computing the Newton step and is needed to determine convergence, a very good approximation is needed. Hence, a very small value of is needed. The numerical precision of computers actually limits how small this number can be before round-off errors accumulate and diminish its accuracy. In theory, the optimal value of is roughly the square root of machine precision, which is the spacing of floating point numbers. For pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi double precision, 2:22 10 – 16 ¼ 1:5 10 – 8 . In some situations, the first derivative may be available but not the second derivative. Or, if the first derivative is not available, then it is very likely that the second derivative is not available either. Thus, an approximation for the second derivative is needed as well. A finite-difference approximation of the second derivative may be computed by finite differencing the first derivative, f 0ðxk Þ

f 9ðxk þ Þ – f 9ðxk Þ f ðxk þ 2Þ – 2f ðxk þ Þ þ f ðxk Þ 2

ð19Þ

Even more accurate central difference formulas may be derived by centering on xk and evaluating f (xk ) as well. Here are two central difference formulas for the first and second derivatives: f ðxk þ Þ – f ðxk – Þ f ðxk þ Þ – 2f ðxk Þ þ f ðxk – Þ f 0ðxk Þ 2

f 9ðxk Þ

518


These formulas have second-order accuracy. As a final point to be considered for univariate optimization, we point out that the iterative process of Newton’s method is typical for these sorts of algorithms. We generate a sequence of points that hopefully get closer to the solution. But it is not guaranteed that each step does get closer. What if one or more steps actually move us away from our goal? There are two cases that must be considered. First, the local model must itself be a proper surrogate model for the problem. If the local quadratic model has a maximum, then the Newton step (18), will move to the maximum of this model. It turns out that this concept is an important restriction that limits a straightforward implementation of secant methods for optimization. This restriction has to do with ensuring that the local model has a minimizer. Second, if the local model is a good approximation very near to xk and yet the Newton step overshoots this region, then one must scale back in some manner. One simple idea is to backtrack along the step already computed by scaling the step sk by some < 1, xkþ1 ¼ xk þ sk

ð20Þ

Both of these cases will be discussed more thoroughly next in the context of multivariate optimization. 1.16.4.3

Minimization in Multiple Variables

After witnessing some key principles on one-dimensional examples, we now proceed to functions with more than one variable. Many of the basic concepts still apply, but the mathematics become more complex in multiple dimensions. In one dimension, the basic procedure was to (1) construct a quadratic local model of the function using the first and second derivatives of f (x), or their approximations, (2) solve this model exactly for the Newton step sk, and (3) add this step to the current estimate xk to improve (hopefully) the estimate of the minimizer. This basic framework still applies; what changes is the use of multivariable calculus to calculate derivatives and linear algebraic techniques to solve the local model. Now we consider a continuous function f (x1, x2, . . ., xn) that returns a single value, f : R n ! R. We typically just write the collection of n variables together as a column vector, x ¼ (x1, x2, . . .,xn)T. The gradient is the vector of first partial derivatives of f with respect to the n variables. By convention, we use the symbol ‘r’ to represent the multidimensional partial differential operator and order these n partials in a column vector: 1 qf B qx1 C C B C B B . C rf ðxÞ ¼ B .. C C B C B @ qf A qxn 0

The ith element of the gradient rf (x) measures the sensitivity of f to component xi. In the univariate case, rf ðx Þ ¼ df =dx. The Hessian is the matrix of second partial derivatives of f with respect to n2 variables. We use the symbol ‘r2’ to represent the multidimensional second partial differential operator and order these n2 partials in a matrix: 0

q2 f B qx 2 B 1 B B q2 f B B B qx2 qx1 2 r f ðxÞ ¼ B B B .. B . B B B 2 @ qf qxn qx1

q2 f qx1 qx2 q2 f qx22 .. . q2 f qxn qx2

1 q2 f qx1 qxn C C C C 2 qf C C qx2 qxn C C C .. C . C C C C 2 qf A qxn2


519

The i, jth element of the Hessian r2f (x) measures the curvature of f with respect to component xi and xj . In the univariate case, r2 f ðx Þ ¼ d2 f =dx 2 . We say that the function f is differentiable if all first derivatives of f exist and that f is continuously differentiable if these derivatives are also continuous. Moreover, f is twice differentiable if all second derivatives exist and twice continuously differentiable if these derivatives are also continuous. These definitions may also apply over some region of x. When f is twice continuously differentiable, the Hessian matrix is symmetric because q2 f q2 f ¼ qxi qxj qxj qxi

for all

i;j ¼ 1;2; . . .; n

This fact makes the linear algebra of working with the Hessian simpler. One other definition is needed for multivariate optimization. It is the directional derivative. If f is continuously differentiable and p P R n is a nonzero perturbation vector, then the directional derivative of f in the direction of p is defined by qf f ðx þ pÞ – f ðx Þ U lim ¼ rf ðx ÞT p !0 qp

ð21Þ

The directional derivative is needed to determine whether the Newton step (or some variant) points in a ‘downhill direction’. In fact, a class of algorithms called descent methods are characterized by their choice of p that points ‘downhill’. When applied to maximization problems, often these same methods are called hill-climbing methods. As we discovered for univariate functions, a local minimizer may be found at the critical points of f, such that f 9(x ) ¼ 0. This is a necessary condition, but, as we learned, it is not sufficient by itself. In addition, it is sufficient if f 0(x ) > 0 and necessary that f 0(x ) 0. By extension, for a multivariate function, the equivalent set of statements is that a minimum may be found where the gradient of f is the zero vector, rf (x ) ¼ 0. A point with zero gradient may be a minimum, a maximum, or a saddle point. The first two cases are straightforward: moving in any direction from x will move us away from optimality, so we are either in a depression or on a peak. The saddle point is a special case. In two dimensions, it can be visualized as the lowest point on a saddle between two mountain tops: moving in one coordinate direction will move us up, while in the other it will move us down. This analogy may be extended to more dimensions. These statements may be expressed in linear algebra by examining the quantity d T r2 f ðx Þd

ð22Þ

for all nonzero directions s P R . If Equation (22) is always positive, then x is a minimum. If Equation (22) is always negative, then x is a maximum. And if Equation (22) is sometimes positive and sometimes negative, then x is a saddle point. In linear algebra terms, these mathematical statements may be replaced by a statement about the ‘definiteness’ of the Hessian matrix. Let A P Rnn be symmetric. If sTAs > 0 for all nonzero s P R n , then A is positive definite. If sTAs 0 for every nonzero s P R n , then A is positive semidefinite. Similarly, A is negative definite or negativesemidefinite if sTAs < 0 or sTAs 0, respectively. Finally, A is indefinite if it is neither positive semidefinite nor negative semidefinite. It can be shown that A is positive definite if, and only if, all of its eigenvalues i are positive. Consequently, if A is positive definite, then so is A1 because A1 has eigenvalues i– 1 . In summary, Table 3 lists the necessary and sufficient conditions for unconstrained multivariate minimization. Hence, given these conditions, x is a local minimizer if rf (x ) ¼ 0 and r2f (x ) is positive definite. The reader should notice that these conditions in the multivariate case still apply to the univariate case. A zero gradient in multiple dimensions reduces to a zero slope in one dimension, and a positive definite Hessian n

Table 3 Necessary and sufficient conditions for multivariate minimization First-order necessary condition Second-order necessary condition Second-order sufficient condition

rf (x ) ¼ qf/qx ¼ 0 r2f (x ) ¼ q2f/qx2 positive semidefinite r2f (x ) ¼ q2f/qx2 positive definite

520


in multiple dimensions reduces to positive curvature in the one-dimensional case (d2f/dx2s2 > 0 for any nonzero s P R is equivalent to d2f/dx2 > 0). So, indeed, the conditions in Table 3 hold for any number of dimensions. It so happens that in the univariate case there are fewer subtleties, which makes the math simpler. 1.16.4.3.1

Newton’s method We now revisit the iterative paradigm of Newton’s method that we had considered for univariate minimization in Section 1.16.4.2. To reiterate, the idea is to start from some initial guess x0 and solve a series of easier subproblems on local models to determine a sequence of steps that (hopefully) lead to increasingly better estimates of the minimizer, x . To construct our local model of f, we revisit the Taylor series approximation for multivariate functions. Here, we write the Taylor series expansion of f in terms of some known point x plus some arbitrary step s P Rn : 1 f ðx þ sÞ ¼ f ðxÞ þ rf ðxÞT s þ s T r2 f ðxÞs þ 2

This Taylor series expansion involves vector and matrix algebra because the gradient is a vector and the Hessian is a matrix; (see Figure 5). The higher-order derivatives of f in the Taylor series are nth order tensors, so the math to compute these terms would involve tensor–vector products. We construct a local model m(xk þ s) by truncating the Taylor series approximation to include nothing higher than the quadratic terms in s, 1 mðxk þ sÞ ¼ f ðxk Þ þ rf ðxk ÞT s þ s T r2 f ðxk Þs 2

ð23Þ

In principle, because this is a quadratic model, it may be possible to determine the exact minimizer, if one exists, with some simple linear algebra. According to our necessary and sufficient conditions in Table 3, a local minimizer exists where the gradient is zero and the Hessian is positive definite. The second condition on the Hessian is what makes life more difficult for multivariate optimization, as we shall see. To begin, we wish to solve a local subproblem min mðxk þ sÞ s

where m(xk þ s) is defined by Equation (23). If m(xk þ s) has a unique local minimizer, then we find s by differentiating with respect to s, setting it equal to zero, and solving the resultant system of linear equations for s. According to multivariable calculus, this differentiation is simply rs mðxk þ sÞ ¼ rf ðxk Þ þ r2 f ðxk Þs ¼ 0

Thus, the basic outline of Newton’s method for minimization of a multivariate function is as follows: solve for sk in the linear system: r2 f ðxk Þsk ¼ – rf ðxk Þ xkþ1 ¼ xk þ sk

ð24Þ ð25Þ

Figure 6 shows one iteration of this process for a general two-dimensional problem. The function f (x) and local model m(xk þ s) are represented by level sets, which are contours in the space of x such that f (x) or m(xk þ s) has the same value. We digress, for a moment, to discuss an expedient way to solve the linear system in (24), by factorizing the matrix into upper and lower triangular factors and using these to solve two linear systems by successive

f(x + s) ≈ f (x )

+

+

∇f (x)T

1 2

sT

s

Figure 5 First three terms of the Taylor series expansion of f.

∇2 f (x )

s


521

x2

sk xk

x1 Figure 6 A general nonlinear function in two variables. Level sets of f (x) are solid lines and level sets of the local model at xk are dashed lines.

substitution. Specifically, for a general linear system Ax ¼ b, one computes the LU factorization of A, such that A ¼ LU, where L is lower triangular and U is upper triangular.2 This factorization costs n3/3 operations. Then the solution is found by solving first for y and then for x in the following two linear systems: Ly ¼ b Ux ¼ y

Triangular systems such as these are easy to solve. Overall, the dominant cost is the factorization, which is O (n3). By comparison, computing the inverse of A and multiplying it with b to get x would cost O (n4). However, for a symmetric matrix APRnn , it is better to compute the Cholesky factorization of A, such that A ¼ LLT, where L is lower triangular.2 This factorization exploits the symmetry of the matrix, costing n3/6 operations and also using less memory. Similarly, the solution is found by solving first for y and then for x in the following two linear systems: Ly ¼ b LT x ¼ y

Once again, the dominant cost is the factorization, which is O (n3). The algorithm in steps (24)–(25) shares many characteristics with the method already discussed for univariate minimization. At this point, it is necessary to discuss some important considerations for multivariate minimization that were alluded to previously in the single variable case. At each iteration, Newton’s method is solving a local quadratic model for a critical point. At this stage, there is nothing in the algorithm to guarantee that it will move toward the minimizer of f. This is a consequence of the fact that the Hessian matrix at each iterate, rf (xk), may not always be positive definite. Indeed, it could be negative definite, which means that our local model has a maximum and the step subsequently computed leads to a local maximum and, most likely, away from a minimum of f. Thus, it is imperative that we modify the algorithm if the Hessian r2f (xk) is not sufficiently positive definite. A proven approach is to change the local model so that it has a local minimizer and use this minimizer as our step sk. In other words, we need to modify the Hessian in a principled manner to make it positive definite. As mentioned previously, a property of symmetric positive-definite matrices is that their eigenvalues are all larger than zero. Knowing this fact, we may use the following theorem: Theorem:2 Let matrix APR nn have eigenvalues i, i ¼ 1, . . ., n. Then the eigenvalues of A þ I, for PR, are i þ , i ¼ 1, . . ., n. This theorem indicates that if the eigenvalues of the Hessian are not all positive (i.e., r2f (xk) is not positive definite), then we may add some positive multiple of the identity matrix I to the Hessian to make it positive definite.

522


Unfortunately, computing the eigenvalue decomposition is relatively expensive, so another approach would be helpful. It turns out that the Cholesky factorization can be used to determine whether a matrix A is positive definite and how to modify A if it is not, which is the idea behind the so-called modified Cholesky factorizations. The diagonal elements of L will all be positive if and only if A is positive definite. Thus, in the process of computing the Cholesky factorization of the Hessian, one may keep a tally of how much ‘extra’ is needed to keep the diagonal values of L positive and use that measure as a gauge for I to make A þ I positive definite. In practice, this means that we must factorize the Hessian matrix once to see if it is safely positive definite (according to some threshold) or, if it is not, increase the diagonal elements of the Hessian to ensure that the diagonal elements of L are sufficiently positive. If the diagonal elements were modified, then we use these values to determine a value for that we add to the Hessian along its diagonal to make it safely positive definite and re-factorize. In either case, we are guaranteed that r2f (xk) þ I for 0 is positive definite. By using the modified Cholesky factorization on the Hessian matrix, we can ensure that the local model has a minimizer. This modification to Newton’s method is summarized in Algorithm 1. The modified Cholesky factorizations of Schnabel and Eskow3 and Gill et al.4 are two popular algorithms. Algorithm 1: Modified Newton’s method (1) Apply a modified Cholesky factorization to r2f (xk) to find the Cholesky factors of Hk ¼ r2f (xk) þ I, where ¼ 0 if r2f (xk) is sufficiently positive definite, or > 0 such that Hk is sufficiently positive definite. (2) Solve for sk in the linear system: Hk sk ¼ – rf ðxk Þ ð3Þ Update x: xkþ1 ¼ xk þ sk

ð26Þ ð27Þ

In effect, when the Hessian is not positive definite, the modified Newton’s method computes a constant > 0 that biases the Newton step toward the steepest descent direction, rf (xk).

1.16.4.3.2

Quasi-Newton methods There are situations when a procedure for computing the Hessian is not available or when dealing with problems containing a large number of variables (in which case it may not be practical to compute exact Hessians and/or solve linear systems involving the exact Hessian). In such cases, the algorithms of Section 1.16.4.3.1 need to be modified in the same spirit as secant methods for nonlinear equations, which were described in Section 1.16.4.1. These methods are called quasi-Newton methods because they use the same algorithmic framework of Newton’s method, except that they use secant approximations for the Hessian matrix. The fundamental idea is that instead of recomputing a fresh Hessian at every iteration, we use the most recently observed information of f (x) to update our best approximation of the Hessian thus far. Quasi-Newton methods belong to a class of methods that do not require second partial derivatives, yet still have respectable performance. The most popular of the quasi-Newton methods is the BFGS method, so named after its founders Broyden, Fletcher, Goldfarb, and Shanno.5 Another popular updating formula is the symmetric rank-1 (SR1) update5 (see also Khalfan et al.6). We will derive a close relative of the BFGS method, called the DFP method, named after Davidon, Fletcher, and Powell5, and describe its implementation and extension as the BFGS method. We begin with an approximate local model of the objective function f that does not involve the Hessian matrix, 1 mðxk þ sÞ ¼ f ðxk Þ þ rf ðxk ÞT sk þ skT Hk sk 2

ð28Þ

where Hk PR nn is a symmetric positive-definite matrix that approximates the exact Hessian r2f (xk) in some manner. The basic premise of secant methods for unconstrained optimization is that at each iteration, we use curvature information measured from the most recent point, xk1, to update the previous estimate Hk1. There are a number of requirements that we must impose to construct the local model (28) from the previous local model at xk1. The first condition is that the gradient of m(xk þ s) should match the gradient of f at the two most recent points, xk and xk1. The first of these is satisfied automatically because rm (xk) ¼ rf (xk). Assuming


523

that sk1 ¼ xk xk1, the second condition is that the gradient of the current model at the previous point is equal to the actual gradient, rmðxk – 1 Þ ¼ rmðxk – sk – 1 Þ ¼ rf ðxk Þ – Hk sk – 1 ¼ rf ðxk – 1 Þ

After rearranging the terms, we obtain Hk sk – 1 ¼ rf ðxk Þ – rf ðxk – 1 Þ

ð29Þ

yk ¼ rf ðxk Þ – rf ðxk – 1 Þ

ð30Þ

Hk sk – 1 ¼ yk

ð31Þ

To simplify the notation, let

then Equation (29) becomes

We will refer to Equation (31) as the secant equation for the multivariate minimization problem. Given the previous step sk1 and the difference in gradients yk, we wish to find a symmetric Hk that obeys the secant equation constraint (31). The problem, however, is that there are an infinite number of solutions because this system is underdetermined – there are n(n þ 1)/2 unknowns in the symmetric matrix and only n equations represented in Equation (31). We must impose additional constraints for a fully determined problem. One reasonable approach is to find the Hk that is, in some sense, closest to its predecessor, Hk1. It is reasonable to determine the ‘closeness’ in terms of the Frobenius norm, but other norms could be chosen, particularly if scaling is an issue (see, e.g., the weighted Frobenius norm in Powell7). All together, we may pose this update problem as an optimization problem, min jjHk – Hk – 1 jjF Hk

subject to Hk ¼ HkT ;

Hk sk – 1 ¼ yk

ð32Þ

where yk satisfies Equation (30) and Hk1 is symmetric and positive definite. It turns out that the unique solution to problem (32) is Hk ¼

yk s T sk – 1 y T yk y T I – T k – 1 Hk – 1 I – T k þ T k y k sk – 1 y k sk – 1 y k sk – 1

ð33Þ

This formula is called the symmetric positive-definite update or the DFP update. The reader can verify that Equation (33) obeys Hksk1 ¼ yk and that Hk Hk1 is a symmetric matrix of rank at most two. Following the previous discussion of Newton’s method, the minimizer of the local model (28) is the solution sk to the linear system Hk sk ¼ – rf ðxk Þ

ð34Þ

which we use to update xk to xkþ1. An important consideration of these low-rank secant update formulas is that they can apply to either Hk or to the inverse of Hk (or even the Cholesky factorization, Hk ¼ Lk LT k ). This is important because the quasi-Newton method only needs Hk in the linear system (34). Therefore, if instead of Hk we have Hk– 1 , then the solution of this linear system reduces to a fast and easy matrix–vector product. This is the idea of the BFGS update, which is considered to be the best of all quasiNewton updates, and is very closely related to the symmetric positive-definite update in Equation (33). Let Bk be the inverse of an approximation of the Hessian matrix, Hk. If Hk is positive definite, then its inverse Bk ¼ Hk– 1 will also be positive definite. Following problem (32), we wish to find the inverse Hessian approximation Bk that is closest to its predecessor, Bk1, in the Frobenius norm. We pose this problem mathematically as min jjBk – Bk – 1 jjF Bk

subject to Bk ¼ BkT ;

Bk yk ¼ sk – 1

ð35Þ

524


where yk satisfies Equation (30) and Bk1 is symmetric and positive definite. The unique solution to problem (35) is Bk ¼

sk – 1 y T yk s T sk – 1 s T I – T k Bk – 1 I – T k – 1 þ T k – 1 sk – 1 yk sk – 1 yk sk – 1 yk

ð36Þ

This formula is called the BFGS update.5 The reader can verify that Equation (36) obeys Bkyk ¼ sk1 and that Bk Bk1 is a symmetric matrix of rank at most two. To complete the quasi-Newton algorithm, one final piece of information is needed – the initial approximation H0 for Equation (33) or B0 for Equation (36). Usually, this initial estimate is either a finite-difference approximation to the Hessian (or its inverse) or some multiple of the identity. As a final note, the BFGS update is useful for large-scale applications where memory and computational cost become prime considerations for performance. For large-scale problems, the Hessian matrix may be too large to fit in memory, or, more commonly, the solution to the linear system involving the explicit Hessian matrix may be too costly to obtain using standard techniques. The limited-memory BFGS method (L-BFGS) addresses this issue by only calculating an approximate inverse Hessian.8,9 L-BFGS applies the standard BFGS update formula for m 1 recent steps to update a simple initial matrix B0 that approximates the inverse of the Hessian. When k > m, Bk is obtained by applying m BFGS updates to B0 using information from the m previous iterations. In this manner, only a small multiple of m vectors are needed to describe Bk.

1.16.5 Globalization Strategies In our discussion of optimization methods up to this point, it was assumed that the step calculated by the various methods would make progress toward a local minimizer. This is not guaranteed in practice, however. Steps computed by Newton’s method or, especially, the quasi-Newton methods may actually diverge from the solution and increase f (x). This is something that we wish to avoid. Given any starting point x0, we would like the optimization method to return a valid answer and not just head off in the wrong direction to infinity or get lost in the process. We start with a discussion on local versus global convergence. Essentially, a method is locally convergent if it is guaranteed to arrive at a solution when xk is local to a region containing x (i.e., xk is close enough to x ). Global convergence, on the other hand, does not have this local restriction – it can converge to a solution from anywhere. It is important to understand that global convergence has nothing to do with global optimization and finding the global optimum. The language was precise that we may converge to a solution and not the solution. Global optimization will be considered briefly in Section 1.16.7. The distinction between local and global convergence is important. In theory, Newton’s method is locally convergent and exhibits quadratic convergence, but this happens only when the current estimate is close enough to the solution. If the current estimate is far from the solution, then the steps might actually overshoot the solution or point in a poor direction. The radius of this local region of convergence depends on the nature of the particular function, for example, how bumpy or smooth it is, which depends greatly on higher-order derivatives of f (x). Explicitly quantifying this distance or determining whether xk is in this region are things that we wish to avoid. Instead, we would prefer a general optimization method that may start from any given point and return a solution (or at least state that no minimum could be found if the function is not bounded from below). That is, we want a globally convergent strategy coupled with a fast local optimization method. They should be coupled in a way that we make progress toward a solution when xk is outside of that local region and, yet, we still get fast convergence when xk is in that local region. This section discusses two globally convergent modifications for Newton’s method for handling spurious steps that would otherwise lead us away from the solution. These situations arise when the local model of the objective function does not accurately characterize f (x) very far from xk and that the step sk calculated from this model either overshoots or points in a poor direction. Both strategies still rely on the local quadratic model m(xk þ s), but they use this model in different ways.


525

The first strategy is the line search method, which uses the local model to compute a search direction and then turns to the actual function for identifying a step length along this direction to make satisfactory progress. The second strategy is the trust region method that uses the local model to generate simultaneously an optimal search direction and step length based on how much ‘trust’ we place in the local model. If this step is not satisfactory on the actual function, then the radius of trust in the local minimizer is reduced and a new step is calculated. The two global strategies rely on the concept of a descent direction, which was introduced in Section 1.16.4.3.1. Hence, before discussing these ideas, it is helpful to discuss the steepest descent method. 1.16.5.1

Steepest Descent

The most basic of algorithms for unconstrained optimization is called the steepest descent method. The idea is simple and intuitive, but it turns out that it is not very good in practice, which is why it was not discussed earlier. The steepest descent method has linear convergence, which is far slower than Newton’s method. So far, we have only considered the Newton step, sk ¼ – Hk– 1 rf ðxk Þ, where Hk is either r2f (xk) or an approximation to it. This step is a descent direction on f if and only if its directional derivative (21) is negative, rf ðxk ÞT sk ¼ – rf ðxk ÞT Hk– 1 rf ðxk Þ < 0

which is true if Hk– 1 is positive definite. By the definition of positive-definite matrices, if Hk is positive definite, then so is Hk– 1 , which further explains why we always enforce Hk to be positive definite. However, it may come as a surprise that the direction of the Newton step does not decrease f the most rapidly from x (We define the direction of any step d as d =jjd jj2 . When we refer to a direction of some vector d, we consider this step that is normalized to have unit length.). To determine the direction that has the most rapid descent on f, we must examine the directional derivative of some step that is normalized to unit length. Specifically, we are interested in the following optimization problem: minn rf ðxÞT d d R

subject to jjd jj2 ¼ 1

which has the solution d ¼ – rf ðxÞ=jjrf ðxÞjj2 and is called the steepest descent direction. Stated in a different way, the gradient rf (x), by definition, provides the direction of steepest ascent on the function at the point given by x. That is, among all possible directions from x, it is the direction that most rapidly increases f. Hence, it stands to reason that if our goal is to minimize f, we proceed in the opposite direction, rf (x), which would be the direction of steepest descent. It is important to note, however, that rf (x) does not say anything about how far we should go in this direction. Therefore, our iterative method, based on steps (24)–(25), would be xkþ1 ¼ xk – rf ðxk Þ

ð37Þ

where in this case sk ¼ r f (xk). Algorithm 2 shows a version of the classic minimization algorithm called the method of steepest descent. The computation of each step contains a one-dimensional minimization problem. Some approaches for approximately solving this problem are explained in Sections 1.16.5.2 and 1.16.5.3. Algorithm 2: Steepest descent method (1) For k ¼ 1, 2, . . ., kmax, do: (a) Calculate the steepest descent direction sk ¼ rf (xk). (b) Find a step length that (approximately) minimizes f (xk rf (x)). (c) Set xkþ1 ¼ xk þ sk. This algorithm raises a basic question: how far do we move in the direction of rf (xk)? In other words, what is ? One approach is to search along this direction with different values of for a new point with a lower function value. This is the idea behind the line search global strategy, and there are a number of ways to choose an

526


appropriate step length , which we discuss in Section 1.16.5.2. Another approach is to find the optimal step length that minimizes f (xk þ sk) or possibly the local model m(xk þ sk). The latter forms a piece of the trust region global strategy, which we discuss in Section 1.16.5.3.

1.16.5.2

Line Search Methods

Each iteration of a line search method computes a step sk and then finds a value of for which the new point has a lower function value. The iteration is given by xkþ1 ¼ xk þ sk

ð38Þ

where > 0 is the line search parameter and determines the step length. A general line search implementation requires sk to be a descent direction, such as the modified Newton step (26) or the steepest descent direction, which guarantees that f ðxkþ1 Þ < f ðxk Þ

ð39Þ

as ! 0. However, it turns out that the condition (39) is not sufficient to guarantee that the iterates converge to a minimizer of f. It could be that the sequence of iterates continually decreases f (x) but approaches an asymptotic limit that is not a minimizer. This rarely happens in practice, but it is not precluded by theory and, therefore, something that we must guard against. Instead, to encourage convergence to a minimizer, we must require the notion of sufficient decrease. Furthermore, we must also introduce a curvature condition to prevent the steps from becoming too small. There are a number of popular sets of these conditions. Here we consider the traditional Armijo and Wolfe conditions because they are easy to understand and simple to implement. We will consider separately two situations that illustrate these conditions. First, we do not want very small decreases in function values for long step lengths. The implication is obvious. If a step is a descent direction on f but never makes substantial progress toward the minimizer by decreasing f, then the algorithm will stall and never reach the minimizer. We can fix this by requiring that the decrease from f (xk) to f (xk þ 1) be at least proportional to the step length and the initial rate of decrease in the given direction (i.e., the directional derivative at xk). That is, we only accept the step if f ðxk Þ – f ðxkþ1 Þ – 1 rf ðxk ÞT sk

ð40Þ

f ðxkþ1 Þ f ðxk Þ þ 1 rf ðxk ÞT sk

ð41Þ

or, equivalently, if

where 1 P (0,1) is a parameter that typically is set to 104. This inequality is usually known as the sufficient decrease condition and is sometimes called the Armijo condition based on the work of Armijo.10 Second, we do not want steps that are too short relative to the initial rate of decrease of the function. This is the counter argument to the first condition. It makes sense that if the step is getting progressively shorter relative to the projected decrease, then we will not reach a minimizer either. We can fix this by requiring that the rate of decrease in the direction sk at the current point xkþ1 be proportionally larger than the rate of decrease at the current point xk. That is, we only accept the step if rf ðxkþ1 ÞT sk 2 rf ðxk ÞT sk

ð42Þ

where 2 P ( 1, 1) is an algorithmic parameter. This equation is known as the curvature condition. Collectively, the sufficient decrease and curvature conditions (41) and (42), are known as the Wolfe conditions.11,12 These conditions are represented graphically in Figure 7, which shows the region of acceptable choices of .


527

f(x k + αs k ) f (x k ) + τ1α ∇f (x k )Ts k

τ2 ∇f (x k )Ts k ∇f(x k )Ts k Acceptable

α

Figure 7 Acceptable choices of step length according to the Wolfe conditions (41) and (42).

Together, the Wolfe conditions ensure that the optimization algorithm makes sufficient progress toward the minimizer. However, by designing an appropriate line search method, it turns out that under mild assumptions one can satisfy the Wolfe conditions without explicitly checking condition (42) at each trial point. This is accomplished with a backtracking scheme that tries the longest step first against condition (41) and systematically shortens any subsequent steps. In this manner, the backtracking strategy avoids excessively long steps and the second condition (42) can be ignored. If the full Newton step is tried first, then the fast quadratic convergence properties of Newton’s method can be preserved in a strategy for global convergence. This is an important consideration that provides both speed and robustness. A basic backtracking strategy is to first try the full step (Newton or some other descent direction) and then repeatedly shorten the step length by half until condition (41) is satisfied. This strategy works reasonably well on most problems, but more sophisticated schemes can provide even better performance. The problem is that if the full step is especially poor and exceedingly long (which can occur, e.g., when the Hessian is nearly singular), then this basic strategy may try a large number of trial steps before cutting down to a reasonable size before condition (41) even has a chance of being satisfied. A better and more popular backtracking scheme is based on polynomial interpolation. After computing a step sk and computing the function at the first trial point, we can begin to construct a polynomial model of the function in the direction of sk. A common approach is to model the one-dimensional function

ðÞ U f ðxk þ sk Þ

by a quadratic polynomial and find its minimum in . Initially, we have two pieces of information about (),

ð0Þ ¼ f ðxk Þ

and

9ð0Þ ¼ rf ðxk ÞT sk

If the full step sk with ¼ 1 does not satisfy (41), then we have a third piece of information,

ð1Þ ¼ f ðxk þ sk Þ

These three pieces of information are enough to construct a quadratic model of (), qðÞ ¼ ð0Þ þ 9ð0Þ þ ð ð1Þ – ð0Þ – 9ð0ÞÞ2

ð43Þ

The minimum of q() occurs at the point ¼

– 9ð0Þ 2ð ð1Þ – ð0Þ – 9ð0ÞÞ

ð44Þ

This new step length must be safeguarded to prevent excessively small steps of the sort that might fail condition (42) or steps so large that too many backtracks are needed before condition (41) is met. Therefore, in practice, we bound the reduction of the step length, ¼ new =old , to the interval [low, high]. If the minimizer

528


falls outside of this interval, then we reduce by low or high instead. Note that the minimum of Equation (43) is 12, which effectively provides an implicit upper bound of high ¼ 12. A practical lower bound low might be 1/10. If the trial point fails and further backtracking is necessary, then two options are available. The quadratic interpolating polynomial may be updated with the last trial point information, or a cubic polynomial model of

() may be constructed from the last two trial points. The benefit of using a cubic polynomial is that it can more accurately model situations where f has negative curvature. However, in most situations a quadratic interpolating model is generally fine and has a good balance between performance and coding complexity. We continue with the quadratic model; more information on a cubic interpolating polynomial is available in Dennis et al.13 or Kelley.14 To simplify the algorithmic framework for subsequent reductions, a new quadratic model q() may be constructed on the interval P [0, 1], where represents the fraction to reduce and ¼ 1 is the most recent trial point. The quadratic model is identical to Equation (43), and the minimizer of q() may be determined from the same Equation (44). The complete algorithmic framework of a backtracking line search strategy is shown in Algorithm 3. Algorithm 3: Backtracking line search strategy (1) Given step sk, which is a descent direction on f, and line search parameters 1 P [0, 12], 0 < low < high < 1: (a) Set ¼ 1. (b) Evaluate f (xk þ sk). (c) While f (xk þ sk) f (xk) þ 1rf (xk)Tsk, do i. Update for some P [low, high] (d) Set xkþ1 ¼ xk þ sk.

1.16.5.3

Trust Region Methods

The other global strategy that we consider is a trust region method, which employs a different philosophical model than a line search method. Whereas line search methods deal with finding an acceptable step length in a given search direction (usually the Newton direction) and search exclusively on that line, trust region methods allow the direction to change as the steps are shortened. Trust region methods even allow the use of complete Hessian information. The idea is rather simple. We define as the distance from xk in which the local model 1 mðxk þ sÞ ¼ f ðxk Þ þ rf ðxk ÞT s þ s T Hk s 2

can be trusted to adequately model f. The trust region is defined as the ball about xk such that jjx – xk jj2 ¼ jjs jj , where is called the trust region radius (Trust region methods can handle the case Hk ¼ r2f (xk), even if the Hessian is not positive definite, but here we assume that the model Hessian Hk is symmetric and positive definite.). We desire the optimal point that best minimizes the local model to the extent that we have established some level of trust in the model, represented by . Mathematically, we want the (approximate) minimizer of 1 mins mðxk þ sÞ ¼ f ðxk Þ þ rf ðxk ÞT s þ s T Hk s 2 subject to jjs jj2

ð45Þ

We use this solution as the step toward our next trial point, but we do not scale the step length as we have with line search methods. Instead, we test both the step and the trust region radius and decide whether to accept the trial point, the trust region radius, or both. This basic framework of the trust region strategy is given in Algorithm 4.


529

Algorithm 4: Trust region strategy (1) Given an initial trust region radius 0 > 0: (a) Initialize the trust region radius . (b) Do until xkþ1 is an acceptable next point: (i) Compute an approximate solution sk to problem (45) for the current trust region radius k. (ii) Set xkþ1 ¼ xk þ sk. (iii) Decide whether xkþ1 is acceptable and/or calculate a new value of k. (c) Set kþ1 ¼ k. The solution to Problem (45) is sk ðÞ ¼ – ðHk þ I Þ – 1 rf ðxk Þ

ð46Þ

such that the step length jjsk ðÞjj equals for the unique 0, unless jjsk ð0Þjj < , in which case ¼ 0. Unfortunately, there is no finite method for determining the unique > 0 in the first case. Therefore, problem (45) is solved only approximately, which turns out to have perfectly acceptable performance in practice. There are many algorithms for obtaining an approximate solution to problem (45). For example, the locally constrained optimal (or ‘hook’) step13 finds a such that jjsk ðÞjj and sets xkþ1 ¼ xk þ sk () (see Figure 8). The one that we will discuss in detail is the dogleg step, which solves problem (45) along a piecewise linear path that approximates the parametric curve sk(). Figure 8 shows the path sk() traced out as ¼ 1 ! 0. When ¼ 0, the optimal solution is the Newton step. As approaches 1, the step retreats back to the steepest descent direction in the asymptotic limit. The dogleg step is a step that spans these two directions. Specifically, as the trust region radius expands from zero, the dogleg step proceeds in the steepest descent direction to a point and then changes direction to end up at the Newton direction. The question, though, is how far should one go in the steepest descent direction? Unlike the Newton step, there is no ‘natural’ step length associated with the steepest descent direction, rf (xk). As discussed previously in Section 1.16.5.1, one approach is to find the optimal step length that minimizes the local model m(xkrf (xk)), where the s is now the steepest descent direction. The minimization problem is then 1 min mðxk – rf ðxk ÞÞ ¼ f ðxk Þ – jjrf ðxk Þjj22 þ 2 rf ðxk ÞT Hk rf ðxk Þ 2

which has the unique solution ¼

jjrf ðxk Þjj22 rf ðxk ÞT Hk rf ðxk Þ

This particular point is called the Cauchy point, which we label as xkCP ¼ xk – rf ðxk Þ x2

–∇f(x k )

xk

s(μ ) – H k–1 ∇f(x k )

x1 Figure 8 The locally constrained optimal (or ‘hook’) step, sk ().

530


The dogleg curve is the piecewise linear path from the current point xk to xkCP and then changing directions to the Newton point, xkN ¼ xk – Hk– 1 rf ðxk Þ

Figure 9 shows this path, which is an approximation to the exact path defined by Equation (46). A trust region method that restricts its solution to the dogleg path is much easier to solve. It requires only computations of the Newton and Cauchy points and then a determination of where the step length jjs jj2 equals . This calculation may be handled more easily by first determining the line segment to which the trust region step belongs. If < xkCP – xk , then the step is along the steepest descentdirection. Otherwise, the step is somewhere along the path from the Cauchy point to the Newton point. If > xkN – xk , then the chosen step is the Newton step. A slightly more complicated path is the double dogleg step, which consists of three line segments, depicted in Figure 10. This path has an extra ‘kink’ on the way from the Cauchy point to the Newton point, to bias the step more aggressively to the Newton direction and take advantage of the faster convergence properties of Newton’s method. This extra point lies in the Newton direction such that the step length increases monotonically along the double dogleg path. The remaining pieces of the trust region strategy left to discuss are step acceptance and the procedure for updating the trust region radius. These are usually tested simultaneously. The basic philosophy is that we examine how well the local model agrees with the function after testing a trial point. There are a number of schemes for deciding this and we consider just one. If the trial point xkþ1 is not accepted according to the Armijo condition (41) (and hence the local model in the trust region has poor agreement with f ), then we reduce the trust region radius and compute another step by minimizing the same quadratic model on the smaller trust region. The approach to updating is similar to a x2

–∇f (x k ) x kCP

xk –H k –1 ∇f (x k )

x1 Figure 9 The dogleg step.

x2

–∇f (x k ) x kCP

xk – H k–1 ∇f (x k )

x1 Figure 10 The double dogleg step.


531

backtracking line search, except that we consider the model instead of the step. If the trial point xkþ1 is accepted, then we must decide whether should be increased, decreased, or kept the same for the next iteration. We do this by considering the actual reduction in f, ared X f (xk) f (x k þ 1), relative to the predicted reduction by the model, pred X – rf ðxk ÞT s – ½s T Hk s, for some trust region control parameters, low < high. If the actual reduction in f agrees very well with the predicted reduction (ared/pred > high), then we gain trust in our model and may consider increasing the trust region radius for the next iteration. If the agreement is poor (ared/pred < low), then we reduce . Otherwise, we do not change . The expansion or reduction of by a simple factor of 2 is enough to give global convergence, and typical values for low and high are 0.25 and 0.75, respectively.

1.16.6 Constrained Optimization Methods This section builds upon the optimization methods of Section 1.16.4 by considering feasibility constraints on the variables. A general formulation for these problems is min

xPR n

subject to

f ðxÞ ci ðxÞ ¼ 0;

i PE

ci ðxÞ 0;

i PJ

ð47Þ

where f and the functions ci are all smooth (twice continuously differentiable) functions on a subset of R n , and E and J are two finite sets of indices. Let m ¼ jE j þ jJ j. We call the equations ci ; iPE , the equality constraints and the inequalities ci ; iPJ , the inequality constraints. The inequality constraints may be simple bound constraints on individual variables, such as x1 0, or they may be more complex nonlinear constraints involving multiple variables, such as 2 – x12 – x22 0. An important definition that is used throughout constrained optimization is the Lagrangian function, L ðx; Þ ¼ f ðxÞ –

m X

i ci ðxÞ

ð48Þ

i¼1

where i is called the Lagrange multiplier for the constraint ci (x). The derivative ofPthe Lagrangian function with respect to x is denoted by rx L (x, ). We point out that rx L ðx; Þ ¼ rf ðxÞ – mi¼1 i rci ðxÞ. We say that a constraint is active if equality holds at a particular point x. Hence, the active set A (x) is the union of E with the subset of J that has equality at x, A ðxÞ ¼ E [ fiPJ jci ðxÞ ¼ 0g

ð49Þ

We now describe the first-order optimality conditions required for a particular point x to be a local solution to problem (47). The first-order optimality conditions for unconstrained optimization examined the gradient of f, but for constrained optimization we must consider even more conditions. First, there is a linear independence constraint qualification (LICQ) that states that the set of active constraint gradients {rci(x ), i P A (x )} is linearly independent. Second, if the LICQ holds at x and there is a Lagrange multiplier , with components i ; iPE [ J , then the following conditions are satisfied at a local solution of problem (47): rx L ðx ; Þ ¼ 0

ð50Þ

ci ðx Þ ¼ 0;

iPE

ð51Þ

ci ðx Þ 0;

iPJ

ð52Þ

i 0; i ci ðx Þ ¼ 0;

iPJ iPE [ J

ð53Þ ð54Þ

532


These conditions are known as the Karush–Kuhn–Tucker conditions, or simply just the KKT conditions. The last condition is referred to as the complementarity or ‘slackness’ condition. It says that if an inequality constraint is not active, then its Lagrange multiplier must be zero. All together the KKT conditions deal with stationarity and feasibility. The second-order conditions are a bit more complex and deal with curvature of f while also considering the constraints. The second-order sufficient condition for optimality requires that d T r2x L ðx ; Þd > 0;

d PF

where F is the subset of directions that tend to stick to the active inequality constraints for which the Lagrange multiplier is positive, as well as to the equality constraints. A more rigorous mathematical definition is given in Nocedal and Wright.7 Like unconstrained optimization, there are a number of different methods for constrained optimization, but the mathematics and underlying theory are even more complicated. Here we consider an overview of some of the more common methods and refer the interested reader to Nocedal and Wright7 and Gill et al.4 for a more indepth treatment.

1.16.6.1

Penalty, Barrier, and Augmented Lagrangian Methods

We start with a class of methods for constrained optimization that finds the solution using the algorithmic framework previously established for unconstrained optimization. These approaches solve a sequence of unconstrained optimization problems that incorporate the deviation from the constraints in the objective function. These methods are attractive because they simplify a more complex, constrained problem to one that uses the simpler algorithms of unconstrained optimization. However, the resulting function is often difficult to minimize and is not appropriate for all applications. 1.16.6.1.1

Quadratic penalty function The first straightforward approach to solving constrained optimization problems is to replace the constrained problem by a penalty function that considers the original objective function plus an extra, nonnegative term for each constraint that is violated. By separately weighting the contributions from the original objective and the constraint violations, we may define a sequence of unconstrained optimization problems that increasingly penalizes the constraint violation relative to the original objective. As the constraints are enforced more heavily at each iteration, the minimizer of the penalty function is driven to the feasible region of the constrained problem while simultaneously minimizing the objective function. An obvious and straightforward implementation of this idea is the quadratic penalty function, which involves adding the square of the constraint violations to the objective f. For this discussion, we consider the equalityconstrained problem min f ðxÞ;

xPRn

subject to ci ðxÞ ¼ 0;

i PE

ð55Þ

The corresponding quadratic penalty function is Q ðx; Þ X f ðxÞ þ

1 X ðci ðxÞÞ2 2 iPE

ð56Þ

1 where > 0 is called the penalty parameter. The coefficient 2 is used for mathematical convenience. By solving a sequence of unconstrained optimization problems on Q(x ; ) for a sequence {k} and successively forcing k > k þ 1 ! 0 as k ! 1, the relative importance of the penalty term increases, and the solution tends to approach the actual solution to problem (55). By choosing the sequence of penalty parameters wisely, the solution for k is likely to be a good initial guess for the next problem, k þ 1 < k, so it is possible that each unconstrained optimization subproblem will not require many iterations. In fact, the sequence may be chosen adaptively, choosing the next k þ 1 based upon the difficulty of the current problem, k.


533

While the quadratic penalty method is appealing, it has limited usefulness in practice for two reasons. First, the minimization of Q (x; k) becomes increasingly difficult when k becomes small because the Hessian r2xx Q ðx; Þ becomes ill-conditioned near the minimizer. This quality adversely affects the steps computed by quasi-Newton methods and lowers the accuracy of the step computed by Newton’s method in floating-point arithmetic. Fortunately, there exists an alternative formulation of the Newton equations that avoids this illconditioning; see Nocedal and Wright.7 Second, the Taylor series approximation (and, hence, the local model) to Q (x ; k) is a good approximation to the original problem only in a small region around x. Hence, one should expect small steps and slow convergence. 1.16.6.1.2

Augmented Lagrangian method A closely related approach to penalty methods is the augmented Lagrangian method,15 which is also known as the method of multipliers. Here, we consider the equality-constrained problem (55). Inequality constraints may be converted to equality constraints by replacing ci(x) 0, i P J , with ci ðxÞ – zi ¼ 0;

zi 0;

iPJ

where zi is a slack variable. While we have introduced more variables for the optimizer to handle, these slack variables are bound constrained, which are easier constraints to handle. The penalty function method that was discussed previously has a drawback in that the constraints do not quite satisfy ci(x) ¼ 0, i P E . Instead, the constraints depend on the final penalty parameter and approximately satisfy ci ðxÞ ¼ i ;

iPE

Granted, these constraints will approach zero as ! 0, but it may take many iterations to reduce sufficiently close to zero. The idea with the augmented Lagrangian method is to make the minimizers satisfy the equality constraints more quickly. This is accomplished by including each i as a variable in the objective function. Let the augmented Lagrangian function be defined by L A ðx; ; Þ X f ðxÞ –

X

i ci ðxÞ þ

iPE

1 X ðci ðxÞÞ2 2 iPE

ð57Þ

Note that the augmented Lagrangian function is a combination of Equation (56) and (48). The derivative of L A (x, ; ) is given by rx L A ðx; ; Þ ¼ rf ðxÞ –

X 1 i – ci ðxÞ rci ðxÞ iPE

According to the KKT conditions, a minimizer of problem (55) occurs when rx L ¼ 0. Comparing rL (x, ) with rL A (x, ; ) reveals that i ki –

ci ðxk Þ k

ð58Þ

This observation suggests an approach for finding the minimizer of problem (55) using the augmented Lagrangian. By solving a series of unconstrained optimization subproblems with fixed and , min L A ðx; ; Þ

xPRn

we may drive k to zero and obtain increasingly better estimates of the Lagrange multipliers, i. The update formula for i is suggested above in Equation (58), kþ1 ¼ ki – i

ci ðxk Þ ; k

i PE

Convergence of this method may be achieved without decreasing to a very small value, unlike the penalty method. Hence, ill-conditioning is less of a concern than in the quadratic penalty method. The augmented Lagrangian method is the basis for the software implementation of LANCELOT by Conn et al.15,16

534


1.16.6.1.3

Logarithmic barrier method Another closely related approach to penalty methods is a barrier method. Just as with penalty methods, the idea is to solve a sequence of unconstrained optimization problems with small adjustments to a parameter to influence the minimizer so that it progressively obeys the constraints. Here, we consider the inequality-constrained problem min

xPRn

f ðxÞ;

subject to ci ðxÞ 0;

i PJ

ð59Þ

The logarithmic barrier function for the inequality constraints is –

X

logci ðxÞ

iPJ

This function has the property that it is defined only in the feasible region, specifically when ci (x) > 0. As x approaches the boundary of the feasible region, the barrier function approaches 1. Hence, we must start from some point x0 that is feasible for this function to be defined. To convert problem (59) to an unconstrained optimization problem, we add a multiple of the logarithmic barrier function to the objective function, Pðx; Þ X f ðxÞ –

X

logci ðxÞ

iPJ

where > 0 is called the barrier parameter. The barrier function erects a steep wall that prevents descent on f from ever reaching the boundary, but can be lowered as ! 0. By solving a sequence of unconstrained optimization problems on P (x ; ) for a sequence {k}, and successively forcing k > k þ 1 ! 0 as k ! 1, the iterates are allowed to get closer to any active inequality constraints without ever violating them. Hence, the solution tends to approach asymptotically the actual minimizer to problem (59). If k is not too different from k1, then the solution for k1 is likely to be a good initial guess for the next problem with k. By choosing the sequence of barrier parameters wisely, each unconstrained optimization subproblem might not require many iterations. The sequence may be chosen adaptively, based upon the difficulty of the previous problem, k1. As with the quadratic penalty function, minimization of P (x ; ) presents some difficulties. The Hessian matrix r2xx Pðx; Þ becomes increasingly ill-conditioned near the minimizer as ! 0. Fortunately, there is also a way to reformulate the Newton equations that avoids this problem (see Nocedal and Wright15). Finally, one must be careful with a straightforward implementation of one of the unconstrained optimization algorithms from Section 1.16.4. These algorithms have no safeguards for staying within some region where the objective is defined. If a trial point is outside of the feasible region, then the logarithmic barrier function is not defined.

1.16.6.2

Active Set Methods

An entirely different paradigm for solving the unconstrained optimization problem (47) is an active set method. Instead of arriving at a solution from along some path that might not be feasible, the idea is to remain feasible among a specified subset of constraints on the way to finding the minimizer. Small changes are made to the list of constraints that are likely to be active at the solution. By progressively refining this list and making progress on the objective function, we may reach a local optimum. 1.16.6.2.1

Linear programming and the simplex method An optimization problem with a linear objective function and linear constraints is called a linear program (LP). Linear programming was developed in 1940 by Dantzig and has thrived in many communities, particularly in economics and business, where linear models are prevalent.7 Consequently, a whole field has matured separate from nonlinear optimization, and, as such, the standard notation and terminology is somewhat different. To


535

remain consistent with previous concepts in this chapter and avoid any possible confusion, we will retain much of the nonlinear optimization notation that we have used thus far. Typically, LPs are written as min

xPR n

subject to

cTx ð60Þ

Ax ¼ b x0

where cPR n ; bPR m , and APR mn . We assume that m < n, otherwise some constraints are redundant or infeasible, or the constraints define a unique point. Moreover, we assume that A has full row rank (i.e., it has linearly independent rows). We call problem (60) the standard form of an LP. If a given formulation is not in standard form, then some amount of algebra and other tricks may transform it into standard form. This may involve, for example, adding slack variables to change inequality constraints into equality constraints or doubling the number of unbounded variables to make corresponding bounded variables (i.e., let xþ ¼ max (x, 0) 0 and x ¼ max (x, 0) 0). The feasible region is bounded by the linear equality and bound constraints. Because the objective is linear, the gradient is constant and has level sets with constant spacing. Figure 11 shows this geometrical relationship for a simple two-dimensional example with several constraints. It is easy to see in this case that a unique optimal solution will be at a vertex of the constraints. The optimality conditions for problem (60) follow from the KKT conditions for general nonlinear problems, Equation (54). Only the first-order conditions are needed because the objective f (x) ¼ cTx is linear, in which case the Hessian of the Lagrangian is zero. We can write the Lagrangian of problem (60) as T L ðx; e ; b Þ ¼ c T x – T e ðAx – bÞ – b x

where we have split the Lagrange multipliers into two vectors e and b corresponding to the equality constraints and bound constraints, respectively. From the general KKT condition (54), the first-order conditions for x to be a solution of problem (60) are AT e þ b ¼ c

ð61Þ

Ax ¼ b

ð62Þ

x 0

ð63Þ

b 0

ð64Þ

bi xi ¼ 0;

i ¼ 1; 2; . . .; n

ð65Þ

x2 –∇ f (x) = –c

x∗

x1 Figure 11 A linear program with three linear inequality constraints and bound constraints on x. The level sets of f (x) ¼ cTx are straight lines with constant spacing, and the constraints bound a feasible region. A unique optimal solution is found at an intersection of constraints, which in this case will be one of the five corners of the feasible polygon. The optimal solution is indicated by x .

536


The feasible set defined by the linear equality constraints is a polytope, a polygon in higher-dimensional space. A point that uniquely satisfies the first-order optimality conditions must lie on the vertices of the feasible polytope. Other optimal solutions may lie on a line or higher-dimensional subspace, but these solutions are not unique and not considered here. Hence, a unique solution lies only on the corner of a polygon that is defined by the constraints. This geometrical understanding suggests an algorithm: choose a vertex and move to a neighboring vertex until the objective function no longer improves. This is the basic idea of the simplex method. All iterates of the simplex method are feasible points (i.e., corners on the feasible polytope). In the language of linear programming, these points are more specifically called basic feasible points. Each step in the simplex method moves from one basic feasible point to another. The objective function value usually decreases. The simplex method uses an accounting system that divides the elements of x into two sets: so-called basic and nonbasic variables. We define the basic set B as a subset of m indices from {1, 2, . . ., n}, and we define the nonbasic set N as the complement of B , that is, N ¼ f1; 2; . . .; ngnB

The solution to problem (60) lies on a vertex of the polytope, which, for m constraint equations, requires m variables to uniquely specify. The basic set B contains the indices of m variables that define a basic feasible point. According to the first-order optimality conditions (61)–(65), it may be shown that the remaining n m variables are equal to zero, whose indices we store in the nonbasic set N . At each iteration in the simplex method, one exchanges an index in B for one in N . Before proceeding further, some notation is helpful. We partition the vectors x, c, and b according to the index sets B and N , which we denote by xB ¼ ½xi iPB ; xN ¼ ½xi iPN cB ¼ ½ci iPB ; cN ¼ ½ci iPN B ¼ ½ bi iPB ; N ¼ ½ bi iPN

We define the matrix B as the submatrix consisting of the m columns of A with column indices iPB . Similarly, the matrix N consists of the n m columns of A with column indices iPN . Thus, we can rewrite problem (60) as min

xPRn

subject to

cBT xB þ cNT xN Ax ¼ BxB þ NxN ¼ b xB 0

ð66Þ

xN 0

We start the simplex method by choosing a basic feasible point such that xB ¼ B1b 0 and xN ¼ 0 (by our assumptions on A, we know that B is nonsingular). We then use the KKT conditions to solve for the remaining variables and to determine optimality. Thus far, we have satisfied the equality constraints and nonnegativity condition on x in the KKT conditions. We satisfy the complementarity condition (65) by setting bi ¼ 0; iPB . To find the remaining Lagrange multipliers, we use the first KKT condition (61), which we rewrite as B T e þ B ¼ cB N T e þ N ¼ cN

Because B is square and nonsingular and bi ¼ 0; iPB , we may solve for the remaining Lagrange multipliers, e ¼ B – T cB N ¼ cN – N T e ¼ cN – ðB – 1 N ÞT cB

The remaining KKT condition for us to consider, b 0, determines whether our basic feasible point is optimal. If it is not optimal, then one must select an index from the nonbasic set N and exchange it with one from B and try again.


537

We start by identifying a variable in the nonbasic set with a negative Lagrange multiplier. Let q be an index in N for which bq < 0. Then we determine the effect of increasing xq on the current set of basic variables xB, while still staying feasible with respect to the equality constraints. This corresponds to a move along an edge of the polytope in the direction s ¼ B1Aq (which is a descent direction on f ) until a new bound constraint becomes active, at which point we have reached a new vertex. The first basic variable xp ; pPB , to decrease to zero is the one we remove from B . This index p is determined by the smallest step length such that xB þ s has a new zero element and is calculated by ðxB Þp sp < 0 ¼ min – p sp

For the last steps of the simplex method, we update xB ¼ xB þ s and set xq ¼ , and exchange p and q in the sets B and N . For an efficient and robust implementation of the simplex method, there are many important details that we omit here, such as choosing the first basic feasible point (which is not trivial), maintaining factorizations of B for efficient linear solves, selecting a good entering index q from N , and avoiding other potential pitfalls in the algorithm, such as degenerate steps and cycling. For further details on these issues and other algorithms, we refer the reader to Nocedal and Wright,7 Todd,17 Wolfe,18 and references therein. 1.16.6.2.2

Quadratic programming We now increase the complexity slightly over linear programs by considering quadratic objective functions, which are called quadratic programs (QPs). The general QP is stated as 1 qðxÞ ¼ hT x þ x T Gx 2

min

xPRn

subject to

aiT x ¼ bi ;

iPE

aiT x bi ;

iPJ

ð67Þ

where G is a symmetric n n matrix; h, b, and ai, i ¼ 1, 2, . . ., m, are n-vectors; and E and J are sets of unique indices with cardinality m ¼ jE j þ jJ j. We consider the case where G is positive semidefinite, in which case we call problem (67) a convex QP. If G is indefinite, then we say that program (67) is a nonconvex or indefinite QP, which is difficult to solve and can have multiple stationary points and local minima. We start with equality-constrained QPs, which we write as 1 qðxÞ ¼ h T x þ x T Gx 2 Ax ¼ b

min

xPR n

subject to

ð68Þ

where A ¼ ½ai T iP is an m n matrix. We assume that A has full row rank (i.e., no redundant or linearly dependent constraints). The KKT conditions applied to problem (68) yield Gx þ h – AT ¼ 0 Ax ¼ b

which we may write in matrix form as "

G

– AT

A

0

#"

x

#

" ¼

–h b

# ð69Þ

538


This system of equations is called the KKT system, and the matrix is called the KKT matrix. For computational purposes, it is often convenient to rewrite Equation (69) by expressing x as x ¼ x0 þ s, where x0 is a starting point and s is a step. Then, with some rearrangement, we may write (69) as "

G

AT

A

0

#"

–s

#

¼

" # g c

ð70Þ

where s ¼ x – x0 ;

g ¼ Gx0 þ h;

c ¼ Ax0 – b

We call Equation (70) the shifted KKT system. The matrix is symmetric, which is a benefit to some algorithms for solving this system. The optimal solution to problem (68) is found in one step: an explicit solution to Equation (69) or a solution to Equation (70) plus some extra algebra for the step computation. No iterating is required for the equalityconstrained problem. There are a number of ways to solve Equation (69) or (70). They include direct factorization, the null space method, and the range space method. We briefly discuss these methods here and mention when to use one or the other. More specific details may be found in Nocedal and Wright.7 A direct factorization of the shifted KKT matrix may be used to solve the linear system. While the matrix is symmetric, a Cholesky factorization cannot be used because the matrix is indefinite. Instead, a symmetricindefinite factorization is required, such as the Bunch–Parlett factorization.19 These methods factorize a general symmetric matrix K into the form P T KP ¼ LBLT

where P is a permutation matrix, L is a lower triangular matrix, and B is a block diagonal matrix with blocks of size 1 1 or 2 2. The direct factorization approach is best when G and A are large, sparse matrices because one can try to maintain sparsity in L. The null space method rearranges the shifted KKT system into a smaller system by first choosing an initial point x0 that satisfies Ax0 ¼ b (using, e.g., the basic method of linear programming) and then solving for a step s that lies in the span of the null space of A. That is, let s ¼ Zu nn –m

where ZPR is a basis for the null space of A (i.e., AZ ¼ 0) and uPR n –m is a vector of unknowns. Then the problem is to minimize 1 qðx0 þ ZuÞ ¼ qðx0 Þ þ ðGx0 þ hÞT Zu þ uT ZT GZu 2

by solving the unconstrained minimization over all u. The matrix (ZTGZ) is called the reduced Hessian, and if it is positive definite, then the unique minimizer u is the solution to the linear system ðZT GZÞu ¼ – ZT ðGx0 þ hÞ

The null space method is most efficient if (n m) n, but a drawback is the computation to get Z, which will most likely be dense. The range space method (sometimes called the dual) rearranges the shifted KKT system into two smaller systems and solves first for the Lagrange multipliers . Specifically, we solve for in the linear system ðAG – 1 AT Þ ¼ AG – 1 g – c ¼ AG – 1 h – b

and then solve for s in the first row of Equation (70), Gs ¼ AT – g

This method is most useful when G is easy to factorize or G1 is known explicitly (e.g., through a quasi-Newton updating formula, such as BFGS) and m n so that the matrix (AG1AT) is not too large.


539

After discussing the equality-constrained quadratic programming problem, we now turn to the inequalityconstrained problem using classical active-set methods. Other classes of methods exist for solving the inequality-constrained problem, including gradient-projection and interior-point methods, but active-set methods can be applied to both convex and nonconvex problems and have been in existence since the 1970s. Active-set methods usually work well for small to medium-size problems. With active-set methods, we iterate to find the set of inequality constraints that are active at the solution. This procedure is accomplished in a manner similar to linear programming by moving inequality constraints in and out of a set of active constraints. This set is called the working set and is denoted by W k at the kth iteration. W k consists of all equality constraints in E plus any number of the inequality constraints from J that are deemed to be active at the optimal point x . We start the process by choosing an initial working set W 0 so that fai jiPW 0 g are linearly independent and then find a feasible x0. Then, given an iterate xk, we seek a step s that solves the following equality-constrained QP: min

sPRn

1 qðxk þ sÞ ¼ qðxk Þ þ ðGxk þ hÞT s þ s T Gs 2

subject to aiT ðxk þ sÞ ¼ bi;

iPW

ð71Þ

k

or, alternatively, min

sPR n

subject to

1 ðGxk þ hÞT s þ s T Gs 2 aiT s ¼ 0;

iPW

ð72Þ

k

We denote the solution by sk. If G is positive definite, then all of the methods previously discussed for solving the equality-constrained QP may be used to solve this QP subproblem. Note that since the constraints are aiT s ¼ 0, we remain feasible among the constraints in W k as we move along the step sk. However, we must check the constraints i ˇ W k, which are any remaining inequality constraints in W . We set xkþ1 ¼ xk þ ksk, where k is chosen to be the largest step length such that all constraints are satisfied. The optimal step length is given by this formula bi – aiT x T a k ¼ min 1; min sk < 0 iˇW k aiT sk i

ð73Þ

which checks only the inequality constraints not in the working set that are being approached. Since we are assuming G is positive definite, then the function is convex and any k P (0, 1] is decreasing the objective q(x). If k < 0, then the constraints i ˇ W k for which Equation (73) is minimum are called blocking constraints. One of these blocking constraints is added to W k to form a new working set W kþ1. Note that it is possible for k to be zero in Equation (73) because xk could be on an active constraint that is not in W k, in which case we add that blocking constraint to W k and solve the QP again. We continue to iterate in this manner until the solution to the W k problem is sk ¼ 0, at which point we have solved the inequality-constrained QP for all of the constraints in W k. However, we still need to check the overall KKT conditions for program (67) because one or more constraints in W k might be unnecessary. We examine the Lagrange multipliers for i P W k \ J . If one of the multipliers is negative, then this constraint may be dropped from W k and we solve a new QP for W kþ1. We continue to iterate by adding to or subtracting from the current working set W k. Once sk ¼ 0 and all i 0, then we have solved the general inequality-constrained QP (67). We point out that every point xk is feasible and each iteration improves the objective, q(xkþ1) q(xk) (and q(xkþ1) < q(xk) if sk 6¼ 0 and k > 0). 1.16.6.2.3

Sequential quadratic programming We now consider the most general (and difficult) of optimization problems, so-called nonlinear programs (NLPs). We consider the general constrained optimization problem in problem (47), where the objective function f (x) may be any smooth nonlinear function (twice continuously differentiable) and each constraint ci(x) is a smooth nonlinear function. The main idea behind sequential quadratic programming (SQP) is to approximate problem (47) at each iterate xk with a local model that is represented by a simpler QP.

540


Recall that the Lagrangian for problem (47) is L ðx; Þ U f ðxÞ – T cðxÞ

Let W(x, ) denote the Hessian of the Lagrangian, W ðx; Þ U r2xx L ðx; Þ

Let A(x) denote the Jacobian matrix of the constraints, 2 6 6 AðxÞ U 6 6 4

rc1 ðxÞT .. .

3 7 7 7 7 5

rcm ðxÞT

We approximate the general NLP with a QP based upon a local model at the current iterate xk, 1 min rf ðxk ÞT s þ s T W ðxk ; k Þs s 2 subject to Aðxk Þs þ cðxk Þ ¼ 0

ð74Þ

Provided that A(xk) has full row rank and that the reduced Hessian of the Lagrangian, W(xk, k), is positive definite, then problem (74) has a unique solution. We solve the shifted KKT system "

W ðxk ; k Þ

Aðxk ÞT

Aðxk Þ

0

#"

– sk kþ1

#

" ¼

rf ðxk Þ

#

cðxk Þ

ð75Þ

for sk and kþ1. We use sk as the step to update xk. This framework can be extended easily to include inequality constraints. We still iterate on a local QP that approximates the objective and the corresponding constraints at xk, except that we solve an inequalityconstrained QP, which in turn requires iterating on the working set of active constraints, W . We may use the final working set from the previous QP as a start for the next QP to help speed up the algorithm. As the outer QP iterations approach the solution, changes to the working set in each QP subproblem will be minimal, and each QP subproblem will act more like an equality-constrained QP, which requires just one linear solve. We point out that instead of solving an inequality-constrained QP at each iteration, where the active inequality constraints are determined in the subproblem, another approach is to solve an equality-constrained QP and then move consideration of the inequality constraints (i.e., the management of the working set W ) to the outer SQP loop. The basic SQP algorithm for the general nonlinear programming problem is given in Algorithm 5. We point out that this algorithm may need a global strategy to achieve convergence from any starting point, especially if the assumptions on the Hessian of the Lagrangian or the Jacobian of the constraints do not hold at a particular iterate xk. Line search and trust region strategies are two popular approaches and are similar in spirit to those previously described in Section 1.16.5.5. Algorithm 5: Sequential quadratic programming (1) Given initial x0, 0, iterate for k ¼ 0, 1, 2, . . .: (a) Compute rf (xk), A (xk), c (xk), W (xk, k). (b) Solve the following inequality-constrained QP to obtain sk and kþ1: min s

1 rf ðxk ÞT s þ s T W ðxk ; k Þs 2

subject to rcðxk ÞT s þ cðxk Þ ¼ 0; T

rcðxk Þ s þ ci ðxk Þ ¼ 0;

(c) Set xkþ1 xk þ sk. (d) Return (xkþ1, kþ1) if satisfactory.

iPE iPJ


541

For a simple example that illustrates some of the concepts in constrained optimization, we now consider a variation of the nonnegative matrix factorization problem that was presented in problem (8). Suppose we have a nonnegative matrix Y that we wish to approximate as uvT, where u and v are nonnegative and their elements sum to one. To simplify some of the derivatives in the SQP formulation, we modify the objective function in problem (8) and consider the equivalent objective 2 1 Y – uvT 2 2

min u;v

ð76Þ

subject to u 0; v 0; jjujj1 ¼ jjvjj1 ¼ 1

For our particular problem, we consider the 2 2 matrix " Y ¼

0:12

0:18

0:28

0:42

#

for which the solution to problem (76) is " u ¼

0:3 0:7

#

" ; v ¼

0:4

#

0:6

To proceed with the algorithms outlined above, it helps to rewrite the problem using our familiar notation. We consider the unknowns as one long vector x, such that x U

" # u v

We keep this change of variables in mind when we calculate any derivatives with respect to x and refer back to the original variables u and v. Hence, we may rewrite the objective function of problem (76) as f ðxÞ ¼

2 X 2 1X ðyij – ui vj Þ2 2 i¼1 j ¼1

The Lagrangian function is L ðx; Þ U f ðxÞ – T cðxÞ ¼

1XX ðyij – ui vj Þ2 – 1 ðu1 þ u2 – 1Þ – 2 ðv1 þ v2 – 1Þ 2 i j

We omit the bound constraints on u and v and check them separately. The gradient of f is computed as 2

– v1 ðy11 – u1 v1 Þ

6 6 – v1 ðy21 – u2 v1 Þ 6 rf ðxÞ ¼ 6 6 – u1 ðy11 – u1 v1 Þ 4 – u1 ðy12 – u1 v2 Þ

– v2 ðy12 – u1 v2 Þ

3

7 " # – v2 ðy22 – u2 v2 Þ 7 – ðY – uvT Þv 7 ¼ 7 – u2 ðy21 – u2 v1 Þ 7 – ðY – uvT ÞT u 5 – u2 ðy22 – u2 v2 Þ

The Hessian of the Lagrangian is W ðx; Þ U r2xx L ðx; Þ 2 ðv12 þ v22 Þ 6 6 0 6 ¼6 6 – y11 þ 2u1 v1 4 – y12 þ 2u1 v2

0

– y11 þ 2u1 v1

ðv12 þ v22 Þ

– y21 þ 2u2 v1

– y21 þ 2u2 v1

ðu12 þ u22 Þ

– y22 þ 2u2 v2

0

– y12 þ 2u1 v2

3

7 – y22 þ 2u2 v2 7 7 7 7 0 5 2 2 ðu1 þ u2 Þ

542

Constrained and Unconstrained Optimization " ¼

vT vI

–Y þ 2uvT

ð –Y þ 2uvT ÞT

uT uI

#

Due to the linear constraints, W does not depend on . The Jacobian of the constraints is " Aðxk Þ ¼

rc1 ðxÞT rc2 ðxÞT

#

" ¼

1 1 0 0

# ð77Þ

0 0 1 1

Similarly, due to the linear constraints, A(xk) is constant and does not depend on xk. With each of these quantities defined, we proceed with Algorithm 5. Suppose we start with the initial guess x0, such that u¼

" # 1 1

; v¼

" # 1 1

This solution does not satisfy the norm constraints jjujj1 ¼ jjvjj1 ¼ 1, but it does satisfy the bound constraints u, v 0. The corresponding derivative computations are 2

1:7

3

2

7 6 6 1:3 7 7 6 rf ðx0 Þ ¼ 6 7; 6 1:6 7 5 4 1:4

2

0

6 6 0 2 6 W ðx0; Þ ¼6 6 1:88 1:72 4 1:82 1:58

1:88 1:82

3

7 1:72 1:58 7 7 7 2 0 7 5 0 2

The solution of the equality-constrained QP (68), provides a new approximate solution " u¼

0:4495 0:5505

#

" ; v¼

0:4740

#

0:5260

We see that the equality constraints are satisfied in a single iteration. The subsequent steps in the SQP algorithm are given in Table 4. We point out that all subsequent iterations satisfy the constraints and that a quadratic convergence rate is evident in the objective function values.

1.16.7 Discussion We conclude this chapter with a few remarks. We have described a broad array of techniques for solving unconstrained and constrained optimization problems. We limited the discussion to the simpler cases to facilitate a discussion on the underlying principles of optimization and the algorithmic frameworks of many methods. As one might guess, as restrictions are lifted on the type of functions and constraints allowed, the problem becomes more difficult to solve, the prevailing methods become more complex, and the theory grows increasingly rich.

Table 4 SQP iterations of the nonnegative matrix factorization problem, problem (76) k

u

v

f (xk)

0 1 2 3 4

(1,1) (0.4495, 0.5505) (0.2924, 0.7076) (0.2998, 0.7002) (0.3000, 0.7000)

(1,1) (0.4740, 0.5260) (0.3862, 0.6138) (0.3999, 0.6001) (0.4000, 0.6000)

1.1508eþ00 1.4607e02 1.5049e04 4.0620e08 2.9436e15


543

Other factors may affect the choice of algorithmic options. In particular, problems that violate simplifying assumptions pose additional challenges. Usually these assumptions include that the Hessian is positive definite or that constraints are linearly independent. If these assumptions do not hold true, then other techniques that rely on more advanced theory must be used to make forward progress. Furthermore, more advanced techniques have been developed to deal with large-scale problems. The point at which a problem is defined as large is usually dependent on the type of problem and the algorithm – in general, unconstrained optimization methods can handle slightly larger problems than constrained optimizations. At some point, a different approach is necessary, and typically this approach involves solving the local model only approximately with fast iterative solvers, such as conjugate gradient. The optimization algorithms that we have described in this chapter require a computation of the gradient, either analytically or through finite differencing techniques. Without a gradient, these methods are stuck. The design of optimization algorithms that do not require gradients is an active area of research today. Direct search and pattern methods are two classes of methods that are especially useful when derivatives are not available or the function is noisy. See Powell20 and Kolda et al.21 for a survey of these and other methods. Finally, this chapter has dealt solely with the case of identifying a local minimizer. Global optimization techniques seek to find the best minimizer among all local minima. This is a very challenging problem because one cannot be sure that the true global optimum has been found without exploring the whole parameter space. However, with some assumptions on the function, there are some techniques that can identify the global optimum. Branch and bound, simulated annealing, and genetic algorithms are among the many techniques for solving the global optimization problem. A comprehensive reference is Neumaier.22 In summary, optimization theory follows very closely to that of solving nonlinear equations. The first-order necessary conditions require that the gradient equals zero at a minimizer. The general approach in all of these methods is to create a simple model that locally approximates the real function and is simple enough that by solving a linear system the local model may be minimized exactly. The resultant step moves us closer to the actual minimum on f, provided that the local model matches f sufficiently well. If it does not, then globalization strategies keep the step bounded so that each iteration still makes progress toward the solution. This overall approach is a strategy that works well on a variety of challenging problems and is the engine behind today’s most complicated optimization software packages.

Acknowledgments This work was funded by Sandia National Laboratories, a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration under Contract DE-AC04-94AL85000.

References 1. Harshman, R. A. Foundations of the PARAFAC Procedure: Models and Conditions for an ‘Explanatory’ Multi-Modal Factor Analysis. UCLA Working Papers in Phonetics 1970, 16, 1–84. 2. Golub, G. H.; Van Loan, C. F. Matrix Computations; Johns Hopkins University Press: Baltimore, 1996. 3. Schnabel, R. B.; Eskow, E. A New Modified Cholesky Factorization. SIAM J. Sci. Comput. 1991, 11, 1136–1158. 4. Gill, P. E.; Murray, W.; Wright, M. H. Practical Optimization; Academic Press: London, 1981. 5. Dennis, J. E.; More´, J. J. Quasi-Newton Methods, Motivation and Theory. SIAM Rev. 1977, 19 (1), 46–89. 6. Khalfan, H. F.; Byrd, R. H.; Schnabel, R. B. A Theoretical and Experimental Study of the Symmetric Rank One Update. SIAM J. Optim. 1993, 3, 1–24. 7. Nocedal, J.; Wright, S. J. Numerical Optimization; Springer: New York, 1999. 8. Nocedal, J. Updating Quasi-Newton Matrices with Limited Storage. Math. Comput. 1980, 35, 773–782. 9. Liu, D. C.; Nocedal, J. On the Limited Memory BFGS Method for Large Scale Optimization. Math. Prog. 1989, 45, 503–528. 10. Armijo, L. Minimization of Functions Having Lipschitz-Continuous First Partial Derivatives. Pacific J. Math. 1966, 16, 1–3. 11. Wolfe, P. Convergence Conditions for Ascent Methods. SIAM Rev. 1969, 11, 226–235. 12. Wolfe, P. Convergence Conditions for Ascent Methods II: Some Corrections. SIAM Rev. 1971, 13, 185–188. 13. Dennis, J. E. Jr.; Schnabel, R. B. Numerical Methods for Unconstrained Optimization and Nonlinear Equations; Prentice-Hall: Englewood Cliffs, NJ 1983; Reprinted by SIAM Publications, 1996. 14. Kelley, C. T. Iterative Methods for Optimization; SIAM: Philadelphia, PA, 1999.

544


15. Conn, A.; Gould, N.; Toint, P. A Globally Convergent Augmented Lagrangian Algorithm for Optimization with General Constraints and Simple Bounds. SIAM J. Num. Anal. 1991, 28 (2), 545–572. 16. Conn, A.; Gould, N.; Toint, P. LANCELOT: A FORTRAN Package for Large-Scale Nonlinear Optimization (Release A); SpringerVerlag: New York, 1992. 17. Todd, M. J. The Many Facets of Linear Programming. Math. Prog. 1989, 91, 417–436. 18. Wolfe, P. The Composite Simplex Algorithm. SIAM Rev. 1965, 7 (1), 42–54. 19. Bunch, J. R.; Parlett, B. N. Direct Methods for Solving Symmetric Indefinite Systems of Linear Equations. SIAM J. Num. Anal. 1971, 8, 639–655. 20. Powell, M. J. D. Direct Search Algorithms for Optimization Calculations. Acta Numerica 1998, 7, 287–336. 21. Kolda, T. G.; Lewis, R. M.; Torczon, V. Optimization by Direct Search: New Perspectives on Some Classical and Modern Methods. SIAM Rev. 2003, 45 (3), 385–482. 22. Neumaier, A. Complete Search in Continuous Global Optimization and Constraint Satisfaction. Acta Numerica 2004, 13, 271–369.


Biographical Sketch

Brett W. Bader received his B.S. and M.S. degrees in chemical engineering from the Massachusets Institute of Technology. Subsequently, he worked in the chemical industry, developing mathematical models of chemical plants for online, real-time optimization. He received his Ph.D. in computer science from the University of Colorado at Boulder, studying methods for optimization and solving systems of nonlinear equations. He now develops multiway data analysis and machine learning techniques.

545


1.17

Sequential Optimization Methods

B. Dejaegher and Y. V. Heyden, Vrije Universiteit Brussel, Laarbeeklaan, Brussels, Belgium ª 2009 Elsevier B.V. All rights reserved.

1.17.1 1.17.2 1.17.2.1 1.17.2.2 1.17.2.2.1 1.17.2.2.2 1.17.2.2.3 1.17.2.2.4 1.17.2.2.5 1.17.3 1.17.4 References

Introduction Sequential Optimization Methods Fibonacci Numbers Simplex Methods Introduction The (basic) simplex method The variable-size or modified simplex method Discussion Case studies Mixed Sequential–Simultaneous Optimization Methods Conclusions

Symbols B

b ¼ [x1b,x2b]

b ¼ [x1b,x2b, . . ., xfb]

bi

vertex with the best response in the simplex procedure for two or f factors vector notation of vertex B in the simplex procedure for two factors vector notation of vertex B in the simplex procedure for f factors general notation for the coefficients of the model

cW,1 D

di

E1

y ¼ b0 þ b1 x1 þ b2 x2

bij

BNW

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi b21 þ b22

CR,1

cR,1 CW,1

interaction coefficient between variables xi and xj representation of initial simplex length of the line segment g in the steepest ascent method procedure contraction vertex toward vertex R1 in the modified simplex procedure vector notation of vertex CR,1 contraction vertex toward vertex W in the

e1 Eover

Eover – 1

Ewithin

549 551 551 555 555 555 558 561 566 569 572 572

modified simplex procedure vector notation of vertex CW,1 combined response (0 < D < 1), i.e., geometric mean from Derringer desirability functions desirability value (0 < di < 1) from Derringer desirability function expansion vertex in the modified simplex procedure vector notation of vertex E1 last experiment where the optimum was overshot in the steepest ascent method procedure last-but-one experiment where the optimum is not yet overshot in the steepest ascent method procedure point that falls within the limits of the experimental domain in cases where the limits of the domain are reached in the

547

548


f f(x) g

L

M

Mopt

N

n ¼ [x1n,x2n]

ni ¼ ½x1ni ; x2ni ; . . . ; xfni ] Ni

P1

p1 R R1

steepest ascent method procedure number of factors function of x variable(s) line segment, determining the slope or the direction of movement in the steepest ascent method procedure real length of a step (L ¼ constant x g) in the steepest ascent method procedure coefficient determining the situation of a given point in the super modified simplex procedure optimal value of the coefficient M in the super modified simplex procedure vertex with the next-tobest response in the simplex procedure for two factors vector notation of vertex N vector notation of vertex Ni vertexes (i ¼ 1, 2, . . . ,f-1) with responses between the best and the worst in the simplex procedure for f factors centroid of the line segment BN in the simplex procedure for two factors or centroid of the hyperspace remaining after deleting vertex W in the simplex procedure for f factors vector notation of centroid P1 the response at a vertex of the simplex reflection vertex or mirror image of vertex W in the simplex procedure for two or f factors

R91

r1 r91 Rg Ropt

tn w

W

W

w ¼ [x1w,x2w]

w ¼ [x1w,x2w,. . .,xfw]

x, xi y y0i

Y1,opt

reflection vertex or mirror image of vertex N from the simplex BNCR,1 in a modification of the modified simplex procedure vector notation of R1 vector notation of R91 response at the gravity center of the simplex the point from the second-order polynomial model through W, R1, and P1, with maximal or minimal response, depending on what is desired, in the supermodified simplex procedure the (n þ 1)th number in the Fibonacci series width of the optimal region in the Fibonacci procedure Fibonacci procedure: width of original search region simplex procedure for two or f factors: vertex with the worst response vector notation of vertex W in the simplex procedure for two factors vector notation of vertex W in the simplex procedure for f factors variables or factors response y response at the center point of the 22 full factorial design the point of the Gaussian model through W, R1, and P1, with maximal or minimal response, depending on what is desired, in a modification to the supermodified simplex procedure

Sequential Optimization Methods 549

the contraction coefficient, representing the contraction of the simplex (0 < < 1), in the modified simplex procedure

the expansion coefficient, representing the expansion of the simplex ( > 1), in the modified simplex procedure

1.17.1 Introduction Several strategies can be applied to optimize analytical methods. Often, the method is complex, and different parameters or factors can affect its analytical results. When several factors need to be optimized, the possible strategies can be divided into univariate and multivariate approaches.1 In each approach, the influence of a number of selected factors on the method response(s) is examined. A univariate method varies only one factor at a time, while a multivariate procedure varies several factors simultaneously. The multivariate approaches can be further divided into sequential optimization methods and simultaneous optimization strategies.1,2 In the latter, a predefined number of experiments is performed according to a well-defined experimental setup, that is, an experimental design. Experimental designs are thoroughly discussed in Chapters 1.09–1.15. In sequential optimization strategies, the topic of this chapter, initially only a few experiments are performed and their results are sequentially used to define the next experiment(s).1,3 Although the definitions allow a clear distinction between the different approaches, both the univariate and the experimental design approaches can also be applied in some sequential procedures, as explained below. A classical univariate approach is the one-variable-at-a-time (OVAT) procedure.1 It can be applied to examine sequentially the influence of several factors on the response(s), in order to obtain the optimum method conditions. This procedure evaluates the influence of one factor by varying its levels, while keeping the levels of all other factors constant. The results after varying one factor are then evaluated, to decide on the best level setting. The level of this factor is then fixed at its optimum, and a next factor is optimized.1,3 Although a following experiment is not defined based on the results of the former, and the method is not a sequential optimization method by definition, the OVAT approach can, in fact, be considered a semisequential optimization strategy, because the different factor levels are optimized sequentially. For example, in Figure 1(a), the optimization of two factors, x1 and x2, is shown. Isoresponse lines are displayed to give an idea about the, in practice unknown, behavior of the response. Suppose that the optimal response is when it is maximal. With the OVAT approach, first, a number of experiments are carried out at different levels of x1, while keeping the level of x2 constant at a given value (points 1–5 in Figure 1(a)). The best setting for x1 is defined as x1,optimum and was found with experiment 3. Subsequent optimization of x2 at value x1 ¼ x1,optimum (points 3 and 6–9 in Figure 1(a)) leads to x2,optimum at experiment 8 as best level for x2. For this particular example, this approach results in finding the desired optimum (point 8 (x1,optimum, x2,optimum)). However, the above approach does not allow selecting the optimum of the response function shown in Figure 1(b). After the two steps, performed as described above, point 7 incorrectly would be chosen as optimum. The conditions of point 7 are in fact rather far from the optimum. A further optimization by repeating the univariate approach, that is, varying the levels of x1 a second time at x2 ¼ x2,optimum (points 7 and 10–13 in Figure 1(b)), would lead to an experimental optimum, closer to the real optimum. The univariate procedure could be repeated, until the real optimum is reached. However, this would require a high number of experiments to be performed, which makes this approach less attractive from a practical point of view. Moreover, if more than two factors were to be optimized, the situation would have been even worse. The main difference between the situations, shown in Figures 1(a) and 1(b), is the absence or presence of an interaction effect between x1 and x2, respectively.1,3 An interaction between two factors is present when the

550


(a)

(b)

x2

x2

Optimum

20

Optimum

40

20 40

9

1

2

3 7

8

60

60 8

x2,optimum

9

x2,optimum 4

10

11

12

1

2

3

7

13

5 4

5

6

6 x1

x1,optimum

x1,optimum

x1

Figure 1 One-variable-at-a-time optimization procedure for two factors with absence (a) or presence (b) of an interaction effect between the factors.

x2 Optimum

20

(30) 1

(10) 4

40 60

(10) 2

(35) 3 x1

Figure 2 Response with an interaction effect between the factors x1 and x2.

effect of one factor depends on the level of the other. This is illustrated in Figure 2, where the response was measured at four points (1–4). When evaluating the difference between points 1 and 2, the response increases when increasing the value of x2, while when looking at the difference between points 3 and 4, the response increases when decreasing the value of x2. Thus, the effect of x2 on the response depends on the value of the second factor, x1. For more information on interactions, we would like to refer to the chapter on experimental designs (Chapters 1.09–1.15). In case interactions occur, the results of the OVAT approach will depend on the executed sequence of the experiments, that is, on the sequence of the factors chosen to be optimized,1,3 and on the level of the fixed factor in the first step. The optimum will not necessarily be found when each factor is only evaluated once as was done in Figure 1(a).


Summarizing, the OVAT procedure is efficient when no interactions occur, as is the case in Figure 1(a), but much less when they are present, as in Figure 1(b). Nevertheless, in method optimization, these interactions occur regularly and cannot be neglected. A second disadvantage of the OVAT approach is that a high number of experiments are required when the number of factors increases. For the reasons mentioned above, the use of multivariate approaches seems better and usually also is in practice.1,3 In experimental design methodology, a semisequential strategy can also be applied during method optimization.1,4 First, the effects of a (relatively) high number of factors can be examined by applying a screening design, in order to determine those factors with the most important influence on the method response(s). The latter factors can then further be evaluated using a response surface design methodology (RSM), in order to model, from the obtained results, the response as a function of these most influencing factors. Thus, the optimal factor level settings are obtained from the response surface model. This will not be further elaborated, because experimental designs and their approaches are thoroughly discussed in Chapters 1.09–1.15. The mixed sequential–simultaneous optimization approaches, which are discussed in Section 1.17.3 of this chapter, are methodologies where experimental designs are used in a sequential optimization method. In this chapter, sequential optimization methods are described. First, the approach based on the Fibonacci numbers is explained, which can only be used when the effect of only one factor on one response needs to be evaluated. Second, the simplex methods, which are the best-known sequential optimization methods, are discussed. They allow evaluating the influences of several factors on one response. Two different approaches, that is, the (basic) simplex and the modified simplex methods, are explained in detail and clarified with examples from the literature. In Section 1.17.3, mixed sequential– simultaneous approaches, such as the steepest ascent method and the evolutionary operation (EVOP), are discussed. All these methods, in general, consider only one response or optimization criterion. To optimize several responses simultaneously, multicriteria decision-making (MCDM) strategies should be used, which are thoroughly discussed in Chapter 1.19. They are mainly used to transform the multiple responses into one new response.

1.17.2 Sequential Optimization Methods In sequential optimization methods, only a restricted number of experiments, usually one more than the number of selected factors, are performed, and from these results, the next experiment is defined. Then, some results are further used to select the following experiment to be performed, and so on.1,3 Two approaches are described and discussed, one based on the Fibonacci numbers and the other on the simplex methods, which are the most widely applied. The former can be used when the influence of only one factor needs to be examined, whereas the latter allows evaluating the effects of several factors on one response. For both methods, the response is supposed to be unimodal, that is, having only one optimum. In case of a multimodal response, the total region should be divided into unimodal regions.2

1.17.2.1

Fibonacci Numbers

Fibonacci numbers are named after the mathematician Leonardo of Pisa, who was also called Fibonacci. Each number of the Fibonacci series is the sum of the two preceding numbers.1,2 Thus, Fibonacci numbers are defined by the recursive relationship tnþ2 ¼ tn þ tnþ1

ð1Þ

where t0 ¼ 1 and t1 ¼ 1, and n ¼ 0, 1, 2, . . . . Therefore, the Fibonacci series begins as t0 ¼ 1, t1 ¼ 1, t2 ¼ 2, t3 ¼ 3, t4 ¼ 5, t5 ¼ 8, t6 ¼ 13, t7 ¼ 21, t8 ¼ 34, t9 ¼ 55, t10 ¼ 89, t11 ¼ 144, t12 ¼ 233, . . . .

552


yopt

y

y4 y3

y1

y2

xi

x3

x4 x1 xopt

x2

x

xj

Figure 3 Response y as a function of factor x. Example of the first steps of the Fibonacci search procedure.

These numbers can be used for a restricted region search, that is, the limits of the region to be evaluated are a priori known. As mentioned above, this method is only valid for unimodal single-factor functions. This search method sequentially eliminates parts of the region under investigation, thereby narrowing at each cycle the region in which the optimum is situated.1,2 The Fibonacci search procedure can be explained as follows.1,2 Consider a unimodal response y, which is a function of only one factor x, as shown in Figure 3. Suppose one seeks to find the maximum of the response y ¼ f(x) in the region [xi, xj]. The maximum to be found is situated at x ¼ xopt. First, two experiments are performed at x ¼ x1 and x ¼ x2, and the resulting responses are y1 and y2, respectively. The values x1 and x2 are chosen in such a way that the distance between xi and x1 is equal to that between xj and x2. As y1 > y2, it is concluded that the maximum is not situated in [x2, xj], allowing to eliminate this region for further investigations, and to only further focus on the [xi, x2] region. In the latter region, already one experimental result, y(x1) ¼ y1, is available. The procedure is repeated, and one additional experiment is performed at x ¼ x3, for which the distance between xi and x3 is equal to the one between x1 and x2. The resulting response is y3. As y1 > y3, this leads to the elimination of the region [xi, x3], and further optimization will concentrate on the region [x3, x2]. Subsequently, one experiment at x ¼ x4 is performed, for which the distance between x3 and x4 is equal to the one between x1 and x2, and so on. How the Fibonacci numbers are used in the search procedure is explained in the following.1,2 First, the width (w) of the optimal region, which the experimenter will accept compared to the one of the original search region (W), is chosen. For example, when examining the effect of varying the pH from 1 to 11 and when the optimum should be determined within a region of 0.5 pH units, W ¼ 10 and w ¼ 0.5. Then the Fibonacci number that is immediately larger than W/w is selected from the Fibonacci series. This number is tn, which is the (n þ 1)th number in the series, and n steps need to be performed in the search procedure. The first step requires two experiments, x ¼ x1 and x ¼ x2, whereas the next steps all require only one additional experiment, x ¼ x3 to x ¼ xnþ1. For example, when W/w is 20, as in the pH example given above, the smallest Fibonacci number higher than 20 is 21, which is t7, the eighth number in the series, and thus, seven steps requiring eight experiments will be performed. The first n – 1 steps are defined and performed as follows. Consider W ¼ W1 the length of the original search region, that is, W ¼ W1 ¼ xj – xi , and w1 the distances between the experiments at x ¼ x1 or x ¼ x2 and the limits xi and xj, respectively, that is, w1 ¼ x1 xi ¼ xj x2. Equation (2) is a general equation to determine generally wk with k ¼ 1, 2, . . ., n 1. Defining wk according to Equation (2) results in an optimal number of experiments. wk ¼

t½n – ðkþ1Þ Wk t½n – ðk – 1Þ

ð2Þ

with Wk the width of the search region remaining at the kth step of the procedure. In Equation (3), k ¼ 2, 3, . . ., n. Wk ¼ Wk – 1 – wk – 1 ¼ Wk – 1 –

tðn – kÞ

t½n – ðk – 2Þ

Wk – 1

¼

t½n – ðk – 1Þ Wk – 1 t½n – ðk – 2Þ

ð3Þ

For the example given above, for step 1, w1 ¼ ðtn – 2 =tn ÞW1 = 8/21 W1 . Thus, the first two experiments are defined as x1 ¼ xi þ ð8=21Þ xj – xi and x2 ¼ xj – ð8=21Þ xj – xi . As explained earlier, then the


responses are measured at x ¼ x1 and x ¼ x2, and consecutively the interval either [xi, x1] or [x2, xj] is eliminated, depending on whether y1 < y2 or vice versa. Suppose y1 > y2, as in Figure 3, then [x2, xj] is eliminated and the region [xi, x2] or W2 is further examined. The result at x ¼ x1 is retained, and a second length w2 should be determined in such a way that for the new experiment x3, w2 ¼ x3 xi ¼ x2 x1 (step 2). Therefore, the general equations, that is, Equation (3) to determine Wk and Equation (2) to determine wk, can be applied. These equations are then used to further define steps 2 till (n 1), describing the experiments that need to be performed at x ¼ x3 till x ¼ xn. For the example given above, the values of Wk and wk, defining the steps, and thus consecutively the experiments, are given in Table 1. In the last-but-one (n 1)th step, the ratio wn – 1 =Wn – 1 equals, according to Equation (2), wn – 1 =Wn – 1 ¼ t½n – ðn – 1þ1Þ =t½n – ðn – 1 – 1Þ ¼ t0 =t2 ¼ 1=2, as also can be seen from Table 1. Thus, the distance between the last-but-one experiment at x ¼ xn and the boundaries, wn1, is equal to half of the length or is situated at the center of the remaining search region, Wn1. Finally, for the last nth step, the last experiment at x ¼ xnþ1 is prescribed1,2 at one of the limits of the remaining search region W n ¼ 1=2W n – 1 (Equation (3)); thus wn ¼ 0. However, as thus no new information would be gained, the last experiment is performed at the smallest distance, which is supposed to give a measurable difference in the response.1,2 The Fibonacci procedure leads to a small region, in which the optimum is situated. At the beginning of the last step, the length of the region was already within the range, in which one wants to define the optimum. The steps for the above example of pH (¼x) are also shown in Table 1. When assuming that the response decreases continuously in the interval (1,11) and that the highest is the optimum, in each step, the upper interval is discarded, in order to minimize the region where the optimum is situated. For this situation, the Fibonacci search procedure is represented in Figure 4. When following the procedure as described above and in Massart et al.,1,2 the experiments should be conducted at x1 ¼ 4.81, x2 ¼ 7.19, x3 ¼ 3.38, x4 ¼ 2.43, x5 ¼ 1.95, x6 ¼ 1.48, x7 ¼ x6, and x8 ¼ x7 ¼ x6. However, then the 7th experiment equals the 6th experiment, and comparing the response values for both thus does not make sense. Therefore, it seems more logical to execute an experiment at the remaining search region limit in step 6, in order to further minimize the search region. Then, the region selected after this step is smaller than the user-defined w ¼ 0.5. As shown in Figure 4, the 7th experiment could then be situated at x7 ¼ xi ¼ 1, because this limit has not been performed yet. When y7 ¼ yi > y6, the region [x6, x5] is eliminated and [x7 ¼ xi, x6] further examined. For the last experiment, it seems more logical to choose

Table 1 The n ¼ 7 steps of the Fibonacci search procedure for the situation W/w ¼ 20, with k ¼ 1, 2, . . ., n – 1 for wk, and k ¼ 2, 3, . . ., n for Wk Theoretical values k

Wk

1

W1 ¼ xj – x i

2

W2

3

W3

4

W4

5

W5

6

W6

7

W7

Numerical values wk

13 ¼ W1 21 8 ¼ W2 13 5 ¼ W3 8 3 ¼ W4 5 2 ¼ W5 3 1 ¼ W6 2

8 W1 21 5 ¼ W2 13 3 ¼ W3 8 2 ¼ W4 5 1 ¼ W5 3 1 ¼ W6 2 ¼0

w1 ¼ w2 w3 w4 w5 w6 w7

Wk

wk

10.00

3.81

6.19

2.38

3.81

1.43

2.38

0.95

1.43

0.48

0.95

0.48

0.48

0.00

In the last two columns, the results are given for the example, where the effect of varying the pH from 1 to 11 (W ¼ 10) is examined, until an optimal range of at the most 0.5 pH units is found (w ¼ 0.5).

554


x1

xi

1

x2

xj

W = W1 = 10 w1 = 3.81 x3

xi

2

w1 = 3.81 x1

x2

W2 = 6.19 w2 = 2.38

w2 = 2.38 x4

xi

3

x3

x1

W3 = 3.81 w3 = 1.43

w3 = 1.43 xi

4

x5 x4

x3

W4 = 2.38 w4 = 0.95 w4 = 0.95 xi

5

x6 x5 x4 W5 = 1.43 w5 = 0.48

x7 = xi x6 x5

6

W6 = 0.95 w6 = 0.48 x7 = xi x8 x6

7

W7 = 0.48

Figure 4 Fibonacci procedure for a continuously decreasing response with W ¼ 10 and w ¼ 0.5.

the center of the remaining region to further fine-tune the selection of the optimal region, instead of performing an experiment at the ‘‘smallest distance which is supposed to give a measurable difference in the response’’.1,2 Then, the last experiment would be performed at x8 ¼ 1.24, as shown in Figure 4. It can be remarked that the user-defined interval w, in which the optimum is to be situated, is already found in the one-but-last step. In the literature, a case study is found in Xiaoning et al.,5 where the Fibonacci procedure is used for the linearization of atomic absorption calibration curves. Two parameters, that is, the proportion of stray light s and a coefficient p, are individually optimized with the Fibonacci procedure. The considered response is the quality coefficient (QC)1 of the calibration line. When QC is minimal, the optimal factor settings are found. Then the straight line model will describe the data best. The Fibonacci search procedure is very effective concerning the number of necessary steps, that is, only a small number of experiments are required, especially when the optimal value must be indicated very accurately.1,2 For example, compare the Fibonacci approach with a preplanned regular interval design, where the response is determined at regular intervals of the factor levels. To obtain a region including the optimum, which is equal to one tenth of the original search region (e.g., 0–10 M natrium hydroxide), the Fibonacci approach requires only six steps or seven experiments because t6 ¼ 13 > W/w ¼ 10, whereas the regular interval design needs 11 experiments. The difference becomes even more obvious when the region including the optimum is smaller. For example, when obtaining a region equal to one thousandth of the original search region is required, 16 steps or 17 experiments are needed with the Fibonacci search procedure because t16 ¼ 1597 > W/w ¼ 1000, whereas the regular interval designs requires 1001 experiments, which is a very large number. Thus, the Fibonacci search procedure is very effective, especially when W/w becomes larger.


A disadvantage of the Fibonacci approach is that the procedure works only in the absence of or with a small experimental error. When the experimental error is large, this could lead to the exclusion of the wrong region.

1.17.2.2

Simplex Methods

1.17.2.2.1

Introduction A simplex is a geometric figure, defined by a number of points or vertices (corners) equal to one more than the number of factors examined in the optimization strategy. Thus, when optimizing f factors, f þ 1 points define the simplex in that factor space. The dimension of the simplex is equal to the number of factors. The vertices of the simplex each correspond to a set of experimental conditions.1–3,6,7 In the simplex methods, the simplex sequentially moves through the experimental domain. The next simplex to be performed is based on the results of the previous, and is defined according to specific rules. The movement of the simplex in the domain is in the direction of the optimum of the response. Two approaches can be distinguished, the (basic) simplex method, which was proposed by Spendley et al.,6 and the modified simplex method by Nelder and Mead.8 In the (basic) simplex method (Section 1.17.2.2.2), only a reflection of the simplex is performed to obtain the next simplex, and the size of the simplexes remains the same throughout the sequential procedure.1–3,6,7 In the modified simplex method (Section 1.17.2.2.3), a reflection, an expansion, or a contraction of the executed simplex occurs to obtain the next simplex.1–3,8,9 Thus, because in expansion or contraction steps the size changes, the simplex is modified. 1.17.2.2.2

The (basic) simplex method

Two-dimensional simplex method When applying the (basic) simplex method for the optimization of two factors (f ¼ 2), the simplex is an equilateral triangle (f þ 1 ¼ 3 points). This most simple situation will be used to explain and illustrate the method. Consider the imaginary response surface of a method, shown in Figure 5 as contour plot, which represents the response as a function of two factors (x1 and x2). The highest response value is considered to be the optimum.

1.17.2.2.2(i)

1.17.2.2.2.(i)(a)

Initial simplex

First three experiments (points 1, 2, and 3 in Figure 5) will be performed, according to the experimental conditions defined by the initial simplex (simplex 1).1–3,6,7 The size of the simplex is arbitrarily chosen by the analyst .

x2

9 S7

7 S5

5 S3 1 S1 2

S2

10

S4 4

S8

S6 6

8

S9 100

S 10

12

11 80 60

40

20

3 x1

Figure 5 Two-dimensional simplex: illustration of the (basic) simplex method. Si ¼ consecutively selected simplexes.

556


x2 R1 N

P1 B

W x1 Figure 6 Two-dimensional (basic) simplex BNW: B ¼ vertex with the best response, W ¼ vertex with the worst response, N ¼ vertex with the next-to-best response, P1 ¼ centroid of the remaining hyperspace (here, the line segment BN), and R1 ¼ reflected vertex.

Written in vector notation, the initial simplex is called BNW (Figure 6). In this simplex, B stands for the vertex with the best response, W for the one with the worst response, and N for the one with the next-to-best response. The points of the simplex B, N, and W can be represented by the vectors b, n, and w, that is, b ¼ [x1b , x2b], n ¼ [x1n, x2n], and w ¼ [x1w, x2w]. Based on the obtained results, the position of the second simplex in the experimental domain will be determined. 1.17.2.2.2.(i)(b)

Successive simplexes: calculation and evaluation of the reflected vertex

The (basic) simplex procedure is described by the following four rules.1–3,6,7 Rule 1: The new simplex is formed by keeping the two vertices from the preceding simplex with the best results, and replacing the rejected vertex, that is, the one with the worst result in the preceding simplex, with its mirror image across the line defined by the two remaining vertices.

Written in vector notation, as W yields the worst result from the initial BNW simplex, W is eliminated and replaced by its mirror image R1. If p1 (Equation (4)) is the vector representing the centroid P1 of the line segment BN, the coordinates of the new vertex are obtained by adding the vector (p1 – w) to p1. Thus, the vector r1 representing point R1 is defined as in Equation (5). 1 ½ðx1n þ x1b Þ;ðx2n þ x2b Þ p1 ¼ ðn þ bÞ ¼ 2 2

r1 ¼ p1 þ p1 – w ¼ 2p1 – w

ð4Þ ð5Þ

In the initial stages of a method optimization, the experiment, performed according to the experimental conditions defined by this new vertex, will usually lead to better results than at least one of the two remaining vertices of the preceding simplex, because the simplexes tend to move toward the optimum. By applying rule 1 to the example of Figure 5, the vertex that gives the worst response (point 2) is rejected, and the simplex is moved into an adjacent area, by projecting the rejected vertex through the average of the remaining vertices (points 1 and 3) to create a new vertex (point 4) on the opposite side of the simplex. The two remaining vertices (points 1 and 3) and the new vertex (point 4) then form the new simplex (simplex 2). An experiment is then run at the experimental conditions defined by this new vertex (point 4), and the procedure is repeated. For simplexes 2–6, defined according to rule 1, each time the new experiment yielded better results than at least one of the two remaining experiments of the preceding simplex. However, in case the new vertex does not yield better results, that is, does not move in the general direction toward the optimum, a change in the progression axis is necessary. In this situation, applying rule 1 is impossible, because this would lead to a reflection back to the vertex that was rejected from the preceding simplex. This is the case for simplex 7, which was formed by replacing the rejected point 6 with the new vertex (point 9) and keeping points 7 and 8 from the preceding simplex 6. In the new simplex 7, point 9 yields the least


desirable response. Applying rule 1 would lead to rejection of point 9 and a replacement by point 6. Further applying rule 1 would only lead to an oscillation between two simplexes (simplexes 6 and 7). In these situations, rule 2 is applied. Rule 2: In case the new vertex in a simplex corresponds to the worst result, rule 1 is not applied. In this situation, the vertex corresponding to the second lowest response is eliminated from this simplex and its mirror image across the line defined by the two remaining vertices is defined as the new vertex to form the new simplex.

Applying rule 2 results in a change of the direction of progression toward the optimum. This occurs most often in the region of the optimum. If a vertex near the optimum has been obtained, all new vertices will be situated further from the optimum, resulting in less desirable response values. Then, a change in direction is indicated. The consequence is that in the region of the optimum, the effect of applying rule 2 is that the consecutive simplexes will circle around the provisional optimal point. In the example of Figure 5, applying rule 2 to simplex 7 leads to a rejection of the vertex with the second lowest response (point 7). This point is replaced by its mirror image, that is, point 10. As can be seen from Figure 5, point 10 leads to the lowest result in simplex 8, defined by points 8, 9, and 10. Applying rule 2 leads to rejecting point 9 to replace it by point 11, and simplex 9 is then defined by points 8, 10, and 11. In the latter simplex, point 10 yields the worst result. Applying rule 1 leads to point 12, which in turn is again the vertex with the worst result in the new simplex 10, defined by points 8, 11, and 12. Thus, rule 2 would be applied, resulting in a replacement of point 11, that is, the second worst result in simplex 10, and thus leading back to point 6. As can be seen from the above and from Figure 5, point 8 is retained in successive simplexes, indicating that this point is as near to the optimum as one can get with the size of the initially chosen simplex and with the initially chosen start conditions. However, in practice, when the response surface is unknown, the above decision might be based on an erroneously high response obtained at point 8, and therefore, rule 3 is applied. Rule 3: When a certain point is retained in three (f þ 1) successive simplexes, the response at this point or vertex is redetermined. If the newly obtained result is the best, that is, the highest or the lowest depending on what is the most desired situation, compared to the points from the last three (f þ 1) simplexes, it is considered the best optimum that can be obtained with simplexes of the chosen size. On the other hand, if the obtained result is not the best, the simplex has become stuck to a false optimum, and then, it is better to start again.

One difficulty that needs to be resolved is what to do in practice when a situation occurs in which the experimental conditions, defined by a new vertex, are situated outside the examined or feasible experimental domain. To avoid problems, one identifies the constraints or the boundaries between which the simplex may move. For example, when the two factors (x1 and x2) represent concentrations, usually only values in a certain range are possible. The lower limit is then usually set at 0 (i.e., absence of reagent) and the highest is usually the highest concentration, which may be present or which is considered feasible. Once these boundaries have been set, rule 4 can be applied in given situations. Rule 4: If a point falls outside the boundaries or the chosen range of the factors, an artificially worst response, i.e., a low or a high response, depending on what is the least desired, should be assigned to it and one proceeds further with rules 1–3. This will force the simplex back into the boundaries.

Applying rule 4 leads to automatically rejecting the outlying point without bringing the succession of simplexes to an end. An example in which rule 4 had to be applied is shown in Figure 7, where an undesirable response is assigned to point 10. Thus, point 10 is considered the worst vertex of simplex 8. Therefore, rule 2 is applied, and the next-to-best point 9 is reflected and replaced by point 11, leading to simplex 9. In the next step, rule 1 is applied, and the worst point 10 is replaced by point 12, resulting in simplex 10.

Three- or more-dimensional simplex method The abovedescribed (basic) simplex method for two factors can be generalized to an f-factor case. Although the two-dimensional simplex can be visualized geometrically, this is no longer possible for three- or more-dimensional simplexes. Nevertheless, the 1.17.2.2.2(ii)

558


x2 11

•

S 10

S9

10

•

80

•12 60

40

8

•

S8

•9

20

6

100

•

S6

S7

•

4

•

S4

S5

7

• 5

•

S2

S3 1•

3

S1

•2 x1 Figure 7 Two-dimensional (basic) simplex: example in which rule 4 had to be applied.

principle remains exactly the same. When examining f factors, the simplex contains f þ 1 vertices or points. After determination of the vertex to be rejected, the coordinates of the new vertex are obtained as follows. The coordinates of the f retained vertices are summed for each factor and divided by f/2. From the resulting values, the coordinates of the rejected point are subtracted, yielding the coordinates of the new vertex. The above are generalizations of Equations (4–5). In vector notation, when examining f factors, the f þ 1 points of the initial simplex, B, N1, N2, . . ., Nf–1, and W can be represented by the vectors b, n1, n2, . . ., nf–1 and w, that is, b ¼ [x1b, x2b, . . ., xfb], n1 ¼ [x1n1; x2n1; . . .; xfn1 ], n2 ¼ [x1n2; x2n2; . . .; xfn2 ], . . ., nf –1 ¼ [x1nf – 1; x2nf – 1; . . .; xfnf – 1 ], and w ¼ [x1w, x2w, . . ., xfw]. B and W are the best and the worst results, respectively. N1, N2, . . ., Nf1 are neither the best nor the worst results. The centroid P1 of the hyperspace remaining after deleting the vertex giving the worst response W, and represented as vector p1, is then defined as given in Equation (6), and the coordinates of the new vertex R1, represented as vector r1, are given by Equation (5). Furthermore, the same rules, as given above, are valid.

n1 þ n2 þ ::: þ nf –1 þ b p1 ¼ f

x1n1 þ x1n2 þ þ x1nf 1 þ x1b ; x2n1 þ x2n2 þ þ x2nf 1 þ x2b ;:::; xfn1 þ xfn2 þ þ xfnf 1 þ xfb ¼ f

1.17.2.2.3

ð6Þ

The variable-size or modified simplex method In the (basic) simplex method described above, as proposed by Spendley et al.,6 the step size is fixed, that is, the size of the initially chosen simplex determines that of all successive simplexes. However, when the initial size is too small, finding the optimum might require too many experiments, while, when the step size is too large, the optimum might be insufficiently approached. Indeed, as mentioned above, only that point, as near to the optimum as can be obtained with the size of the initially chosen simplex, is found. In the latter situation, a new simplex could be started around the provisional optimum. The new simplex should then have a smaller step size.10 This is represented in Figure 8, where after simplex 7, the step size is decreased and the simplex procedure is repeated (simplexes 8–14). Nevertheless, the modified simplex method, initially proposed by Nelder and Mead,8 offers a better alternative. In this method, the step size is variable throughout the whole procedure. Furthermore, the


x2

100 S12 S13 S14 8 S11 S10 S9 S8

• •

9

80

• • •

60

40 20

6

•

S6

S7

•

7

4

•

S4

S5

•

•

S2

S3

5

1•

3

S1

•2 x1 Figure 8 Example where the step size of the (basic) simplex procedure is decreased and the procedure repeated, to approach the optimum of the method more accurately.

principles and the rules of the (basic) simplex method are retained. Additionally, information concerning the expansion or contraction of simplexes is provided. In search directions that seem favorable, the size of the simplex is expanded to accelerate the procedure. On the contrary, in directions that seem unfavorable, the size of the simplex is contracted to slow down the procedure. Thus, besides reflection, two additional operations, that is, expansion and contraction, are possible in the modified simplex procedure.1–3,8,9,11,12 The modified simplex method for two-dimensional simplexes is described below. For three- and more-dimensional simplexes, the method can be adapted, as was also the case for the (basic) simplex method.

1.17.2.2.3(i) Initial simplex When examining two factors, again the simplex is a triangle, but now one that is not necessarily equilateral anymore (Figure 9).1–3,8,9,11 Three experiments are performed according to the experimental conditions of the vertices. In vector notation, the points of the simplex BNW can again be represented by the vectors b, n, and w, with b ¼ [x1b, x2b], n ¼ [x1n, x2n], and w ¼ [x1w, x2w]. As for the (basic) simplex method, B stands for the vertex with the best response, W for the one with the worst response, and N for the one with the next-to-best response. Successive simplexes: calculation and evaluation of the reflected vertex The vertex W, corresponding to the worst response, is rejected and reflected through the centroid P1, represented by the vector p1 (Equation (4)), to obtain point R1, represented by the vector r1 (Equation (5)). Then, the experiment is performed according to the experimental conditions of this new point R1, and depending on the result of the experiment at this reflected vertex, three situations are possible.1–3,8,9,11 In the following, it is again assumed that the best response corresponds to the highest, that is, the optimum is at maximal response values. In the first situation, the response in R1 is higher than that in B, in the second it is lower than or equal to that in B but higher than or equal to that in N, and in the third and last situation the response in R1 is lower than that in N. Depending on the situation, different steps are undertaken.1–3,8,9,11 1.17.2.2.3(ii)

560


x2 E1

R1 CR,1

N

P1 Cw,1

B

W x1 Figure 9 Two-dimensional modified simplex BNW: B ¼ vertex with the best response, W ¼ vertex with the worst response, N ¼ vertex with the next-to-best response, P1 ¼ centroid of the remaining hyperspace (here, the line segment BN), R1 ¼ reflected vertex, E1 ¼ expansion vertex ( ¼ 2), CR,1 the contraction vertex on the reflection side ( ¼ 0.5), and CW,1 = the contraction vertex on the worst side ( ¼ 0.5).

1.17.2.2.3(ii)(a)

Situation 1: response at R 1 > response at B

This implicates that the simplex moves in a favorable direction. An attempt to expand the simplex is made, by generating the expansion vertex E1, represented by vector e1, and defined as follows:

e1 ¼ p1 þ p1 – w

ð7Þ

where is the expansion coefficient, representing the expansion of the simplex, and > 1. The larger the value of , the more the simplex will be expanded. Usually is chosen equal to 2, as in Figure 9. When the response in E1 is also better than or equal to that in B, the expansion is considered successful and the simplex to decide on the next experiment is BNE1. On the contrary, if the response in E1 is worse than that in B, the expansion has failed and BNR1 is taken as the simplex to decide on the next experiment. Furthermore, the usual procedure is followed, that is, the worst vertex of the new simplex is rejected and reflected, and so on. To select the simplex to be used to decide on the next experiment, a small modification to the above was introduced in Morgan et al.,9 and Aberg and Gustavsson,13 where, instead of comparing E1 with B, the result at the expansion vertex, E1, is compared with that at reflection vertex, R1, and the one with the most desirable result is retained. In our opinion, this modification seems quite logical. 1.17.2.2.3(ii)(b)

Situation 2: response at N response at R 1 response at B

Neither expansion nor contraction is considered, and the next simplex is determined from BNR1, using the classic rules. 1.17.2.2.3(ii)(c)

Situation 3: response at R 1 < response at N

This implicates that the simplex has moved too far. It should thus be contracted. Two possibilities are distinguished. In the first, the response at R1 is better than or equal to that in W, and the new vertex CR,1, represented by vector cR,1 and situated nearer to R1 than to W (Equation (8)), is selected (Figure 9).

cR;1 ¼ p1 þ p1 – w

ð8Þ

is the contraction coefficient, representing the contraction of the simplex, and 0 < < 1. The smaller the value of , the more the simplex will be contracted. Usually, is chosen equal to 0.5, as in Figure 9. The response at this vertex is then determined. When a better or equal response is obtained in CR,1 than in R1, the contraction is considered successful, and the simplex to determine the following experiment is BNCR,1.


Otherwise, when a worse response is found, the contraction is considered a failure, and the simplex from which the next experiment is determined remains BNR1.9 Furthermore, the usual classic procedure is followed, that is, the worst vertex of the new simplex is rejected and reflected, and so on. The second possibility is that the response at R1 is worse than that in W, and then the new vertex CW,1, represented by vector cW,1 and situated nearer to W (Equation (9)), is selected.

cW;1 ¼ p1 – p1 – w

ð9Þ

The response at this vertex is then determined. When the response obtained in CW,1 is better than or equal to the one in R1, the contraction is found successful, and the simplex for further use is BNCW,1. On the contrary, when the obtained response is worse, the contraction is said to have failed, and the simplex for further use remains BNR1.9 Furthermore, the usual procedure is followed, that is, the worst vertex of the new simplex is rejected and reflected, and so on. Furthermore, if a certain point is retained in f þ 1 successive simplexes, the response at this point or vertex is redetermined.3,11 If again the most desirable result is obtained, compared to the points from the last f þ 1 simplexes, it is considered to be near the optimum. On the contrary, if the replication does not confirm the desirable result, the simplex has been misled because of the erroneously high result. Then several possibilities exist to proceed, for example, one can take the most recent response at the successively retained vertex,3 the average result of both responses at this vertex,11,13 or a weighted average of both responses at this vertex3 as the response for this vertex. When using a weighted average, the most recent response is given the highest weight. A difficulty is to define a criterion to stop the (modified) simplex procedure. Besides the budget which should be taken into account when planning and performing experiments,3 possibilities to stop are when the predefined aim of the optimization or when a given improvement has been achieved.3,9 In case the analyst has no idea of what the optimum should be, a termination criterion could be used.3,9,11,13 Possible criteria are the step size, which becomes smaller than a predefined value, for example, 1% of the domain of each examined factor,9,11 or the differences in response, which approach the experimental error. When the latter actually is unknown, it is simply set at a predefined value.3,8,9,11 Another possibility is that two coupled convergence criteria are both fulfilled.9,13 The first criterion is that the standard deviation (SD) of all vertex responses of a simplex should be below a preset value. The second is that the root-mean-squared (RMS) deviation of the vertex responses from the response of the gravity center of the simplex should be below the same preset value as the SD (Equation (10)). "P RMS ¼

R – Rg f þ1

2 #1=2 ð10Þ

In Equation (10), R is the response at a vertex, f the number of examined factors, and Rg the response at the gravity center of the simplex. Furthermore, when the boundaries of the experimental domain are exceeded, an undesirable response value is assigned to that vertex, as with the (basic) simplex method. This forces the simplex back inside the experimental domain, defined by the limits of the factors.

1.17.2.2.4

Discussion The simplex methods are based on the hypothesis that most experimental response surfaces are smooth and that a sequential approach thus can be used to find the optimum. The simplex methodology is often used, both in numerical and in experimental optimization. In the latter, it is a very useful tool for evaluating a single performance criterion, because one moves rapidly toward the optimum.1–3,7,9 It is also a very useful tool when improvement, but not necessarily optimization, that is, determination of the global optimum, is wanted.1 The method is so simple that the analyst might forget the difficulties in its use and think that it always leads straight to the optimum. On the contrary, several analysts are not in favor of using simplex methods because, according to them, they only offer information about the location of the optimum, but not about the influence of

562


the factors involved.2 For statistical significance testing of the factor effects, experimental design approaches indeed seem more appropriate. There are also several disadvantages related to the use of simplex methods. 1. A first difficulty arises from the selection of the factors to be examined in the simplex approach. Preferably, only the most important factors are to be selected. This choice can be based on prior knowledge of the method or on the results of preliminary experiments,7 for example, from performing a screening design. Although evaluating two factors results in a method that is easily graphically illustrated, considering two factors should only be done when indeed only two are important or when a third important factor has a much lower influence on the response.7 In fact, it is best to include as many factors as can be handled conveniently.14 For analytical methods, the concentration of the component to be determined should best not be selected as factor, because often the maximum response will proceed in the direction of increasing concentration.7 This is, for example, the case when absorbance, peak area, or peak height is selected as response. In fact, in later application of the method, the concentration of the component is the response to be determined, which is modeled through the measured signal, such as peak area. Therefore, this concentration is to be considered a response, instead of a factor. The same remark is valid in experimental design approaches. The simplex methods cannot be applied to examine noncontinuous variables.2 When examining a factor with only discrete levels, an experimental design approach should be used. However, in the latter, modeling as a function of the factor levels is also not possible. In general, modeling from simplex approaches is not recommended either (see 1.17.2.2.4, disadvantage 6). When two factors are dependent, only one can be chosen to be examined.7 This is, for example, the case for the fractions of solvents in the mobile phase of chromatographic methods. The general rule here is that when a mixture consists of p components, only p – 1 can be examined during optimization. However, this is also the case for experimental design approaches, thus not really a problem that is inherent in the simplex methods. 2. A second difficulty, besides the selection of the factors, is the determination of the first simplex (size, orientation, and location).2 Two setup simplexes, that is, a cornered and a tilted simplex, are proposed to determine the experiments of the vertices of the initial simplex.3 The tilted simplex is the regular equilateral simplex, first presented in Spendley et al.,6 and which has been considered in the above (basic) simplex method. It is illustrated in Figure 10(a). The cornered simplex is an isosceles triangle with one angle of 90 between two sides of equal length. It fits into the corners defined by the factor axes when translated to the origin. It is illustrated in Figure 10(b). The placement and size of the initial simplex depends on the purpose of the optimization.3 In cases where the acceptable experimental conditions are already more or less known, the simplex approach is used to improve the performance of the method. In this situation, small simplexes are desirable, and to avoid that small simplexes become large ones, preferably the (basic) simplex method is applied. On the contrary, in situations where less is known concerning the best experimental conditions, the simplex method is used for the purpose of optimization. Thus, large simplexes are desirable, and preferably the modified simplex procedure is applied, to enable a change from larger to smaller simplexes. The importance of the first simplex is evaluated for both the basic and the modified simplex methods.15 The influence of the shape of the first simplex was evaluated with three different setups, that is, cornered, tilted, and modified optimal simplex. The modified optimal simplex algorithm uses a D-optimal linear design matrix to

x2 (a) Tilted

(b) Cornered

(c) Optimal

x1

Figure 10 Shape of the initial simplex: (a) tilted, (b) cornered, and (c) modified optimal simplex.


define the shape of the first simplex.15 This is illustrated in Figure 10(c). When the response surface studied is a second-order model, the optimal first simplex is superior, and the tilted simplex performs better than the cornered simplex. 3. The simplex can move in wrong directions, especially when the imprecision of the measurement, because of natural variability, large random errors, or system noise, is relatively large and the slope of the response surface studied is small.1,7,9 Although the wrong move often can be corrected in a subsequent move, it makes the method inefficient, because more experiments are required. 4. The simplex approaches will find the global optimum when only one optimum is present. With different local optima, the simplex methods will only find one of the optima, that is, a local optimum, which will not necessarily be the best one or the global optimum.1,2 In many situations, one knows that there is only one optimum, but situations exist where several optima are found. The most commonly known example occurs in the optimization of chromatographic or electrophoretic separations. When, in the examined factor interval, the elution order of two components X and Y changes, there will be a local optimum for the elution sequence XY and one for YX. To examine whether only one optimum exists, the (simplex) optimization should be performed twice from different experimental starting conditions. When the same optimum is obtained, one can be reasonably sure that only one exists. 5. Determining the optimum is not always evident when more than two factors are examined, because from higher-dimensional simplexes it is not always clear when the optimum has been reached.7 6. When not only the optimum of a single performance criterion is required but the response also should be modeled, the sequential optimization strategies are not appropriate, because the simplex methods do not provide much information about the response surface.1 In theory, the experimental results could be used to model the surface, but the points are chosen in a haphazard way. Therefore, it is not evident to map the whole experimental area of interest from the performed experiments, and it is probable that the experimental setup, defined by these experiments, will not correspond to the optimality criteria of Chapter 1.12. For the purpose of modeling a response, it is better and advised to apply response surface designs.1 These designs are described in Chapter 1.12. 7. When evaluating more than one performance criterion, that is, with probably their optima at different locations, the simplex procedure, as discussed above, cannot be applied. A possibility to find the optimum or the best compromise between the multiple responses is to combine the simplex method with an MCDM method,1 such as Derringer desirability functions.16 Some other MCDM methods, such as Pareto optimality methods,17 cannot be combined with the simplex method, because they do not result in one response value. The MCDM methods have been thoroughly described in Chapter 1.19. In the Derringer approach, the different response values are first transformed to desirability values (di), all situated between 0 and 1. These are then combined to a D-response, also situated between 0 and 1, representing a very undesirable and a very desirable situation, respectively. Nevertheless, in situations where several responses need to be optimized simultaneously, a more common approach would be the execution of a response surface experimental design, combined with an MCDM technique, such as Pareto optimality methods17 or Derringer desirability functions.16 When comparing the (basic) simplex and the modified simplex methods, some remarks can be made. A disadvantage of the (basic) simplex method, as was already indicated higher, is that there is no possibility to accelerate the progress along the response surface.7,9,12 The number of experiments is largely determined by the size of the initial simplex. Especially when the starting conditions in the optimization procedure are situated well away from the optimum, this can cause problems when using the (basic) simplex method. When, in those situations, a very small size is chosen, too many experiments need to be performed.3 This problem does not occur when using the modified simplex method, because with this method, the size is variable throughout the whole procedure. On the contrary, a disadvantage of the modified simplex method is that the simplicity of the calculations, as is the case for the original basic simplex method, no longer exists.1,2 In the literature, several modifications to the basic or modified simplex procedures have been suggested in order to approach the optimum more rapidly, accurately, or reliably. Below, a short overview is given. 1. A difference in opinions exists to what should be done when the contraction of the simplex has failed in the modified simplex procedure (see the above situation 3). A further contraction, called massive contraction,

564


x2 N

B

W x1 Figure 11 Massive contraction in the two-dimensional modified simplex procedure.

has been suggested in Nelder and Mead,8 reducing the simplex even more. This is illustrated in Figure 11. It can be observed that the contraction is different from that considered in the modified simplex procedure of Figure 9. However, two disadvantages should be considered for this approach. First of all, it requires performing f new experiments, according to the experimental conditions defined by the new vertices. Second, because the simplex is considerably contracted, this might give rise to a premature convergence or stopping of the procedure, in the presence of experimental error.7 The massive contraction principle was also performed in the basic simplex approach of Figure 8 (S8), in order to reduce the simplex size. Instead of making a massive contraction when a failed contraction occurs in the modified simplex approach, the vertex with the next-to-best response, N, should be rejected from the simplex and reflected, instead of the one with the worst desirable response, W.9,13 This is similar to rule 2 of the (basic) simplex method. Thus, to deal with a failure of the contraction toward the reflection vertex, N is rejected from the simplex BNCR,1 and replaced by its reflection vertex R91, represented by vector r91, to obtain the new simplex BCR,1R91 (see Figure 9). Similarly, to deal with a failure of the contraction toward the worst vertex, N is rejected from the simplex BNCW,1, and replaced by its reflection vertex R91, represented by vector r91, to obtain the new simplex BCW,1R91 (see Figure 9). 2. Another small modification to the modified simplex procedure was performed.9,13 It was found that comparing the result at the expansion vertex, E1, with that at reflection vertex, R1, and retaining the one with the most desirable result performed better than comparing E1 with the best vertex of the simplex, B, as was proposed in the modified simplex method. This modification was already mentioned earlier. 3. The supermodified simplex was proposed in Routh et al.,18 and evaluated in Morgan et al.19 After rejecting the worst vertex W and replacing it by its reflected vertex R1, in this approach, the optimum of a second-order polynomial model through W, R1, and P1 is determined (Figure 12).18,19 An additional experiment at the centroid P1 thus should be carried out, to obtain the three results. In general, the coordinates of each point, situated on the axis formed by W, R1, and P1, can be described with Equation (11). r ¼Mp1 þð1 – M Þw

ð11Þ

The M value in Equation (11) is a coefficient, determining the situation of a given point. For instance, the previously considered reflection vertex is found at M ¼ 2, the expansion vertex at M ¼ 3, and the contraction vertices toward R1 or W at M ¼ 1.5 and M ¼ 0.5, respectively. The optimal point Ropt is then predicted as the point which maximizes or minimizes the response, depending on what is the most desired situation. Ropt, represented by vector ropt (Equation (12)), is thus considered the new vertex of the simplex BNRopt, and the result of the experiment, conducted at the experimental conditions defined by this vertex, is evaluated.18,19 This is represented in Figure 12.

ropt ¼Mopt p1 þ 1 – Mopt w

ð12Þ


x2 R1,opt

R1

N P1 B

W x1 Figure 12 Two-dimensional supermodified simplex BNW: B ¼ vertex with the best response, W ¼ vertex with the worst response, N ¼ vertex with the next-to-best response, P1 ¼ centroid of the remaining hyperspace (here, the line segment BN), R1 ¼ reflected vertex, and R1,opt ¼ vertex corresponding to the optimum of a second-order polynomial model through W, R1, and P1.

The optimal M value, Mopt in Equation (12), is determined from Equation (13). Mopt ¼

w – p1 w – 2p1 þ r1

þ 0:5

ð13Þ

When the response needs to be maximized, the requirement w – 2p1 þ r1 < 0 should be fulfilled.19 In this supermodified simplex approach, a second modification has been made to the simplex procedure, that is, what to do when the boundaries of the factors, defining the experimental domain, are violated.18,19 The supermodified simplex deals with the crossing of a limit in a completely different way than described earlier. This has been described in Morgan et al.19 An M value corresponding to the crossed boundary is calculated and then used to calculate the conditions for both boundary-crossing reflection and optimal vertices.18 The supermodified simplex method allows following the response surface more closely and finds the optimum faster. Thus, this method is claimed to lead to an increased utility, reliability, and efficiency of the simplex procedure.18 4. Weighted centroid methods were proposed in Ryan et al.,20 and are based on the modified simplex method. The centroid of the remaining hyperspace after elimination of the worst vertex is weighted according to the values of the responses at the remaining vertices. The worst point is then replaced by its symmetrical point through this weighted centroid, and thus, a new simplex is obtained. A new experiment is then performed, according to the experimental conditions, defined by this reflected vertex. The approach is shown in Figure 13. x2

N

R1

PW B

W x1 Figure 13 Two-dimensional simplex BNW of the weighted centroid method: B ¼ vertex with the best response, W ¼ vertex with the worst response, N ¼ vertex with the next-to-best response, PW ¼ weighted centroid of the remaining hyperspace (here, the line segment BN), and R1 ¼ reflected vertex.

566


As weighted centroid methods successfully predict gradient directions but tend to become degenerate in following them, two variants are proposed to solve this degeneracy problem, that is, the controlled weighted centroid method and the orthogonal jump weighted centroid method.13,20 Both methods apply a different algorithm to solve degeneracy. Advantageous is that these methods allow approaching the optimum faster. For more information, we refer to Aberg and Gustavsson13 and Ryan et al.20 5. Further modifications are described in the literature. However, they are often more complex. For instance, the applied modification to the supermodified simplex approach determines the optimum of a Gaussian model through W, R1, and P1, instead of from a second-order polynomial model.21 Then lnðwÞ – 2 ln p1 þ lnðr1 Þ < 0 is the requirement in order to find a value for Y1,opt, which maximizes the response. The Gaussian model is claimed to be more efficient in approaching the optimum than the second-order model.19,21 Boundaries are set to M1,opt, that is, –1 M1,opt 3, in the supermodified simplex, to allow the simplex to grow maximally twice the size of the initial simplex.22 When no restrictions are set, a minor change in a response might cause a large effect on the size of the simplex. In this modification, also further restrictions are set to deal with the boundary violations. It is suggested that the M value is only accepted, when it is larger than or equal to 1.5, otherwise an M value of 0.5 is applied. Second, the estimation of M1,opt depends on an even spacing of R, P1, and W. When the boundaries have been violated, this should be taken into account to calculate M1,opt. For the latter situations, an adjustment of the equations to calculate M1,opt, for both the second-order and the Gaussian Y1,opt estimates, is required. More detailed information is described in Morgan et al.,19 and van der Wiel et al.22 An extension of the modified simplex is presented that is able to deal with constrained nonlinear optimization problems.23 A method, called the advanced regular simplex method, is explained by Shao.24 Hedlund and Gustavsson25–27 examined different modifications to the modified simplex method, in order to improve the speed, reliability, and convergence ability of the procedure. The reflection, expansion, and contraction coefficients are further optimized in Hedlund and Gustavsson.25 Both a simplification of the degeneracy control and a correction for the vertices located outside a variable boundary back to within the boundaries are examined in Hedlund and Gustavsson.26 A degeneracy constraint is evaluated in Hedlund and Gustavsson.27 For more information, we refer to the references given above. 1.17.2.2.5

Case studies Many case studies have been described in the references mentioned earlier. Walters et al.,3 contains a chapter overviewing the simplex optimization evolution from 1962 to 1990. In Morgan and Deming,11 the simplex optimization of analytical chemical methods is discussed, and in Berridge,12 a review on the simplex optimization of high-performance liquid chromatography (HPLC) separations is given. For chromatographic separations, a problem for simplex optimization is that often several responses, for example, retention times and resolutions between peaks, are examined. To obtain a single response, describing the quality of a global chromatogram, different chromatographic response functions (CRFs) have been proposed.12 An alternative is the use of the Derringer’s desirability functions, as already discussed above. Below, some optimizations from the literature are given, in which a simplex procedure was applied. Mainly publications from the last 10 years were reviewed. The list of publications is certainly not exhaustive. For each case study, the selected factors, the applied simplex procedure, and the examined response or evaluation criterion are given. Only a few examples using the (basic) simplex procedure were found.28,29 The modified simplex method30–44 or some modification,45 on the contrary, is more often applied. In some publications, the type of simplex procedure is not specified and cannot be derived from the paper by Dol and Knochen.46 The (basic) simplex procedure was used to optimize the derivatization reaction (esterification of phenoxy acids) prior to the determination of herbicides in soil samples with gas chromatography (GC).28 Examined factors are methylation temperature and time, shaking time, and solvent volume. The evaluated criterion is the peak area ratio (derivatized compound/internal standard). A sequential injection analysis (SIA) method was optimized to determine two pharmaceutical analytes, that is, amoxicillin and clavulanic acid.29 First, a fractional factorial design was performed to examine the effects of six factors, in order to determine the starting conditions of the initial simplex. Then, the simplex approach is used to further optimize the six factors. No reduction in the number of factors was made, based on the results of the fractional factorial design, as one might have expected. Examined factors are solvent volume, sample volume, concentration of acid and base, and flow rate. Evaluated criteria are the lack-of-fit of the model,


distances between the peaks, correlations between spectra, and quantification errors. The Derringer desirability functions were used to obtain one response to be maximized, that is, the overall desirability function D. The modified simplex procedure was applied for the optimization of the lipase activity from Rhizopus species in the triglyceride hydrolysis of olive oil,30 for the off-line optimization of temperature-programmed GC separations of a test mixture containing 16 components with different functionality,31 and for the optimization of a flow injection method to determine furfural.32 For the latter, different responses were combined with an algebraic function into one response. Also several purge-and-trap extraction methods on soil samples, performed during the determination of a mixture of phenols33 and of nine volatile organic compounds,34 with GC-FID (flame ionization detection) were optimized using the modified simplex procedure and the fuzzy set theory.1 This was also the case for the solidphase microextraction, performed prior to the GC-MS (mass spectrometry) determination of polycyclic aromatic hydrocarbons, polychlorinated biphenyls, and phthalates in water samples.39 The fuzzy set theory method is an MCDM strategy that combines different responses into one joint response, with values between 0 and 1. The higher the joint response (called the membership value), the closer the optimum (which represents a suitable compromise between the different responses) is approached. The most important variables for the determination of fluticasone propionate with flow injection analysis (FIA) were first selected with a screening experimental design – here an asymmetrical design where some factors are evaluated at two levels and others at more.35 Two significant factors are then further optimized with the modified simplex procedure. The evolution of the simplex can be followed in Table 2 and Figure 14. The assays of some heavy metals with ion-pair reversed-phase HPLC,36 of the antihypertensive drug naftopidil with a luminescence technique (phosphorescence),37 of 4-methylpropranolol in cerebrospinal fluid, serum, and urine with a nonprotected fluid room-temperature phosphorescence method,38 of a mixture of polycyclic aromatic hydrocarbons with HPLC,40 of the nonsteroidal anti-inflammatory drug (NSAID) nabumetone in pharmaceutical preparations,41 and of its major metabolite in human urine,42 all were optimized with the modified simplex approach. Also the growth and carotenoid production in Rhodotorula glutinis43 was optimized using this approach. Both responses, cell growth and carotenoid production, were transformed to one response function, which was applied for optimization. Another example is found in the optimization of a capillary zone electrophoresis method for the analysis of amphetamine, amphetamine analogues, cocaine, and heroin.44 First, a full factorial design was applied to screen three variables, and then, the modified simplex method was used for the final optimization of the two most important factors.

Table 2 Evolution of the modified simplex approach for the determination of fluticasone propionate with flow injection analysis Factors

Response

Vertex

Experiment and step type

x1

x2

Peak height

1-2-3 1-2-3 1-2-4 1-2-4 2-4-6 2-4-6 2-4-6 2-6-10 2-6-10 6-10-12 6-10-13

1 2 3 4 (R:3) 5 (E:3) 6 (R:1) 7 (E:1) 8 (R:2) 9 (CW:2) 10 (R:4) 11 (R:2) 12 (CW:2) 13 (R:12) 14 (R:10)

62.0 66.7 63.2 65.5 66.7 70.2 74.4 69.1 67.3 71.4 75.0 68.8 72.9 71.7

0.580 0.586 0.604 0.562 0.542 0.568 0.562 0.545 0.576 0.592 0.574 0.583 0.577 0.553

391066 498882 358817 531687 0 642704 0 0 498807 592610 0 570506 638893 630364

x1, temperature ( C); x2, fraction methanol.

568


0.61 3

0.60 10

0.59 2 12

1

0.58

13

x2

9

0.57

11

6 4

0.56

7 14

0.55 8 5

0.54 0.53 60

62

64

66

68 x1

70

72

74

76

Figure 14 Evolution of the modified simplex approach for the determination of fluticasone propionate with flow injection analysis. 1, 2, 3 ¼ initial simplex, and 4, 5, . . ., 14 ¼ sequentially selected vertices. Reproduced from Vannecke, C.; Nguyen Minh Nguyet, A.; Bloomfield, M. S.; Staple, A. J.; Vander Heyden, Y.; Massart, D. L. Development and Optimisation of a Flow Injection Assay for Fluticasone Propionate Using an Asymmetrical Design and the Variable-Size Simplex Algorithm. J. Pharm. Biomed. Anal. 2000, 23, 291–306.

The selected factors and the examined responses of the optimizations described above, all using the modified simplex approach, are summarized in Table 3. Not only chemical variables but also instrumental variables were optimized with the modified simplex procedure.37,38,41,42 The response for the chemical variables was the phosphorescence intensity, for the instrumental variables it was the signal-to-background noise ratio.

Table 3 Selected factors and examined responses from case studies using the modified simplex procedure References Factors pH Temperature Concentration of reagent Salt concentration Buffer concentration Ion-pairing agent concentration Acid or base concentration ( pH) Sodium sulfite concentration (for redox reaction) Surfactant concentration Thallium nitrate concentration % solvent Composition of the mobile phase at beginning and end of gradient Slope of the gradient Length of time interval at final mobile phase composition in gradient elution Initial or midpoint column temperature Oven temperature rate (slope of temperature gradient) Oven temperature Heating time Purge time Extraction time and temperature of SPME

44 35 32 33,34,39 36,44 36 37,38,41,42 37,38,41,42 41,42 41,42 35,36 40 40 40 31,40 31 33,34 33,34 33,34 39 (Continued )

Sequential Optimization Methods 569 Table 3

(Continued) References

Desorption time and temperature of SPME Length of the reaction coil Injection volume Flow rate Instrumental variables for phosphorescence (voltage of photomultiplier detector, delay time, gate time, time between flashes) Hydrolysis parameters (pH, temperature, colloid concentration, emulsifier concentration) Fermentation parameters Responses Hydrolysis rate Chromatographic response function (CRF) Algebraic response function Peak area Peak height Joint response from fuzzy set theory Intensity of phosphorescence Signal-to-background noise ratio

39 32 32 32,40 37,38,41,42 30 43 30 31,36,40,44 32,43 33,34,39 35 33,34,39 37,38,41,42 37,38,41,42

SPME, solid-phase microextraction.

A supermodified simplex procedure was used for the separation and quantification of some psychotherapeutic benzodiazepines with HPLC.45 A second-order or a Gaussian model was fitted through the vertices W, P1, and R1, in order to predict the point with the most optimal response. Selected factors were the composition of the mobile phase, the flow rate, and the sample volume. The examined response was a CRF. A sequential simplex method was used for the optimization of an FIA method to determine salbutamol.46 Nevertheless, it is neither specified whether the basic or modified simplex approach was followed nor specified whether modifications were applied. Selected variables were lengths of the reaction coils and reagents concentrations. The examined response was peak height.

1.17.3 Mixed Sequential–Simultaneous Optimization Methods The mixed sequential–simultaneous optimization methods combine a simultaneous method, for example, an experimental design approach, with a sequential one. An example is the steepest ascent method.1,2,47 The steepest ascent method tries to find the optimum of a response surface without modeling it mathematically. In fact, only the gradient along the surface is described, and after determining the direction of the gradient, experiments are carried out in a sequential way along this line of steepest ascent.2 Local full factorial designs can be applied as hill-climbing methods.1 In the following, this method is explained for the optimization of two factors affecting one response. However, the method can be extended to more than two factors. Consider the four experiments from a 22 full factorial design (Figure 15). The response y can be described as a function of the factors x1 and x2, y ¼ b0 þ b1 x1 þ b2 x2

ð14Þ

with b0, b1, and b2 the model coefficients.1 For simplification, the interaction term x1x2 with coefficient b12 was neglected. The center point of the design has been replicated, yielding y01, y02, and y03, which allows carrying out a curvature check. If there is no curvature, Equation (14) describes the response at the center point well. The thus defined response surface is a plane and the best response value in the examined domain must be situated on the boundaries defined by the first four experiments.

570


x2

+1

y3 y0

0 –1

y4

y1 –1

y2 0

+1

x1

2

Figure 15 A 2 full factorial design with center point. y0 is the center point result and yi are the design results.

x2 b1 g

y0 b2

x1 Figure 16 Direction of the gradient in the steepest ascent method, given by a line segment g ¼

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi b 21 þ b 22 .

Suppose that the highest response is the optimum and that both b1 and b2 are negative, as in Figure 16. Possibly, an optimum is situated outside the examined domain, in the direction of lower values of the x1 and x2 variables. Thus, it would be logical to perform an experiment in that region of the experimental domain.1 The slope in the x1 direction is proportional to b1 ¼ ½ðy2 þ y4 Þ – ðy1 þ y3 Þ. Similarly, the slope in the x2 direction is proportional to b2 ¼ ½ðy3 þ y4 Þ – ðy1 þ y2 Þ. The direction of the slope in the steepest ascent method procedure represented in Figure 16. The direction of movement is given by a line segment g, with a length ffiffiffiffiffiffiffiffiffiffiffiffiffi ffi pis equal to b12 þ b22 .2 The path of the steepest ascent defines that for every unit of movement in the direction of x1, one should move b2/b1 units in the direction of x2.1 When the chosen real length of the step is L ¼ constant g, the displacement in the x1 direction ispðbffiffiffiffiffiffiffiffiffiffiffiffiffi 1 =gÞL,ffi and in the x2 direction ðb2 =g ÞL.2 The length of the step L can be based on intuition, equal to Lp¼ffiffiffiffiffiffiffiffiffiffiffiffiffi g ¼ ffi b12 þ b22 as was used in the examples below, or defined as proposed in Brooks,47 that is, L ¼ 2g ¼ 2 b12 þ b22 . For the latter situation, the displacement in the x1 direction then is 2b1, and in the x2 direction 2b2. Additional experiments are then carried out with steps of equal length, until the results of the last experiment (Eover) indicate that the optimum was overshot.1 Then, a new 22 full factorial design is performed around the provisional optimum, defined by the one-but-last experiment (Eover – 1). By replicating this new center point (Eover – 1), the curvature of the response surface again can be evaluated, which, in this situation, corresponds to evaluating whether the optimum is situated in this experimental domain. When it is decided that the response is curved in this region of the experimental domain, it is concluded that the optimum is situated in this region.1 The use of the steepest ascent method is represented in Figure 17. The step length L was chosen equal to pffiffiffiffiffiffiffiffiffiffiffiffiffiffi g ¼ b12 þ b22 . Suppose the response increases between experiments 5 and 7, and then decreases between experiments 7 and 8. When the highest response is optimal, the result from experiment 8 indicates that the optimum is overshot, and thus, a new full factorial design is performed around experiment 7. The four experiments of the new design are points 9, 10, 11, and 12, while experiment 7 is the new center point. The center point is then repeated, and a check for curvature is performed. When curvature is present in this region of the experimental domain, it is concluded that the optimum can be found in this region.


x2 y3

y4 b1

y0

g

b2

y1

y2 y5

y6 y11

y12 y7

y9

y10

y8 x1 Figure 17 The steepest ascent method with a 22 full factorial design. Step length L ¼ g ¼

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi b 21 þ b 22 .

When the limits of the experimental domain are reached, the step length L can be decreased, to obtain a point (Ewithin) that falls within the limits of the domain. As the boundaries are reached, further movement should be made in another direction, which is determined from a new 22 full factorial design, that has the same size as the original full factorial design and for which the point Ewithin is one of the four design experiments. The procedure described above is then repeated this new design.2 This is illustrated in Figure 18. Suppose the initial step pffiffiffiffiffiffiffiffiffiffiffiffiffiwith ffi 2 2 length L is equal to g ¼ b1 þ b2 . The conditions for experiment 8, according to the step length L, are situated outside the experimental domain, defined by the boundaries of the examined factors. Therefore, the step length

x2

y3

y4 b1 g

y1

y0 b2 y2

y5 y6 y10

b′2 y7

y11

b′1

y12 y ′8 y8

g′

y9 x1

Figure 18 The steepest ascent method with a situation where the limits of the experimental domain are reached:qdecrease ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi of the step length, Ewithin ¼ point 89, and change in direction for further movement (g9). Initial step length L ¼ g ¼ b 21 þ b 22 .

572


x2

20

40

60 4

5

3 2 1

x1 2

Figure 19 Evolutionary operation (EVOP) procedure with a 2 design.

is decreased, to obtain a point that is situated within the domain, for example, Ewithin ¼ point 89. As the direction of the movement needs to be changed, a new 22 design with the same size is performed. The four experiments of the new design are defined by points 89, 9, 10, 11, and the new center point is experiment 12. The above procedure is then repeated and moves into the direction of the new steepest ascent. Another mixed sequential–simultaneous optimization procedure, called the EVOP and proposed first by Box,48 has mainly been used for the optimization of industrial processes. The procedure is illustrated in Figure 19. Around the starting point chosen somewhere in the experimental domain, a factorial design is performed, for example, a 22 full factorial design in Figure 19. This design is used to estimate the coefficients of the model, described in Equation (14). From the model, the direction of the steepest ascent is derived. Instead of performing additional experiments along the line of the steepest ascent, a new factorial design is performed. The new design uses one or some of the experiments from the former design (Figure 19). In this example, the highest response is the optimal. The same procedure is then repeated for each factorial design. The end point of this method is found similar to the one for the steepest ascent method. When the optimum is overshot, the curvature of the response surface is evaluated for the one-but-last factorial design, that is, it is evaluated whether the optimum is situated in this region. When it is decided that curvature is present in this region of the experimental design, it is concluded that the optimum is situated in this region. When second-order equations are preferred to model the response, Doehlert designs can be used sequentially in a similar way. An example of the latter is found in Garcıá Campanã et al.,49 where three factors of a micelleenhanced fluorescence method to determine germanium were optimized using sequential Doehlert designs.

1.17.4 Conclusions In this chapter, sequential optimization methods, such as Fibonacci numbers and simplex procedures, have been discussed. For the latter, the two main approaches, that is, basic simplex and modified simplex methods, have been discussed and illustrated with examples. Several modifications to these methods have been discussed. Finally, mixed sequential–simultaneous optimization methods, such as the steepest ascent method and EVOP, have been reviewed.

References 1. Massart, D. L.; Vandeginste, B. G. M.; Buydens, L. M. C.; De Jong, S.; Lewi, P. J.; Smeyers-Verbeke, J. Handbook of Chemometrics and Qualimetrics: Part A; Amsterdam: Elsevier, 1997. 2. Massart, D. L.; Dijkstra, A.; Kaufman, L. Evaluation and Optimization of Laboratory Methods and Analytical Procedures, a Survey of Statistical and Mathematical Techniques; Amsterdam: Elsevier, 1978.

Sequential Optimization Methods 573 3. Walters, F. H.; Parker, L. R.; Morgan, S. L.; Deming, S. N. Sequential Simplex Optimization, a Technique for Improving Quality and Productivity in Research, Development, and Manufacturing; Boca Raton, FL: CRC Press, 1991. 4. Vander Heyden, Y.; Perrin, C.; Massart, D. L. Optimization Strategies for HPLC and CZE. Handbook of Analytical Separations, 1, Separation Methods in Drug Synthesis and Purification; Amsterdam: Elsevier, 2000; pp 163–212. 5. Xiaoning, W.; Smeyers-Verbeke, J.; Massart, D. L. Linearization of Atomic Absorption Calibration Curves. Analusis 1992, 20, 209–215. 6. Spendley, W.; Hext, G. R.; Himsworth, F. R. Sequential Application of Simplex Designs in Optimisation and Evolutionary Operation. Technometrics 1962, 4, 441–461. 7. Burton, K. W. C.; Nickless, G. Optimisation via Simplex, Part I. Background, Definitions and a Simple Application. Chemom. Intell. Lab. Syst. 1987, 1, 135–149. 8. Nelder, J. A.; Mead, R. A Simplex Method for Function Minimization. Comput. J. 1965, 7, 308–313. 9. Morgan, E.; Burton, K. W.; Nickless, G. Optimization Using the Modified Simplex Method. Chemom. Intell. Lab. Syst. 1990, 7, 209–222. 10. Long, D. E. Simplex Optimization of the Response from Chemical Systems. Anal. Chim. Acta 1969, 46, 193–206. 11. Morgan, S. L.; Deming, S. N. Simplex Optimization of Analytical Chemical Methods. Anal. Chem. 1974, 46, 1170–1181. 12. Berridge, J. C. Simplex Optimization of High-Performance Liquid Chromatographic Separations. J. Chromatogr. 1989, 485, 3–14. 13. Aberg, E. R.; Gustavsson, A. G. T. Design and Evaluation of Modified Simplex Methods. Anal. Chim. Acta 1982, 144, 39–53. 14. Yarbro, L. A.; Deming, S. N. Selection and Preprocessing of Factors for Simplex Optimization. Anal. Chim. Acta 1974, 73, 391–398. 15. O¨berg, T. Importance of the First Design Matrix in Experimental Simplex Optimization. Chemom. Intell. Lab. Syst. 1998, 44, 147–151. 16. Derringer, G.; Suich, R. Simultaneous Optimization of Several Response Variables. J. Qual. Technol. 1980, 12, 214–219. 17. Smilde, A. K.; Knevelman, A.; Coenegracht, P. M. J. Introduction of Multi-Criteria Decision Making in Optimization Procedures for High-Performance Liquid Chromatographic Separations. J. Chromatogr. 1986, 369, 1–10. 18. Routh, M. W.; Swartz, P. A.; Denton, M. B. Performance of the Super-Modified Simplex. Anal. Chem. 1977, 49, 1422–1428. 19. Morgan, E.; Burton, K. W.; Nickless, G. Optimization Using the Super Modified Simplex Method. Chemom. Intell. Lab. Syst. 1990, 8, 97–107. 20. Ryan, P. B.; Barr, R. L.; Todd, H. D. Simplex Techniques for Nonlinear Optimization. Anal. Chem. 1980, 52, 1460–1467. 21. van der Wiel, P. F. A. Improvement of the Super-Modified Simplex Optimization Procedure. Anal. Chim. Acta 1980, 122, 421–433. 22. van der Wiel, P. F. A.; Maassen, R.; Kateman, G. The Symmetry-Controlled Simplex Optimization Procedure. Anal. Chim. Acta 1983, 153, 83–92. 23. Subrahmanyam, M. B. An Extension of the Simplex Method to Constrained Nonlinear Optimization. J. Optim. Theory Appl. 1989, 62, 311–319. 24. Shao, M. H. Techniques and Performance of the Advanced Regular Simplex Method. Anal. Chim. Acta 1991, 248, 31–39. 25. Hedlund, P.; Gustavsson, A. Design and Evaluation of Modified Simplex Methods Having Enhanced Convergence Ability. Anal. Chim. Acta 1992, 259, 243–256. 26. Hedlund, P.; Gustavsson, A. Design and Evaluation of an Improved Simplex Method. Anal. Chim. Acta 1998, 371, 9–21. 27. Hedlund, P.; Gustavsson, A. Design and Evaluation of an Effective Modified Simplex Method. Anal. Chim. Acta 1999, 391, 257–267. 28. Santos-Delgado, M. J.; Crespo-Corral, E.; Polo-Dıéz, L. M. Determination of Herbicides in Soil Samples by Gas Chromatography: Optimization by the Simplex Method. Talanta 2000, 53, 367–377. 29. Pasamontes, A.; Callao, P. Fractional Factorial Design and Simplex Algorithm for Optimizing Sequential Injection Analysis (SIA) and Second Order Calibration. Chemom. Intell. Lab. Syst. 2006, 83, 127–132. 30. Kolossva´ry, G. J. Optimization of Lipase Activity from Rhizopus sp. in Triglyceride Hydrolysis Using a Modified Simplex Method. Process Biochem. 1996, 31, 595–600. 31. Snijders, H.; Janssen, H. G.; Cramers, C. Optimization of Temperature-Programmed Gas Chromatographic Separations, II. OffLine Simplex Optimization and Column Selection. J. Chromatogr. A 1996, 756, 175–183. 32. Vereda, E.; Rios, A.; Valva´rcel, M. Quality Compromises Incorporated in Simplex Optimization of a Flow Injection System. Anal. Chim. Acta 1997, 348, 129–134. 33. Zuloaga, O.; Etxebarria, N.; Fernańdez, L. A.; Madariaga, J. M. MultiSimplex Optimisation of Purge-and-Trap Extraction of Phenols in Soil Samples. J. Chromatogr. A 1999, 849, 225–234. 34. Zuloaga, O.; Etxebarria, N.; Fernańdez, L. A.; Madariaga, J. M. MultiSimplex Optimisation and Comparison of Different Purgeand-Trap Extractions of Volatile Organic Compounds in Soil Samples. Anal. Chim. Acta 2000, 416, 43–53. 35. Vannecke, C.; Nguyen Minh Nguyet, A.; Bloomfield, M. S.; Staple, A. J.; Vander Heyden, Y.; Massart, D. L. Development and Optimisation of a Flow Injection Assay for Fluticasone Propionate Using an Asymmetrical Design and the Variable-Size Simplex Algorithm. J. Pharm. Biomed. Anal. 2000, 23, 291–306. 36. Srijaranai, S.; Burakham, R.; Deming, R. L.; Khammeng, T. Simplex Optimization of Ion-Pair Reversed-Phase High Performance Liquid Chromatographic Analysis of Some Heavy Metals. Talanta 2002, 56, 655–661. 37. Murillo Pulgarıń, J. A.; Alanõń Molina, A.; Alanõń Pardo, M. T. The Use of Modified Simplex Method to Optimize the Room Temperature Phosphorescence Variables in the Determination of an Antihypertensive Drug. Talanta 2002, 57, 795–805. 38. Murillo Pulgarıń, J. A.; Alanõń Molina, A.; Alanõń Pardo, M. T. Determination of 4-Methylpropranolol in Cerebrospinal Fluid, Serum, and Urine by Nonprotected Fluid Room-Temperature Phosphorescence Using Simplex Optimization. Anal. Biochem. 2002, 306, 270–277. 39. Cortazar, E.; Zuloaga, O.; Sanz, J.; Raposo, J. C.; Etxebarria, N.; Fernańdez, L. A. MultiSimplex Optimisation of the Solid-Phase Microextraction-Gas Chromatographic-Mass Spectrometric Determination of Polycylic Aromatic Hydrocarbons, Polychlorinated Biphenyls and Phthalates from Water Samples. J. Chromatogr. A 2002, 978, 165–175. 40. Kuppithayanant, N.; Rayanakorn, M.; Wongpornchai, S.; Prapamontol, T.; Deming, R. L. Enhanced Sensitivity and Selectivity in the Detection of Polycyclic Aromatic Hydrocarbons Using Sequential Simplex Optimization, the Addition of an Organic Modifier and Wavelength Programming. Talanta 2003, 61, 879–888.

574


41. Murillo Pulgarıń, J. A.; Alanõń Molina, A.; Alanõń Pardo, M. T. Simplex Optimization and Kinetic Determination of Nabumetone in Pharmaceutical Preparations by Micellar-Stabilized Room Temperature Phosphorescence. Anal. Chim. Acta 2005, 528, 77–82. 42. Murillo Pulgarıń, J. A.; Alanõń Molina, A.; Alanõń Pardo, M. T. Simplex Optimization of the Variables Affecting the MicelleStabilized Room Temperature Phosphorescence of 6-Methoxy-2-Naphtylacetic Acid and Its Kinetic Determination in Urine. Anal. Biochem. 2005, 339, 157–164. 43. Tinoi, J.; Rakariyatham, N.; Deming, R. L. Simplex Optimization of Carotenoid Production by Rhodotorula glutinis Using Hydrolyzed Mung Bean Waste Flour as Substrate. Process Biochem. 2005, 40, 2551–2557. 44. Dahleń, J.; von Eckardstein, S. Development of a Capillary Zone Electrophoresis Method Including a Factorial Design and Simplex Optimization for Analysis of Amphetamine, Amphetamine Analogues, Cocaine, and Heroin. Forensic Sci. Int. 2006, 157, 93–105. 45. Sultan, S. M.; El-Mubarak, A. H. High Performance Liquid Chromatographic Method for the Separation and Quantification of Some Psychotherapeutic Benzodiazepines Optimized by the Modified Simplex Procedure. Talanta 1996, 43, 569–576. 46. Dol, I.; Knochen, M. Flow-Injection Spectrophotometric Determination of Salbutamol with 4-Aminoantipyrine. Talanta 2004, 64, 1233–1236. 47. Brooks, S. H. A Comparison of Maximum-Seeking Methods. Oper. Res. 1959, 7, 430–457. 48. Box, G. E. P. Evolutionary Operation, a Method for Increasing Industrial Productivity. Appl. Stat. 1957, 6, 81–91. 49. Garcıá Campanã, A. M.; Cuadros Rodrı´guez, L.; Lupian˜ez Gonza´lez, A.; Ale´s Barrero, F.; Romań Ceba, M. Sequential Response Surface Methodology for Multioptimization in Analytical Chemistry with Three-Variable Doehlert Designs. Anal. Chim. Acta 1997, 348, 237–246.



Bieke Dejaegher is a postdoctoral Fellow of the Research Foundation – Flanders (FWO) at the Vrije Universiteit Brussel, Belgium, Department of Analytical Chemistry and Pharmaceutical Technology, working on chemometrics and experimental designs.

Yvan Vander Heyden is a professor at the Vrije Universiteit Brussel, Belgium, Department of Analytical Chemistry and Pharmaceutical Technology, and heads a research group on applied chemometrics and separation science.


1.18 Steepest Ascent, Steepest Descent, and Gradient Methods R. G. Brereton, University of Bristol, Bristol, UK ª 2009 Elsevier B.V. All rights reserved.

1.18.1 1.18.1.1 1.18.1.2 1.18.1.3 1.18.1.4 1.18.2 1.18.2.1 1.18.2.2 1.18.2.3 1.18.2.4 1.18.3 1.18.3.1 1.18.3.2 1.18.3.3 1.18.4 References

Introduction Formulation of Problem Literature Applications in Chemometrics Notation Method One Variable More than One Variable Influence of the Parameters Limitations Examples Algebraic Numerical Literature Conclusion

Symbols a b1 i t xi xit

stepsize parameter estimated coefficient for factorial design factor number iteration number value of factor i (any iteration) value response for factor i and iteration t

577 577 578 579 579 580 580 581 582 584 584 584 585 587 589 589

xt y y yt

response vector for iteration t response vector (t iterations) value of response (any iteration) response for iteration t distance between two levels in one factor design

1.18.1 Introduction 1.18.1.1

Formulation of Problem

The steepest ascent (or steepest descent or gradient) method is used to determine optimum conditions. One of the best-cited early descriptions in the literature was by Box and Wilson in 1951,1 who were working in ICI in the UK, Box by training a statistician and Wilson a chemist. A major driving force for fundamental work in the area of optimization was post–World War II industry, with a requirement for very efficient processes as part of economic reconstruction. Chemists, in particular, encounter multifactorial problems where it is necessary to optimize, for example, a yield or the quality of a product as a function of many factors. A traditional approach to searching for optima is to perform a grid search, for example, to run experiments or processes over a series of conditions, for example, if there are three factors, a 10 10 10 grid would involve performing 1000 experiments under different conditions. The problems with grid search approaches are numerous. The first is that the

577

578

Steepest Ascent, Steepest Descent, and Gradient Methods

60

Gradient small and negative: Slow movement in negative direction. Close to optimum.

Gradient small and positive. Slow movement in positive direction. Close to optimum.

50

40 Gradient large and positive. Rapid movement in positive direction. Far from optimum.

30

20

10 Direction of gradient

0 0

0.5

1

1.5

2

2.5

3

3.5

4

Figure 1 Illustration of the principle of steepest ascent.

number of experiments can be prohibitive if there are several factors. The second is that there may be significant experimental error meaning that if experiments are repeated under identical conditions, different answers might be obtained; hence, just choosing the best point on the grid may be misleading especially if the optimum is quite flat. The third is that the initial grid may be sparse in order to make the number of experiments feasible, and could miss features that are close to the optimum or find false (local) optima. The steepest ascent is based on the premise that the response changes fastest in the direction of the optimum, so it is mainly necessary to determine the gradient and use this to determine the direction of the next point. This method requires only the gradient to be calculated and a step size that relates to how far away the new point is from the old one, although the original variables may have to be coded in the experimental space. Figure 1 illustrates the principle for a simple optimization of a parabolic function. At the bottom left, the gradient is positive and large, so the gradient method tells the optimizer to move by a large amount in the positive direction. In contrast at the top right, the gradient is small and negative, telling the optimizer to move in a negative direction by a small amount. Of course in real situations, we do not know the shape or reproducibility of the curve, and as such steepest ascent methods provide pointers as to what conditions are necessary to perform the next optimization experiment. The methods described in this chapter are sequential methods in that single experiments are performed in sequence.

1.18.1.2

Literature

There is a large literature on this method. The most recognized and best-cited early reference in the statistics and chemometrics literature is by Box and Wilson.1 As often happens, however, there are earlier and less wellcited papers on the same theme often in diverse literature, for example Booth in the late 1940s published numerous papers on this method, which he called ‘steepest ascents’ primarily in crystallography.2,3 Mathematicians have often attributed the method of steepest descent to the physicist Debye,4 who in 1909 worked it out in an asymptotic study of Bessel functions. Debye himself commented that he had borrowed the idea of the method from a paper of Riemann published in 1863, and the first paper on the method is generally


579

accepted to be from Cauchy in the 1820s5 in the context of solving integrals. A good historical perspective has been published in 1997.6 At the time of the 1940s and 1950s, it was less common for there to be cross-disciplinary communication, especially because journals were less openly accessible and probably most read by specialist peer groups. For example, at the time of writing, Booth’s paper in Nature2 has been cited only 30 times and primarily in the crystallographical literature, whereas Box and Wilson’s paper over 1100 times. Most of the subsequent papers were published in the statistical literature and can be quite daunting for chemists. Brooks and Mills published the first paper with steepest ascent in the title.7 It is widely reported in most classical statistical textbooks on experimental design and optimization.8–10 In the mainstream chemical literature, this approach was slow to become recognized, in contrast to the simplex method in the 1970s, when analytical chemists in particular started to recognize the importance of optimization. This is largely due to advocacy in that there were powerful advocates of the simplex methods, especially Deming and coworkers, and these were incorporated into software. An ISI survey of papers published in 2006 with keywords steepest ascent/descent/descents revealed 122 papers published in that year, of which only 8 had any chemical content, primarily at the interface between chemistry and other disciplines. Of these, 5 use the classically accepted and defined steepest ascent algorithm.11–15 The few papers using the steepest ascent or descent method in experimental applications relating to analytical chemistry were mainly published in the mid-1990s of which three are cited in this chapter.16–18 In contrast, 26 out of 117 paper in simplex optimization/optimization were published in journals with chemical content over that period, including mainstream analytical chemistry, chemometrics, chromatography, and mass spectrometry journals. Part of this discrepancy is likely due to readily available software for laboratory-based optimization. In fact steepest ascent methods are commonly incorporated into algorithms being very simple and widely available but chemists who are laboratory based and not fundamentally programmers are less aware of the potential of this approach for optimization. 1.18.1.3

Applications in Chemometrics

There are two very distinctly different reasons why a chemometrician may wish to use the methods described in this chapter. The first reason is for computational optimization. This does not involve performing experiments and may involve trying to find a solution to a numerical problem, for example, in curve resolution, or for optimizing a model. This could be quite useful for example if the relationship between a response and the value of several factors is known, for example by using an experimental design such as a central composite design, and from this relationship the optimal conditions are obtained. This type of application is the most common and involves primarily incorporation into algorithms, and could be used to obtain the minimum or maximum of a response surface. The second, and less common, application is as an aid to experimental optimization. In this case, steepest ascent (or descent) methods can be used to determine the conditions for a sequence of experiments. Normally, a small experiment is performed (such as a two-level factorial design) to obtain some feel for the relationship between the response and the experimental conditions, and after this, steepest ascent suggests new conditions for the next set of experiments and so on. 1.18.1.4

Notation

In this chapter, we will employ the following notation. We will exemplify the methods using a single response vector y, which is a function of one or more (¼I) factors to be optimized, denoted by x1, . . . . xi . . . . xI or in vector form x. Each step in the optimization will be denoted by t so that yt is the tth value of y, xt of the factors, and xit of factor i. Note that a scalar value of y without a subscript does not refer to a specific step, similarly a scalar value of x with a single subscript. When a specific step is referenced, then y is subscripted by the iteration number and the scalar x has two subscripts, and the vector x one subsript. The symbol q denotes a partial derivative, so that qy/qxi is the partial derivative of the response with the ith factor. The notation y implies the vector of partial derivatives of y with respect to x so that y ¼ [qy/qx1 . . . qy/qxi . . . qy/qxI]. Other variables will be defined where necessary.

580


1.18.2 Method 1.18.2.1

One Variable

The method for one variable is straightforward. The first step is to choose (a) starting condition of the factor to be optimized, x11 (factor 1 and iteration 1, in this case I ¼ 1) and (b) a step size parameter a, which determines how fast the optimization progresses. It is then very straightforward for determining the condition for the next step: x12 ¼ x11 þ a dy/dx1 for finding a maximum and x12 ¼ x11 a dy/dx1 for finding a minimum. The gradient of the response is obtained at the new value of x12 and the procedure continues. Most of the original applications of this approach were in numerical computing and simulations usually trying to find the optimum of a function efficiently. Under such circumstances, it is easy to numerically or algebraically calculate a derivative. If the equation of the function is known, this can be performed algebraically. Otherwise, a large variety of approaches are available, one of the simplest and fastest being the Savitzky–Golay method.19 In experimental work however, this is not straightforward and one commonly recommended approach is to perform a small factorial design at each point. For example, we could perform a one-factor, two-level design at each point x1t by recording the responses at x1t and x1t þ . These can then be modeled by the equation y ¼ b0 þ b1x1 so the derivative dy/dx1 ¼ b1. Therefore, the next point for determining a maximum is x1tþ1 ¼ x1t þ ab1. A quadratic model would be possible if three experimental points were measured, of the form y ¼ b0 þ b1x1 þ b11x21 giving a new point at x1tþ1 ¼ x1t þ a(b1 þ 2x1t). This would add to the amount of experiments required; however, even if the overall response curve is quadratic or higher in nature, often a linear local approximation is likely to be adequate in many cases providing the step size (a) is sufficiently small. The benefits of a better local approximation using more experiments and a higher order model, probably resulting in faster progress toward the optimum, should be balanced against the extra experimentation required to achieve this improved model. As an example of a single-parameter model, Figure 2 shows the first eight steps of a steepest descent optimization for a single-factor response, where the stating point is x1 ¼ 0.6, the true optimum is x1 ¼ 2.33, the

110 100 90

y

80 70 60 50 40 0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

x1 Figure 2 Illustration of steepest descent for a single-parameter optimization, involving eight steps, a value of a of 0.1, of of 0.05, and a starting point of x1 ¼ 0.6.


581

Table 1 Main adjustable parameters Starting condition(s)

x1 (one factor); x1 (several factors)

Rate of optimization Window size for determination of local gradient Stopping criterion

a (usually only necessary for experimental implementation) " (could relate to difference in response or in the values of the factors)

value of a ¼ 0.1, and the gradient at each point in the graph is estimated by a linear approximation involving performing experiments at x1t 0.05. There are various stopping rules to determine whether an optimum is found, but in numerical optimization usually a small value " is set up in advance and convergence is assumed when |x1tþ1 x1t| < " or the value of the optimum does not vary much. For experimental optimization, this is probably best determined by eye, when it looks as if the process has converged to the satisfaction of the experimenter, and the convergence can be influenced by noise: If there is a high noise level, often the optimum will oscillate around a central point. Hence oscillation is also a sign that the optimum has been found providing the rate of oscillation does not change much. Sometimes, these new conditions could lead to a new experimental optimization around the point determined by simplest ascent, or performing the optimization in a second step using fresh optimization parameters (usually values of a). The procedure will be illustrated numerically in Section 1.18.3.2. The main parameters are summarized in Table 1.

1.18.2.2

More than One Variable

The equations when I > 1 are a relatively straightforward extension of the method described above. The first step is to choose (a) starting conditions for the factors to be optimized, x1, (b) a step size parameter a, which determines how fast the optimization progresses. In experimental situations, the factors should be coded to ensure that they are on a similar scale and to produce a sensible experimental design at each point in the optimization. This is because of the parameter discussed above; for example, if the original parameters are recorded on very different scales, for example pH and temperature, a 1 change in temperature may have much less influence than a change in pH of 1 unit. For numerical computing, this may not matter so much especially in the case of optimization of functions. The equations then become x2 ¼ x1 þ a y ¼ x1 þ a[qy/qx11 . . . qy/qxi1 . . . qy/qxI 1], where xt is the vector of conditions for step t, for finding a maximum, with an equivalent equation for a minimum. Stopping rules similar to the one-factor example can be obtained, with the Euclidean distance between the new and old vector of conditions and also between the new and old responses being the common convergence criterion. As usual if this is to be performed experimentally, at each new point in the optimization, several experiments are required in order to determine the gradient. A recommended simple strategy involves a two-level, twofactor design consisting of four experiments, the distance between the levels being defined by . This allows the response to be fitted to the equation y ¼ b0 þ b1x1 þ b2x2 þ b12x1x2. Sometimes, the interaction term can be neglected. Note that even if the overall response surface has obvious curvature, a local linear model is usually adequate, as the aim of the model is to determine the direction and rate of the search for the next point and not as an overall model for the response surface. Note that there are several alternative approaches in the literature, especially where there are several factors, one being to perform a factorial (or fractional factorial) design to get the coefficients of a model that are used to determine the gradient for several steps (e.g., five or six) and then at the new optimum perform a similar design again to obtain a revised model and then perform further optimization and so on. This modification may require more steps to reach an optimum than the alternative of performing a factorial design at each step, as the experimental model for determining the gradient is not estimated at each step, but this is balanced against the number of experiments required to obtain an accurate measure of the gradient at

582


1

16

10

10

12

14

9

11

0.8

9

7 9

6

7 6

12

0.4

10

8

10

0.6

5.5

8

5.5

5

5

11

3

4

–0.6

–0.4

4. 5

5 4.5

6

4

–0.2

3.5

0 x1

0.2

4

0.4

7

5 5 5. 6

–0.8

7

10

11

–1 –1

2.6

8 3

8

9

–0.8

2.

2 2.5 .4

5 4.5

3.5

–0.6

3.5

2.3

3 2.8

–0.4

2.4

5.5

6 5.5

2.6

5 2.

4

–0.2

3

7

2.8

3.5

10

8

6

9

5

0

4.

3.5

x2

7

4.5

4

0.2

12

11

8

0.6

0.8

1

Figure 3 Illustration of steepest descent for a single-parameter optimization, involving 14 steps, a value of a of 0.05, of of 0.05, and a starting point of x1 ¼ [0.7 0.8].

each step, and for more than two factors it is preferable to reduce the number of factorial designs at the cost of slightly less efficient convergence to the optimum owing to inaccuracies in the estimation of the gradient at every step. The progress of a typical optimization for two factors is given in Figure 3. In this case, a two-level, two-factor design is performed at each experimental point.

1.18.2.3

Influence of the Parameters

The most crucial parameter is a, which should be chosen with some care. The influence of a on optimization is illustrated in Figure 4 all starting at the same point (Table 2). It can be seen that there is only a quite small region of values for which optimization is possible. It is particularly important to note how different the behavior of the method is when a is increased from 0.1 to 0.115. In the latter case the method diverges and will never find an optimum, whereas in the former it oscillates around the optimum but finds it within 10–15 steps, dependent on stopping criteria. Note that a value of a equal to 0.05 is extremely efficient, but the problem is that the response surface is not known in advance and as such it is not possible to know what the correct value is in advance of experimentation or computation. In a computational algorithm, often some checks are needed to see whether it looks as if the optimum is converging and if so how fast, and then perhaps modify a if there is a problem. In experimental science, it should be obvious if there are difficulties. If the value of the response is getting worse, this is likely to be a sign of lack of convergence (although a noisy response surface may cause confusion, hence some idea of the experimental error is useful). If the value of the response is very slowly improving, possibly a is too small. In contrast, the starting conditions are not too crucial, even if they are very wide off the optimum (see Figure 5) and it normally just takes a few extra steps in the optimization procedure. This in practice is because steepest ascent (or descent) methods do not have fixed step sizes unlike methods such as simplex optimization, so can move rapidly toward the minimum providing the value of a is set sensibly.


(a) 110

(b) 110

100

100

90

90

Start

Start

80

80

70

70

60

60

50

0

0.5

1

1.5

2

2.5

3

3.5

4

50

(c)

(d)

110

110

100

100

90

0

0.5

1

1.5

2

2.5

3

3.5

4

2

2.5

3

3.5

4

6

7

90

Start

Start 80

80

70

70

60

60

50

583

0

0.5

1

1.5

2

2.5

3

3.5

4

50

(e)

(f)

110

250

0

0.5

1

1.5

100 200 90

Start

80

150

70 100 60 Start

50

0

0.5

1

1.5

2

2.5

3

3.5

4

50 –2

–1

0

1

2

3

4

5

Figure 4 Influence of the value of a on optimization: (a) 0.001, (b) 0.01, (c) 0.05, (d) 0.1, (e) 0.11, and (f) 0.115. All iterations start at a value of x1 ¼ 0.6. Note that the scale for (f) is different to illustrate divergence. The horizontal axis represents x1 and the vertical axis y.

584


Table 2 Influence of a on the optimization of Figure 4 Value of a

Comments

0.001 0.01 0.05 0.1 0.11 0.115

Is unlikely to attain the optimum within a realistic timescale Slowly finds optimum within 10–15 steps dependent on stopping criterion Very rapidly finds optimum within 3–4 steps Oscillates around optimum but finds it within 10–15 steps Wide oscillation around optimum, very slow convergence Does not converge, oscillates wildly

(a)

(b)

110

110 Start

100

100

90

90

80

80

70

70

60

60

50 0

0.5

1

1.5

2

2.5

3

3.5

4

50

Start

0

0.5

1

1.5

2

2.5

3

3.5

4

Figure 5 Influence of starting conditions on the progress of the optimization, using a ¼ 0.1, and a value of (a) x1 ¼ 0 and (b) x1 ¼ 01.5.

1.18.2.4

Limitations

Although this method may appear simple, a problem is that it can be quite slow for convergence, and will often perform many small steps when going down a valley to a minimum. It is also quite strongly dependent on the choice of a. For fairly straightforward optimizations it is adequate, but for more elaborate optimizations it can be slow and sometimes never reach an optimum. There are various modifications but the conjugate gradient method20 is probably the most common.

1.18.3 Examples 1.18.3.1

Algebraic

The simplest type of example involves the optimization of a function. In chemometrics, it is often possible to determine a relationship between the level of factors that determine a response and the response itself. As an example, consider trying to find the minimum of the function y ¼ 3 – 3x1 – 2x2 þ 2x12 þ 7x22 – 2x1 x2

We can calculate the derivatives qy=qx1 ¼ – 3 þ 4x1 – 2x2 qy=qx2 ¼ – 2 þ 14x2 – 2x1


585

Table 3 Numerical example of optimization of the function y ¼ 3 – 3x1 – 2x2 þ 2x12 þ 7x22 – 2x1 x2 Iteration

x1

x2

y

qy/qx1

qy/qx2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

0.000 0.300 0.520 0.648 0.735 0.788 0.824 0.845 0.860 0.869 0.874 0.878 0.880 0.882 0.883

0.000 0.200 0.180 0.232 0.237 0.252 0.257 0.262 0.264 0.266 0.267 0.268 0.268 0.269 0.269

3.0000 2.0400 1.6604 1.5079 1.4462 1.4211 1.4109 1.4067 1.4050 1.4043 1.4040 1.4039 1.4039 1.4039 1.4039

–3.000 –2.200 –1.280 –0.872 –0.533 –0.351 –0.219 –0.142 –0.090 –0.058 –0.037 –0.024 –0.015 –0.010 –0.006

–2.000 0.200 –0.520 –0.048 –0.155 –0.044 –0.052 –0.023 –0.019 –0.010 –0.007 –0.004 –0.003 –0.002 –0.001

The calculations are presented in Table 3 for a value of a ¼ 0.1. It can be seen that the response has effectively converged within 10 iterations, although there is still a slight improvement in the optimal values of x1 and x2 as the optimum is probably very flat. Whereas in the numerical example above, it is possible to obtain the value of the optimum by simple differentiation of the original response function without the need for steepest descent methods, where there are several factors and several terms including high-order interactions and functions such as higher order polynomials or exponentials, it may be hard or impossible to solve the equations algebraically and the methods described in this chapter provide valuable alternatives.

1.18.3.2

Numerical

In most practical cases, the underlying relationship between the response to be optimized and the levels of the factors is unknown, and optimization consists of performing a series of experiments in sequence until an optimum is found. In order to understand this, we will illustrate by a simple one-factor example. If we use a numerical approach for optimization (which could be performed experimentally as well as by computational modeling), a typical procedure might be as follows. 1. Choose a starting value of x1. 2. In order to obtain an approximate value of the gradient at this starting point, some tests need to be performed. A simple one is to perform a two-factor factorial design,21 involving, in this case, just two experiments at x1 þ and x1 . The values of the response y can be determined and a linear model of the form y ¼ b0 þ b1x1 obtained. 3. The value of b1 is the value of the gradient and so a new value of x1 þ ab1 is used for the next step. 4. The response of this new value is then determined in the next step for an estimate of the gradient at x1 þ ab1 and so on. Note that it is not actually necessary to perform an experiment at x1 to determine the gradient in the region of the new response. Note also that a factorial design is very simple and approximates to a linear model, even though the response surface is likely to be multilinear (and cannot be linear if there is an optimum): This is an approximation to the gradient. Better approximations are possible if more experiments are performed at each stage, but this requires more time and effort, and in most cases the estimate of the gradient will differ by only a small amount; hence there is a balance between performing an elaborate design and a very basic experiment that may involve an approximation but is faster.

586


Table 4 Numerical example: noise free, ¼ 0.05, a ¼ 0.02 x1

y

x1 –

y

x1 þ

y

b1

y – b1x1

y þ b1 x1

1.000 1.312 1.577 1.776 1.908 1.988 2.033 2.057 2.071 2.077 2.081 2.083 2.084 2.084 2.085

70.971 75.515 78.626 80.280 80.983 81.234 81.314 81.337 81.344 81.346 81.346 81.347 81.347 81.347 81.347

0.950 1.262 1.527 1.726 1.858 1.938 1.983 2.007 2.021 2.027 2.031 2.033 2.034 2.034 2.035

70.186 74.839 78.111 79.927 80.758 81.093 81.223 81.275 81.297 81.307 81.312 81.314 81.315 81.316 81.316

1.050 1.362 1.627 1.826 1.958 2.038 2.083 2.107 2.121 2.127 2.131 2.133 2.134 2.134 2.135

71.744 76.166 79.104 80.587 81.157 81.320 81.347 81.341 81.331 81.325 81.321 81.319 81.318 81.317 81.317

15.583 13.275 9.930 6.598 3.992 2.263 1.232 0.655 0.344 0.179 0.093 0.048 0.025 0.013 0.007

70.192 74.852 78.129 79.950 80.784 81.121 81.252 81.305 81.327 81.337 81.342 81.344 81.345 81.346 81.346

71.750 76.179 79.122 80.610 81.183 81.348 81.376 81.370 81.361 81.355 81.351 81.349 81.348 81.347 81.347

Italicized observations are never obtained and are given for reference as discussed in the text.

An example of a simulated and noise-free numerical optimization is given in Table 4. The starting point of this optimization is x1 ¼ 1. A value of ¼ 0.05 is chosen and two experiments are performed at x1 ¼ 0.95 and 1.05 giving values of the response y of 7.186 and 7.744, respectively. Using a simple liner model, the value of the gradient is b1 ¼ 15.583, so the next value of x1 ¼ 1 þ a 15.583. Setting a as 0.02, we have a new value of x1 ¼ 1.312 and so test the response at x1 ¼ 1.262 and x1 ¼ 1.362 and so on. The columns in italics are not directly observed, but are presented for reference. The response at the central value of x1 can be calculated as can the values of y b1, which are the values that would be obtained at the factorial points if the linear model was perfect, which is not the case because the surface is curved. The first six steps of this optimization are illustrated in Figure 6, with the six values of x1 for each step represented using squares and the factorial points that are actually measured represented using circles. In normal practice, there is noise imposed upon an experiment, and the understanding of the optimization process becomes more complex. Table 5 illustrates a similar optimization to that of Table 4 except that there is some noise in all the measurements. We can see that the values of the gradient are quite different in this case, and the progress of the optimization is at a different rate. When near the optimum, it oscillates because the

82

Response (y )

80 78 76 74 72 70 0.8

1

1.2

1.4 1.6 Value of x1

Figure 6 Numerical optimization of one-factor response – noise free.

1.8

2

2.2


587

Table 5 Numerical example: noise free, ¼ 0.05, a ¼ 0.02 x1

y

x1 –

y

x1 þ

y

b1

y – b1 x 1

y þ b1 x1

1.000 1.376 1.656 1.799 1.988 2.024 2.071 2.117 2.051 2.058 2.064 2.122 2.084 2.084 2.070

70.974 76.421 79.402 80.390 81.251 81.310 81.256 81.380 81.418 81.266 81.259 81.437 81.385 81.280 81.309

0.950 1.326 1.606 1.749 1.938 1.974 2.021 2.067 2.001 2.008 2.014 2.072 2.034 2.034 2.020

69.926 75.656 79.003 80.038 80.938 81.198 81.111 81.357 81.184 81.345 81.181 81.392 81.284 81.387 81.329

1.050 1.426 1.706 1.849 2.038 2.074 2.121 2.167 2.101 2.108 2.114 2.172 2.134 2.134 2.120

71.806 77.054 79.722 80.982 81.117 81.431 81.345 81.024 81.219 81.377 81.467 81.204 81.287 81.314 81.227

18.801 13.982 7.185 9.443 1.792 2.323 2.339 –3.329 0.356 0.322 2.862 –1.880 0.028 –0.730 –1.020

70.034 75.722 79.042 79.918 81.161 81.194 81.139 81.546 81.400 81.250 81.116 81.531 81.383 81.317 81.360

71.914 77.120 79.761 80.863 81.340 81.426 81.372 81.213 81.436 81.282 81.403 81.343 81.386 81.244 81.258

Italicized observations are never obtained and are given for reference as discussed in the text.

estimation of the gradient is extremely sensitive to noise; hence there appear quite large gradients, both positive and negative close to the optimum. This is illustrated in Figure 7 for the first six steps. Note that the two factorial points are sometimes completely out of line with the central point. This is because of the influence of noise, which can have the effect of shifting these points either both above or below the curve, especially when it is quite flat, often resulting in quite false estimates of the gradient and hence oscillation. A comparison of the change in the value of x1 for both the noise-free process and noisy process at each successive step is given in Figure 8.

1.18.3.3

Literature

There are relatively few examples of this technique reported in the literature that may be of interest to chemometricians, but we will review one application in this section.16 This work was used to optimize the separation of two enantiomers chromatographically on a chiral stationary-phase column using super critical fluid chromatography (SFC), and is exemplified by the separation 82 80

Response (y)

78 76 74 72 70 68 0.8

1

1.2

1.4 1.6 Value of x1

1.8

2

Figure 7 Numerical optimization of one-factor response in the presence of noise.

2.2

588


2.2 Noisy 2.0

Value of x1

1.8 1.6 Noise free 1.4 1.2 1.0 1

2

3

4

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 Step

Figure 8 Value of x1 for each successive step of optimization for the noisy and noise-free numerical examples: the underlying process has the same behavior.

of ethyl-(2R,3S)-dihydroxyoctanoate and ethyl-(2S,3R)- dihydroxyoctanoate. Three factors were considered, namely (1) start density of the mobile phase, (2) temperature, and (3) density gradient. The aim was to obtain a good chiral resolution (CR) given by CR ¼

tR2 – tR1 –1 wh2 þ wh1

where tR is the retention time and wh the peak width at half-height for each compound whereby both peaks are resolved (set by the authors at around 0.274) while minimizing the retention time, so achieving a good resolution at maximum speed. 1. The first stage was to perform a half-factorial design with central points (five experiments) to determine conditions that would lead to the desired resolution of 0.274, using coded values as in Table 6, which were found by the model to be x1 ¼ 0.344 g ml1, x2 ¼ 52 C, and x3 ¼ 0.013 g ml1 min1 giving an experimental value of tR2 ¼ 12.251 min and CR ¼ 0.281 (owing to experimental uncertainty, it is not exactly equal to the estimated value). 2. Once this was achieved, another half-factorial design (this time with just four points) was obtained around the optimum of step 1, to determine the rate of change with respect to both the resolution and the retention time (RT) of the slowest elute compound, which gives a model that can be used to determine the derivatives for the steepest ascent. Note that in this application the authors do not perform factorial designs afresh at each step of the optimization. 3. Six experiments were performed along the line of steepest ascent using the model of step 2 for computing the derivatives to give an optimum value of x1 ¼ 0.311 g ml1, x2 ¼ 58 C, and x3 ¼ 0.019 g ml1 min1 with a value of CR ¼ 0.255 and tR2 ¼ 10.704 min. 4. Steps 2 and 3 are repeated again to give final values of x1 ¼ 0.276 g ml1, x2 ¼ 62 C, and x3 ¼ 0.022 g ml1 min1 with a value of CR ¼ 0.265 and tR2 ¼ 10.293 min. Table 6 Three variables for worked example of Section 1.18.3.3 coded for the first half-factorial design Variable

–1

0

þ1

x1, start density, g ml–1 x2, temperature, C x3, density gradient, g ml–1 min–1

0.270 47 0.007

0.300 50 0.010

0.330 53 0.013


589

1.18.4 Conclusion The steepest ascent (or descent) method is one of the oldest and most straightforward approaches for optimization based on approaches first described almost two centuries ago. This method is quite commonly used in computational optimization algorithms, whereas it is relatively rare in chemometrics, probably largely due to the lack of publicity in the literature, as compared to other methods such as simplex optimization. It does have some limitations, and is very sensitive to the value of a, which relates to the step size of the optimization; however, other approaches such as simplex also have such limitations. When used experimentally or numerically, there needs to be a way of estimating gradients, which may involve performing an experimental design, such as a factorial design, at each successive step. Nevertheless, this is a simple approach with a long vintage and as such is worth considering as one of the tools of the chemometrician.

References 1. Box, G. E. P.; Wilson, K. B. On the Experimental Attainment of Optimum Conditions. J. R. Stat. Soc. B Stat. Methodol. 1951, 13, 1–45. 2. Booth, A. D. Application of the Method of Steepest Descents to X-Ray Structure Analysis. Nature 1947, 160, 196–196. 3. Booth, A. D. The Refinement of Atomic Parameters by the Technique Known in X-Ray Crystallography as the Method of Steepest Descents. Proc. R. Soc. Lond. A Math. Phys. Sci. 1949, 197, 336–355. 4. Debye, P. Naeherungsformeln fuer die Zylinderfunktionen fuer große Werte des Arguments undunbeschraenkt veraenderliche Werte des Index. Math. Ann. 1909, 67, 535–558. 5. Cauchy, A. L. Memoire sur divers points d’analyse. Mem. Acad. (France) 1829, 8, 130–138. 6. Petrova, S. S.; Solov’ev, A. D. The origin of the Method of Steepest Descent. Historia Math. 1997, 24, 361–375. 7. Brooks, S. H.; Mickey, M. R. Optimum Estimation of Gradient Direction in Steepest Ascent Experiments. Biometrics 1961, 17, 48–56. 8. Davies, O. L., Ed. Design and Analysis of Industrial Experiments. Hafner: New York, 1954. 9. Box, G. E. P.; Hunter, W. G.; Hunter, J. S. Statistics for Experimenters; Wiley: New York, 1978. 10. Bayne, C. K.; Rubin, I. B. Practical Experimental Designs and Optimization Methods for Chemists. VCH Publishers: Deerfield Beach, FL, 1986. 11. Aguilar-Mogas, A.; Gimenez, X.; Bofill, J. On the Implementation of the Runge–Kutta–Fehlberg Algorithm to Integrate Intrinsic Reaction Coordinate Paths. Chem. Phys. Lett. 2006, 432, 375–382. 12. Ma, H. T.; Liu, X. J.; Bian, W. S.; Meng, L. P.; Zheng, S. J. A Theoretical Study of the Mechanism and Kinetics of FþN-3 Reactions. ChemPhysChem. 2006, 7, 1786–1794. 13. Liu, T.; Ye, L.; Chen, H. J.; Li, J. Y.; Wu, Z. H.; Zhou, R. H. A Combined Steepest Descent and Genetic Algorithm (SD/GA) Approach for the Optimization of Solvation Parameters. Mol. Simul. 2006, 32, 427–435. 14. Boye-Peronne, S.; Gauyacq, D.; Lievin, J. Vinylidene–Acetylene Cation Isomerization Investigated by Large Scale ab initio Calculations. J. Chem. Phys. 2006, 124, 214305. 15. Mezey, P. G. Rules on the Changes of Approximate Symmetry Measures along Reaction Paths. Mol. Phys. 2006, 104, 723–729. 16. Petersson, P.; Lundell, N.; Markides, K. E. Chiral Separations in Supercritical Fluid Chromatography: A Multivariate Optimization Method. J. Chromatogr. A 1992, 623, 129–137. 17. Duineveld, C. A. A.; Bruins, C. H. P.; Smilde, A. K.; Bolhuis, G. K.; Zuurman, K.; Doornbos, D. A. Multicriteria Steepest Ascent. Chemom. Intell. Lab. Syst. 1994, 25, 183–201. 18. Ghosh, B.; Agarwal, D. C.; Bhatia, S. Synthesis of Zeolite A from Calcined Diatomaceous Clay: Optimization Studies. Ind. Eng. Chem. Res. 1994, 33, 2107–2110. 19. Savitzky, A.; Golay, M. J. E. Smoothing þ Differentiation of Data by Simplified Least Squares Procedures. Anal. Chem. 1964, 36, 1627–1639. 20. Fletcher, R.; Reeves, C. M. Function Minimisation by Conjugate Gradients. Comp. J. 1964, 7, 149–154. 21. Brereton, R. G. Chemometrics: Data Analysis for the Laboratory and Chemical Plant; Wiley: Chichester, 2003; Chapter 2.

590


Biographical Sketch

Richard Brereton is Professor of Chemometrics and Director of the Centre for Chemometrics in Bristol. He obtained his BA, MA, and Ph.D. from the University of Cambridge and was subsequently employed by the University of Bristol. He is Fellow of the Royal Society of Chemistry, a Chartered Chemist, Fellow of the Royal Statistical Society, and Fellow of the Royal Society of Medicine. He was awarded the Royal Society of Chemistry’s 2006 Theophilus Redwood lectureship. He is the author of the book Chemometrics: Data Analysis for the Laboratory and Chemical Plant (Wiley 2003) and the author of Applied Chemometrics for Scientists (Wiley 2007), and was fortnightly Feature Writer in Chemometrics for the Alchemist and Associate Editor of Chemometrics and Intelligent Laboratory Systems. He is Chemometrics editor for Chemistry Central Journal. By early 2008, he published 311 articles, 136 of which are refereed papers and 6 edited/authored books. His work has been cited in 1792 publications. He has given 142 invited lectures in 23 countries worldwide and supervised 113 coworkers in Bristol. His current research interests include pattern recognition in forensics, biology, pharmaceuticals, medicine, and plastics using data obtained from chromatography, thermal analysis, acoustic spectroscopy, and mass spectrometry.

1.19

Multicriteria Decision-Making Methods

M. Pavan, Joint Research Centre, European Commission, Ispra, Italy R. Todeschini, University of Milano–Bicocca, Milan, Italy ª 2009 Elsevier B.V. All rights reserved.

1.19.1 1.19.2 1.19.3 1.19.4 1.19.4.1 1.19.4.2 1.19.4.3 1.19.4.3.1 1.19.4.3.2 1.19.4.4 1.19.4.4.1 1.19.4.4.2 1.19.4.4.3 1.19.4.5 1.19.4.6 References

Introduction Basic Notation Illustrative Example Multi-Criteria Decision Making Methods Pareto optimality Simple Additive Ranking Multiattribute Value Theory Intracriteria information (scores) Inter-criteria information (weights) Outranking PROMETHEE ELECTRE ORESTE Hasse Diagram Technique Goal Programming

591 594 595 596 597 598 599 599 601 605 606 609 614 619 625 625

1.19.1 Introduction Multi-criteria decision making (MCDM) is a discipline in its own right, which deals with decisions involving the choice of a best alternative from several potential candidates in a decision, subject to several criteria or attribute that may be concrete or vague. When making decisions, decision makers (DMs) always try to choose the optimal solution. Unfortunately an optimal solution exists only in case of one single criterion; in real decision situations almost any decision involves some conflicts or dissatisfaction. Typically, a decision problem is a situation where an individual has alternative courses of action available and has to select one of them, without an a priori knowledge of which one is the best. A decision process can be organized in three phases: problem identification and structuring, which consists of identification of the purpose of the decision, recognition of the problem to be solved, diagnosis of the cause–effect relationships for the decision situation, and identification of the judgment criteria. The second phase is the so-called model development and use, which includes the development of formal models DM preferences, values, trade-offs, goals to compare the alternatives or actions under consideration with each other in a systematic and transparent way. The third phase is the development of action plans since the analysis does not solve the decision. The decision process, which results in the selection of the best solution, that is the solution where the positive outcomes outweigh possible losses, is efficient if the procedure to reach the solution is optimal. The aims of a decision process are to generate information effectively on the decision problem from available data, to generate solutions effectively, and to provide a good understanding of the structure of a decision problem. MCDM methods are used to help people make their decision according to their preferences, in cases where there is more than one conflicting criterion, finding the optimal choice among the alternatives. Another term, which is often used to emphasize that the methods should support DMs in making the best decision is multiple criteria decision analysis (MCDA), but the methods themselves cannot make the final decisions. In fact, the

591

592


principal aim of MCDA is to help DMs synthesize the information so that they feel comfortable and confident about making a decision. MCDA integrates objective measurements with value judgments and makes explicit subjectivity. Making a decision is not just a question of selecting a best alternative. Often the need is to prioritize all the alternatives for resource allocation, or to combine the strengths of preferences of individuals to form a collective preference. Mathematics applied to decision making provides methods for quantifying or prioritizing personal or group judgments that are typically intangible and subjective. Decision making requires comparing different kinds of alternatives by decomposing the preferences into the many properties that the alternatives have, determining their importance, comparing and obtaining the relative preference of alternatives with respect to each property, and synthesizing the results to get the overall preference. Therefore, the strategy consists in breaking a complex problem into its smaller components and establishing importance or priority to rank the alternatives in a comprehensive and general way to look at the problem mathematically. The key starting point of MCDM lies in attempting to represent often intangible goals in terms of number of individual criterion. In fact, there are two main issues to be faced in a multicriteria problem: how to measure what is known as intangibles, and how to combine their measurements to produce an overall preference or ranking; and then, how to use it to make a decision with the best available mathematics. The identification of the set of alternatives is a crucial task. A differentiation is often made in MCDM theory1 between cases in which the set of alternatives is defined explicitly by a finite list of alternatives (called multiattribute decision making), and those in which the set of alternatives is defined implicitly by a mathematical programming structure (called multiobjective optimization theory). Another challenging feature of MCDM methods is the identification of the set of criteria by which alternatives are to be compared. Criteria selection is part of the modeling and problem formulation, a significant phase often underemphasized. Criteria are often developed in a hierarchical way, starting from general and vague goals and refined into more detailed subgoals.2 A useful general definition of a criterion is the one provided by Bouyssou3 as a tool allowing comparison of alternatives according to a particular axis or point of view. It is generally assumed that each criterion can be represented by a surrogate measure of performance, represented by some measurable attribute of the consequences arising from the achievement of any particular decision alternative. In identifying the criteria, some considerations are to be taken in account: their value relevance, that is their link with the DM concept of their goals; their understandability and their measurability, that is the performance of the alternative against the criteria should be measurable; their non-redundancy to avoid the concept they represent is attributed undue importance; their judgmental independence, that is the preferences with respect to a single criterion should be independent of the level of another; and their balancing between completeness and conciseness. Subjectivity is intrinsic in all decision making and in particular in the choice of the criteria on which the decision is based on and in their relative weight. MCDA does not dissolve subjectivity, but it makes the need for subjective judgments explicit and the whole process by which they are considered is made transparent. The general structure of the MCDM process is shown in Figure 1. According to Stewart,2 MCDM methods can be used in two contexts. In the first context, DM would be a single individual or a homogenous group of people, trying to make a decision that does not seriously affect, and therefore necessitate justification to, other parties. In this specific case, the methods can be pretty informal and the foundation of the decision adopted does not need to be extensively documented. In the second context, DM has to make decision on behalf of a large group of people. In such a case, the rationale of the choices should be clearly documented and proper consideration of each interest needs to be demonstrated. Therefore, the use of more formal methods is required. A distinction between methods relevant in this perspective is the one between ‘prior’ and ‘progressive’ (also named ‘interactive’) articulation of preferences.1,4 In case of prior articulation of preferences, the DM has to specify value judgments in isolation from the particular choices available, and these are then translated to the particular choices from the set of alternatives that are consistent with these preferences. This kind of approach is considered advantageous when full justification and rationale for decisions are required. The methods of

Multicriteria Decision-Making Methods 593

Optimality Condition

Criteria Goal Alternatives

Weights

MCD method

Best alternative

Alternatives ranking

Figure 1 General structure of the multicriteria decision-making (MCDM) process.

progressive articulation of preference give the DM the opportunity to explore the decision space systematically, without the need to specify any prior preferences. Often this approach is considered to be more efficient; it requires less comprehensive assumptions on preferences structures. However, it is more subjective to manipulation by skilled users, and therefore is less defensible in cases where solutions have to be justified and rationalized. Over the years, several MCDM methods have been proposed5 in different areas, with different theoretical background and facing different kind of questions and providing different kind of results.6 Some of these methods have been developed to fulfill the need of specific problems, whereas other methods are more general and have been used in different areas. The different MCDM methods are distinguished from each other in the nature of the model, in the information needed, and in how the model is used. They have in common the aim to create a more formalized and better informed decision-making process, the need to define alternatives to be considered, the criteria to guide the evaluation, and the relative importance of the different criteria. The different methods developed to solve multicriteria decision problems can be divided into several groups.7 The mostly known approaches to multicriteria decisions are Pareto optimality, outranking method based on ordinal comparison of concordance and discordance, partial order ranking (POR) theory, utility and multiattribute value theory based on the use of lottery comparisons, goal programming (GP) that is basically a modified version of linear programming, priority theory of the analytic hierarchy process, regime method, convex cone theory, hierarchical interactive method, visual reference direction approach, aspiration level interactive method, fuzzy set theory, and Bayesian theory based on probabilities. MCDM methods have been classified into generating techniques and preference techniques by Cohon and Marks.8 Generating techniques do not use any prior information to define the importance of the criteria. A point is considered noninferior solution to the MCDM problem if there exists no other point in the design space, which provides an improvement in one criterion without causing degradation in the other. Weighting and decisions making are performed after the complete set of noninferior solutions is developed. The first multicriteria optimization techniques were based on this principle. Examples of generating techniques are the overlay of contour plots, the methods based on the Pareto optimality concept.9 In these methods, alternative settings of independent variables can only be excluded on the basis of inferiority. Techniques based on prior judgment or those based on preference functions give a more complete ordering of the total set of alternatives. These techniques can be subclassified into those using a target solution (GP, visual reference direction approach) and those based on subjective preference of the DMs (utility and MAVT). Another type of classification is the one based on the techniques that are used with sequential or simultaneous optimization.

594


1.19.2 Basic Notation Before moving to the description of the multicriteria decision-making methods, it is useful to introduce the basic notations for MCDM. Let A be the finite set of n alternatives (actions, objects, scenarios, events, cases, and so on) that has to be ranked; the consequences of the actions are analyzed by a set F of p criteria (variables, attributes, properties, and so on): A ¼ {a1, a2, . . ., an} or, in a simpler way, as A ¼ {a, b, c, . . .} and F ¼ {f1, f2, . . ., fp}. Therefore, alternatives a and b are p-dimensional vectors as

a ¼ a1 ; a2 ; . . .; ap ;

b ¼ b1 ; b2 ; . . .; bp

ð1Þ

where p is the number of selected criteria. Each ith alternative (i ¼ 1, . . .,n) assumes for the jth criterion (j ¼ 1, . . ., p) the actual value fij . The values of the jth criterion for the n alternatives can be collected in a n-dimensional vector as

f j ¼ a1j ; a2j ; . . .; anj

ð2Þ

For simplicity, in this chapter, the vectorial notation will be used only if necessary. Each criterion can then be weighted to take into account the different importance of the criteria in the decision rule. For this purpose, each criterion can be weighted. The weight vector is a p-dimensional vector defined as w ¼ ðw1 ; w2 ; . . .; wp Þ

ð3Þ

with the following constraints: 0 wj 1;

p X

wj ¼ 1;

j ¼ 1; . . .; p

ð4Þ

j ¼1

For the purpose of exposition, and without loss of generality, we shall suppose that the criteria are defined in increasing sense, that is the DM prefers large to smaller values for each fj . We can also suppose, without loss of generality, that the actual values of each criterion can be transformed into values between 0 and 1 by a transformation function t, and the optimal values of each criterion can be assumed as the highest ones. Thus, each transformation function t can be defined as

tij fij with the constraint 0 tij fij 1

ð5Þ

As example, a classical linear transformation is the following:

tij fij ¼

fij – minðf j Þ

; max f j – min f j

0 tj fij 1

ð6Þ

A ranking function R is a function that works on the n alternatives and associates to each alternative its rank, that is its position within the n-dimensional set of alternatives. As stated above, it can be assumed that the best alternative has rank n and the worst alternative the rank 1. Given a vector x of n values, a ranking function R can be defined as RðxÞ: Rn ! r PR;

1r n

ð7Þ

The value produced by a ranking function is an integer if no object with the same rank is found, but is a real value if the average ranking is calculated for objects having the same rank.


Therefore, a ranking function gives to the ith object value xi a ranking value ri with respect to the n values, as Rðxi j xÞ ¼ ri ;

1 ri n

ð8Þ

Thus, the term rij indicates the rank of the ith object for the jth criteria. Furthermore, if some alternatives have the same x values, they should have the same rank, calculated as average rank.10 The average rank is computed as the average of the initial not averaged ranks. An example may be of help: in the case of four alternatives with same x values that have ranks 3, 4, 5, and 6, the average rank is (3 þ 4 þ 5 þ 6)/4 ¼ 4.5, and it is assigned to all the four alternatives. For each criterion a preference structure, also named a weak order, on the set A is defined as

Pj ða; bÞ X Fj aj ; bj ; wj

ð9Þ

and it is then used to define the global preference function associated to each pair of alternatives:

ða; bÞ X F ða; b; wÞ X F a1 ; a2 ; . . .; aj ; b1 ; b2 ; . . .; bj ; w1 ; w2 ; . . . ; wj

ð10Þ

Pþ(a, b) is the set of criteria where a dominates b, that is where a is better than b, P0(a, b) is the one where a and b are equal (or indifferent), and P–(a, b) is the set of criteria where a is dominated by b, defined as follows:

P þ ða; bÞ X 8j j faj > fbj

P 0 ða; bÞ X 8j j faj ¼ fbj

P – ða; bÞ X 8j j faj < fbj

ð11Þ ð12Þ ð13Þ

1.19.3 Illustrative Example To understand the methods better, the following example is used in the whole chapter. Consider the decision problem to select the best alternative (e.g., the best car) out of the three alternatives whose performances judged by five criteria are illustrated in Table 1. Let c1 and c5 be the criteria to be minimized (e.g., price and maintenance), and c2, c3, and c4 the criteria to be maximized (e.g., performance, economy, and appeal). Information about the criteria, their relative importance, and their ranges is provided in Table 2.

Table 1 Values of alternatives for criteria Alternatives

c1

c2

c3

c4

c5

1 2 3

3400 1880 4000

190 200 186

30 24 28

45 35 50

1400 2800 1680

Table 2 Criteria, weights, and ranges Ranges Criteria c1 c2 c3 c4 c5

Price (Euro) Performance (horsepower) Economy (km l–1) Appeal (1 ¼ ugly; 5 ¼ beautiful) Maintenance (Euro per year)

Weights

Minimum

Maximum

0.35 0.23 0.18 0.12 0.12

1000 150 20 30 1000

5000 200 40 60 3000

596


1.19.4 Multi-Criteria Decision Making Methods Once the decision problem identification phase has generated a set of alternatives, which can be a discrete list of alternatives as well as be defined implicitly by a set of constraints on a vector of decision variables, and once the set of criteria against which the alternatives have to be analyzed and compared, then a decision model has to be built to support DMs in searching the optimal or the set of satisfactory solutions to the multicriteria decision problem. The decision model needs to reflect the DM preferences and value decisions. It is therefore made of two main components as described by Belton and Stewart:11

Preferences in terms of each individual criterion, that is, models describing the relative importance or desirability of achieving different levels of performance for each identified criterion. In addition, for each criterion it is necessary to ascertain explicitly if the best condition is satisfied by a minimum or a maximum criterion value, and the trend from the minimum to the maximum must also be established. Criteria setting is a crucial point since it requires the mathematization of decision criteria which are often not completely defined or explicit. An aggregation model, that is a model allowing intercriteria comparisons (such as trade-offs), to combine preferences across criteria. Criteria are not always in agreement, they can be conflicting, motivating the need to find an overall optimum that can deviate from the optima of one or more of the single criteria. Multicriteria decision-making methods are often based on an aggregation function of the criteria fj, where j ¼ 1, . . ., p : X gðf 1 ; f 2 ; . . .; f p Þ

ð14Þ

Thus, if an alternative is characterized by p criteria, then a comparison of different elements needs a scalar function, that is an order or ranking index, to sort them according to the numerical value of . Several evaluation methods that define a ranking parameter generating a total order ranking have been proposed in the literature. The modeling in multicriteria decision analysis differs significantly from the modeling of external realities because in the latter case the true reality is, in principle, observable as a standard against which the model can be tested and validated. In the case of multicriteria decision modeling, there is no observable reality; the decision modeling procedure aims to help DMs in understanding what they really want to do and identifying the options which best satisfy their objectives. Thus, the purpose of the decision model is to create a view of DM preferences based on a defined set of assumptions and to guide the DM in the optimum solution search. Before reviewing different multicriteria decision-making methods, some further terms and basic principles are introduced. A p-dimensional system is generally considered, with an associated (n p) data matrix X. To each of the n alternatives, a set of p criteria relevant to the decision-making procedure is associated. Each criterion can then be weighted to take account of the different importance of the criteria in the decision rule. The strategies to reach the optimal choice require the development of a ranking of the different options. Within a set of alternatives A (a, b, c, d), a ranking (order) on A is a relation with the following properties: a a ðreflexivityÞ

ð15Þ

a b and b a ) b ¼ a ðantisymmetryÞ

ð16Þ

a b and b c ) a c ðtransitivityÞ

ð17Þ

A set A equipped with the relation is said to be an ordered set. A multicriteria decision-making method can generate

a complete or total ranking: a > b > c > d (or linear order) the best alternative: a > ðb; c; d Þ


a set of acceptable alternatives: ða; b; cÞ > d an incomplete ranking of alternatives a > ðb; c; d Þ or ða; bÞ > ðc; d Þ

The correct definition of a criterion may imply a more or less objective ordering of the alternatives according to this criterion, the orientation of the criterion, that is, the direction of preference for the criterion. For each criterion it has to be explicitly established whether the best condition is satisfied by a minimum or maximum value of the criterion. Where this well-defined measure of performance exists, the performance level or attribute value of the alternative a according to criterion j can be represented by faj . To simplify the discussion, all criteria are supposed to be defined in such a way that increasing values are preferred. The preference function is a measure of performance for criterion j, and is a partial preference function in the sense that alternative a is strictly preferred to b in terms of criterion j if and only if faj > fbj . These preference functions may correspond to natural attributes on a cardinal scale or may be built on ordinal or categorical scales. The only property these functions need to satisfy is the ordinal preferential independence, that is, it must be possible to rank order alternatives on one criterion independently of performances in terms of the other criteria. Once partial preference functions have been associated with each criterion, a check is performed on the existence of any pairs of alternatives a and b for which a is at least as good as b on all criteria (i.e., faj fbj for all j ), and is strictly preferred to b on at least one criterion (i.e., faj > fbj for at least one j ). In this case, the vector of performance measures fa is said to dominate fb .

1.19.4.1

Pareto optimality

Criterion 2

Pareto optimality is a multicriteria decision-making method introduced into chemometrics by Smilde et al.12–16 If for two alternatives a and b, faj fbj for all the criteria (1 j p), with at least one strictly inequality, then we say that the alternative a dominates b. Alternatives that are not dominated by any other are termed Pareto optimal (PO) points (efficient points). The Pareto frontier is the set of PO points. The Pareto optimality technique selects the PO points, and the points that are not PO points are inferior to the PO points with respect to at least one criterion. Figure 2 illustrates a two-dimensional criterion space. A point corresponds to one setting of two criteria, the criterion values of which are plotted against each other. The space around the point P can be divided in four quadrants. In the case of two criteria, both to be maximized, the points in the first quadrant are inferior to point P, whereas points in the fourth quadrant are superior to point P. The points in the second and third quadrants are incomparable with point P since they are superior to P for one criterion and inferior for the other. By definition, a PO point is superior to all other comparable points; thus

2

4 P

1

3

Criterion 1 Figure 2 Representation of the four quadrants in a two-dimensional criterion space around point P.

598


a

f2

c

b

f1 Figure 3 Bivariate representation of the criteria f1 and f2. Points a and c are Pareto optimal points; the continuous line is the Pareto frontier.

in the case of Figure 3 representing the space of two criteria f1 and f2, both to be maximized, a point a is superior to another point b if the following conditions are verified: fa1 > fb1

and

fa2 > fb2

ð18Þ

fa1 > fb1

and

fa1 ¼ fb1

ð19Þ

fa1 ¼ fb1

and

fa2 > fb2

ð20Þ

or

or

In other words, a point is a PO point if no other points are found in the upper-right quadrant. According to Pareto optimality, at least one point must be PO, and all the noninferior and incomparable points together form a set of PO points. If the system under study is described by more than two criteria, the p-dimensional criterion space (p > 2) containing the PO points can be projected onto a two-dimensional plane. Through principal component analysis (PCA) of the matrix containing the PO points and following projection of the scores, it is possible to investigate the criterion space graphically.

1.19.4.2

Simple Additive Ranking

The simple additive ranking (SAR) method17–19 is a very intuitive approach to MCDM, based on the ranking of the alternative with respect to each criterion separately, and the subsequent aggregation of the weighted ranks by arithmetic mean, finally normalized. The resulting score, which defines the performance of the alternative, is computed as Pp Si ¼

j ¼1

wj rij

n

;

1 Si 1 n

ð21Þ

with the following constraints, 0 wj 1;

p X

wj ¼ 1;

j ¼ 1; . . .; p

ð22Þ

j ¼1

where rij is the rank of the ith alternatives Pp for the jth criterion and the number n of alternatives, a normalization factor, under the weight constraint j ¼1 wr ¼ 1. It is assumed that the best and the worst values are rank n and rank 1, respectively.

Multicriteria Decision-Making Methods 599 Table 3 Estimated simple additive ranking (SAR) scores Alternatives

SAR score

1 2 3

0.650 0.580 0.270

Finally, to obtain scores between 0 and 1, a further scaling can be performed as S9 i ¼

Si – ð1=nÞ ; 1 – ð1=nÞ

0 S9i 1

ð23Þ

Being based on the separate ranking of each criterion, this method is robust with respect to anomalous values obtained for some alternatives in some criterion. The results of the SAR method applied on the illustrative example data of Tables 1 and 2 are provided in Table 3.

1.19.4.3

Multiattribute Value Theory

Multiattribute value function methods provide a synthesis of the performances of alternatives against individual criteria, together with intercriteria information that reflect the relative importance of the different criteria, to provide an overall estimate of each alternative reflecting the DMs’ preferences. Therefore, the purpose of this approach is to associate a number, a global ranking index, to each alternative, to produce a preference order of the alternatives according to the numerical value of the ranking index. Therefore, a number (score) Sa is associated to each alternative a, in such a way that a is judged to be preferred to b (a > b), taking into account all criteria if and only if Sa > Sb , which implies indifference between a and b if and only if Sa ¼ Sb . As pointed out by Belton and Stewart,11 the preference order implied by any value function should provide a complete or total order. This means that preferences are complete, that is for any pair of alternatives, either one is strictly preferred to the other or there is indifference between them. In addition, preferences and indifferences are transitive, that is for any three alternatives a, b, and c, if a b and b c then a c. Once the initial model structure has been defined together with the set of alternatives, then the next step consists in drawing the information required by the model. In the case of multiattribute value function models, two types of information are needed, sometimes referred to as intracriterion information and intercriterion information, or as scores and weights, respectively. 1.19.4.3.1

Intracriteria information (scores) Scoring is the process of assessing the partial value functions for each criterion in the decision model. Partial value functions are defined so that alternative a is preferred to b in terms of criterion j if and only if faj > fbj , whereas indifference between a and b in terms of this criterion exists if and only if faj ¼ fbj . The score values need to be defined on an interval scale of measurement, on which the key factor is the difference between points. Therefore, a ratio of values will only have meaning if the zero point on the scale is absolutely and unambiguously defined. Thus, to build such a scale it is necessary to define two reference points and to distribute numerical values to these points. The reference points are often taken at the bottom and top of the scale, to which are assigned values such as 0 and 100, or 0 and 1. The minimum and maximum points on the scale can be defined on a local scale or on a global scale. The local scale is defined by the set of alternatives under consideration and the minimum and maximum points correspond to the alternative, which does least well on a particular criterion (score ¼ 0) and the one which does the best (score ¼ 1). All other alternatives obtain intermediate scores (0 < score < 1), according to their performance relative to the two reference points. The use of local scales has the advantage that allows a quite quick assessment of values and, thus, it is useful under time constraints.

600


A global scale is defined by taking into account a wider set of possibilities. The reference points correspond to the ideal and the worst possible performance on each specific criterion, or by the best and worst performance that could realistically take place. The definition of a global scale requires additional information and in general more work than a local scale. However, the advantage of the global scale is that it is more general and therefore it can be defined before the examination of the specific alternatives. Another way to describe a global scale consists in specifying reference points in terms of neutral and good performance levels. More details on this approach can be found in Bana e Costa and Yansnick.20 Both in cases of local and global scales, it is important that all subsequent analysis, like the assessment of the weights, is consistent with the chosen scaling. For the illustrative example of Tables 1 and 2, the local scale for the first criterion is 1880 and 4000, that is the minimum and maximum values of the actual data, whereas the global scale is 1000 and 5000, that is the minimum and maximum values corresponding to the ideal and the worst possible performance on that criterion as defined by the DM. Once the reference points have been defined, the next step, which consists in assigning the other scores, can be performed in three ways:

defining a partial value function building a qualitative value scale providing a direct rating of the alternatives

Defining a partial value function To define a partial value function, a measurable attribute scale closely related to the DM values need to be identified. The partial value function can be assessed directly or by using indirect questioning. In case of direct assessment, which is often supported by a visual representation, the DM should evaluate whether

1.19.4.3.1(i)

The value function is monotonically increasing against the natural scale: the highest value of the criterion is most preferred, whereas the lowest least preferred. The value function is monotonically decreasing against the natural scale: the lowest value of the criterion is most preferred, whereas the highest is least preferred (cost criteria). The value function is nonmonotonic: an intermediate point of the scale corresponds to the most preferred or least preferred point (pH ¼ 7).

As demonstrated by Stewart,21,22 the final results of the decision-making analysis are strongly dependent on these assumptions and the defaults assumption of linearity, which is commonly made, may generate confusing results. Examples of monotonically increasing, monotonically decreasing, and nonmonotonic value functions are illustrated in Figure 4. The partial value functions can be assessed also indirectly, assuming that the value function is monotonically increasing or decreasing. Two main methods are widely used for this purpose, called ‘bisection’ and ‘difference’ as described by von Winterfeldt and Edwards23 and Watson and Buede.24 Using the bisection method, the DM has to identify the point on the criterion scale which is halfway, in value terms, between the two minimum and maximum reference points. The difference approach includes a set of methods all requiring the DM to consider increments on the objectively measured scale and to relate them to differences in value. One of these methods, described by Watson and Buede,24 require the partition of the criterion scale into a defined number of equal intervals. The DM has to then rank the specified differences according to increase in associated value. Another indirect approach, described by von Winterfeldt and Edwards,23 consists in defining a unit level on the criterion scale (suggested between one-tenth and one-fifth of the difference between the minimum and maximum reference points) and questioning on the unit levels. Examples of these approaches are also described in Belton and Stewart.11 1.19.4.3.1(ii) Building a qualitative value scale An appropriate qualitative scale is needed in all those circumstances where it is not possible to find a measurable scale for an identified criterion. Like for the other


Linear function 1.2 1 0.8 0.6 0.4 0.2 0 –0.2

Sigmoid function 1.2 1 0.8 0.6 0.4 0.2 0 –0.2

Inverse linear function 1.2 1 0.8 0.6 0.4 0.2 0 –0.2

Step function 1.2 1 0.8 0.6 0.4 0.2 0 –0.2

Triangular function 1.2 1 0.8 0.6 0.4 0.2 0 –0.2

Logarithmic function 1.2 1 0.8 0.6 0.4 0.2 0 –0.2

Step function 1.2 1 0.8 0.6 0.4 0.2 0 –0.2

Inverse triangular function 1.2 1 0.8 0.6 0.4 0.2 0 –0.2

Parabolic function 1.2 1 0.8 0.6 0.4 0.2 0 –0.2

Figure 4 Examples of monotonically increasing, monotonically decreasing, and nonmonotonically value functions.

approaches, at least the two reference points (minimum and maximum) have to be already defined. Qualitative scales should show evidence of some requirements:11

Operational: they should permit the DMs to rate alternatives not used in the definition of the scale. Reliable: two independent ratings of an alternative should lead to the same score Value relevant: they should be related to the DM objective Justifiable: they should be capable of convincing an independent observer that the scale is reasonable.

An example of this process is described in Belton and Stewart.11 1.19.4.3.1(iii) Direct rating of the alternatives The direct rating approach is another way which can be used for the construction of a value scale; it requires only the definition of the reference points of the scale. Both local and global scales can be used. In the case of a local scale, the best alternative is given the highest rate (usually 100, 10, or 1) and the worst alternative is given a score of 0. All the other alternatives are then located directly on the scale reflecting their performance with respect to the two reference points. The main disadvantage in using local scales is that if new alternatives are subsequently introduced into the decision problem, then the scales have to be redefined, and consequentially the weighting criteria.

1.19.4.3.2

Inter-criteria information (weights) An important piece of information, which also strongly affects the final results of the decision-making process, is the one related to the relative importance of the criteria. The weight assigned to a criterion j is essentially a scaling factor which relates scores on that criterion to scores on all other criteria. Thus if to a criterion j is assigned a weight which is twice that of another criterion k this should be interpreted that the DM values 10 value points on criterion j the same as 20 value paints on criterion k. These weights are referred to as swing weights to make a distinction to the concept of importance weights.11 It is rather common to make the error of assuming that weights are independent of the measurement scales used, while the effect of the weight parameter wj is directly linked to the scaling used for the partial preference function and the two are closely connected. To assign swing weights, the method commonly followed is that from the worst value to the best value on each criterion. The DM may be asked to consider all bottom-level criteria at the same time. The swing which gives the greatest increase in overall value is the one which will have the highest weight. The process is then

602


applied on the remaining set of criteria and proceeds until a ranking of the criteria weights has been determined. Once a ranking has been defined, then they need to be assigned values. One of the possible ways to do so is by directly asking the DM to compare each criterion with the highest ranked one. Thus, the DM is asked to define the increase in overall value resulting from an increase from score 0 to a score of 100 on the selected criterion as a percentage of the increase in overall value resulting in an increase from score of 0 to 100 on the highest ranked criterion.11 Weights are then generally normalized to sum up to 1 or 100. The normalization allows an easier interpretation of the importance of the criteria. Note. Without loss of generality, it can be assumed that the p criteria are ranked from 1 to p, where the criterion 1 is ranked 1, criterion 2 is ranked 2, and criterion p is ranked p. If the DM is able to rank the criteria, a simple consistent weighting of the criteria can be obtained with respect to the first one as wj ¼ w1 =rjk . The normalized weights for ranked criteria can be calculated as Q =rjk wj ¼ Pp k j ¼1 Q =rj

ð24Þ

where rj is the jth criterion rank, k (0) is parameter, and Q is defined as Q ¼

p Y j ¼1

rjk

¼ exp

p X

k ln rj

! ð25Þ

j ¼1

The smoothing parameter k influences the relative differences between the criterion weights; in particular, for k ¼ 0 all the criteria are equally weighted, whereas with increasing k, the weight of the first criterion becomes more and more relevant with respect to the other lower ranked criteria, and the weight of the second criterion becomes more and more relevant with respect to the other lower ranked criteria, and so on. In practice, increasing k increases the differences of the criterion weights. It must be noted that the criteria ranks can be also defined as average ranks. Some examples of weights are given in Table 4 for different k values and criterion ranks, for four criteria (p ¼ 4). 1.19.4.3.2(i) Sensitivity and robustness Once the intracriterion information and intercriterion information have been defined, a good practice is to check whether the preliminary assumptions are robust or whether they are sensitive to changes in the model. Technically, a sensitivity analysis consists in examining the effect on the output of a model resulted by changes in input parameters of the model. In the case of decisionmaking model, the parameters are the partial value functions and the scores and weights assigned by the DM, whereas the output is the overall evaluation of the alternatives. Therefore, a sensitivity analysis is useful to identify which, in case there is any, of the input parameters provides a crucial influence on the overall evaluation. The sensitivity analysis is also helpful for the DM to confirm his understanding of the problem and when it is performed in a group context it provides the opportunity to consider and/or explore alternative views of the problem. Table 4 Examples of weight calculations from ranks K

Criterion

1

2

3

4

0.5

Rank Weight Rank Weight Rank Weight Rank Weight Rank Weight

1 0.359 1 0.480 1 0.598 1.5 0.348 1.5 0.390

2 0.254 2 0.240 2 0.212 1.5 0.348 1.5 0.390

3 0.207 3 0.160 3 0.115 3 0.174 3.5 0.110

4 0.180 4 0.120 4 0.075 4 0.130 3.5 0.110

1.0 1.5 1.0 1.5


Different methods belong to the multiattribute value function approach; the three most commonly applied methods are illustrated in the following sections. More detailed discussions on the value measurement theory can be found in Keeney and Raiffa,25 Roberts,26 von Wintefeldt and Edwards,27 French,19 Keller and Massart,28 Hendriks et al.,29 and Lewi et al.30 Several variants of the multiattribute value function approach have been proposed; among these there is the so-called interactive methods based on value function approach, which comprises methods based on trade-off information,31 methods using direct comparisons,32,33 and the so-called convex cone approach.34 1.19.4.3.2(i)(a)

Utility and desirability

Utility functions and desirability functions are well-known multicriteria decision-making methods. The approach is the form most simply and easily understood by DMs from a variety of backgrounds, since it does not require any stronger restrictions on the preferences structures than the aggregation formula. They are based on the definition of a partial value function, that is, a transformation function t, for each criterion to standardize the partial value functions transforming values of the criteria to the same scale. Typically, the best and worst conditions need to be defined for each criterion. This can be done locally, taking simply the best and worst of the available alternatives, or more generally as the best and worst possible conditions in similar contexts. For this purpose, different kinds of functions can be used, the more common ones being linear, sigmoid, logarithmic, exponential, step, normal, parabolic, Laplace, triangular, and box (Figure 4). Each criterion is independently transformed into a utility/desirability tij by an arbitrary function, which transforms the actual value fij of each ith alternative for the jth criterion into a value between 0 and 1. Once the kind of function and its trend for each criterion has been defined, the overall utility/desirability of each ith alternative is computed. Utility and desirability functions differ only for the aggregation form of the overall utility U and desirability D. The overall utility Ui of each ith alternative is defined for the unweighted and weighted cases, as arithmetic mean: Pp Ui ¼

j ¼1 tij

p

; Ui ¼

p X

wj tij ;

0 Ui 1

ð26Þ

j ¼1

A particular case of the utility function is the so-called simple additive weighting method,33 which consists in the utility function approach performed by using a linear transformation on a local scale for each jth criterion. In the case of the desirability method, first presented by Harrington35 and then generalized by Derringer and Suich,36 the overall desirability Di of each ith alternative is defined, for the unweighted and weighted cases, as geometric mean: Di ¼

p ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi w p ti1 ti2 . . . tip ; Di ¼ ti1w1 ti2w2 . . . tip p ;

Pp

0 Di 1

ð27Þ

In all the cases, the weight constraint is assumed: j ¼1 wj ¼ 1. It can be noticed that the overall desirability is calculated more severely than the utility: in fact, if an element is poor with respect to one criterion, its overall desirability will be poor. If any desirability di is equal to 0 the overall desirability Di will be zero, whereas the Di will be equal to one only if all the desirabilities have the maximum value of one. Once the overall utility Ui or desirability Di for each alternative has been calculated, all the alternatives can be totally ranked according to their U or D values and the element with the highest U or D can be selected as the best one, if its value is considered acceptable. A desirability scale, shown in Table 5, was developed by Harrington.35 Both utility and desirability functions are affected by arbitrariness related to the a priori selection of the partial value functions and corresponding upper and lower limits. Moreover, these functions are very easy to calculate and no specific software is required. The critical feature of these approaches to multicriteria decision-making problems is the establishment of the relation between criteria and partial value functions values, which must be performed by the DM. The simplicity of the additive aggregation makes the utility function approach particularly appealing. Only relatively minor assumptions are needed and these are primarily related to the criteria definition and to the

604


Table 5 Harrington qualitative definition of the Desirability scale Scale of D

Quality evaluation

1.00 1.00–0.80 0.80–0.63 0.63–0.40 0.40–0.30 0.30–0.00 0.00

Improvement beyond this point has no preference Acceptable and excellent Acceptable and good Acceptable but poor Borderline Unacceptable Completely unacceptable

interpretation of partial value functions and weights. Three additional requirements have been illustrated by Belton and Stewart,11 derived by simple algebraic properties of the additive form, such as preferential independence, interval scale property, and weights as scaling constants. 1.19.4.3.2(i)(b)

Dominance

The dominance function method is based on the comparison of the state of the different criteria for each pair of alternatives. This approach does not require the transformation of each criterion into a quantitative partial value function; it only requires establishing whether the best condition is satisfied by a minimum or maximum value of the selected criterion. For each pair of alternatives (a, b), three sets of criteria are determined: Pþ(a, b) is the set of criteria where a dominates b, that is, where a is better than b, P0(a, b) is the one where a and b are equal, and P(a, b) is the set of criteria where a is dominated by b. The dominance function between two alternatives a and b is calculated considering – separately – the weights for the criteria in the Pþ and P sets as follows: Cða; bÞ ¼

1þ 1þ

P þ ða;bÞ

wj

j PP – ða;bÞ

wj

Pj PP

;

0:5 Cða; bÞ 2

ð28Þ

Pp Being P þ ða; bÞ [ P – ða; bÞF and with the usual constraint j ¼1 wr ¼ 1: A C(a, b) value equal to 1 means equivalence of the two alternatives; C(a, b) > 1 means that the alternative a is, on the whole, superior to the alternative b, whereas C(a, b) < 1 means that the alternative a is, on the whole, inferior to the alternative b. The obtained values can be normalized according to C9ða; bÞ ¼

Cða; bÞ – 0:5 ; 2 – 0:5

0 C9ða; bÞ 1

ð29Þ

0 a n – 1

ð30Þ

A global score of the alternative a is then calculated as a ¼

n X

C9ða; iÞ;

i 6¼ a;

i¼1

and the corresponding scaled value is 9a ¼

a ; n–1

0 9a 1

ð31Þ

A total ranking is obtained on 9 and the highest values are the best alternatives. The results of desirability, utility, and dominance functions applied to the illustrative example of Tables 1 and 2 on the global scale and by using linear transformations for all the partial value functions are illustrated in Table 6. When the local scale is used instead of the global one, the following results illustrated in Table 7 are provided. It can be noticed that more severity is provided by the desirability approach with respect to the utility. The second and third alternatives are judged as not desirable because of their low performance on the third and first criterion, respectively, no matter their performances on the other criteria. The dominance results are independent from the scale used, being based on a pair comparison approach.

Multicriteria Decision-Making Methods 605 Table 6 Estimated values of alternatives for criteria computed on global scale Alternatives

Desirability

Utility

Dominance

1 2 3

0.545 0.420 0.439

0.570 0.571 0.484

0.526 0.408 0.165

Table 7 Estimated values of alternatives for criteria computed on local scale

1.19.4.4

Alternatives

Desirability

Utility

Dominance

1 2 3

0.453 0 0

0.541 0.567 0.336

0.526 0.408 0.165

Outranking

The concept of outranking relations was born in France from difficulties encountered with diverse concrete problems.20–40 Numerous applications of the concept have then been developed by Barda et al.,41 Climaco et al.,42 Martel and Nadeau,43 Maystre and Simos,44 Parent and Schna˜bele,45 Rabeharisoa,46 Renard,47 Roy et al.,48,49 and Slowinski and Treichel.50 A comprehensive treatment of the outranking concept is given by Roy and Bouyssou51 and Roy.52 Outranking models are applied directly to partial preference functions, which have been defined for each criterion.11 The outranking approach is based on a generalization of the concept of dominance, according to which if for two alternatives a and b, faj fbj for all criteria, with strict inequality faj > fbj for at least one criterion, then it can be concluded that a should be preferred to b. In this case, the evidence that alternative a is as good as or better than alternative b is incontestable and a is said to dominate b. More generally, it is said that a outranks alternative b if there is ‘sufficient evidence’ to support the conclusion that a is at least as good as b taking all criteria into account. Two key features make the outranking definition different from the preference relationships corresponding to value functions. The first feature is related to the fact that the importance is on strength of evidence for the statement that ‘a is at least as good as b’ rather than on strength of preference itself. As a consequence, it is possible that strong preferences are identified, even though the evidence at an earlier stage of building preferences might be highly conflicting, leading to ‘no outranking’. On the other hand, the evidence for preferring one alternative over another may be certain (one alternative outranks the other), even though preferences are weak. This result is obtained in the case that one alternative dominates another, but none of differences are considered significantly important. The second feature that makes the outranking definition different from the preference relationships is that even when neither a nor b outranks the other, a state of indifference is not inevitably implicit. Therefore, the comparison of two alternatives can lead to four situations (despite the three implied by full adherence to value function): (1) a definite preference for alternative a over b, (2) a definite preference for alternative b over a, (3) indifference, which corresponds to the case of lack of decisive evidence, or (4) incomparability, which corresponds to the case of contradictory evidence. The outranking approach starts considering all individual partial preference functions. The common assumption is that the preference functions are usually rather imprecise measures and therefore alternative a can be considered conclusively preferred to b in terms of criterion j if the evidence by which faj exceed fbj is above the ‘indifference threshold’. In more details, two thresholds can be defined to distinguish between weak and strict preference, respectively. As already pointed out, equal increments in the preference function values are not necessarily of equal importance; therefore, the threshold for weak and strict preference may depend on the actual value of faj . In the outranking method,11 two thresholds sj (strong preference) and qj (weak preference) are defined for each criterion j so that:

Alternative b is weakly preferred to alternative a in terms of criterion j if fbj – faj > qj

ð32Þ

606


Alternative b is strictly preferred to alternative a in terms of criterion j if fbj – faj > sj

ð33Þ

where for consistency sj >qj In the case that faj – fbj qj , then in terms of the criterion j there is indifference between a and b, that is, there is no evidence to support preference for either of the alternatives. The preferences in terms of each criterion are modeled by partial preference functions and the indifference and preference thresholds, then the evidence is aggregated across criteria to summarize the discrimination between the alternatives. This process is clearly dynamic and different rules of evidence have been defined leading to the existence of many outranking methods for MCDA. However, two recurring themes used in most of the methods have been identified:11

The concordance principle: if a is demonstrated to be as good as or better than b according to a sufficiently large weight of criteria, then this is considered to be evidence of a outranking b. The discordance principle: if b is very strongly preferred to a on one or more criteria, then this is considered to give evidence against a outranking b.

The concordance principle is generally applied by some form of weighted pairwise comparison procedure. An importance weight is associated to each criterion and then for any pair of alternatives a and b, each criterion votes for or against the statement that ‘a is at least as good as b’. Different outranking methods have been proposed according to how the vote is applied. The weight interpretation in the outranking methods differ from the one in the value function because they do not represent trade-offs or scaling factors introduced to guarantee commensurability between criterion measures. The weights provide a measure of the influence each criterion should have in supporting the assertion that one alternative is at least as good as another, and are not affected by monotonic transformations of the preference functions. The two most known outranking methods are the ELECTRE family of methods, developed by Roy and his group at the Laboratoire d’Analyse et Mode´lisation de Systemes pour l’Aide a la Decision (LAMSADE), University of Paris Dauphine, and PROMETHEE, proposed by Brans from the Free University of Brussels. Roy was the one who started and developed the initial and major theory on outranking methods; he was motivated by the need to develop an approach, which was not affected by the strong requirements of the value function approach that all the available alternatives are comparable. He proposed the ELECTRE method with the purpose to provide a weaker, poorer decision-making model than a value function model but built with less effort, and requiring fewer hypotheses, and not always allowing a final conclusion. The outranking approach was described by Brans and Vincke7,53 as an enrichment of the dominance relation, which is not extreme, as with value function approach, but realistic. One of the main advantages of all of the outranking methods is the prevention to the overly restrictive assumptions involved in the value measurements approaches. All the outranking methods are based on the pairwise comparison of the alternatives; their major drawback concerns the rather many nonintuitive inputs that are required. The impacts of the several inputs are difficult to understand intuitively, and the algorithms tend to be complex to be understood by DMs. Results can be nonintuitive, with unexpected changes in rank ordering as a consequence of changes in the threshold levels. Because of these drawbacks, outranking methods are sometimes considered more convenient in cases of advanced analyses performed by analysts and support staff to the final DMs. In fact, in that case they can provide a valuable understanding of the arguments that can arise for and against the available alternatives. 1.19.4.4.1

PROMETHEE The PROMETHEE function ranking method was developed by Brans and co-workers.53–55 This approach uses subjective preference functions for each separate criterion to rank the different alternatives. Unlike the desirability and utility functions, the preference function trend does not directly model the alternative values for each criterion, but it models the difference values between each pair of alternatives. Despite the ELECTRE method, rather than the specification of indifference and preference thresholds, the intensity of preference for alternative a over alternative b, Pj(a, b) is described by a function of the differences in performance levels on that criterion for the two alternatives, that is faj fbj .


The preference function defines the degree to which, with regard to each jth criterion, alternative a is preferred to alternative b and is defined as53 ( Pj ða;bÞ ¼

0

if

faj fbj

pj ðfa ; fb Þ

if

faj > fbj

ð34Þ

where pj( fa, fb) is a function of the type

pj ðfa ; fb Þ ¼ g faj – fbj

ð35Þ

with g an arbitrary function depending on the difference between the values faj and fbj . Six types of function (Figure 5) were identified to cover most of the cases; for each criterion a maximum of two parameters need to be defined by the DM. It can be noticed that Type 1 and Type 3 generalized criteria can be derived from the Type 5 generalized criterion, setting the parameters q ¼ s ¼ 0 and q ¼ 0 and s 6¼ 0, respectively. For s ¼ q 6¼ 0, Type 2 could also be derived from Type 5 . For the Type 6 function, the only required parameter is the standard deviation , accordingly to the user’s experience with the Normal distribution. Then for each pair of alternative a, b P A, a global preference index for alternative a over alternative b, over all criteria, is defined as ða; bÞ ¼

p X

wj Pj ða; bÞ;

0 ða; bÞ 1

ð36Þ

j ¼1

Pp where p is the total number of criteria and wj the weight for the jth criterion with j ¼1 wj ¼ 1. This index is a measure of the preference of a to b for all criteria: the closer to 1, the greater the preference. This index defines a valued outranking relation, which is then used to determine an ordering of the alternatives. The outranking relation can be used to solve both ranking and choice problems. In the first case, the DM wants to rank all the alternatives from the best to the worst and for this purpose the outranking relation is used to build a total ranking on the n alternatives. In the case of a choice problem, the DM wants to select the best alternative in A, and since there is in general no best solution in multicriteria problem, the outranking relation will identify a set of good alternatives.

P(fa – fb)

P(fa – fb)

P(fa – fb)

1

1

1

0

fa – fb

0 q

Type 1: Usual criterion

fa – fb

Type 2: Quasi-criterion

0

P(fa – fb)

P(fa – fb)

1

1

1

q

s

fa – fb

Type 4: Level-criterion

0

q

s

fa – fb

Type 5: Criterion with linear preference

Figure 5 Examples of PROMETHEE preference functions.

fa – fb

Type 3: Criterion with linear preference

P(fa – fb)

0

s

0

f a – fb

Type 6: Gaussian criterion

608


To solve these problems, two additional indices, called the

positive outranking flow and the– negative

þ – outranking flow need to be introduced. The positive þ ab ;ba and negative outranking flows ab ;ba are defined as follows: þ ab ¼ ab– ¼

X

ða; bÞ;

Pþ

X P–

þ ba ¼

ðb; aÞ; ba– ¼

X

ðb; aÞ

ð37Þ

ða; bÞ

ð38Þ

Pþ

X P–

where Pþ is the set of criteria favorable to a (or b), and P— is the set of criteria not favorable to a (or b). Each of these indices defines a complete preorder of the alternatives, the intersection of which generates a partial order as follows:

a outranks b if þ þ ab > ba

and

ab– < ba–

ð39Þ

þ þ ab > ba

and

ab– ¼ ba–

ð40Þ

þ þ ab ¼ ba

and

ab– < ba–

ð41Þ

þ þ ab ¼ ba

and

ab– ¼ ba–

ð42Þ

þ þ ab > ba

and

ab– > ba–

ð43Þ

þ þ ab < ba

and

ab– < ba–

ð44Þ

or

or

a is indifferent to b if

a and b are incomparable if

or

This approach described is also known as PROMETHEE I partial relation. It offers the DM a solution where some alternatives are comparable, whereas others are not. In the extended PROMETHEE II approach, a complete or total order of the alternatives is derived from the ‘net flow’ for each alternative, as – a ¼ þ a – a

ð45Þ

where the positive and negative outranking flows for the alternative a are calculated as þ a ¼

n X

ða; iÞ;

i 6¼ a

ð46Þ

ði; aÞ;

i 6¼ a

ð47Þ

i¼1

a– ¼

n X i¼1

Then a outranks b if a > b , with indifference if a ¼ b . The net flow outranking index can then be normalized according to 9a ¼

a þ ðn – 1Þ ðn – 1Þ þ ðn – 1Þ

ð48Þ

where n is the total number of alternatives, and (n 1) and (n 1) are respectively the maximum and minimum values of a .


Comparing the PROMETHEE I method with the PROMETHEE II, Brans and Vincke53 commented that the latter, providing a complete order, can be agreeable to the DM; however, it is more questionable because some useful information about incomparabilities is lost. The PROMETHEE method combines the simplicity and transparency of the early ELECTRE methods with an increased sophistication of the preference modeling built in the ELECTRE III. On the other hand, similar to the ELECTRE III, the needed distillation procedure for PROMETHEE I can provide nonintuitive results. The PROMETHEE method is implemented in PROMCALC,56 developed by Brans and Mareschal, and in Decision Lab 2000 software,57 developed by the U.L.B. team under an exclusive agreement with Visual Decision. 1.19.4.4.2

ELECTRE The ELECTRE approach includes a number of methods that differ in the degree of complexity, the detail of information required, and the nature of the underlying problem. It tends to be based on a limited number of criteria, about 6–10. The ELECTRE methods are based on the evaluation of two indices, namely concordance index and discordance index, which are defined for each pair of alternatives a and b. The concordance index C(a, b) provides a measure of the strength of support in the information given in favor of the hypothesis that a is at least as good as b. The discordance index D(a, b) is a measure of the strength of evidence against this hypothesis. 1.19.4.4.2(i) ELECTRE I The first ELECTRE method proposed is the so-called ELECTRE I. The concordance index used in ELECTRE I is defined as Cða; bÞ ¼

X

wj ;

0 Cða; bÞ 1

ð49Þ

j PP þ ða;bÞ

Pp Under the usual constraint j ¼1 wr ¼ 1, where Pþ(a, b) is the set of criteria for which the alternative a is equal or preferred (at least as good as) the alternative b and is defined as

P þ ða; bÞ ¼ 8j j faj > fbj

ð50Þ

Therefore, the concordance index can be interpreted as the sum of criteria weights allocated to the set of criteria for which a is equal to or preferred to b. The index ranges from 0 to 1 so that higher values indicate stronger evidence in support of the fact that a performs as good as b on all criteria (a dominates or is equivalent to b). The discordance index D(a, b) used in ELECTRE I is defined as Dða; bÞ ¼

maxj PP – ða;bÞ fbj – faj ; maxj maxc;d PA fcj – fdj

0 Dða; bÞ 1

ð51Þ

where c and d are any two alternatives, A is the set of all alternatives and P(a, b) is the set of criteria for which the alternative b is strictly preferred to the alternative a and is defined as

P – ða; bÞ ¼ 8j j faj < fbj

ð52Þ

The discordance index for a compared to b represents the maximum value by which b is better than a, expressed as a proportion of the maximum difference between any two alternatives on any criterion. It ranges from 0 to 1 so that higher values indicate that at least on one criterion b is better than a. From the definition of this index, it can be observed that it requires that all evaluations be made on an ordinal scale and the weights render scales comparable across criteria, which are quite restrictive assumptions. To use the concordance and discordance indices for each pair of alternatives to build an outranking relation, the global concordance and discordance thresholds, C and D , need to be specified. From the above threshold definition, it follows that alternative a outranks alternative b if the concordance index C(a, b) is greater than or equal to the threshold C and the discordance index D(a, b) if less than or equal to D . The values of C and D depend on the specified outranking relation and they can be tuned to give more or less severe outranking relations. The higher the value of C and the lower the value of D , the more severe the outranking relation, which means that it is more difficult for one alternative to outrank another. Depending on how the thresholds

610


are defined, the outranking relation can become too severe, with the result that almost all pairs of alternatives will be ‘incomparable’ or not severe enough with too many alternatives outranking too many others. Thus, the key point is to identify the right values for the thresholds, to make C large enough (but not too large) and D small enough (but not too small) to define the useful and informative outranking relation. For clarification, the results of the method applied to the illustrative example of Tables 1 and 2 are presented here. Table 8 provides the row-normalized data derived from Table 1. The concordance indices are then obtained as follows: Cð1;2Þ ¼ 0:18 þ 0:12 þ 0:12 ¼ 0:42;

Cð2;1Þ ¼ 0:35 þ 0:23 ¼ 0:58

Cð1;3Þ ¼ 0:35 þ 0:23 þ 0:18 þ 0:12 ¼ 0:88;

Cð3;1Þ ¼ 0:12

Cð2;3Þ ¼ 0:35 þ 0:23 ¼ 0:58;

Cð3;2Þ ¼ 0:18 þ 0:12 þ 0:12 ¼ 0:42

and represented in the concordance matrix of Table 9. The discordance indices are then computed as follows: 0:366 – 0:203 ¼ 0:685; 0:238 0:366 – 0:293 ¼ 0:307; Dð2;1Þ: c3 ¼ 0:238 0:476 – 0:238 Dð2;1Þ: c5 ¼ ¼1 0:238 0:385 – 0:346 Dð1;3Þ: c4 ¼ ¼ 0:164 0:238 0:431 – 0:366 ¼ 0:273; Dð3;1Þ: c1 ¼ 0:238 0:366 – 0:341 ¼ 0:105; Dð3;1Þ: c3 ¼ 0:238 0:341 – 0:293 Dð2;3Þ: c3 ¼ ¼ 0:202; 0:238 0:476 – 0:286 Dð2;3Þ: c5 ¼ ¼ 0:798 0:238 0:431 – 0:203 Dð3;2Þ: c1 ¼ ¼ 0:958; 0:238 Dð1;2Þ: c1 ¼

0:347 – 0:330 ¼ 0:071 0:238 0:346 – 0:269 Dð2;1Þ: c4 ¼ ¼ 0:324 0:238 Dð1;2Þ: c2 ¼

0:330 – 0:323 ¼ 0:029 0:238 0:286 – 0:238 Dð3;1Þ: c5 ¼ ¼ 0:202 0:238 0:385 – 0:269 Dð2;3Þ: c4 ¼ ¼ 0:487 0:238 Dð3;1Þ: c2 ¼

Dð3;2Þ: c2 ¼

0:347 – 0:323 ¼ 0:101 0:238

and represented in the discordance matrix of Table 10. Table 8 Normalized data for alternatives and criteria Alternatives Criteria

1

2

3

c1 c2 c3 c4 c5

0.366 0.330 0.366 0.346 0.238

0.203 0.347 0.293 0.269 0.476

0.431 0.323 0.341 0.385 0.286

Table 9 Concordance matrix Alternatives

1

2

3

1 2 3

– 0.58 0.12

0.42 – 0.42

0.88 0.58 –

Multicriteria Decision-Making Methods 611 Table 10 Discordance matrix Alternatives

1

2

3

1 2 3

– 1 0.273

0.685 – 0.958

0.164 0.798 –

1

2

3 Figure 6 Graph derived from ELECTRE I applied to the illustrative example, the global concordance and discordance thresholds being equal to 0.60 and 0.4, respectively.

Assuming a global concordance and discordance thresholds equal to 0.60 and 0.4 respectively, the graph of Figure 6 is derived. Thus, the first alternative outranks the third one, whereas the second alternative is incomparable with the others. An alternative approach to the definition of the discordance term is the one based on the use of a veto threshold for each criterion j, say vj , so that a cannot outrank b, if the score for b on any criterion exceeds the score for a on that criterion by a certain amount equal to or greater than its veto threshold: ( Dða;bÞ ¼

fbj – faj > vj for any j

1

if

0

otherwise

ð53Þ

ELECTRE I was developed to assist in the identification of a preferred alternative. This is generally performed by identifying the set of alternatives, referred to as the kernel, that fulfill the following two properties:

Any alternative not in the set is outranked by at least one alternative which is in the set. All alternatives in the set are comparable.

If there is a cycle in the outranking relation, which means that a outranks b and b outranks a, there is more than one kernel set. To solve this problem the ELECTRE method was modified in its later versions. Clearly, the changes in the threshold values used to define the outranking relation strongly affect the final result of the analysis, and the sensitivity and robustness analysis play an important role in the decision process. Unluckily, sensitivity and robustness analysis are ad hoc examination of the effect of changing the threshold values, which is generally not performed in an automated way. ELECTRE II The first modification of the ELECTRE I method was implemented in the socalled ELECTRE II method with the aim to produce a ranking of alternatives rather than to indicate only the most preferred. In ELECTRE II, outranking relation is built mainly in the same way as for ELECTRE I. The main difference is related to the fact that outranking relations are built using different pairs of concordance and discordance thresholds. The relations are called strong and weak outranking relations, the former having a higher concordance threshold and a lower discordance threshold. In addition to this, a further constraint is introduced in the outranking test to reduce the possibility of two alternatives each outranking the other (incomparability): in addition to C(a, b) > C it is also required that C(a, b) C(b, a). The procedure uses the two outranking relations to determine two rankings of the alternatives. The first starts with the ‘best’ alternatives and works downward, giving the descending order, whereas the second starts

1.19.4.4.2(ii)

612


with the ‘worst’ and works up, giving the ascending order. The ELECTRE II method can be summarized concisely in the following main steps, as illustrated by Belton and Stewart:11 1. Specify concordance and discordance thresholds C , D for the strong outranking relation, and C, D for the weak outranking relation, defined such that C > C and D > D. 2. Let A be the set of all alternatives 3. Determine the set of alternatives, M A which are not strongly outranked by any other alternative in A. 4. Within M determine the subset of alternatives, say M9, which is not weakly outranked by any other member of M. This defines the first set of the descending ranking. 5. Delete the alternatives in M9 from A, and repeat the procedure from step 3, continuing until all alternatives have been classified. This generates the descending order. 6. Start again with A being the full set of alternatives. 7. Determine the set of alternatives say G A, which does not strongly outrank any other alternative. 8. Within G determine the subset of alternatives, say G9, which does not weakly outrank any other member of G. This defines the first set of the ascending ranking. 9. Delete the alternatives in G9 from A, and repeat the procedure from step 7, continuing until all alternatives have been classified. This generates the ascending order. The ascending and descending orders should strictly be called complete preorders, or weak orders, because only the sets are strictly ordered, and no preference ordering is already established between alternatives in the same set. Once the two preorders have been defined, they need to be compared. If significant differences are identified, this means unusual alternatives. The comparison of the two orders is then performed to give a single ranking. Different approaches have been proposed for this. The most commonly used one consists in determining the partial order (allowing for incomparabilities) defined by the ‘junction’ of the ascending and descending preorders. The junction of two outranking relations is defined such that aFb, which means that a outranks b according to relation F if and only if a outranks or is in the same set as b according to the preorders corresponding to both outranking relationships. To clarify the process better, an example is provided in Table 11. The junction of the two preorders can then be refined to give a total ranking (removing incomparabilities). ELECTRE III Both ELECTRE I and ELECTRE II are based on the assumption that any difference in performance as measured by the faj implies a corresponding difference in preference. Thus, indifference corresponds only to the case of two alternatives performing identically on a given criterion. ELECTRE III introduces the concepts of indifference and preference thresholds. The preference and indifference thresholds, qj and uj, respectively, are used to construct a concordance index Cj(a, b) for each criterion defined by

1.19.4.4.2(iii)

( Cj ða; bÞ ¼

1 if fbj – faj uj 0 if fbj – faj qj

ð54Þ

or by linear interpolation between 0 and 1 when faj þ uj < fbj < faj þ qj . In this way, a is defined to be at least as good as b according to the jth criterion only if there is no even weak preference for b to a. If b is strictly preferred to a on criterion j, then a definitely does not outrank b according to criterion j. When b is weakly preferred to a, then there is an ambiguous situation and the value of concordance is set between 0 and 1. Table 11 Example of ‘junction’ of the ascending and descending preorders Descending order

Ascending order

Partial order resulted from the junction

a f d B, e c g

a f d, e C g, b

a f d e b, c g


Then, the overall concordance is described as Cða; bÞ ¼

p X

wj Cj ða; bÞ;

0 Cða; bÞ 1

ð55Þ

j ¼1

Also, the discordance is defined using a veto threshold for each criterion j, vj, so that a cannot outrank b, if the score for b on any criterion exceeds the score for a on that criterion by a certain amount equal to or greater than its veto threshold, fbj faj vj for any j: ( Dj ða; bÞ ¼

0

if fbj – faj qj

1 if fbj – faj vj

ð56Þ

with linear interpolation between 0 and 1 when faj þ qj < fbj < faj þ vj . Once the concordance and discordance indices have been defined, they are combined in the outranking relation. The alternative a outranks alternative b with credibility B(a, b) defined as follows: 8 Cða;bÞ >
: Cða;bÞ j PJ ða;bÞ ð1 – Cða;bÞÞ

if Dj ða;bÞ Cða;bÞ 8j otherwise

ð57Þ

where J(a, b) is the set of criteria for which Dj(a, b) > C(a, b). If there is no discordant criterion, which means that the discordance index is lower than the overall concordance index, then the credibility is equal to the overall concordance index. On the other hand, when there are discordant criteria, the overall concordance index is modified to reach a credibility index that is lower in value. The interpretation of the credibility index is not straightforward. Roy and Bouyssou51 advised that B(a, b) > B(c, d) should not necessarily mean that there is a stronger support that a outranks b than that c outranks d, because this conclusion can be derived only if B(a, b) is significantly larger than that B(c, d). They also suggested that if that B(a, b) ¼ , then the conclusion that a outranks b can be considered more reasonable than that c outranks d if B(c, d) – s, where s ¼ 0.3 – 0.15 . This rule was proposed by Roy and Bouyssou to define the so-called -preference, included in the procedure for the outranking relation in the following way: a is -preferred to b if ð1 – sÞBða; bÞ > Bða; bÞ

and

Bða; bÞ >

ð58Þ

where s ¼ 0.3 0.15 . Similar to ELECTRE II, ELECTRE III provides a descending and ascending order of the alternatives by the so-called descending and ascending distillation procedure; the two rankings are then compared and combined. The descending distillation procedure, as described by Belton and Stewart,11 consist of the following steps: 1. Determine the maximum value of the credibility index, max ¼ max B(a, b), where the maximization is taken over the current set of alternatives under evaluation. 2. Set ¼ max (0.3 0.15 ). 3. For each alternative determine its -strength, that is, the number of alternatives in the current set to which it is -preferred to use ¼ . 4. For each alternative determine its -weakness, that is, the number of alternatives in the current set which are -preferred to it using ¼ . 5. For each alternative determine its qualification, which is its -strength minus its -weakness. 6. The set of alternatives having the largest qualification is called the first distillate D1. 7. If D1 has more than one member, repeat the process on the set D1 until all alternatives have been classified; then continue with the original set minus D1 repeating until all alternatives have been classified. The ascending distillation procedure is obtained in the same way as the descending distillation except that at step 6 above, the set of alternatives having the lowest qualification forms the first distillate. Some disadvantages of the ELECTRE III method have been later identified by Vincke7 and Roy and Bouyssou.51 First, the indifference, preference and veto thresholds do not have a clearly defined physical or

614


psychological interpretation; therefore DMs are poorly guided in the interpretations of their preferences in terms of preference and veto thresholds. Second, the aggregation procedure is a little bit unclear and it is based on a rather complex algorithm difficult to be understood by DMs. A connected drawback is that the distillation procedure can provide some very strange and nonintuitive results and, finally, the method needs to be implemented in effective software to be used in practice. Some variations of ELECTRE III have therefore been proposed; among them, the most known are the ELECTRE IV and ELECTRE TRI. ELECTRE IV was developed by Roy and Hugonnard58 for the cases in which it is not possible to specify criteria weights. The approach follows the same initial preference modeling as for ELECTRE III, defining indifference, preference, and veto thresholds for each criterion. Outranking relations are then defined by direct reference to the performance levels of alternatives. A strong and weak outranking relation could be defined as follows:

1.19.4.4.2(iv)

Electre IV

a strongly outranks b if there is no criterion on which b is strictly preferred to a and the number of criteria on which b is weakly preferred to a is less than the number on which a is strictly preferred to b. a weakly outranks b if: there is no criterion on which b is preferred (strictly or weakly) to a, or b is preferred to a (strictly or weakly) on only 1 criterion, a is preferred (strictly or weakly) to b on at least half of the criteria, and b does not veto a on any criteria.

The development of the outranking relation can then proceed in descending and ascending distillation processes as for ELECTRE III. More details on the method can be found in the original paper by Roy and Hugonnard58 and in Roy and Bouyssou.51 1.19.4.4.2(v) ELECTRE TRI ELECTRE TRI was developed to handle classification problems. The original procedure was designed to assign alternatives to one of three categories (acceptable, unacceptable, or indeterminate) and further developments of the method extended it to problems with more than three different categories. Different approaches have been proposed; the one described in Roy and Bouyssou51 is based on categories ordered and defined by a set of reference actions, or limiting profiles, c0, c1, . . ., cx, where cx represents a hypothetical alternative with performance levels fj(cx) for j ¼ 1, . . ., p. The category Cx is bounded below and above by the limiting profile cx1 and cx, and defines a higher level of performance than category Cx1. Then, an alternative a for classification is compared to each reference profile in succession to determine:

The highest limiting profile such that a outranks cx (i.e., B(a, cx) , where the threshold is a specified value such that 0 1), which determines the pessimistic allocation of a to Cxþ1. The lowest limiting profile such that cy is preferred to a (cy outranks a and a does not outrank cy), which determines the optimistic allocation of a to category Cy (y x þ 1). If y ¼ x þ 1, a is unambiguously allocated to that category. Where y > x then both: a is indifferent to all intermediate limiting profile. If y ¼ x þ 2 then a outranks cx, is indifferent to cxþ1, and is outranked by cxþ2. The case of an alternative indifferent to more than one limiting profile means that the categories are too narrowly defined. a is incomparable to all intermediate profiles.

The method has been further developed, and an extended review and examples of the modified methods is provided by Perny.59

1.19.4.4.3

ORESTE The Organisation, Rangement Et Synthe`se de donneés relaTionElles (ORESTE) method is an operations research method that can be applied to optimize simultaneously different performance goals. It was presented as a general method that can be applied also in situations where no quantitative data are available and avoids the necessity of determining criteria weights. The ORESTE method was originally proposed by Rubens60 for optimizing quality in operations research and was then applied in that field.60–62 The procedure was explained in a tutorial article60 and applied in a case study of a car,61 computer selection systems,63 and in experimental


chemistry.64 The method was implemented in a decision support package developed at the Royal Military Academy (Brussels) for personal computers.65,66 ORESTE operates in the opposite way with respect to the ELECTRE method: in a first phase a complete preorder of the alternatives is constructed and in a second phase some parts of the complete preorder are invalidated by conflict introducing indifference and incomparability thresholds. Two alternatives a and b are considered indifferent if alternative a is slightly better than alternative b for some criteria whereas the opposite is true for other criteria. In this case, one alternative cannot be considered to be globally better than the other by means of a preference relation and both alternatives are considered as equally good. Conflicting situations arise when alternative a is much better than alternative b for one or some criteria but much worse for others. This is the situation, called incomparability, of two alternatives too different to be compared with each other. The first phase of the ORESTE method can be subdivided into three steps: 1. Ranking of alternatives and criteria. A weak order of the criteria is first performed, being a weak order or complete preorder a preference structure in which the relative importance of the criteria is defined, taking into account indifference (I) and preference (P) relations. In the same way, for each criterion a weak order of the alternatives is defined. Then a global weak order is constructed assigning a rank to each alternative of a weak order by using the average rank. Average ranks are applied to give each alternative a rank in the weak order of the alternatives for each criterion and to give each criterion a rank in the weak order of all criteria. To better understand the approach, a simple example is provided. Let us consider four alternatives (a, b, c, d) and four criteria (1, 2, 3, 4), criterion 1 being indifferent to 2, which is more important than criterion 3, which is more important than criterion 4. For each criterion, the weak orders for the alternatives are the following: 1: 2: 3: 4:

a preferred to b, b indifferent to c, c preferred to d b preferred to a, a preferred to d, d preferred to c c indifferent to d, d preferred to a, a preferred to b b preferred to a, a preferred to d, d indifferent to c

The average ranks arbitrarily assigned to the criteria in this example are r 91 ¼ 1:5, r 29 ¼ 1:5, r 39 ¼ 3, and r 94 ¼ 4, where r 9j is the average rank of the jth criterion in the weak order of the criteria. The average ranks for the alternatives obtained for the different criteria are illustrated in Table 12. The matrix of the relative positions, named position matrix, is then built as shown in Figure 7.

Table 12 Average ranks for the alternatives in the Oreste example Criteria Alternatives

1

2

3

4

a b c d

1 2.5 2.5 4

2 1 4 3

3 4 1.5 1.5

2 1 3.5 3.5

1

Criteria importance 2 3 4

a Performance of the alternatives

b a

b, c

b c, d

d

a

c

b

a d, c

d Figure 7 Position matrix of the ORESTE example.

616


2. Aggregation of the position table of the alternatives. An origin is introduced from which distances are calculated to each alternative in the position table. These distances are used to compare alternatives taking into account the relative importance of the criteria at the same time. As an example, it allows the comparison of the best alternative for the second important criterion with the second alternative for the most important criterion. The smaller the distance to the origin, the better the alternative. Being the best alternative near to the origin of the axes, the origin can be assumed in (0, 0) in place of the best alternative coordinates. Using the Manhattan distance, the following distance between the origin and the ith alternative for the jth criterion is calculated as Dj ðo; iÞ ¼ r j9 þ rij

ð59Þ

It can be noticed that the ranks of the criteria in the weak order is as equally important as the ranks of the alternatives for a certain criterion; subsequently, the loss of one rank position of an alternative is completely rewarded by the increase of one rank position of a criterion. In case it is required to treat the ranks of criteria differently from one of the alternatives, a weighted sum of both ranks was proposed: Dj ðo; iÞ ¼ ð1 – Þr j9 þ rij ;

1 i n;

01

ð60Þ

where r9 is the rank of the criterion, rij the rank of the ith alternative based on the jth criterion, and is a weighting factor. If is set equal to 0.5, then again the ranks of criteria and of alternatives for criteria are equally important and it corresponds to a linear orthogonal projection of the position table on its principal diagonal so that better positions are located to the left and the worse ones to the right. Some modifications of this equation have been proposed by Van Velthoven.61 The distances resulting from the linear orthogonal projection of the above example are illustrated in Table 13. For example, when ¼ 0.5 the score of the alternative a for criterion 2 is obtained as 1.5/2 þ 2/2 ¼ 1.75. 3. Ranking of the projections of the whole table entries by means of average ranks. The distances of Table 13 are ranked as shown in Table 14. A global (weak) order is then derived by summing the global ranks for each alternative: ri ¼

p X

ð61Þ

rij

j ¼1

where p is the number of criteria, and rij is the global rank of the ith alternative for the jth criterion. Table 13 Distances derived from the linear orthogonal projection Criteria Alternatives

1

2

3

4

a b c d

1.25 2 2 2.75

1.75 1.25 2.75 2.25

3 3.5 2.25 2.25

3 2.5 3.75 3.75

Table 14 Average ranks of the linear orthogonal projections Criteria Alternatives

1

2

3

4

a b c d

1.5 4.5 4.5 10.5

3 1.5 10.5 7

12.5 14 7 7

12.5 9 15.5 15.5


In the example above, the sums of the global ranks of each alternative are ra ¼ 29:5;

rb ¼ 29;

rc ¼ 37:5;

rd ¼ 40

which provides the following total order: b preferred to a; a preferred to c; c preferred to d

. The order obtained in this way does not consider either indifference or incomparability. In the second phase of the ORESTE method, both indifference and incomparability are identified and used to construct an incomplete preference structure. For this purpose, preference intensities are computed as C9ða; bÞ ¼

X

rbj – raj

ð62Þ

j PP þ ða;bÞ

where C9(a, b) is preference intensity of alternative a to b and it measures the degree of which an alternative is preferred to another. The preference intensities are then normalized, dividing them for an upper bound C9upper bound ¼ ðn – 1Þp2

ð63Þ

where n is the number of alternatives and p is the number of criteria. After the normalization, the intensities range from 0 to 1. The normalized preference intensities of the example above are shown in Table 15. The preference intensities are then used to identify conflictual situations. When C(a, b) is almost equal to C(b, a), indifference arises when both intensities are small, whereas a conflictual situation, that is incomparability, arises when both the intensities are large. This happens when a is better than b on a subset of criteria and b is better than a on another subset of criteria. To differentiate among indifference, incomparability, and preference, three thresholds, , C , and need to be computed. Threshold allows the distinction between indifference and preference, C performs the discrimination between indifference and incomparability, whereas discriminates between preference and incomparability and therefore can be interpreted as the risk that the DM selects one alternative irrespective of the conflicting situations. To interpret the thresholds better, a normalized preference plot was introduced. This plot is a triangular area with coordinates C(a, b) and C(b, a) in which four subzones are identified, namely indifference, incomparability, preference of a to b, and preference of b to a. A detailed mathematical interpretation of the thresholds is provided by Roubens,60,63 who showed that and C are upper bounded as follows: 1 ðn – 1Þp

ð64Þ

C 1 2ðn – 1Þ

ð65Þ

ðp – 2"Þ 4"

ð66Þ

whereas is lower bounded:

Table 15 Normalized preference intensities Alternatives

a

b

c

d

a b c d

0.00 0.10 0.11 0.11

0.09 0.00 0.15 0.15

0.27 0.32 0.00 0.07

0.33 0.38 0.13 0.00

618


where " is the degree of perturbation. The DM is asked to define the condition in which he or she accepts incomparability or preference of two alternatives. In case of unequal number of criteria, the number of criteria is increased by one for parity reasons. The tests are performed according to the steps illustrated in Figure 8. First, the difference in normalized preference intensities between two alternatives a and b with respect to is investigated. If the difference is smaller than or equal to then either indifference or incomparability can occur, but definitely one alternative is not preferred to the other. To solve whether it is indifference or incomparability, the normalized preference intensities of the two alternatives a and b are compared with C . If both are smaller than or equal to C then there is indifference between a and b, otherwise incomparability. If the difference in normalized preference intensities between two alternatives a and b is greater than , then either preference or incomparability may occur. If the relation of the smallest normalized preference intensity to the difference in normalized preference intensities is larger than or equal to then incomparability is assessed, else there is preference. The final outranking relation can then be represented by a matrix or a dominance graph. The dominance graph in Figure 9 is constructed so that an arrow from alternative a to b means that alternative b is dominated by alternative a, whereas incomparable alternatives are not linked together. Therefore in the following dominance graph, alternative a is superior to b and alternative c is superior to alternative b, whereas alternatives a and c are incomparable. According to Pastijn,65 the main advantage of the ORESTE method is that it is appropriate to handle complicated MCDM problems where criteria cannot be expressed or weighted numerically. The method is strongly discriminatory about conflictual alternatives and clearly identifies incomparabilities. The DM needs to define some technical parameters and although these parameters are easy to understand, he or she has to be guided in this process.

C(a,b) – C(b,a) ≤ β Y C(a,b) ≤ C*

and C(b,a) ≤ C*

N

Y

C(a,b) > C(b,a)

aIb

Y

C(b,a) ≥γ C(a,b) – C(b,a)

Y

N

N N

a b

C(a,b) ≥γ C(b,a) – C(a,b)

Y

aPb

N

bPa

Figure 8 Decision-making process about incomparability, indifference, and preference. Y ¼ Yes; N ¼ No; || ¼ incomparability.

a

c b

Figure 9 Example of dominance graph.


1.19.4.5

Hasse Diagram Technique

The Hasse diagram technique (HDT) belongs to the POR methods, which are vectorial approaches that recognize that different criteria are not always in agreement, but can be conflicting, which means that not all the alternatives can be directly compared with others. An example is often used to better clarify the meaning of ‘conflicts’. Consider a system made up of five, not perfectly correlated, alternatives (a, b, c, d, e), each described by two criteria f1 and f2; the aim is to discover which alternative performs better than the other with respect to all the criteria. The alternatives are sorted, arranging them according to f1 and f2 in the permutation diagram67 or by parallel coordinates68 with a vertical orientation, as shown in Figure 10. This representation highlights the inversions between the two criteria. Alternatives mutually exchange their position according to the criterion used to sort them. The higher the number of criteria, the higher the probability that contradictions in the ranking exist. The partial ranking approach not only ranks alternatives but also identifies contradictions in the criteria used for ranking: some residual order remains when many criteria are considered and this motivates the term partial order. Thus the more known concept of order is the one demanding that all alternatives be comparable, that is, linear or total order, whereas partial order is the one in which alternatives can be ‘not comparable’. If many alternatives are to be investigated, and particularly if many criteria are considered, the parallel coordinates become complex and confusing. The Hasse diagram is a means of illustrating POR proposed by Hasse in 1952.69 It was introduced in environmental sciences by Halfon70 and refined by Bru¨ggemann.71 It is based on a specific order relation, named product order, and it provides a diagram, which visualizes the results of the sorting. In this approach, the basis for ranking is the information collected in the full set of p criteria, which is called the information basis of the comparative evaluation of alternatives. According to the POR, which the HDT is based upon, two alternatives a and b are comparable if for all fj P F either faj fbj or fbj faj. If faj fbj for all fj P F then a b. The request ‘for all’ is very important and is called the generality principle: a; bPA;

a b , fa f b

fa fb , faj fb

for all fj PF

ð67Þ ð68Þ

If there are some fj, for which faj < fbj and some others for which faj > fbj then a and b are not comparable, and the common notation is a k b. If only one criterion is used or all the criteria are perfectly rank correlated, then a complete or total order is obtained, and all the alternatives are comparable. Partial order is determined by the actual information base; thus by changing the information base different partial orders arise. Partial order sets can be developed easily with the HDT, comparing each pair of alternatives and storing this information in the Hasse matrix which is a (n n) antisymmetric matrix. For each pair of elements a and b, the entry hab of this matrix is 8 þ1 if faj fbj for all fj PF > > < hab – 1 if faj < fbj for all fj PF > > : 0 otherwise

ð69Þ

and hba is equal to –hab or to zero if they are incomparable. Thus according to the so-called cover relation, if there is no alternative a of A, for which a b c, b 6¼ a, c and a 6¼ c, then a is covered by c and c covers a.

f1

f2

a c e b d

a e b c d

Figure 10 Alternatives arranged in two sequences according to two different criteria.

620


The results of the POR is visualized in a diagram where each alternative is represented by a small circle, and within each circle the alternative name, or the equivalence class, is given. Equivalent alternatives are different alternatives that have the same numerical values with respect to a given set of criteria. The equality according to a set of criteria defines an equivalence relation. The diagram is then a kind of dominance diagram, where if an order or cover relation exists then a line between the corresponding pairs of alternatives is drawn; the alternatives belonging to an order relation are ‘comparable’. The diagram has orientation; consequently a sequence of lines can only be read in one direction either upward or downward. In case a b and b c then a c according to the transitivity rule; however, a line between a and c is not drawn because this connection can be deduced from the lines between a and b and b and c. Incomparable alternatives are not connected by a line and are located at the same geometrical height and as high as possible in the diagram, resulting in a structure of levels. Alternatives belonging to the same level are incomparable and represent a Pareto frontier at that level; alternatives belonging to the highest level constitute the classical Pareto frontier. However, a location of alternatives at different levels does not imply comparability. According to the Hasse diagram terminology, the alternatives at the top of the diagram are called maximals whereas those that have no alternatives below are called minimals and they do not cover any further alternative. In the environmental field, where the Hasse technique was first applied, the criteria describe the alternatives in terms of environmental hazard. The main assumption is that the lower the numerical value the lower the hazard. If a high numerical value of a criterion corresponds to low hazard, the criterion values must be multiplied by –1 to invert their order. Therefore, by this convention, the maximal alternatives are the most hazardous, and are selected to form the set of priority alternatives. Alternatives that are not comparable with any other alternative are called isolated alternatives, and can be seen as maximals and minimals at once: according to the caution principle, they are located at the top of the diagram within those elements that require priority attention. A chain is a set of comparable alternatives; therefore, levels can be defined as the longest chain within the diagram. An antichain is a set of mutually incomparable alternatives. The height (longest chain) and width (longest antichain) of an order set are indicators of the relative number of comparable pairs of alternatives compared to the total number of pairs. A typical Hasse diagram is the one illustrated in Figure 11. The key diagram interpretation is provided by the meaning of chain and antichain. A chain indicates that the values of the criteria increase synchronously, whereas antichains correspond to diverse patterns. The Hasse diagram can be characterized by some key parameters,72–74 which count the number of equivalent classes with more than one alternative, the width of a Hasse diagram, that is the maximum number of alternatives that can be found in an antichain, the number of levels, that is the number of alternatives in the longest chain, the capability of discriminating alternatives, the diagram stability, complexity, selectivity, etc. For the example illustrated in Tables 1 and 2, the three alternatives are incomparable, that is an antichain is obtained. In recent years, the HDT has been widely applied for different purposes in several fields: evaluation of aquatic toxicological tests;75,76 analysis of waste disposal sites;77 ranking chemicals for environmental hazard;70,78 comparison among ecosystems;79–83 chemicals prioritization;84 evaluation of online databases;72–86 ranking of contaminated sites,87,88 and evaluation of materials in car production.80 A new, recently presented, application of HDT in chemistry is that of reaction diagrams of progressive substitution on a fixed molecular skeleton, forming Hasse diagrams for a partially ordered set of substituted structures.89 An interesting example of the potentiality of the HDT is the one provided by its application to effect– concentration (EC) curves, which describe the mode of action of a toxicant. Typically, the entire curve is drawn Hazard a

b

e

a, b, e: Maximals d: Minimals

c

a, b, e: Incomparable alternatives a, c, d: Chain and d ≤ c ≤ a

d Figure 11 Typical Hasse diagram.

b, c, d: Chain and d ≤ c ≤ b


from the EC01, EC10, EC50, and EC90 values, which are the concentrations at which 1, 10, 50, and 90% of the test population are affected, respectively. If the common assumption that similarly acting substances show similar EC curves, whereas dissimilarly acting substances show dissimilar EC curves, characterized by many crossing, is accepted then the EC curves have to be compared to establish whether the mixture components act with the same mechanism of action. HDT being a multivariate explorative method is not limited to one level of biological response and thus it seems suitable for comparison of concentration–effect curves, highlighting different EC curve shapes. This method has been performed to compare EC curves of both similarly and dissimilarly acting substances, the aim being to evaluate the Hasse diagram capability of comparing toxicity profiles and provide diverse results for EC curves of substances with the same mode of action from the ones of substances with different mode of action. The HDT has been applied on two datasets: 12 phenylureas as similar acting chemicals (Table 16) and 21 diverse chemicals as dissimilar acting chemicals (Table 17).

Table 16 Toxicity values of 12 phenylureas log(1/EC) ID

Substance

EC01

EC10

EC50

EC90

1 2 3 4 5 6 7 8 9 10 11 12

Buturon Chlobromuron Chlortoluron Diuron Fenuron Fluometuron Isoproturon Linuron Metobromuron Metoxuron Monolinuron Monuron

1.877 3.244 2.523 3.071 0.927 2.030 2.226 3.155 1.430 2.320 2.058 2.569

0.897 2.059 1.576 2.223 0.137 0.772 1.363 1.990 0.630 1.209 0.920 1.367

0.111 1.222 0.815 1.538 –0.633 –0.173 0.642 1.056 –0.019 0.319 0.007 0.402

–0.390 0.824 0.332 1.109 –1.214 –0.849 0.166 0.463 –0.490 –0.249 –0.575 –0.212

Table 17 Toxicity values of 21 dissimilar-acting substances log(1/EC) ID

Substance

EC01

EC10

EC50

EC90

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

Aclonifen 8-Azaguanine Azaserin Carbonyl cyanide m-chlorophenylhydra zone (CCCP) Chloramphenicol Fenfuram 5-Fluoruracil Fusidic acid Kresoxim methyl Metalaxyl Metazachlor Metsulfuron methyl Nalidixic acid Norflurazon Paraquat n-Propyl gallate Pyrimethamin Seanine Steptomycin sulfate Terbuthylazine Triadimenol

2.455 2.347 2.592 0.979 –0.483 1.106 1.983 0.588 0.793 –0.784 2.268 1.366 –0.700 2.249 0.668 –1.441 –0.218 1.155 –0.213 2.243 0.496

2.022 1.815 0.927 0.488 –0.721 0.139 1.582 0.247 0.296 –1.633 1.182 0.273 –1.419 1.959 0.417 –1.845 –1.157 0.924 –0.676 1.642 –0.109

1.523 1.150 –0.250 –0.092 –1.100 –0.635 1.013 –0.256 –0.310 –2.312 0.775 –0.604 –1.994 1.699 0.107 –2.214 –1.711 0.635 –1.099 1.161 –0.539

1.067 0.134 –0.873 –0.653 –1.955 –1.129 –0.059 –0.906 –0.925 –2.745 0.587 –0.162 –2.362 1.428 –0.292 –2.583 –2.008 0.187 –1.523 0.852 –0.76

622


The Hasse diagram derived for the 12 phenylureas as similar-acting chemicals, shown in Figure 12, is a quite simple diagram arranged in seven levels. It identifies two maximals: chlobromuron and diuron, selected as the most toxic. They are incomparable because some contradictions exist among their EC values: chlobromuron is more toxic than diuron on 01 and 50 concentration levels, but it is less toxic on 10 and 90 concentration levels. Fenuron is the less toxic substance, as it is characterized by the lowest effect concentration values. As the 12 phenylureas are similar-acting substances, not too many contradictions in their EC values exist. Thus, as expected, the Hasse diagram is characterized by a few number of incomparabilities: 18 over 132 comparisons. In Figure 13, the phenylurea EC curves are illustrated: the EC curves are quite similar and only a few crossing, corresponding to the above incomparabilities, can be identified. Therefore, the analysis performed not only allows to rank the substances according to their toxicity values but also provide a clear and simple way to detect EC curves crossing.

Toxicity

2

4

8 3

12

7

10

11

1

6

9 5

Figure 12 Hasse diagram of 12 phenylureas.

100

4 2 83 7

10 1 9 5 11 6 12

Inhibition of algal repoduction (%)

90 80 70 60

9, 11

50 40

2, 4

30

6, 9 1, 11

20

7, 12

10 0 0.001

0.01

0.1 1 Concentration [μmol l–1]

Figure 13 Effect–concentration curves of 12 phenylureas.

10

100


The Hasse diagram derived for the 21 diverse chemicals as dissimilar-acting chemicals shown in Figure 14 is a quite complex diagram arranged in nine levels. It identifies three maximals, aclonifen, azaserin, and norflurazon, and two minimals metalaxyl and n-propyl-gallate. The 21 substances being characterized by a diverse mechanism of action several contradictions exist among their EC values. The diagram identifies 76 incomparabilities over 420 comparisons. In Figure 15, the EC curves of 21 substances are illustrated: the EC curves are pretty diverse and many crossing, corresponding to the Hasse diagram incomparabilities, can be identified.

Toxicity 3 2

1

14

11

20

77

18

4

12

15

8

6

9

21

19 5

17 13

10

16

Figure 14 Hasse diagram of 21 dissimilar-acting substances.

100

14

90

19 17 1316

1 2011 18 7

10

4

Inhibition of algal reproduction (%)

80

7, 11

70

2 5, 17

5

60 50

15

40

2, 11

30

10

10, 16

3

20

1, 14

0 0.001

0.01

0.1 1 10 Concentration [μmol l–1]

Figure 15 Effect–concentration curves of 21 dissimilar-acting substances.

100

1000

624


The results obtained on the two datasets investigated highlight that the HDT is an efficient and simple tool not only to rank the substances according to their toxicity values but also to detect the similarity of the mechanism of action of the toxicants. In fact, the higher the number of incomparabilities in the diagram, the higher the number of crossing in the EC curves, and thus the higher the dissimilarity in the mechanism of action of the substances under study. Thus, the complexity and the number of incomparabilities provided by partial ranking analysis on the EC values of substances with an unknown mode of action can be used as a measure of the dissimilarity degree in their mechanism. As pointed out by Bru¨ggemann et al.,90 the HDT has some relevant advantages: its evaluation can be represented as a graph; the mathematics is very simple; it can easily manage criteria of different scales (linguistic, ordinal, and ratio-scaled criteria) because it does not perform any numerical aggregation of the criteria. Nevertheless, there are some severe drawbacks that have been already identified: it is strictly dependent on the clarity of the graphical diagram; diagrams that are too complex or too poorly structured, with more isolated alternatives than comparable alternatives because of conflict, are of little use; if there are too many contradictions, criterion reduction has to be performed by preliminary multivariate statistic techniques, such as PCA or multidimensional scaling; if many alternatives are to be evaluated, preliminary multivariate statistic techniques, such as cluster analysis, are needed to get a readable diagram; the generality principle is very restrictive and requires appropriate data handling. In fact, it must be ensured that any two alternatives ordered by > can be considered as physically and numerically significantly different, that is, they should have numerically significant data differences. Differences within statistical noise, numerical uncertainty, and experimental error are considered physically meaningless, but the HDT considers such alternatives as different. An important approach proposed to make the HDT able to handle complex data is the one based on the use of the concept of linear extensions,91–93 which provide a linear or total order. A linear extension is a projection of the partial order into a total order (a linear rank) that comply with all the relations in the partial order. It is also referred to as order-preserving mapping of the partial order. In a linear extension, the incomparable alternatives are thus given a mutual order. For instance, if a linear extension is made from the Hasse diagram in Figure 11, a must be larger than d (a > d), in any linear extensions because a is larger than d in the Hasse diagram. On the other hand, a can be both larger than or smaller than b, (a > b) or (a < b), depending on the mathematical manipulation applied to obtain the linear order, because these alternatives are incomparable in the Hasse diagram. Unless the partial order is very simple, it is always possible to find several linear extensions. The total set of linear extensions for a partial order can be found by a combinatorial algorithm, where the incomparable alternatives systematically are given an order with respect to each other. Thus, a linear extension can be considered the result of a specific mathematical manipulation for establishment of a total order. If a utility function is applied, the result will always correspond to one of the linear extensions. The total set of linear extensions can thus be considered as the result of a series of different mathematical functions each having a specific outcome, namely one of the linear extensions. If all linear extensions are identified for a partial order, it is possible to count the frequencies, nir , of the occurrence of a specific alternative, i, on a specific rank, r. By dividing the frequencies, nir , by the total number of linear extensions, the ranking probabilities, pir , are obtained. From the Hasse diagram, it is possible to predict the maximum and minimum ranking positions that an alternative can obtain in a linear extension. The highest possible ranking position an alternative can obtain can be calculated as the total number of alternatives minus the number of alternative above that alternative in the Hasse diagram. In the Hasse diagram of Figure 11, for the alternative d which has five alternatives on top of it the highest possible ranking position is 2. In addition, from the linear extensions, the mutual ranking probabilities, pmr, for two incomparable alternatives can be computed in terms of the probability for one alternative to be above or below another alternative. Finally, the average rank of a single alternative in the Hasse diagram can be computed by applying the simple empirical relation proposed by Bru¨ggemann et al. 94 and Carlsen95 as follows: Average Rank ðiÞ ¼ ðn þ 1Þ –

ðSi þ 1Þðn þ 1Þ ðn þ 1 – Ui Þ

ð70Þ

where n is the total number of alternatives, Si is the number of alternatives ranked below the ith alternative (successor of i), and Ui is the number of alternatives incomparable with the ith alternative.


The number of possible linear extensions is closely related to the characteristics of the partial order; thus, the number of linear extensions increases dramatically as the number of incomparable pairs increases. The dramatic increase in the number of linear extensions makes the approach not able to handle data sets of more than 15–20 alternatives. To overcome this inconvenience, a method was developed by Lerche and Sørensen,92,96,97 which estimates the ranking probabilities and the average rank for large partial orders. The average rank is then predicted based on a minor fraction of randomly chosen linear extensions. This extends the manageable number of alternatives from around 20 to 1000. The random linear extensions are created using a Monte Carlo simulation where all incomparable alternatives are given a random order. However, before the method can be applied it is necessary to estimate how many linear extensions are needed to perform an appropriate estimation. Finding the mean square root distance of the estimated ranking probability in one estimation compared with the probabilities found in another estimation when the number of runs is increased with 100, it can be shown that after approximately 1000 runs the random uncertainty stabilizes at around 1.5%. This was considered to be acceptable. The HDT is implemented in WHASSE computer program73,90 and in DART software.98

1.19.4.6

Goal Programming

GP was probably the oldest approach in the field of multicriteria decision making. This approach was described by Charnes and Cooper;99 the reviews of Romero100 and Schniederjans101 provide references dating back to the 1950s. The approach was also the topic of several textbooks, such as the ones by Lee102 and Ignizio;103–105 comprehensive reviews are the ones provided by Lee and Olson,106 Stewart,2 and Belton and Stewart.11 The fundamental principles may be linked to Simon’s107 concept of satisfying, in the sense that particular attention is placed on achieving satisfactory levels of achievement on each criterion, with attention shifting to other criteria once this is achieved. Originally, GP was developed in the context of linear programming problems, but then the principles were applied to nonlinear and nonconvex (including discrete choice) problems. Recently, other variations have been proposed, including several interactive methods and the reference point method as proposed by Wierzbicki,108 which was then termed generalized GP by Korhonen and Laakso.109 In all the methods developed within the GP approach, each criterion needs to be linked with an attribute defined on a measurable scale; these methods are not designed for those situations where one or more criteria are defined on categorical scales. In addition, the DM is asked to express value judgments in terms of goals, or aspiration levels for each criterion, defined in terms of a specific and realistic value (target level) associated with a desired or acceptable level of achievement of the criterion. Generally, the application of these methods is implicitly or explicitly iterative. In fact, according to this approach the natural DM behavior is to concentrate initially on improving what he or she considers the most important or critical criterion until it reaches some satisfactory level of performance. Thereafter, the DM shifts his or her attention to the next most important criterion. In this sense, GP can be considered a learning tool. Clearly, goals can range from highly unrealistic and optimistic to a fixed bottom. Even these methods should be robust to these uncertainties; it appears that they perform best when the set of goal levels are moderately optimistic.

References 1. Hwang, C. L.; Paidy, S. R.; Yoon, K.; Masud, A. S. M. Mathematical Programming with Multiple Objectives: A Tutorial. Comput. Oper. Res. 1980, 7, 5–31. 2. Stewart, T. J. A Critical Survey on the Status of Multiple Criteria Decision Making Theory and Practice. Omega-Int. J. Manage. Sci. 1992, 20, 569–586. 3. Bouyssou, D. Building Criteria: A Prerequisite for MCDA. In Readings in Multiple Criteria Decision Aid; Bana e Costa, C. A., Ed.; Springer-Verlag: Berlin, 1990; pp 58–80. 4. Goicoechea, A.; Hansen, D. R.; Duckstein, L. Multiobjective Decision Analysis with Engineering and Business Applications; Wiley: New York, 1982. 5. Hobbs, B. F.; Horn, G. T. F. Building Public Confidence in Energy Planning: A Multimethod MCDM Approach to Demand-Side Planning at BC Gas. Energy Policy 1997, 25(3), 357–375.

626


6. Hobbs, B. F.; Meier, P. M. Multicriteria Methods for Resource Planning: An Experimental Comparison. IEEE Trans. Power Syst. 1994, 9(4), 1811–1817. 7. Vincke, P. Multicriteria Decision-Aid; Wiley: New York, 1992. 8. Cohon, J. L.; Marks, D. H. A Review and Evaluation of Multi-Objective Programming. Water Resour. Res. 1975, 11, 208–220. 9. Cohon, J. L. Multiobjective Programming and Planning; Academic Press: New York, 1978. 10. Kendall, M. G. The Treatment of Ties in Ranking Problems. Biometrika 1945, 33(3), 239–251. 11. Belton, V.; Stewart, T. J. Multiple Criteria Decision Analysis, An Integrated Approach; Kluwer Academic Publisher: Dordrecht, 2003. 12. Smilde, A. K.; Knevelmann, A.; Coenegracht, P. M. J. Introduction of Multicriteria Decision Making in Optimisation Procedures for High-performance Liquid Chromatographic Separations. J. Chromatogr. 1986, 369, 1–10. 13. Smilde, A. K.; Bruins, C. H. P.; Doornbos, D. A.; Vinck, J. Optimisation of the Reversed-Phase High-Performance Liquid Chromatographic Separation of Synthetic Estrogenic and Progestogenic Steroids Using the Multi-Criteria Decision Making Method. J. Chromatogr. 1987, 410, 1–12. 14. Boer, J. H.; Smilde, A. K.; Doornbos, D. A. Introduction of Multi-Criteria Decision Making in Optimization Procedures for Pharmaceutical Formulations. Eur. J. Pharm. Biopharm. 1988, 34, 140–143. 15. Doornbos, D. A.; Smilde, A. K.; Boer, J. H.; Duineveld, C. A. A. Experimental Design, Response Surface Methodology and Multicriteria Decision Making in the Development of Drug Dosage Forms. In Scientific Computing and Automation (Europe); Karjalainen, E. J., Ed.; Elsevier: Amsterdam, 1990; Chapter 8. 16. Keller, H. R.; Massart, D. L. Program for Pareto-Optimality in Multicriteria Problems. Trends Analyt Chem. 1990, 9, 251–253. 17. Zimmermann, H. J.; Gutsche, L. Multi-Criteria Analyse; Springer: Berlin, 1991. 18. Eisenfu¨hr, F.; Weber, M. Zielstrukturierung: Ein kritischer Schritt im Entscheidungsprozeß. Zeitschrift fu¨r betriebswirtschaftliche Forschung 1986, 38, 907–929. 19. French, S. Decision Theory: An Introduction to the Mathematics of Rationality; Ellis Horwood: Chichester, 1988. 20. Bana e Costa, C. A.; Yansnick, J. C. The MACBETH Approach: Basic Ideas, Software and An Application. In Advances in Decision Analysis; Meskens, N., Roubens, M., Eds.; Kluwer Academic Publishers, 1999; pp 131–157. 21. Stewart, T. J. Use of Piecewise Linear Functions in Interactive Multicriteria Decision Support: A Monte Carlo Study. Manage. Sci. 1993, 39, 1369–1381. 22. Stewart, T. J. Robustness of Additive Value Function Methods in MCDM. J. Multi-Crit. Decis. Anal. 1996, 5, 301–309. 23. von Winterfeldt, D.; Edwards, W. Decision Analysis and Behavioral Research; Cambridge University Press: Cambridge, 1986. 24. Watson, S. R.; Buede, D. M. Decision Synthesis. The Principles and Practise of Decision Analysis; Cambridge University Press: Cambridge, 1987. 25. Keeney, R. L.; Raiffa, H. Decision with Multiple Objectives; Wiley: New York, 1976. 26. Roberts, F. S. Measurement Theory with Applications to Decision Making, Utility and the Social Sciences; Addison-Wesley: London, 1979. 27. von Wintefeldt, D.; Edwards, W. Decision Analysis and Behavioural Research; Cambridge University Press: Cambridge, 1986. 28. Keller, H. R.; Massart, D. L. Multicriteria Decision Making: A Case Study. Chemom. Intell. Lab. Syst. 1991, 11, 175–189. 29. Hendriks, M. M. W. B.; Boer, J. H.; Smilde, A. K.; Doorbos, D. A. Multicriteria Decision Making. Chemom. Intell. Lab. Syst. 1992, 16, 175–191. 30. Lewi, P. J.; Van Hoof, J.; Boey, P. Multicriteria Decision Making Using Pareto Optimality and PROMETHEE Preference Ranking. Chemom. Intell. Lab. Syst. 1992, 16, 139–144. 31. Geoffrion, A. M.; Dyer, J. S.; Feinberg, A. An Interactive Approach for Multicriterion Optimization with An Application to the Operation of An Academic Department. Manage. Sci. 1972, 19, 357–368. 32. Zionts, S.; Wallenius, J. An Interactive Programming Method for Solving the Multiple Criteria Problem. Manage. Sci. 1976, 22, 652–663. 33. Zionts, S.; Wallenius, J. An Interactive Multiple Objective Programming Method for a Class of Underlying Nonlinear Utility Functions. Manage. Sci. 1983, 29, 519–529. 34. Korhonen, P.; Wallenius, J.; Zionts, S. Solving the Discrete Multiple Criteria Problem Using Convex Cones. Manage. Sci. 1984, 30, 1336–1345. 35. Harrington, E. C. The Desirability Function. Ind. Qual. Control 1965, 21, 494–498. 36. Derringer, G. C.; Suich, R. Simultaneous Optimization of Several Response Variables. J. Qual. Technol. 1980, 12, 214–219. 37. Geldermann, J.; Rentz, O. Bridging the Gap Between American and European MADM-Approaches? Presented at the 51st Meeting of the European Working Group ‘Multicriteria Aid for Decisions’, Madrid, March 30–31, 2000. 38. Be´tolaud, Y.; Fe´vrier, R. Conservation des foreˆts suburbaines et passage des autoroutes – L’exemple du trace de l’autoroute A86–A87. Revue Forestie`re Française, 1973, 179–200. 39. Buffet, P.; Gre´my, J. P.; Marc, M.; Sussmann, B. Peut-on choisir en tenant compte de criteres multipes? Une me´thode (ELECTRE) et trois applications. Revue METRA 1967, 6(2), 283–316. 40. Charpentier, A. R.; Jacquet-Lagre`ze, E. La promotion de l’e´lectricite´ dans l’industrie et l’utilisation de me´thodes multicrite`res. In Multiple Criteria Decision Making; Thiriez, H. Zionts, S. Eds.; Lecture Notes in Economics and Mathematical System, Vol. 130, Springer-Verlag: Berlin, 1976; pp 364–377. 41. Barda, H.; Dupuy, J.; Lencioni, P. Multicriteria Location of Thermal Power Plants. Eur. J. Oper. Res. 1990, 45, 332–346. 42. Climaco, J.; Martins, A.; Traca-Almeida, A. On a Multicriteria Based Approach for Energy Planning. Communication to Congre`s EURO IX XXVIII, Paris, 6–8 July 1988. 43. Martel, J. M.; Nadeau, R. Revealed Preference Modeling with ELECTRE II: An Interactive Approach. Communication to Congre`s EURO IX-TIMS XXVIII, Paris, July 6–8, 1988. 44. Maystre, L. Y.; Simos, J. Comment pourrait-on ge´rer les dećhets solides urbains de Gene`ve? In Actes du Colloque AFCET sur le De´veloppement des Sciences et Pratiques de l’Organisation, Theme 1987: ‘L’Aide a la Decision dans l’Organisation, Paris, 1987; pp 253–258. 45. Parent, E.; Schna˜bele, P. Le choix d’un ameńagement aquacole – Exemple d’utilisation de la me´thode ELECTRE III et comparaison avec d’autres methods multicrite´res d’aide a la decision. Document du LAMSADE No. 47, Universite´ de ParisDauphine.

Multicriteria Decision-Making Methods 627 46. Rabeharisoa, V. Une application de l’analyse multicrite`re pour l’evaluation d’alternatives: Technologies propres et curatives de la pollution industrielle. Cahier du LAMSADE No. 85, Universite´ de Paris-Dauphine, February 1988. 47. Renard, F. Utilisation d’ELECTRE dans l’analyse des re´ponses a un appel d’offres: Le cas de la machine de tri paquets a la Direction Geńe´rale des Postes. Cahier du LAMSADE No. 73, Universite´ de Paris-Dauphine, June 1986. 48. Roy, B.; Present, M.; SiIhol, D. A. Programming Method for Determining Which Paris Metro Stations should Be Renovated. Eur. J. Oper. Res. 1986, 24, 318–334. 49. Roy, B. The Outranking Approach and the Foundations of Electre Methods. Theory Decis. 1991, 31, 49–73. 50. Slowinski, R.; Treichel, W. MCDM Methodology for Regional Water Supply System Programming. Communication to Congre`s EURO IX-TIMS XXVIII, Paris, 6–8 July 1988. 51. Roy, B.; Bouyssou, D. Aide Multicritere d’Aide a` la Dećision: Me´thodes et Cas; Economica: Paris, 1993. 52. Roy, B. Multicriteria Methodology for Decision Aiding; Kluwer Academic Publisher: Dordrecht, 1996. 53. Brans, J. P.; Vincke, Ph. A Preference Ranking Organization Method (the PROMETHEE Method for Multiple Criteria Decision Making). Manage. Sci. 1985, 31, 647–656. 54. Brans, J. P.; Mareschal, B.; Vincke, Ph. PROMETHEE a New Family of Outranking Methods in Multicriteria Analysis. In Operational Research; Brans, J. P., Ed.; North-Holland: Dordrecht, 1984; pp 477–490. 55. Brans, J. P.; Vincke, Ph.; Mareschal, B. How to Select and How to Rank Projects: the PROMETHEE Method. Eur. J. Oper. Res. 1986, 24, 228–238. 56. Brans, J. P.; Mareschal, B. The PROMCALC & GAIA Decision Support System for Multicriteria Decision Aid. Decis. Support Syst. 1994, 12, 297–310. 57. Geldermann, J.; Zhang, K. Software Review: Decision Lab 2000. J. Multi-Crit. Decis. Anal. 2001, 10, 317–323. 58. Roy, B.; Hugonnard, J. Ranking of Suburban Line Extension Projects on the Paris Metro System by a Multicriteria Method. Transport. Res. 1982, 16A, 301–312. 59. Perny, P. Multicriteria Filtering Methods Based on Concordance and Non-Discordance Principles. Ann. Oper. Res. 1998, 80, 137–165. 60. Rubens, M. Analyse et aggre´gation des pre´fe´rences: mode´lisation, ajustement et re´sume´ de donneés relationelles. Revue Belge de Statistique, d’Informatique et de Recherche Operationelle 1980, 20, 36–67. 61. Van Velthoven, G. Aggregatie van niet-metrische rangschikkingen via louter preordinale preferentieglobalisatie. Revue X 1980, 2, 11–17. 62. Van Den Berghe, R.; Van Velthoven, G. Selection multicrite`re en matie`re de ree´quipment. Revue X 1982, 4, 1–8. 63. Rubens, M. Preference Relations on Actions and Criteria in Multicriteria Decision Making. Eur. J. Oper. Res. 1982, 10, 51–55. 64. Bourguignon, B.; Massart, D. L. The Oreste Method for Multicriteria Decision Making in Experimental Chemistry. Chemom. Intell. Lab. Syst. 1994, 22, 241–256. 65. Pastijn, H.; Leysen, J. Constructing an Outranking Relation with ORESTE. Math. Comput. Model. 1989, 12, 1255–1268. 66. Pastijn, H. ORESTE; Royal Military Academic: Brussels, Belgium, 1988. 67. Urrutia, J. Partial Orders and Euclidian Geometry. In Algorithms and Order, NATO ASI Series, Series C: Mathematical and Physical Science; Rival, I. Ed.; Kluwer Academic Publishers: Dordrecht, 1987; Vol. 255, pp 387–434. 68. Welzl, G.; Voigt, K.; Rediske, G. Visualisation of Environmental Pollution – Hasse Diagram Technique and Explorative Statistical Methods. In Proceedings of the Workshop on Order Theoretical Tools in Environmental Sciences; Group Pragmatic Theoretical Ecology, Ed.; Berichte des IGB: Berlin, 1998; pp 101–110. 69. Hasse, H. U¨ber die klassenzhal abelscher Zahlko¨per; Akademie Verlag: Berlin, Germany, 1952. 70. Halfon, E.; Reggiani, M. G. On Ranking Chemicals for Environmental Hazard. Environ. Sci. Technol. 1986, 20, 1173–1179. 71. Bru¨ggemann, R.; Bartel, H.-G. A Theoretical Concept to Rank Environmentally Significant Chemicals. J. Chem. Inf. Comput. Sci. 1999, 39, 211–217. 72. Bru¨ggemann, R.; Voigt, K.; Steinberg, C. E. W. Application of Formal Concept Analysis to Evaluate Environmental Databases. Chemosphere 1997, 35, 479–486. 73. Bru¨ggemann, R.; Halfon, E. Theoretical Base of the Program ‘Hasse’; GSF-Bericht 20/95, GSF-Forschungszentrum: Oberschleissheim, Germany, 1995. 74. Pavan, M.; Todeschini, R. New Indices for Analysing Partial Ranking Diagrams. Anal. Chim. Acta 2004, 515, 167–181. 75. Bru¨ggemann, R.; Schwaiger, J.; Negele, R. D. Applying Hasse Diagrams Technique for the Evaluation of Toxicological Fish Tests. Chemosphere 1995, 39, 1767–1780. 76. Bru¨ggemann, R.; Oberemm, A.; Steinberg, C. Ranking of Aquatic Effect Tests Using Hasse Diagrams. Environ. Toxicol. Chem. 1997, 63, 125–139. 77. Halfon, E. Comparison of an Index Function and a Vectorial Approach Method for Ranking of Waste Disposal Sites. Environ. Sci. Technol. 1989, 23, 600–609. 78. Newman, A. Ranking Pesticides by Environmental Impact. Environ. Sci. Technol. 1995, 29, 324–326. 79. Bru¨ggemann, R.; Munzer, B.; Halfon, E. An Algebraic/Graphical Tool to Compare Ecosystems with Respect to their Pollution – The German River ‘Elbe’ as an Example – I: Hasse – Diagrams. Chemosphere 1994, 28, 863–872. 80. Pudenz, S.; Bittner, T.; Bru¨ggemann, R. Comparative Evaluation of Materials in Car Production. In Order Theoretical Tools in Environmental Sciences; Proceedings of the Second Workshop; Special Issue of Analytyca Chimica Acta, Elsevier: Amsterdam 1999; 95–104. 81. Pudenz, S.; Bru¨ggemann, R.; Luther, B.; Kaune, A.; Kreimes, K. An Algebraic/Graphical Tool to Compare Ecosystems with Respect to their Pollution V: Cluster Analysis and Hasse Diagrams. Chemosphere 2000, 40, 1373–1382. 82. Munzer, B.; Bru¨ggemann, R.; Halfon, E. An Algebraic/Graphical Tool to Compare Ecosystems with Respect to their Pollution II: Comparative Regional Analysis. Chemosphere 1994, 28, 873–879. 83. Bru¨ggemann, R.; Pudenz, S.; Voigt, K.; Kaune, A.; Kreimes, K. An Algebraic/Graphical Tool to Compare Ecosystems with Respect to their Pollution IV: Comparative Regional Analysis by Boolean Arithmetics. Chemosphere 1999, 38, 2263–2279. 84. Bru¨ggemann, R.; Mu¨nzer, B. A Graph – Theoretical Tool for Priority Setting of Chemicals. Chemosphere 1993, 27, 1729–1736. 85. Voigt, K.; Welz, G.; Rediske, G. Environmental Approaches to Evaluate Internet Databases. In Order Theoretical Tools in Environmental Sciences; Proceedings of the Second Workshop on Order Theoretical Tools in Environmental Sciences; Special Issue of Analytyca Chimica Acta, Elsevier: Amsterdam, 1999; pp 135–144.

628


86. Voigt, K.; Gastaiger, J.; Bru¨ggemann, R. Comparative Evaluation of Chemicals and Environmental Online and CD-ROM Databases. J. Chem. Inf. Comput. Sci. 2000, 40, 44–49. 87. Bru¨ggemann, R.; Zelles, L.; Bai, Q. Y.; Artmann, A. Use of Hasse Diagram Technique for Evaluation of Phospholipid Fatty Acids Distribution as Biomarkers in Selected Soils. Chemosphere 1995, 30, 1209–1228. 88. Sørensen, P. B.; Mogensen, B. B.; Gyldenkærne, S.; Rasmussen, A. G. Pesticides Leaching Assessment Method for Ranking Both Single Substances and Scenarios of Multiple Substance Use. Chemosphere 1998, 36, 2251–2276. 89. Klein, D. J.; Bytautas, L. Directed Reaction Graphs as Poset. Commun. Math. Comput. Chem. 2000, 42, 261–290. 90. Bru¨ggemann, R.; Bucherl, C.; Pudenz, S.; Steinberg, C. E. W. Application of the Concept of Partial Order on Comparative Evaluation of Environmental Chemicals. Acta Hydroch. Hydrob. 1993, 27, 170–178. 91. Davey, B. A.; Priestley, H. A. Introduction to Lattices and Order; Cambridge Mathematical Textbooks: Cambridge, UK, 1980; ISBN 0070543712. 92. Lerche, D. Priority Setting of Chemical Substances for Environmental Policy. Ph.D. Thesis, University of Copenhagen, Copenhagen, 2003. 93. Lerche, D.; van de Plassche, E.; Shwegler, E.; Balk, F. Selecting Chemical Substances for the UN-ECE POP Protocol. Chemosphere 2002, 47(6), 617–630. 94. Bru¨ggemann, R.; Sørensen, P. B.; Lerche, D.; Carlsen, L. Estimation of Averaged Ranks by a Local Partial Order Model. J. Chem. Inf. Comput. Sci. 2004, 44, 617–625. 95. Carlsen, L. A QSAR Approach to Physico-Chemical Data for Organophosphates with Special Focus on Known and Potential Nerve Agents. Internet Electron. J. Mol. Des. 2005, 4, 355–366. 96. Sørensen, P. B.; Lerche, D.; Carlsen, L.; Bru¨ggemann, R. Statistically Approach for Estimating the Total set of Linear Orders – A possible way for Analysing Larger Partial Order Sets. In Order Theoretical Tools in Environmental Science and Decision Systems, Proceedings of the Third Workshop, Berlin, Germany, 6–7 November 2000; Bru¨ggemann, R., Lu¨hr, H.-P., Eds.; Berlin: LeibnitzInstitute of Freshwater Ecology and Inland Fisheries. Berichte des IGB 14 (Sonderheft IV), 87–97. 97. Lerche, D.; Sørensen, P. B.; Bru¨ggemann, R. Improved Estimation of the Ranking Probabilities in Partial Orders using Random Linear Extensions by Approximation of the Mutual Ranking Probabilities. J. Chem. Inf. Comp. Sci. 2003, 43 (5), 1471–1480. 98. DART Decision Analysis by Ranking Techniques, Talete srl. Available at http://ecb.jrc.it. 99. Charnes, A.; Cooper, W. W. Management Model and Industrial Applications of Linear Programming; Wiley: New York, 1961. 100. Romero, C. A Survey of Generalized Goal Programming (1970–1982). Eur. J. Oper. Res. 1986, 25, 183–191. 101. Schniederjans, M. J. Goal Programming: Methodology and Applications; Kluwer Academic Publishers: Boston, 1995. 102. Lee, S. M. Goal Programming for Decision Analysis; Auerbach: Philadelphia, 1972. 103. Ignizio, J. P. Goal Programming and Extensions; Lexington Books: Lexington, MA, 1976. 104. Ignizio, J. P. Introduction to Linear Goal Programming Sage; Publications: London, 1985. 105. Ignizio, J. P. Generalised Goal Programming: An Overview. Comput. Oper. Res. 1983, 36, 27–35. 106. Lee, S. M.; Olson, D. L. Goal Programming. In Multicriteria Decision Making: Advances in MCDM Models, Algorithms, Theory, and Applications; Gal, T., Stewart, T. J., Hanne, T., Eds.; Kluwer Academic Publishers: Boston, 1999; Chapter 8. 107. Simon, H. A. Administrative Behaviour; The Free Press: New York, 1976. 108. Wierzbicki, A. P. The Use of Reference Objectives in Multiobjective Optimization. In Multiple Criteria Decision Making Theary and Practice; Fandel, G., Gal, T., Eds.; Springer: Berlin, 1980; pp 468–486. 109. Korhonen, P.; Laakso, J. A Visual Interactive Method for Solving the Multiple Criteria Problem. Eur. J. Open. Res. 1986, 24, 277–287.



Manuela Pavan is a chemometrician with 8 years of experience in (Q)SAR and modeling approaches. She received her Ph.D. in Chemical Sciences from the University of Milano – Bicocca in 2003 and is currently working in the Joint Research Centre as scientific and technical support officer. She has rich experience in multivariate statistics, molecular modeling, and ranking strategies supporting multicriteria decision-making processes. Her scientific record includes more than 27 papers in peer-reviewed journals. During the period of her employment in JRC, she coordinated the ranking project titled ‘Investigation of Computational Approaches for the Ranking of Chemicals according to their Environmental and Toxicological Concern’.

Roberto Todeschini is full professor of chemometrics at the Department of Environmental Sciences of the University of Milano-Bicocca (Milano, Italy), where he constituted the Milano Chemometrics and QSAR Research Group. His main research activities concern chemometrics, QSAR, molecular descriptors, multicriteria decision making, and software development. President of the International Academy of Mathematical Chemistry, president of the Italian Chemometric Society, and ‘‘ad honorem’’ professor of the University of Azuay (Cuenca, Ecuador), he is author of more than 150 publications on international reviews and of the books The Data Analysis Handbook by I. E. Frank and R. Todeschini (Elsevier, 1994) and Handbook of Molecular Descriptors by R. Todeschini and V. Consonni (Wiley-VCH, 2000).


1.20

Genetic Algorithms

R. Leardi, University of Genoa, Genoa, Italy ª 2009 Elsevier B.V. All rights reserved.

1.20.1 1.20.2 1.20.3 1.20.4 1.20.5 1.20.5.1 1.20.5.2 1.20.5.3 1.20.6 1.20.6.1 1.20.6.2 1.20.6.3 1.20.7 1.20.8 1.20.9 1.20.10 1.20.10.1 1.20.10.2 1.20.10.3 1.20.10.4 1.20.10.5 1.20.10.6 1.20.10.7 1.20.10.8 1.20.10.9 1.20.10.9.1 1.20.10.9.2 References

Introduction The Evolution Theory How to Transform the Evolution Theory into an Optimization Technique? The Problem of Coding Steps of the GAs Creation of the Original Population Reproduction Mutations Comments about the Parameters of the GAs Population Size Reproduction Mutation Probability Hybrid Algorithms Looking for the Global Maximum: Is it Worthwhile? Applications A Specific Application: Variable Selection in Spectral Data Sets Initiation of Population Reproduction and Mutation Insertion of New Chromosomes Influence of the Different Parameters Check of Subsets How to Reduce the Risk of Overfitting How to Take into Account the Autocorrelation among the Spectral Variables How to Get the Final Model An Example Additive B Additive C

631 632 633 633 635 635 635 638 639 639 640 640 640 641 641 642 644 644 644 645 645 646 646 647 648 648 648 651

1.20.1 Introduction When the complexity of the system under study is not extremely high, then ‘standard’ techniques (such as those of the experimental design) give quite good results, with a limited number of experiments. In contrast, when the problem is extremely complex these techniques cannot detect the global maximum (or minimum), because they are very sensitive to global maxima or because the response surface cannot be modeled by simple empirical models. What is a ‘very complex system’? Basically, the complexity can be ascribed to three basic causes: 1. High number of independent variables: For a linear model without interactions, the Plackett–Burman matrices allow the screening of many variables, with a number of experiments equal to the first multiple of four higher than the number of variables. If one is interested also in studying the interactions among variables and the nonlinear terms, then the mathematical model would be too complex and the number of experiments would become extremely high. 631

This page intentionally left blank 632

Genetic Algorithms

2. Very complex or irregular response surface: This can happen when the mathematical function describing the response contains higher degree terms or trigonometrical functions, or when there is no function that can adequately describe the response. This usually happens when several local maxima are present; in those cases it is not possible to use local search techniques, such as simplex, that would stop at the first local maximum they find. 3. Presence of discontinuities in the experimental domain: This can happen when it is not possible to run experiments in some regions of the experimental domain: If the limitations define a continuous zone, then the ‘classical’ techniques can be applied without any problem, while if the limitations define more than one region, separated by discontinuities, then neither experimental design nor the local search techniques can reach the global maximum (e.g., a simplex cannot get out of the region in which it started). 4. Response to be optimized function of several ‘subresponses’: sometimes it possible that what is defined as ‘response’ to be optimized is instead something much more complex, whose value depends on the value of different responses, each of them describing a specific aspect of the problem. For instance, in the case of a chemical reaction the global ‘response’ can depend on yield, time, percentage of impurities, and what one is looking at is a solution being a good compromise. The desirability functions can be used, but it has to be taken into account the fact that the error of the final ‘response’ is the sum of the errors of the different subresponses. If there are many of them, it is clear that, though each one of them can be predicted with a sufficiently small error, the global function will have a rather low predictive ability. In all these cases, the only way to be sure of finding the global maximum would be a ‘grid search’. Under this strategy, each variable is divided into intervals, and the response is measured in every possible combination. It is easy to realize that the number of measurements increases fast as the number of variables increases: with four intervals (and therefore five levels) for each variable, with n variables 5n experiments must be performed. This means 25 experiments with 2 variables, but with 10 variables it is almost 10 million experiments! Furthermore, if the surface is quite irregular, with thin peaks, the number of intervals must be quite high, not to have peaks located between the points of the grid. It is therefore evident that with very complex problems a systematic search is out of question, because the accomplishment of the required task would be impossible. (Let us suppose to evaluate the response of 100 experiments per second: 10 million experiments would require 100 000 s, i.e., more than 1 day!) A second strategy could be performing experiments at random points, retaining the points giving a good response, and trying to improve the response by using the obtained information. This could be carried out, for instance, by performing a local search around the randomly selected point, or by trying to exploit what the best points have in common.

1.20.2 The Evolution Theory The evolution of the species can be considered by itself a form of optimization, in which the response to be optimized is the fitness to the environment. It is anyway easy to understand that the term ‘fitness to the environment’ is something that cannot be defined in an exact form: each individual has several characteristics and the importance of each of them can be extremely different. For instance, if we consider the human species and take into account the success in life, we can see that there are some fundamental characteristics, without which an individual cannot survive (e.g., the absence of lethal pathologies), important characteristics (intelligence, strength, and beauty), and nonrelevant characteristics (the shoe number or the color of the eyes). It can also be noticed how the same value of the response ‘success’ can be obtained by individuals having totally different characteristics: a scientist winning the Nobel prize, an athlete winning the Olympic games, and a girl winning the Miss World title, all have a very high response. The basic idea of the evolution theory is that the individuals with a greater ‘fitness to the environment’ have a greater probability of surviving and a greater probability of winning the fights for mating. In such a way, the

Genetic Algorithms 633

genetic content of the best individuals will be abundantly present in the following generations, since it will be transmitted by the offspring. It is logical to think that, if the parents have a good ‘response’, their offspring will also have a good one, sometimes even better than that of their parents. Since centuries men applied this idea, to get, for instance, cows producing more milk, or horses running faster, or fruits or flowers with specific characteristics. As can be seen, in these cases the response being considered is not a general ‘fitness to the environment’, but a specific characteristics that has to be maximized. As a consequence, we can see that a ‘guided evolution’ can get results completely different from those of the ‘natural evolution’.

1.20.3 How to Transform the Evolution Theory into an Optimization Technique? Genetic algorithms (GAs) have been proposed by Holland in the 1960s, but it was possible to apply them with reasonable computing times only since the 1990s, when computers became much faster. In 1993 a paper was published in the journal Science1 that gave a general presentation of GAs, some mathematical analysis about how GAs work and how best to use them, and some applications in modeling several natural evolutionary systems, including immune systems. In 1995 an article in Nature2 described a problem of molecular dynamics that had been successfully solved by a GA where conventional techniques had failed. General information on GAs can be found in Goldberg,3 Leardi,4,5 Lucasius and Kateman,6,7 Shaffer and Small,8 and Wehrens and Buydens.9 A wealth of information can also be found on the websites of various organizations.10–17 The basic idea is to perform a computer simulation of what occurs in nature, and the first problem to be solved is how to code the information in such a way that the computer can treat it. It can therefore be said that the fitness to the environment is a function of the genetic material in the same way as the result of an experiment is function of the experimental conditions. Therefore, a correspondence genetic material–experimental conditions can be established. At a lower level, we can say that the genetic material is defined by the genes in the same way as an experimental condition is defined by the values of the variables involved in the experiment. Therefore, a correspondence genes–variables can be established. At an even lower level, we can see that the information contained in each gene is defined by a sequence of nitrogenated bases: since there are four bases, each gene can be considered as a word of variable length, written in a four-letter alphabet. In the same way, we can use the binary code to transform the value of a variable in a word of variable length, written in bits (two-letter alphabet, 0 and 1). Therefore, we have the following correspondences: 1. genetic material ¼ experimental conditions (point in the experimental domain) 2. gene ¼ variable 3. nitrogenated base ¼ bit Finally, we can see how the experimental conditions can be coded by a sequence of 0s and 1s.

1.20.4 The Problem of Coding Let us suppose we are dealing with a chemical reaction, for which we want to maximize the yield, and code the following experimental condition (it has to be clear that this is just a didactical example, since it would be foolish to apply such a complex method as GAs to optimize such a simple problem): 1. reaction temperature: 30 C 2. reaction time: 20 min

634

Genetic Algorithms

3. stirring: yes 4. catalyst: type A (A and B possible catalysts) Four genes will therefore form the corresponding ‘genetic material’ (for simplicity, GAs always work with one single chromosome), each gene containing the corresponding information in bits: 011110 10100 1 0 ðblank spaces have been added only to make genes evidentÞ:

The following two facts can be obcerved: 1. variables of different types can be dealt with at the same time: quantitative variables (time, temperature), qualitative variables (type of catalyst), and variables of type yes/no (stirring); 2. the number of bits for each gene can be very different. It is extremely important that the coding of the quantitative variables takes into account the interval in which the same variable can vary and the fact that the difference between two levels is significant. It is intuitive that if a large number of bits is used to describe a variable, very small variations of the same variable could be studied; these small variations must nevertheless have a real meaning. In the previous example, six bits describe the temperature. This means 1. the range is between 0 and 63 C 2. the difference between two levels is 1 C. This is similar to saying that 1. we are interested in studying the reaction from 0 to 63 C 2. the difference of 1 C is significant (the reaction at 25 C can be different from the reaction at 26 C) and the temperature can be set with the precision of 1 C (for an experiment to be performed at 25 C, one can actually set it between 24.5 and 25.5 C). It is clear that in this case the number of bits used is too large for the characteristics of the variable. If the total number of bits of a chromosome is n, the total number of combinations is 2n (in our example, with 13 bits, the total number of possible combinations is 8192); a number of bits larger than that required has the only effect of increasing the complexity of the system and, therefore, decreasing the efficiency of the search. Taking into account our previous example, let us suppose that the temperature range we are interested in is from 25 to 60 C, with an interval of 5 C: As a consequence, eight levels describe completely our variable, and three bits are sufficient:

000 ¼ level 0 ¼ 25 C 001 ¼ level 1 ¼ 30 C 010 ¼ level 2 ¼ 35 C 011 ¼ level 3 ¼ 40 C 100 ¼ level 4 ¼ 45 C 101 ¼ level 5 ¼ 50 C 110 ¼ level 6 ¼ 55 C 111 ¼ level 7 ¼ 60 C

The information that can be obtained with this coding is comparable to that of the original coding, but just three bits instead of six have been used.

This page intentionally left blank Genetic Algorithms 635

In the same way, if for the variable time we are interested in the range 10–40 min, with an interval between levels of 2 min, 16 levels and therefore 4 bits will be required:

0000 ¼ level 1 ¼ 10 min 0001 ¼ level 2 ¼ 12 min ..................................................... 1110 ¼ level 15 ¼ 38 min 1111 ¼ level 16 ¼ 40 min

The final coding would be 001 0101 1 0, with 9 bits and 512 possible combinations. It is easy to realize that a more clever coding can reduce the search complexity by 16 times, without losing almost any information. After having coded the experimental conditions and having generated the corresponding chromosome, the response (in this case the yield of the reaction) is measured: this response will always be associated to the chromosome.

1.20.5 Steps of the GAs According to the evolution theory, the improvement of a species occurs because, through a very high number of generations, the genetic material of its individuals is constantly improving. The reason for this is the ‘bad’ individuals do not survive and the best ones have a greater probability of spreading their genetic material to the following generation. Beyond this ‘logical’ development, mutations allow the exploration of new ‘experimental conditions’; usually mutations produce bad results (e.g., severe pathologies), but it may happen that these random changes of a nitrogenated basis end up in a better genome. Several GAs have been developed; beyond the common basic idea (mimicking the evolution of a species), they can have relevant differences. All of them have three fundamental steps that can in any case be performed in different ways. These three steps are: 1. creation of the original population 2. reproduction 3. mutations Let us now have a short description of each one of them. 1.20.5.1

Creation of the Original Population

The population size stays constant throughout the elaboration. The number of individuals can be quite different, and usually is in the range 20–500 (we describe the influence of this parameter on the performance of the GAs later). After having decided the population size (p), the genetic material of the p individuals is randomly determined. This means that every single bit of each chromosome is randomly set to 0 or 1. If this chromosome corresponds to a possible experimental condition (i.e., inside the experimental domain), its response is evaluated. 1.20.5.2

Reproduction

After having created the original population (or first generation), the individuals start ‘mating’ and ‘producing offspring’. This is the step in which the different GAs have the greatest variations, though all of them follow the same idea: the probability of the best chromosomes (the ones giving the best responses) of producing offspring is

636

Genetic Algorithms

higher than that of the worst chromosomes, and the offspring originated by their ‘mating’ are a recombination of the parents. Basically, the first step is creating the population of the second generation simply by randomly copying p times a chromosome of the first generation. If the drawing would be totally random, then each chromosome would have the same probability of going to the next generation and, therefore, the average response of the generation n þ 1 would be statistically the same as that of generation n. In nature, each individual always has a possibility of reproducing itself, but the best ones have a greater probability of winning the fights for mating. In the same way, the drawing performed to select the chromosomes that will be copied must take into account the response of the individuals, giving the best ones a higher probability. To do this, a biased drawing is performed, in which the probability of each individual of being selected is the function of its response. To visualize this process in a simple way, it is as performing the selection with a roulette wheel in which the slots corresponding to the best individuals are larger than those corresponding to the worst ones. Several functions have been studied to perform this step: the simplest one is that the probability of each chromosome of being selected is equal to its response divided by the sum of the responses: resp pi ¼ P i resp

As a result of such a drawing , the best individuals will be copied more than once in the population n þ 1, while the worst ones will disappear. It can be easily understood that the average response of generation n þ 1 will be higher than the average response of generation n. Continuing with the previous example, let us simulate this step, supposing that the population size is 10 individuals. The original population is

Chromosome

Experimental conditions

Yield

001 1001 0 1 010 0100 1 1 000 1010 0 0 100 0101 1 1 110 0001 1 0 010 1111 0 1 101 0111 1 1 001 0010 1 0 100 1001 1 0 001 0011 1 1

30 C, 28 min, no, B 35 C, 18 min, yes, B 25 C, 30 min, no, A 45 C, 20 min, yes, B 55 C, 12 min, yes, A 35 C, 40 min, no, B 50 C, 24 min, yes, B 30 C, 14 min, yes, A 45 C, 28 min, yes, A 30 C, 16 min, yes, B

54.9 67.2 66.0 70.3 79.1 62.1 71.3 83.4 89.6 59.7

After sorting the population and computing the selection probability, we have

Chromosome


Yield

Probability

100 1001 1 0 001 0010 1 0 110 0001 1 0 101 0111 1 1 100 0101 1 1 010 0100 1 1 000 1010 0 0 010 1111 0 1 001 0011 1 1 001 1001 0 1

45 C, 28 min, yes, A 30 C, 14 min, yes, A 55 C, 12 min, yes, A 50 C, 24 min, yes, B 45 C, 20 min, yes, B 35 C, 18 min, yes, B 25 C, 30 min, no, A 35 C, 40 min, no, B 30 C, 16 min, yes, B 30 C, 28 min, no, B

89.6 83.4 79.1 71.3 70.3 67.2 66.0 62.1 59.7 54.9

0.127 0.119 0.112 0.101 0.100 0.096 0.094 0.088 0.085 0.078


The average response is 70.36. Let us now assume to draw 10 random numbers between 0.001 and 1.000 (with a ‘precision’ of 0.001): these numbers will decide which chromosomes will be copied. This means that the first chromosome will be selected when a number in the interval 0.001–0.127 will be drawn, the second chromosome will be selected when a number in the interval 0.128–0.246 (0.246 being 0.127 þ 0.119) will be drawn, and so on. Let the 10 numbers be 0:353 0:038

0:367 0:324

0:414 0:903

0:150

0:353 0:428

0:915

These values correspond to chromosomes 3, 1, 4, 3, 4, 9, 2, 3, 4, and 9; in the second generation, chromosome 1 will be copied once, chromosome 2 once, chromosome 3 three times, chromosome 4 three times, and chromosome 9 twice. The second generation will be the following: Chromosome


Yield

100 1001 1 0 001 0010 1 0 110 0001 1 0 110 0001 1 0 110 0001 1 0 101 0111 1 1 101 0111 1 1 101 0111 1 1 001 0011 1 1 001 0011 1 1

45 C, 28 min, yes, A 30 C, 14 min, yes, A 55 C, 12 min, yes, A 55 C, 12 min, yes, A 55 C, 12 min, yes, A 50 C, 24 min, yes, B 50 C, 24 min, yes, B 50 C, 24 min, yes, B 30 C, 16 min, yes, B 30 C, 16 min, yes, B

89.6 83.4 79.1 79.1 79.1 71.3 71.3 71.3 59.7 59.7

The average response of this population is 74.36. With this process, chromosomes existing in the previous population have simply been copied and placed in the following population, without exploring any new experimental condition. To perform this, a reproduction is simulated: the 10 individuals are randomly paired in five pairs, and from each pair (the ‘parents’) two new individuals (the ‘offspring’) will be obtained after a ‘crossover’, by which the genes of the parents will be shuffled. Let us suppose the pairs are 1–10, 2–9, 5–8, 4–6, and 3–7. Let us take into account the first one: 100 1001 1 0 001 0011 1 1

45 C, 28 min, yes, A 30 C, 16 min, yes, B

There are several ways of shuffling the genes of the parents. The two most frequently applied are the ‘single crossover’ and the ‘uniform crossover’. In the former, a ‘breaking point’ in the chromosome is randomly selected: the first offspring will be formed by the genes of parent 1 at the left of the breaking point and by the genes of parent 2 at the right of the breaking point; the other way round for offspring 2. With four genes a number between 1 and 3 is drawn: let us suppose 2: 100 1001 j1 0 001 0011 j1 1


100 1001 j1 1 001 0011 j1 0

45 C, 28 min, yes, B 30 C, 16 min, yes, A

The two offspring will be

638

Genetic Algorithms

It is easy to understand that, with this method, the order of the variables in the chromosome is very important: with g genes, two variables coded as contiguous genes will have a probability of just 1/(g 1) of being transmitted each one to a different offspring, while the first and the last variables will always be transmitted to the first and the second offspring, respectively. In the latter method, for each gene a random number is drawn, determining to which offspring the genes of the parents will be assigned: if the value is < 0.5, then the gene of parent 1 will be given to offspring 1 (and the gene of parent 2 will be given to offspring 2), if it is > 0.5, then the gene of parent 1 will be given to offspring 2 (and the gene of parent 2 will be given to offspring 1). Of course, this drawing will not take place for the genes being the same in both parents. Let us suppose that the values are 0.334 for the first gene, 0.719 for the second and 0.265 for the fourth gene (the third gene is the same in both parents). The two offspring will be

100 0011 1 0 001 1001 1 1


Following the same for all the pairs, the following population is obtained:

100 0011 1 0 001 1001 1 1 001 0011 1 0 001 0010 1 1 110 0111 1 0 101 0001 1 1 101 0001 1 0 110 0111 1 1 101 0111 1 0 110 0001 1 1

45 C, 16 min, yes, A 30 C, 28 min, yes, B 30 C, 16 min, yes, A 30 C, 14 min, yes, B 55 C, 24 min, yes, A 50 C, 12 min, yes, B 50 C, 12 min, yes, A 55 C, 24 min, yes, B 50 C, 24 min, yes, A 55 C, 12 min, yes, B

The 10 individuals that have been obtained after this step are different from the 10 individuals of the first generation, and also different from each other (in some algorithms this is a necessary condition, while in some others ‘twins’ are accepted). Although different individuals have been obtained, by continuing in this way only already tested values of the variables would be used; furthermore, in this case the third gene (stirring) has value 1 in all the population: therefore, an experimental condition without stirring could never again occur.

1.20.5.3

Mutations

To overcome these problems an operator simulating mutations is implemented: in nature, mutations take place with an extremely low probability and have as effect the variation of a ‘letter’ of the word coding the gene; a nitrogenated basis in the DNA, a bit in our chromosome. The main difference between crossover and mutation is that, while the crossover is applied at gene level (it involves all the bits coding the variable), the mutation affects single bits. The usual probability is 1–2%. In our case, having a population of 10 chromosomes, each of them described by nine bits, a mutation probability of 2% would lead to an average of 1.8 mutations per generation.


If the bits affected by a mutation are bit number 4 of chromosome 2 and bit number 3 of chromosome 7, the ‘final’ population for the second generation will be 100 0011 1 0 001 0001 1 1 001 0011 1 0 001 0010 1 1 110 0111 1 0 101 0001 1 1 100 0001 1 0 110 0111 1 1 101 0111 1 0 110 0001 1 1

45 C, 16 min, yes, A 30 C, 12 min, yes, B 30 C, 16 min, yes, A 30 C, 14 min, yes, B 55 C, 24 min, yes, A 50 C, 12 min, yes, B 45 C, 12 min, yes, A 55 C, 24 min, yes, B 50 C, 24 min, yes, A 55 C, 12 min, yes, B

After having evaluated the response and having sorted the chromosomes, we have 101 0111 1 0 100 0011 1 0 110 0111 1 0 100 0001 1 0 001 0011 1 0 110 0111 1 1 101 0001 1 1 110 0001 1 1 001 0010 1 1 001 0001 1 1

50 C, 24 min, yes, A 45 C, 16 min, yes, A 55 C, 24 min, yes, A 45 C, 12 min, yes, A 30 C, 16 min, yes, A 55 C, 24 min, yes, B 50 C, 12 min, yes, B 55 C, 12 min, yes, B 30 C, 14 min, yes, B 30 C, 12 min, yes, B

89.2 86.5 85.8 84.0 83.8 68.9 68.4 67.6 65.0 64.6

The average response of the second generation is higher than that of the first generation; it can be noticed that all the best chromosomes have stirring and catalyst A. The whole process (select copy, crossover, mutations) is repeated on these chromosomes and the third generation is obtained. With this kind of selection, each generation will usually have an average response higher than the previous generation. New generations will be created until a stop criterion is satisfied, the most common of which are: predefined number of generations, predefined time of elaboration, and obtention of a target response value.

1.20.6 Comments about the Parameters of the GAs When describing a GA, the details about the different parameters must be given: they can have very different values and can have a very strong effect on the final result. It has to be well understood that an ‘optimal’ form of the GA does not exist, and that for each problem the best results can be obtained by a specifically designed GA. Basically, the strength of the GAs is in the joint application of two strategies: exploration and exploitation. The former is typical of the random searches, in which different points of the experimental domain are randomly tested: this allows testing points in different regions of the space, without being worried by what happens around them. The latter is typical of the local searches, such as simplex, that try to reach the local maximum closer to the starting point, without caring of what happens in different regions of the experimental domain. The basic problem, and the secret to get a good algorithm, is getting a good balance between exploration and exploitation: In the next subsections the influence of each parameter will be described.

1.20.6.1

Population Size

A population formed by many individuals allows keeping a large difference among the chromosomes, and therefore exploring at the same time several different regions; with a small population it can happen that all the individuals are extremely similar.

640

Genetic Algorithms

In contrast, in the same computing time a greater population will produce a smaller number of generations than a larger population. This means that a very good chromosome found in generation n will need much more time in producing its effects, by generating offspring: this will happen only in generation n þ 1. In the literature, populations ranging between 20 and 500 individuals can be found. To decide the population size, the time required to evaluate the response is also important: If it is quite short, then a larger population can be used, since the time interval between the generations will be short; however, if it is quite long, then it would be better to work with a reduced genetic variability, keeping anyway acceptable the time interval between generations. Generally speaking, it can be said that by increasing the population size an increase in the exploration and a decrease in the exploitation is obtained. 1.20.6.2

Reproduction

In the previous example we saw that, usually, generation n þ 1 has an average value of the response higher than generation n. In the same example it occurred that the best chromosome of generation 2 is worse than the best chromosome of generation 1. This can happen because all the parents ‘die’, and it is not sure that in the following generation chromosomes better than the best of the parents are present. To avoid this problem it is possible to set the ‘elitism’: the k best individuals of generation n go directly to generation n þ 1. Therefore, if the global maximum is found, the corresponding chromosome will never die: it is like saying that the concept of immortality has been added. With a population of c chromosomes, the selectcopy operation, performed on all the c chromosomes, will produce c k parents, and therefore (c k)/2 pairs, and finally c k offspring, to which the k best chromosomes of generation n will be added, to reconstitute a population size of c chromosomes. It has to be noticed that the k elitist chromosomes can also be selected as parents and therefore they can continue producing offspring. As an extreme case, it is possible to copy just two chromosomes, whose offspring will enter the population only if they are better than the worse chromosomes of the previous population: by doing that, the concept of ‘generation’ is completely lost. Basically the elitism has the advantage that, at the end of the elaboration, the k best chromosomes of the population will be the k best chromosomes ever found. Furthermore, the time required for a generation is smaller, and therefore the same result as discussed in the subsection ‘population size’ is obtained. Here too, with a high elitism the risk is that all the chromosomes are quite similar, around a good maximum, and that it will be impossible to get out of that region: the only possibility would be landing by chance on a higher peak, with a higher response. Summarizing, a reproduction without elitism has a higher exploration, while the higher the elitism the higher the exploitation. Here too, the problem is finding a good balance. 1.20.6.3

Mutation Probability

Also in this case, the goal is finding a good balance between exploration and exploitation. A low mutation probability does not allow changing easily the study region, and therefore favors exploitation. With a high mutation probability, changing region is much easier, but the local search is very poor, the process is much more random, and exploration is favored.

1.20.7 Hybrid Algorithms The main difference between the classical techniques and GAs has already been reported: While the former focus on local search, succeeding in the identification of the local maximum, without taking into account what happens in the different regions of the experimental domain, the latter perform a very good exploration of the space, without directly trying to go to the top of the local maxima. It is obvious to think that the application of both techniques could lead to results being better of what could be obtained by each of them separately. One of the problems of the classical techniques is that they need previous knowledge to set their parameters (e.g., for a simplex, experimental domain, step and starting point). This knowledge could instead be obtained by a GA: the best chromosome would be a good starting condition for a classical technique.


Even better results are produced by alternating the two techniques: In this case an ‘hybrid’ algorithm is obtained.18 The first step is a GA; when a predefined stop criterion is satisfied (computation time, number of evaluations, and so on), a local search is performed starting from the best chromosome. The result of this local search will be treated as an offspring, and therefore enter the population. At the end of the local search, a new GA is started, until the next stop, and so on. By joining two totally different approaches the drawbacks of both are reduced, and a new technique with both a good exploration and a good exploitation is obtained.

1.20.8 Looking for the Global Maximum: Is it Worthwhile? Many times the validity of a GA is measured only by taking into account its capability of finding the global maximum and the time required for it. This can be really important only if the global maximum is much better than the other local maxima; in this case, if the GA stops on a different maximum (as in the case of simplex), then a much poorer result is obtained. A different situation occurs when the response corresponding to one or more local maxima is not significantly worse than that of the global maximum. In this case, the detection of the global maximum is much more difficult; furthermore, this is not relevant from the practical point of view, provided that the value found by the GA is not very different. Moreover, when working with chemical data, we are dealing with data affected by an experimental error; therefore, the experimental conditions corresponding to the global maximum could be quite different simply as a consequence of measurement errors. Let us suppose to have two local maxima, with similar height: which one of the two peaks will be the higher will depend only on the data taken into account (and on their experimental errors). What is really important is that at least one of them can be detected. A great advantage of GAs over the classical techniques is that at the end of the elaboration the user is given not just an ‘optimal’ solution, but a population of extremely good solutions, usually having very similar responses. With a complex response surface, with many local maxima, it is possible that the same response can be obtained from completely different experimental conditions. In this case, it is quite common that experimental conditions leading to the same response are not equivalent in what concerns their cost and/or practical operating difficulties. The user can therefore select the ‘best’ chromosome based also on practical aspects.

1.20.9 Applications After having described the general algorithm, now we devote our attention to the applications of GA to real problems in chemistry. Nowadays, GAs have become a rather general tool for optimization, and a bibliographic search would lead to thousands of hits illustrating their use (and sometimes their misuse). Since a complete review of the successful applications throughout more than one decade is clearly out of question, particular attention will be given to some ‘historically relevant’ papers and to some specific areas. It is well recognized that GAs are especially useful when the response hypersurface in which the optimum is searched is of high dimension and has many local optima. The high dimensionality of the problem makes the application of an exhaustive search impossible, while the presence of local optima makes direct optimization methods (e.g., steepest ascent) unreliable, since they can be caught by a local optimum and give a solution far away from the global optimum. These characteristics make GAs particularly suitable for molecular modeling, since the energetic hypersurface is very complex with several local minima. Therefore with the ‘standard’ methods the optimized structure can depend heavily on the chosen starting point. In the case of a medium-size protein (100 residues), if there are no constraints, the number of conformations can be approximated to (5 torsion angles per residue 5 likely values per torsion angle)100 ¼ 25100. In such a large search space, clearly beyond the capacity of any supercomputer, the GA is able to find suboptimal solutions. This means that, although it cannot be proven that the GA has really found the optimal solution, some of the results

642

Genetic Algorithms

obtained by the GA practically surpass any previously known solution. This can be of much help in nonpolynomial complete problems where no analytical solution of the problem is available, as shown by Schulze-Kremer.19 In 1992, Blommers et al.,20 applied a GA to derive the solution structure of the photodimer cis.syn-dUp[]dT. In 1993, McGarrah and Judson21 studied the behavior of the GA method as a global search technique for finding low-energy conformations. They evaluated different hybrid methods that combine coarse GA global search with local gradient minimization, and the effects of the GA parameters on the final results. In the same journal issue, Judson et al.,22 used a GA search procedure for finding low-energy conformations of small/ medium organic molecules (1–12 rotatable bonds) and compared the performances against the CSEARCH algorithm in Sybyl. They demonstrated that for molecules with more than eight rotatable bonds the GA method is more efficient computationally, and as the number of rotatable bonds increases, the relative efficiency of the GA method grows. Later, Jin et al.,23,24 efficiently elucidated backbone conformational features observed in the global minimum energy structure of a pentapeptide and searched the conformation space of [Met]-enkephalin. GAs had already been applied to the same molecule by Lee et al.25 Another field in which GAs are being applied is computer-aided molecular design (CAMD), with the goal of designing molecules with desired properties and activities, for which the traditional approach often requires a trial-and-error procedure involving a combinatorially large number of potential candidate molecules. A presentation of a GA in CAMD is given by Venkatasubramanian and Sundaram,26 while Venkatasubramanian et al.,27,28 apply a GA to polymer design. Other applications are reviewed by Devillers.29 Sundaram and Venkatasubramanian30 show very effectively that since the performance of a GA in CAMD is highly dependent on the parameter setting, an optimal or desirable setting can be defined only on a run-to-run basis. In CAMD the property prediction based on the structural characteristics of the molecular subunits is defined as the forward problem, while the inverse problem is the construction of a molecular structure given a set of desired macroscopic properties. De Weijer et al.,31 solve the forward problem by a neural network-based approach, while the application of a GA was found to be very effective in the inverse problem, that is, in finding hypothetical physical structures of polyethylene terephthalate yarns corresponding to a certain combination of mechanical and shrinkage properties. Jones et al.,32 reported three applications of GAs in chemical structure handling and molecular recognition. In the first application a GA was found to be highly efficient in searching the conformational space of small threedimensional molecules for pharmacophoric patterns, by searching substructures in databases of three-dimensional compounds. In the second, a GA for docking flexible ligands into partially flexible protein sites has been developed, while in the third a GA has been used to superimpose flexible molecules automatically. GAs can be successfully applied in curve fitting, as shown by De Weijer et al.33 They have a good search accuracy (they approach the globally optimal solution irrespective of the diverse starting conditions) and a poor search precision (they often fail in finding the very optimum solution), while the traditional local optimization techniques have a good search precision and a poor search accuracy. Therefore better overall performances can be attained with hybrid algorithms in which, for instance, the starting point for a local search method is given by the best GA solution. In the hybrid algorithm used by Hibbert,34 the GA is iterated with a steepest descent procedure, each providing a starting point for the other. Dane et al.,35 applied GAs to the curve fitting required by glancing incidence X-ray reflectometry, obtaining results better than those obtained with a currently used method, reducing the amount of human effort and expertise required for analyzing reflectivity measurements, and reducing the probability of overlooking feasible solutions.

1.20.10 A Specific Application: Variable Selection in Spectral Data Sets This section describes in detail a specific application, to show how the general GA previously described must be adequately modified to make it as suitable as possible to the real problem to which it will be applied.36 Spectral data consisting of hundreds and even thousands of absorbance values per spectrum can now be routinely collected in a matter of seconds. Methods such as partial least squares (PLS) or principal component


regression, being based on latent variables, allow one to take into account the whole spectrum without having to perform a previous feature selection.37,38 Owing to their capability of extracting the relevant information and producing reliable models, these full-spectrum methods were considered to be almost insensitive to noise. Therefore, it was at first commonly stated that no feature selection was required.38 In a following time, it has instead been gradually recognized that an efficient feature selection can be highly beneficial, both to improve the predictive ability of the model and to greatly reduce its complexity.39 In relatively simple cases, referring to materials with a limited number of components, spectroscopists can select some regions according to the knowledge of where these components are spectroscopically active and probably follow the Lambert–Beer law. When analyzing much more complex materials, the wavelength selection based on spectroscopic considerations is much more difficult. The most important reason for that is that some components of the material can be unknown; furthermore, even in the case of known components, their spectral fingerprint can be changed by variable experimental conditions (e.g., temperature). Even if all the components (and their spectral fingerprints) are known, the relevant overlapping of the different fingerprints, the physical and chemical interactions among the components, and other sources of deviations from the Lambert–Beer law can make this kind of selection extremely difficult. Computer-aided variable selection is important for several reasons. Variable selection can improve model performance, provide for more robust models, and models that can transfer more readily and allow nonexpert users to build reliable models with only limited expert intervention. Furthermore, computer-aided selection of variables may be the only approach for some models, for example, predicting physical properties from spectral data. The selection of variables for multivariate calibration can be considered an optimization problem. GAs applied to PLS have been shown to be very efficient optimization procedures. They have been applied to many spectral data sets and are shown to provide better results than full-spectrum approaches.40 The major concern with using GAs is the problem of overfitting. This problem has been addressed using a randomization test.41,42 For spectral data sets the main goal of variable selection is the elimination of noise, together with the reduction of the complexity of the model. It is also to be realized that spectra have another very relevant peculiarity; the spectral variables have a very high autocorrelation (i.e., variable n is very much correlated with variables n 1 and n þ 1). Hence, spectroscopists will never consider single wavelengths. Instead, their analysis will be in terms of spectral regions, each spectral region being formed by a certain number of adjacent variables. Therefore, when comparing different subsets of variables having the same predictive ability, one should take into account not only the number of retained variables, but also how many spectral regions are involved and how well they are defined. One possible application of variable selection in spectral data sets is the setting of filter spectrometers, these instruments being much cheaper and faster than full spectrum instruments.43 It is easy to understand that in such cases the results of a variable selection are of practical use if only a small number of regions are detected. Last but not the least, the identification of spectral regions clearly involved in the modeling of the response can be of a great help in the interpretation of the spectra,44 because the spectroscopists will have very clear indications of which spectral features are the most relevant to the problem under study. The procedure of variable selection, apparently so simple, is indeed very complicated and needs a careful validation to avoid the risk of overestimating the predictive ability of the selected model. This is mainly due to random correlations and is higher when the variables/objects ratio is very high. In the case of spectral data sets, this problem can be limited by reducing the number of variables. Owing to the relevant autocorrelation, the content of information of a new variable constructed from the average of a small number of adjacent original variables is very similar to that of the original variables themselves. The risk of overfitting is also higher the longer the GA runs (i.e., the more models that are tested); a good solution consists of performing a large number of independent short runs, and obtaining the final model by taking into account the results of all the runs. By doing this, a much more consistent (and less overfitted) solution can be obtained. The coding is very simple: In a data set in which each object is described by v variables, each chromosome is composed by v genes, each gene being formed by a single bit.45 As an example, with 10 variables, a set that only uses variables 1, 5, 8, and 9 will be coded as 1000100110. For a multivariate calibration problem, the response will be the cross-validated variance explained by the selected set of variables.

644

Genetic Algorithms

While most GAs are intended to work in a continuous space, in our case the space under investigation is absolutely discontinuous: when working with v variables, it is like studying only the vertices of a v-dimensional hypercube. Several changes have thus been applied to the simple genetic algorithm (SGA) to best adapt it to this specific purpose. It has to be remembered that there is no ‘ideal’ GA structure; this heuristic is applicable to all sorts of problems. Therefore, the success in solving a particular problem is strictly dependent on how well the GA has been adapted to the problem itself, with a good balance between exploration and exploitation. If the search is too unbalanced toward exploration, the GA will become very similar to a random search. In contrast, if exploitation would strongly prevail, then we would obtain something very similar to a ‘classical’ algorithm that finds a nearby optimum solution. In the case of variable selection, it is better to emphasize exploitation. This produces good results within a rather limited number of evaluations and therefore reduces as much as possible the risk of overfitting. 1.20.10.1 Initiation of Population According to the SGA the value of each bit of the initial population is determined by the ‘toss of a coin’. Under this hypothesis an average of 50% of the variables would be selected in every initial chromosome. Such a situation would lead to two main disadvantages:

the computation of the response for a model containing a higher number of variables requires a much longer time; since PLS has a very low sensitivity to noise, the presence of a few ‘good’ variables in a chromosome (this will almost always happen if half of the variables are selected) may be enough to produce a good response, irrespective of the presence of some ‘bad’ variables; this will mean that the responses of different chromosomes will be very similar and therefore it would be much more difficult to get rid of the ‘bad’ variables.

By setting the probability of selection to n/v, n variables will be selected on average out of the v original variables. The selection of a much smaller number of variables will result in a much faster elaboration time and a greater ease in the detection of the relevant variables. This means that during the first stages ‘bad’ variables can be more easily discarded, since one of them can be enough to worsen the response of a chromosome in which only a few ‘good’ variables are selected. Within each run, the combination of these small, highly informative ‘building blocks’ will lead to a gradual increase in the number of selected variables. To avoid chromosomes that produce models containing too many variables, the maximum number of possible selected variables is also set. This constraint will also be enforced in subsequent phases.

1.20.10.2 Reproduction and Mutation To avoid any influence of the order of the variables, an unbiased uniform crossover is used. Since this mating operator promotes exploration and the goal is to emphasize exploitation, it is used within a nongenerational algorithm. Mutation is then applied to the generated offspring. Then, the response of the two offspring is evaluated so that they can replace certain members of the population.

1.20.10.3 Insertion of New Chromosomes The quality, or fitness, of a subset of variables is determined both by the response it gives and by the number of features it uses. Thus it is rather important to know the best result obtained by using a certain number of variables. This chromosome is highly informative and deserves to be saved, regardless of its absolute position in the ranking of the responses. To do this, such a chromosome is ‘protected’ and cannot be eliminated from the population; its particular condition will end when another chromosome, using at most this number of features, gives a better result.


After evaluating the response of each offspring, it has to be decided whether to insert it into the population and, if so, what chromosome of the population to discard. As stated before, a chromosome using v variables is protected when it gives the best response among all the chromosomes using at most v variables. If the new chromosome is a protected one, then it will be a member of the population and the worst nonprotected chromosome will be eliminated. If the new chromosome is nonprotected, then it will be a member of the population only if its response is better than the response of the worst nonprotected chromosome. In this case, the worst nonprotected chromosome will be eliminated. By doing this, at each moment the population is composed of the best absolute chromosomes and of those chromosomes that are highly informative, since they give the best result with a limited number of variables.

1.20.10.4 Influence of the Different Parameters Four parameters (population size, probability of initial selection, maximum number of variables, and probability of mutation) have to be defined. The study of the influence of these parameters, performed on a very large number of data sets, showed that it is possible to obtain an architecture that is always very valid. This is very important from the practical point of view, since it means that a parameter optimization for each new data set is not required. The lower the population size, the lower the exploration and the higher the exploitation, since the bias toward the best chromosomes being selected for crossover is greater; a population size of 30 chromosomes is a good compromise. Furthermore, a nongenerational algorithm means that once a good chromosome has been found, it immediately enters the population, and can then be immediately picked up as a parent without having to wait for the evaluation of the other chromosomes of its same generation. A low value of the probability of initial variable selection allows one to explore many more combinations in the same amount of time, resulting in information that is much easier to interpret. The algorithm itself will build more complex combinations through the reproduction phase. A good value is the one that selects an average of five variables per chromosome (i.e., 5/v, v being the number of variables). The higher the number of selected variables, the higher the model complexity, and therefore the higher the time required for the evaluation of the response. A maximum value of 30 variables allows the algorithm to obtain very good models without making the computations too heavy. The mutation step allows one to avoid deadlock situations and to ‘jump’ to new zones of the space. A very high value of the probability of mutation disrupts the configuration of the chromosomes too much, with a low probability of obtaining a good result, while a very low value does not give this step the importance it deserves. A good compromise is the probability of 0.01 per gene.

1.20.10.5 Check of Subsets Since the number of variables contained within each chromosome can be from one to the maximum allowed, it is possible that one chromosome represents a subset of another. This is the case when the variables selected by chromosome c2 are a subset of the variables selected by chromosome c1. If c2 has a response higher than c1, the extra variables present in c1 (but not present in c2) bring no information, and simply represent noise. By keeping chromosome c1 in the population, we would reduce the diversity of the population (and therefore the exploration capability of the GA) without adding any supplementary information. As a consequence, chromosome c1 is discarded. The performance of the GA is improved by alternating (hybridizing) it with cycles of backward stepwise selection, which is one of the most commonly used techniques for variable selection. The backward elimination is performed on the best chromosome that has not yet undergone a stepwise selection and if the backward elimination results in a better chromosome, this new chromosome will replace the ‘original’ one. One cycle of backward elimination is performed every 100 evaluations and if the stop criterion is not a multiple of 100, a final cycle will also be performed.

646

Genetic Algorithms

1.20.10.6 How to Reduce the Risk of Overfitting As previously stated, the presence of random correlations is surely the most important factor limiting a generalized and extensive use of GAs,41 and not taking it into account can lead to totally senseless models. For the same reason, the runs must be stopped very early, and a randomization test is used to determine when the runs have to be stopped to prevent overfitting. Since only a minor part of the very complex search domain can be explored in a single run, the results of different runs can be rather different and usually consist in models in which only a very few variables are present. This means that the advantage of using PLS is not fully exploited. Furthermore, two opposite events have to be taken into account; on the one hand nonrelevant variables can be occasionally retained in the final model, while on the other it can happen that some relevant variables are occasionally not included. As previously stated, it has to be kept in mind that overfitting is the greatest risk in applying a GA. This risk increases as the number of tested models within a single run increases, since the probability of finding a model whose good performance is such only by chance (i.e., due to random correlations) becomes greater. Crossvalidation is not a complete protection against overfitting, since the objects on which the performance of the models is tested are the same as those on which the variable selection is performed. This basic consideration greatly influences the architecture of the GA. All the parameters are set in such a way that the highest exploitation is obtained, thereby meaning that the goal of the algorithm is to have a very fast increase in the response and therefore to have a very good solution in the very early stages of the process. This is the reason why a nongenerational algorithm and a rather unusually limited population size have been applied. A drawback of this architecture is the fact that, since only a very small part of the domain is explored, the final result can be strongly influenced by the randomly generated original population, and therefore the variables selected in different runs can be substantially different. It is therefore worthwhile to perform a high number of different runs (e.g., 100) and to try to extract some ‘global’ information by taking into account the top chromosome of each run and computing the frequency with which each variable has been selected.46 The final model is obtained following a stepwise approach, in which the variables are entered according to the frequency of selections (i.e., in the model with n variables, the n most frequently selected variables are present), and evaluating the root mean square error in cross-validation (RMSECV) associated to each model. It can be noticed that usually the RMSECV decreases very fast, until it reaches a minimum or a plateau. A crucial point is the detection of the number of variables to be taken into account. The model corresponding to the global minimum usually retains a rather high number of variables and very often it has the lowest RMSECV only due to some overfitting, without being significantly better than several other models. It can be said that the best model is the most parsimonious model among all the models which is not significantly different from the global optimum. This approach generally leads to models having a slightly lower root mean square error in prediction (RMSEP) and a significantly higher ‘definition’ in terms of selected regions (less and/ or smaller regions) than the models corresponding to the global minimum of RMSECV. The following procedure is followed:44

detect the global minimum of RMSECV; by using an F test (p < 0.1, d.o.f. is number of samples in the training set, both in the numerator and in the denominator) select a ‘threshold value’ corresponding to the highest RMSECV, which is not significantly different from the global minimum; look for the solution with the smallest number of variables having an RMSECV lower than the ‘threshold value’.

1.20.10.7 How to Take into Account the Autocorrelation among the Spectral Variables The techniques of feature selection usually assume that there is no autocorrelation among the variables. While this is true in the case of nonspectral data sets, it does not hold in the case of spectral data. This means that if wavelength n is selected as relevant, wavelengths n 1 and n þ 1 should also have a high probability of being selected. The main feature of the previously described algorithm is the fact that, to further reduce the risk of overfitting, the final model is obtained from the results of 100 independent, very short GA runs, while usually the model is obtained from a single, very long run.


In this process, every single run actually starts from scratch without taking into account the results obtained in the previous runs. This approach, though ensuring complete independence of each run, is a waste of energy. Since the frequency with which the single wavelengths are selected in the best chromosome of each run can give valuable information about the relevance of the corresponding spectral region, the probability of each variable of being selected in the initial population is changed according to the number of times the same variable has been selected in the previous runs. This means that a new run could concentrate its efforts mainly on the most interesting regions, without anyway completely discarding the possibility of a global exploration. After the last run the plot of the frequency of selection may not be as smooth as one would expect from spectral data. Since it is not logical that the relevance of adjacent variables in a spectrum is very different, a smoothing by a moving average (window size 3) is also performed. The final model is obtained by the stepwise approach previously described in which the variables are entered according to the smoothed value of the frequency of selection. Owing to the previously reported modifications, the selected variables detect real spectral regions and the variability of the models obtained by several runs of the program on the same data set is quite limited. Unfortunately, it is always possible that some relevant spectral regions are not selected or, the other way round, some regions whose contribution is nonsignificant are included in the model. 1.20.10.8 How to Get the Final Model To further reduce the variability of the model, the whole procedure is repeated several times (at least five times). The regions selected by each repeat are then visually compared, together with the original spectrum. If a region has been selected in the majority of the processes it means that it is significant, while the regions selected a limited number of times can be said to have been selected mainly by chance. This sequence of operations cannot be fully automated, since the visual analysis comparing the regions selected by each trial with the spectrum is needed. In the case of very broad spectral features, it is possible that the regions selected by the different trials seem to be different, since they cannot be overlapped. If all of them refer to the same spectral feature, they should be considered as equivalent and therefore taken into account in the final model. The previously described procedure, though apparently quite complex, leads to a great improvement in the repeatability of the final model. Tests performed on several data sets showed that the models obtained by running the whole procedure several times are extremely similar in both the selected regions and the predictive ability; minor and nonsignificant differences can be found in the definition of the limits of the different regions. As a rule of thumb, it has been found that the performance of the algorithm decreases when more than 200 variables are used.40 This is due to the fact that a higher variables/objects ratio increases the risk of overfitting and also due to the fact that the size of the search domain becomes too great. However, the number of wavelengths measured in a spectral data set is much greater than this. An acceptable number of variables is obtained by dividing the original spectrum in windows of equal size, each of them made by a number of adjacent wavelengths corresponding to the value obtained by rounding at the upper integer the ratio between the number of wavelengths and 200. Each new variable will be given the value corresponding to the average of the x’s at these wavelengths. For instance, in the case of 1050 wavelengths, the width of the window will be 6 (1050/ 200 ¼ 5.25). One hundred and seventy-five new variables will be obtained, with variable 1 being assigned the average of the x’s of wavelengths 1–6, variable 2 being assigned the average of the x’s of wavelengths 7–12, and so on. (Note: to avoid confusion, from now on the original variables will be referred to as ‘wavelengths’, while the new variables obtained by previously described procedure will be referred to as ‘variables’.) This approach gives no problems at all with near-infrared spectra, being characterized by very broad peaks. Fourier transform-infrared (FT-IR) spectra instead have much thinner peaks, and therefore if the window size used for the averaging is not small enough it is possible that some fine features in the spectra are lost that are potentially important to the calibration model. On these data sets, the number of wavelengths can be reduced to the desired value by removing the least relevant ones by applying a different method. For instance, very good results have been obtained by backward interval PLS.47

648

Genetic Algorithms

The architecture of the GA for variable selection in multivariate calibration is the following:

Maximum number of x variables: 200 Population size: 30 chromosomes On average, five variables per chromosome in the original population Regression method: PLS Response: cross-validated % explained variance (five deletion groups; the number of components is determined by cross-validation) Maximum number of variables selected in the same chromosome: 30 Probability of mutation: 1% Maximum number of components: the optimal number of components determined by cross-validation on the model containing all the variables (no higher than 15) Number of runs: 100 Backward elimination after every 100th evaluation and at the end (if the number of evaluations is not a multiple of 100) Window size for smoothing: 3 Pretreatment: none Scaling: autoscaling

1.20.10.9 An Example As an example, the results on a data set of FT-IR spectra of polymer films are reported. In it, the amount of two additives (B and C) has been measured.44 The spectra have been recorded from 4012 to 401 cm1 with a step of 1.93 cm1, for a total of 1873 wavelengths. The samples were obtained from five production batches, with samples from batches 1–3 being used as calibration set and samples from batches 4 and 5 being used as validation set. For additive B, there are 42 calibration samples and 28 validation samples, while for additive C there are 109 calibration samples and 65 validation samples. Before the application of the GA, a pathlength correction has been applied. Since even under well-controlled situations the film thickness can vary slightly, a correction is made using a polymer peak in the spectrum. The pathlength normalization factor is computed as the average peak height between 2662 and 2644 cm1 (10 data points) minus a baseline value estimated as the average from 2459 to 2442 cm1 (10 data points). The pathlength normalization was computed in this manner because it is known that the peak height of the band at 2662–2644 cm1 is solely related to the polymer and is thus directly proportional to the film thickness. 1.20.10.9.1

Additive B Figure 1 shows the average spectrum of the samples containing additive B. The model based on the region selected by the experts (178 wavelengths, highlighted in the figure) gives an RMSEP of 54 (11 components). Figure 2 shows the regions that have been selected by GA (window size of two points). This model takes into account wavelengths from six different regions, for a total of 60 wavelengths and has an RMSEP of 48 (six components). Inside the region indicated by the experts, two much smaller subregions have been selected, indicating a refinement of where the information is contained. The other regions appear to be related to the polymer. It is known that the catalyst ‘health’ influences the state of this additive, and of course also influences the polymer produced. Therefore, it makes sense that polymer peaks would contribute to modeling this additive. 1.20.10.9.2

Additive C Figure 3 shows the average spectrum of the samples containing additive C (only the part of spectrum relevant to additive C is displayed). The model based on the region selected by the experts (37 wavelengths, highlighted in the figure) gives an RMSEP of 48 (12 components). Figure 4 shows the regions selected by GA (window size of two points). This model takes into account wavelengths from three different regions, for a total of 34 wavelengths and has an RMSEP of 47 (12 components). The second and the third region are both inside the boundaries of the region selected by the

This page intentionally left blank Genetic Algorithms 649

1.3 1.2 1.1 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3

200

400

600

800

1000

1200

1400

1600

1800

Figure 1 Data set additives, response additive B: average FT-IR spectrum and region selected by experts. Reproduced with permission from Leardi, R., Ed. Nature-Inspired Methods in Chemometrics: Genetic Algorithms and Artificial Neural Networks, 1st ed.; Elsevier: Amsterdam, 2003; p. 191 (ISBN-13: 978-0444513502). Copyright Elsevier (2003).

1.3 1.2 1.1 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3

200

400

600

800

1000

1200

1400

1600

1800

Figure 2 Data set additives, response additive B: average FT-IR spectrum and regions selected by GA. Reproduced with permission from Leardi, R., Ed. Nature-Inspired Methods in Chemometrics: Genetic Algorithms and Artificial Neural Networks, 1st ed.; Elsevier: Amsterdam, 2003; p. 192 (ISBN-13: 978-0444513502). Copyright Elsevier (2003).

experts and they perfectly correspond to two small peaks, while the valley between them has not been included. The first region corresponds very well to two small peaks and was also known to be related to the additive, though at the time of the original modeling it was not clear to the experts whether this region should be included or not.

650

Genetic Algorithms

1.4 1.3 1.2 1.1 1 0.9 0.8 0.7 0.6 0.5 0.4

1350

1400

1450

1500

1550

1600

1650

1700

Figure 3 Data set additives, response additive C: average FT-IR spectrum and region selected by experts (only the part of the spectrum related to additive C is shown). Reproduced with permission from Leardi, R., Ed. Nature-Inspired Methods in Chemometrics: Genetic Algorithms and Artificial Neural Networks, 1st ed.; Elsevier: Amsterdam, 2003; p. 193 (ISBN-13: 978-0444513502). Copyright Elsevier (2003).

1.4 1.3 1.2 1.1 1 0.9 0.8 0.7 0.6 0.5 0.4

1350

1400

1450

1500

1550

1600

1650

1700

Figure 4 Data set additives, response additive C: average FT-IR spectrum and regions selected by GA (only the part of the spectrum related to additive C is shown). Reproduced with permission from Leardi, R., Ed. Nature-Inspired Methods in Chemometrics: Genetic Algorithms and Artificial Neural Networks, 1st ed.; Elsevier: Amsterdam, 2003; p. 194 (ISBN-13: 978-0444513502). Copyright Elsevier (2003).

This example shows that a GA can select variables that provide good solutions in terms of both predictive ability and interpretability. When comparing the selected variables with the models proposed by the experts, it can be seen that the GA models contain the suggested bands or part of the suggested bands, plus additional bands. In some cases these


extra regions could be readily interpreted; in other cases they could not. It is logical to say that they contain relevant information since they decrease the RMSEP and they were consistently selected in independent GA elaborations. It has to be emphasized that this method does not require any spectroscopic experience by the user. Using this approach, a nonexpert will be able to efficiently construct reliable calibration models with little or no intervention by an expert. Further, this approach can aid the expert with difficult calibration problems where the variable selection is not obvious.

References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28.

29. 30. 31.

Forrest, S. Genetic Algorithms: Principles of Natural Selection Applied to Computation. Science 1993, 261, 872–878. Maddox, J. Genetics Helping Molecular Dynamics. Nature 1995, 376, 209. Goldberg, D. E. Genetic Algorithms in Search, Optimization and Machine Learning; Addison-Wesley: Berkeley, 1989. Leardi, R. Genetic Algorithms in Chemometrics and Chemistry: A Review. J. Chemom. 2001, 15, 559–569. Leardi, R., Ed. Nature-Inspired Methods in Chemometrics: Genetic Algorithms and Artificial Neural Networks, Vol. 23: Data Handling in Science and Technology. Elsevier: Amsterdam, 2003. Lucasius, C. B.; Kateman, G. Understanding and Using Genetic Algorithms: Part 1. Concepts, Properties and Context. Chemom. Intell. Lab. Syst. 1993, 19, 1–33. Lucasius, C. B.; Kateman, G. Understanding and Using Genetic Algorithms: Part 2. Representation, Configuration and Hybridization. Chemom. Intell. Lab. Syst. 1994, 25, 99–145. Shaffer, R. E.; Small, G. W. Learning Optimization from Nature: Simulated Annealing and Genetic Algorithms. Anal. Chem. 1997, 69, 236A–242A. Wehrens, R.; Buydens, L. M. C. Evolutionary Optimization: A Tutorial. Trends Anal. Chem. 1998, 17, 193–203. LIPS (Laboratory for Intelligent Process Systems), Purdue University. http://cobweb.ecn.purdue.edu. Schulze-Kremer, S. Genetic Algorithms and Protein Folding. Faculty of Technology, University of Bielefeld, 1996. http:// www.techfak.uni-bielefeld.de. Wikipedia. Genetic Algorithm 2007. http://en.wikipedia.org. Wong, T.; Wong, H.; Drossopoulou, S. Genetic Algorithms. Imperial College of Science Technology and Medicine, London, 1996. http://www.doc.ic.ac.uk. Rennard, J.-P. Introduction to Genetic Algorithms, 2000. http://www.rennard.org. Association for the Advancement of Artificial Intelligence. Genetic Algorithm and Genetic Programming, 2006. http:// www.aaai.org. Marczyk, A. Genetic Algorithms and Evolutionary Computation. The TalkOrigins Archive, 2004. http://www.talkorigins.org. Holland, J. Genetic Algorithms. In L. Tesfatsion homepage, Department of Economics, Iowa State University, 2007. http:// www.econ.iastate.edu. Hibbert, D. B. A Hybrid Genetic Algorithm for the Estimation of Kinetic Parameters. Chemom. Intell. Lab. Syst. 1993, 19, 319–329. Schulze-Kremer, S. Genetic Algorithms for Protein Tertiary Structure Prediction. In Parallel Problem Solving from Nature II; Maenner, R., Manderick, B., Eds.; North-Holland: Amsterdam, 1992; pp 391–400. Blommers, M. J. J.; Lucasius, C. B.; Kateman, G.; Kaptein, R. Conformational Analysis of a Dinucleotide Photodimer with the Aid of the Genetic Algorithm. Biopolymers 1992, 32, 45–52. McGarrah, D. B.; Judson, R. S. An Analysis of the Genetic Algorithm Method of Molecular Conformation Determination. J. Comput. Chem. 1993, 14, 1385–1395. Judson, R. S.; Jaeger, E. P.; Treasurywala, A. M.; Peterson, M. L. Conformational Searching Methods for Small Molecules. II. Genetic Algorithm Approach. J. Comput. Chem. 1993, 14, 1407–1414. Jin, A. Y.; Leung, F. Y.; Weaver, D. F. Development of a Novel Genetic Algorithm Search Method (GAP 1.0) for Exploring Peptide Conformational Space. J. Comput. Chem. 1997, 18, 1971–1984. Jin, A. Y.; Leung, F. Y.; Weaver, D. F. Three Variations of Genetic Algorithm for Searching Biomolecular Conformation Space: Comparison of GAP 1.0, 2.0 and 3.0. J. Comput. Chem. 1999, 20, 1329–1342. Lee, J.; Scheraga, H.; Rackovsky, S. New Optimization Method for Conformational Energy Calculation. Polypeptides: Conformational Space Annealing. J. Comput. Chem. 1997, 18, 1222–1232. Venkatasubramanian, V.; Sundaram, A. Genetic Algorithms: Introduction and Applications. In Encyclopedia of Computational Chemistry; Schleyer, P. V. R., Ed.; Wiley: New York, 1998; pp 1115–1127. Venkatasubramanian, V.; Chan, K.; Caruthers, J. M. Computer-Aided Molecular Design Using Genetic Algorithms. Comput. Chem. Eng. 1994, 18, 833–844. Venkatasubramanian, V.; Chan, K.; Caruthers, J. M. On the Performance of Genetic Search for Large-Scale Molecular Design. In Proceedings of the Fifth International Symposium on Process Systems Engineering (PSE’94). Kyongu: S. Korea, 1994, pp 1001–1006. Devillers, J. Genetic Algorithms in Computer-Aided Molecular Design. In Genetic Algorithms in Molecular Modeling; Devillers, J., Ed.; Academic Press: New York, 1996, pp 1–34. Sundaram, A.; Venkatasubramanian, V. Parametric Sensitivity and Search-Space Characterization Studies of Genetic Algorithms for Computer-Aided Polymer Design. J. Chem. Inf. Comput. Sci. 1998, 38, 1177–1191. De Weijer, A. P.; Lucasius, C. B.; Buydens, L. M. C.; Kateman, G. Using Genetic Algorithms for an Artificial Neural Network Model Inversion. Chemom. Intell. Lab. Syst. 1993, 20, 45–55.

652

Genetic Algorithms

32. Jones, G.; Willett, P.; Glen, R. C. Genetic Algorithms for Chemical Structure Handling and Molecular Recognition. In Genetic Algorithms in Molecular Modeling; Devillers, J., Ed.; Academic Press: New York, 1996, pp 211–242. 33. De Weijer, A. P.; Lucasius, C. B.; Buydens, L. M. C.; Kateman, G.; Heuvel, H. M.; Mannee, H. Curve Fitting Using Natural Computation. Anal. Chem. 1994, 66, 23–31. 34. Hibbert, D. B. A Hybrid Genetic Algorithm for the Estimation of Kinetic Parameters. Chemom. Intell. Lab. Syst. 1993, 19, 319–329. 35. Dane, A. D.; Veldhuis, A.; de Boer, D. K. G.; Leenaers, A. J. G.; Buydens, L. M. C. Application of Genetic Algorithms for Characterization of Thin Layer Materials by Glancing Incidence X-Ray Refractometry. Physica B 1998, 253, 254–268. 36. Leardi, R. Genetic Algorithm-PLS as a Tool for Wavelength Selection in Spectral Data Sets. In Nature-Inspired Methods in Chemometrics: Genetic Algorithms and Artificial Neural Networks, Vol. 23: Data Handling in Science and Technology; Leardi, R., Ed.; Elsevier: Amsterdam, 2003; pp 169–196. 37. Geladi, P.; Kowalski, B. R. Partial Least Square Regression: A Tutorial. Anal. Chim. Acta 1986, 185, 1–17. 38. Thomas, E. V.; Haaland, D. M. Comparison of Multivariate Calibration Methods for Quantitative Spectral Analysis. Anal. Chem. 1990, 62, 1091–1099. 39. Thomas, E. V. A Primer on Multivariate Calibration. Anal. Chem. 1994, 66, 795a–804a. 40. Leardi, R. Application of Genetic Algorithm-PLS for Feature Selection in Spectral Data Sets. J. Chemom. 2000, 14, 643–655. 41. Jouan-Rimbaud, D.; Massart, D. L.; de Noord, O. E. Random Correlation in Variable Selection for Multivariate Calibration with a Genetic Algorithm. Chemom. Intell. Lab. Syst. 1996, 35, 213–220. 42. Leardi, R. Genetic Algorithms in Feature Selection. In Genetic Algorithms in Molecular Modelling; Devillers, J., Ed.; Academic Press: London, 1996, pp 67–86. 43. Lestander, T. A.; Leardi, R.; Geladi, P. Selection of NIR Wavelengths by Genetic Algorithms for Determination of Seed Moisture Content. J. Near Infrared Spectrosc. 2003, 11, 433–446. 44. Leardi, R.; Seasholtz, M. B.; Pell, R. Variable Selection for Multivariate Calibration Using a Genetic Algorithm: Prediction of Additive Concentrations in Polymer Films from Fourier Transform-Infrared Spectral Data. Anal. Chim. Acta 2002, 461, 189–200. 45. Leardi, R.; Boggia, R.; Terrile, M. Genetic Algorithms as a Strategy for Feature Selection. J. Chemom. 1992, 6, 267–281. 46. Leardi, R.; Lupiań˜ez Gonza´lez, A. Genetic Algorithms Applied to Feature Selection in PLS Regression: How and When to Use Them. Chemom. Intell. Lab. Syst. 1998, 41, 195–207. 47. Leardi, R.; Nørgaard, L. Sequential Application of Backward Interval PLS and Genetic Algorithms for the Selection of Relevant Spectral Regions. J. Chemom. 2004, 18, 11, 486–497.


Biographical Sketch

Riccardo Leardi was born in 1959 and graduated in Pharmaceutical Chemistry and Technology in 1983. Since then he has been working in the section of Analytic Chemistry of the Department of Pharmaceutical and Food Chemistry and Technologies, Faculty of Pharmacy, University of Genova, and his research field is Chemometrics. His interests are mainly devoted to problems of classification and regression (applied especially to food, environmental, and clinical data), experimental design, process optimization, three-way methods, multivariate process control, and multivariate quality control. His research focused mainly on genetic algorithms, and especially in their application to the problem of variable selection. He is author of more than 90 papers and more than 90 communications in national and international meetings; he is Review Editor of Journal of Chemometrics and Editorial Adviser of Analytica Chimica Acta, and a regular reviewer for several journals. Since November 2002 he has started his activity of chemometric consultancy.


Index to Volume 1 The index is in letter-by-letter order, whereby hyphens and spaces within index headings are ignored in the alphabetization, and it is arranged in set-out style, with a maximum of four levels of heading. Page numbers suffixed by f or t refer to figures or tables respectively.

A acetone, 460t acetonitrile, 460t acetophenone, 460t adjusted boxplots, 194f, 194 air pollution study, 177f, 178f, 176 algorithmic statistical process control (ASPC), 112 alias matrix, 263f, 262t, 261 alternating conditional expectation (ACE) algorithm, 175 analysis of variance (ANOVA) residual verification methods, 136t, 139t validation approaches basic concepts, 45 data arrangement, 46t fixed effects model basic concepts, 47t, 48t, 46 means comparison data, 52t, 51 Newman–Keuls procedure, 51t, 52t, 51 orthogonal contrasts, 50t, 50 power of the ANOVA, 48 results analysis, 48t uncertainty tests, 49t, 49 random effects model confidence interval estimation, 54 general discussion, 53 laboratory applications, 57f power of the ANOVA, 53 robust analysis, 62t, 62 analytical measurement validation approaches, 17–76 analysis of variance (ANOVA) basic concepts, 45 data arrangement, 46t fixed effects model, 47t, 48t, 46 means comparison data, 52t, 51 Newman–Keuls procedure, 51t, 52t, 51 orthogonal contrasts, 50t, 50 power of the ANOVA, 48 random effects model, 53 uncertainty tests, 49t, 49

basic concepts general discussion, 18 mean and variance measure, 19f, 20t, 21f, 21t, 18 precision, 21f, 21t, 21 probability density function (PDF), 19f, 20t, 21f, 21t, 18–19 trueness, 21f, 21t, 21 confidence intervals basic concepts, 23 difference in two means, 26 general discussion, 23 joint confidence intervals, 28 mean of a normal distribution, 23 median measures, 28 ratio of variances of two normal distributions, 27 variance of a normal distribution, 25 hypothesis testing Bartlett test, 62t, 42 basic concepts, 31 chi-square ( 2) test, 44t, 43 Cochran test, 42t, 62t, 41 D’Agostino normality test, 45t, 44 difference in two means, 37t, 36 general discussion, 31 goodness-of-fit tests, 43 independent variances, 42f, 41t, 41 interval-based tests, 38 Levene test, 62t, 43 mean measures, 33t mean of a normal distribution, 35t, 34 parametric tests, 33t ratio of variances of two normal distributions, 40t, 39 simultaneous behavior analysis, 34f, 33 two one-sided test (TOST) procedure, 39 Type I/Type II errors, 32t, 31–32 variance of a normal distribution, 33t, 36 statistical inference accuracy measures, 63 consistency analysis, 58 data analysis, 58

655

656

Index to Volume 1

analytical measurement (continued ) data elimination, 58–59 Grubbs’ test, 59t, 60t, 59–60 Huber estimator, 61f, 61 Plackett–Burman exploratory design, 64t, 65t, 63 precision, 55 robust analysis, 60t, 62t, 61 ruggedness testing, 63 trueness, 54 tolerance intervals basic concepts, 29 -content tolerance intervals, 29 -expectation tolerance intervals, 29 general discussion, 23 independent tolerance intervals, 30 uncertainty tests fixed effects model, 49 means comparison data, 52t, 51 Newman–Keuls procedure, 51t, 52t, 51 orthogonal contrasts, 50t, 50 Type II errors, 49t anisole, 460t A-optimality criterion response surface methodology (RSM), 351 supersaturated designs, 264 Armijo condition, 526 ascorbic acid testing, 152t, 152 As Pontes, Corunã, Spain, 182f, 183f, 181 augmented Lagrangian method, 533 average run length (ARL), 106 Avicel PH 101 component proportions, 432t crushing strength, 436f, 442f, 443f, 430 design coordinates/checkpoints, 432f design point values, 433t disintegration time, 437f, 442f, 443f, 430 drug release, 438f, 442f, 443f, 430 global desirability function, 439 mean diameter, 437f, 442f, 443f, 430 multicomponent systems, 430 porosity, 436f, 442f, 443f, 430 reliability study, 441f, 440 sensitivity case study, 442f, 443f, 441 variance prediction, 433f

B Bandurek supersaturated matrix, 267t, 266 Bartlett test, 62t, 42, 48, 138 Bayesian statistical analysis, 213–245 axiomatic foundations decision theory, 217 general discussion, 216 probability and conditional uncertainty general discussion, 216 physical constant values, 217 prediction methods, 217 probabilistic diagnoses, 216 proportion estimation, 216 regression analysis, 217 background information, 214 basic concepts coherence, 242 generality, 243 objectivity, 242 operational meaning, 243

inference summaries basic concepts, 236 estimation basic concepts, 236 conventional estimation, 237 interval estimation, 238 intrinsic estimation, 237 point estimation, 236 hypothesis testing, 239 methodology asymptotic behavior, 226 Bayes’ theorem, 219 learning process basic concepts, 219 domain restrictions, 223f, 222 nuisance parameters, 221 prior/posterior densities plot, 221f, 223f predictive distributions, 223 regression analysis basic concepts, 225 linear modeling, 226f, 225 notation guidelines, 214 reference analysis basic concepts, 228 frequentist analysis general discussion, 234 interval estimation, 235 point estimation, 234 reference distributions Jeffreys’ prior, 230 limited data, 233 multi-parameter reference priors, 233 nuisance parameters, 231 one-parameter reference priors, 231f, 229 Stein’s paradox, 233 Beer–Lambert law, 642 benzaldehyde tests, 157t, 158f, 157 benzene, 460t benzonitrile, 460t benzyl alcohol, 460t, 465t BFGS (Broyden–Fletcher–Goldfarb–Shanno) method, 522 black smoke study See air pollution study blasthole sampling, 7f, 6 Blomqvist’s median correlation coefficient, 204 Bonferroni inequality, 28 bootstrap validation methods, 171–187 generalized additive model (GAM) air pollution study, 177f, 178f, 176 confidence interval estimation, 176 fitting procedures, 175 generalized linear models (GLMs), 174 interactions basic concepts, 179 environmental applications, 182f, 183f, 181 estimation procedures, 180 multivariate adaptive regression splines (MARS), 179 hypothesis testing basic concepts, 180 environmental applications, 182f, 183f, 181 regression models, 173 resampling approaches asymptotic approaches, 172 basic concepts, 171 inferential methods, 171 B-optimality criterion, 264

Index to Volume 1 box and whisker plots, 67f, 66 Box–Behnken designs, 373t, 371, 470, 470t bromobenzene, 460t Bunch–Parlett factorization, 538 butanol, 460t, 467t

C calibration models definitions, 128 linear regression basic concepts, 129 calibration design, 151t, 150f, 148 correlation coefficients, 133 least squares (LS) method, 130f, 129 maximum likelihood estimation (MLE), 132f, 131 random errors, 131 statistical validation methods functional models analysis of variance (ANOVA), 136t, 139t lack of fit test, 136t, 139t, 135 significance tests (F-tests), 134t, 134 inferential methods cadmium calibration data, 138t, 139f, 139t equal variances (homoskedasticity), 138 normal distribution, 139f, 138 residual verification, 138 serial correlation, 138 residual verification general discussion, 136 graphical methods, 137f, 139f, 137 inferential methods, 138 univariate regression, 127–169 advantages/disadvantages, 164t, 165f, 164 calibration design, 148 capability of detection, 152t, 151 capability of discrimination, 153 confidence intervals analyte concentration determination, 147t, 145 correlation coefficients, 141t degree of linearity, 141t, 141 degrees of freedom, 141t general discussion, 140 intercept/slope/residual variance, 142f, 140 joint confidence region, 142f, 141 prediction models, 145f, 144 significance levels, 140 weighted least squares method, 159f, 154 decision limit, 152 error sources, 165f, 163 generalized least squares method, 154 hypothesis testing general discussion, 140 intersection points, 147 null hypothesis tests, 140 two regression lines, 142 least median of squares (LMS), 161t, 162f, 163f, 160 linear regression, 129 robust regression methods, 159 standard additions method, 153f, 153 statistical validation methods, 134 variance inflation factor (VIF), 151t, 150f, 148–149 weighted least squares method, 157t, 158f, 159f, 154 carbon tetrachloride, 460t, 465t Cauchy point, 529 central limit theorem, 70 chi-square ( 2) fitting basic concepts, 68t, 68 computational aspects, 71, 72

657

hypothesis testing, 44t, 43 response surface methodology (RSM), 363 statistical validation methods, 138 chlorobenzene, 460t chloroform, 460t, 467t Cholesky factorization, 521 chromatographic analysis gas chromatography (GC) gas chromatography–flame ionization detection (GC–FID) method, 567 simplex optimization methods, 566 liquid chromatography (LC) high-performance liquid chromatography (HPLC), 566, 121 multivariate statistical process control (MSPC), 121 cineole, 460t classical least squares (CLS) method robust linear regression basic concepts, 197 outlier map, 200f residual plots, 199f scatter plots, 198f Cochran’s theorem, 36–37 Cochran test, 42t, 62t, 41, 138 computer-aided molecular design (CAMD), 641 computer-processable molecular codes, 3 concordance principle, 606 confidence intervals univariate regression analyte concentration determination, 147t, 145 correlation coefficients, 141t degree of linearity, 141t, 141 degrees of freedom, 141t general discussion, 140 intercept/slope/residual variance, 142f, 140 joint confidence region, 142f, 141 prediction models, 145f, 144 significance levels, 140 weighted least squares method, 159f, 154 validation approaches air pollution study, 177f, 178f, 176 basic concepts, 23 difference in two means, 26 general discussion, 23 generalized additive model (GAM), 176 joint confidence intervals, 28 mean of a normal distribution, 23 median measures, 28 random effects model, 54 ratio of variances of two normal distributions, 27 response surface methodology (RSM), 362 variance of a normal distribution, 25 constrained optimization methods active set methods general discussion, 534 Karush–Kuhn–Tucker (KKT) conditions, 534 linear programming, 535f, 534 quadratic programming, 537 sequential quadratic programming (SQP), 542t, 539 simplex methods, 536 augmented Lagrangian method, 533 basic concepts general discussion, 531 Karush–Kuhn–Tucker (KKT) conditions, 532 Lagrangian function, 531 linear independence constraint qualification (LICQ), 531 general discussion, 542 logarithmic barrier method, 534

658

Index to Volume 1

constrained optimization methods (continued ) quadratic penalty function, 532 Crosier’s formula, 406 CUSUM (cumulative sum) control charts basic concepts, 104 illustrative example, 106 individuals control chart, 107f cyclohexanol, 460t cyclohexanone, 460t cyclopentanone, 460t

D D’Agostino normality test, 45t, 44 degrees of freedom, 141t Derringer’s desirability funcitons, 562–566 DFP (Davidon–Fletcher–Powell) algorithm, 522 dichlorobenzene, 460t dichloroethane, 460t dichloromethane, 460t diethylcarbinol, 460t diethyl carbonate, 460t diethylketone, 460t diglycerides, 445t, 447f, 448f diisopropyl ether, 460t, 467t diisopropyl ketone, 465t dimethyl acetamide, 460t dimethyl formamide, 460t dimethyl sulfoxide, 460t, 465t di-n-butyl ether, 460t dioxane, 460t, 465t diphenyl ether, 460t distribution function, 65 Doehlert designs continuous factor–discrete factor designs, 473f continuous factor–mixture factor designs, 486f ellipsoidal regions, 444t, 445f mixed sequential–simultaneous optimization methods, 572 referential mixture designs, 431t, 430f, 429 response surface methodology (RSM), 371f, 371t, 372t, 370 D-optimality criterion constrained mixture experimental studies, 426t, 427f, 428f, 429f, 423 continuous factor–mixture factor designs, 487t multifactor screening study, 467t response surface methodology (RSM), 351 solvent effects study, 464t, 465t, 466t supersaturated designs, 264 Durbin–Watson test, 138

E ELECTRE function ranking method concordance matrix, 610t discordance matrix, 611t ELECTRE I, 611f, 610t, 611t, 609 ELECTRE II, 612t, 611 ELECTRE III, 612 ELECTRE IV, 614 ELECTRE TRI, 614 general discussion, 609 normalized data, 610t ORESTE average ranks, 615t, 616t basic concepts, 614 decision-making process, 618f distance derivations, 616t dominance graph, 618f linear orthogonal projections, 616t

normalized preference intensities, 617t position matrix, 615f empirical modeling, 348 energy of the lowest unoccupied molecular orbital (ELUMO) correlation measures parametric and nonparametric models, 195f, 195 robust correlation and covariance, 197f, 195 empirical distribution function, 207f, 206 kernel density estimators, 207f locally weighted scatter plot smoothing (LOWESS) method, 203f, 202 location–scale estimation models, 191t, 191–192 robust linear regression, 198f, 197 engineering process control (EPC), 111 engineering statistical process control (ESPC), 112 E-optimality criterion response surface methodology (RSM), 351 supersaturated designs, 264 ethanol mixture studies, 444t properties, 460t ethyl acetate, 460t ethylmethylketone, 460t evolution theory (genetic algorithms) applications molecular models, 641 spectral analysis, 642 background information, 633 basic guidelines general discussion, 635 mutation process, 638 original population, 635 reproduction process, 635 coding processes, 633 general discussion, 632 global maximum, 641 hybrid algorithms, 640 parameters general discussion, 639 mutation probability, 640 population size, 639 reproduction process, 640 EWMA (exponentially weighted moving average) control charts basic concepts, 105 illustrative example, 106 individuals control chart, 107f experimental conditions, 301–344 chemometric analysis, 302 current research areas, 501 error sources, 307 factorial designs catalytic hydrogenation design and experimental response, 324t experimental variables, 323t interaction plots, 325f least squares (LS) fitting technique, 323f, 323 response surface projection diagrams, 326f definitions, 310 enamine reduction techniques 22 factorial designs, 311t, 311f, 310f, 310 effects computations, 313f, 313 estimated effects significance, 318t, 318 response model analysis, 322f, 321t fractional factorial designs 23-1 fractional factorial designs, 330t, 331t, 330 23 factorial designs, 341t, 340 25-1 fractional factorial designs, 337 25-2 fractional factorial designs, 332 block-run experiments, 341t, 340

Index to Volume 1 complementary fractions, 333 confounding pattern analysis, 337t, 331 construction procedures, 330f, 329 fold-over designs, 336t, 337t, 335 limitations, 329 resolution techniques, 333 Willgerodt–Kindler reaction, 339f, 337f, 338t, 337 least squares (LS) fitting technique 24 factorial designs, 323 advantages, 322 estimated model parameter quality, 326f, 328f, 325 joint confidence region, 328f, 327 probability distributions, 326f response model analysis, 319 second-order interaction models, 323f, 323 olefin rearrangement 23 factorial designs, 312t, 311 effects computations, 314f, 314 sign tables, 317t two-level factorial designs (2k) 22 factorial designs, 311t, 311f, 320t, 310f, 310 23 factorial designs, 312t, 311, 341t, 340 24 factorial designs, 312t, 312, 323 effects computations, 313 error variance calculations, 319 estimated effects significance, 318t, 317 general discussion, 310 multifactor generalizations, 315 response model analysis, 322f, 320t, 321t, 319 sign tables, 317t, 316 Yates’ algorithm, 316t, 316 general discussion, 342 goals and objectives, 302 Ishikawa diagrams, 304f, 303, 304f non-classical experimental designs, 453–499 Box–Behnken designs, 470, 470t combined designs continuous factor–discrete factor designs, 471 continuous factor–mixture factor designs, 484 general discussion, 471 constraint-based experimental design study, 470f, 469t, 469, 470t criteria type I criteria, 465f, 468f, 457 type II criteria, 458f, 465f, 457 type III criteria, 459 type IV criteria, 459 general discussion, 453, 494 methodology, 249 information of sufficient quality, 455 most economical set, 455 multifactor screening study D-optimality criterion, 467t experimental design, 469t experimental matrices, 468t general discussion, 466 type I criteria, 468f solvent effects study checkpoint values, 467t D-optimality criterion, 464t, 465t, 466t experimental matrices, 462t general discussion, 460 optimal experimental design, 466f, 465t properties, 460t type I criteria, 465f type II criteria, 465f variable transformations, 462t problem definition, 247–250 factor analysis, 248

general discussion, 302 mapping techniques, 303 methodology, 247 mixture studies, 248 response surface methodology (RSM), 248 ruggedness testing, 303 screening strategies, 303, 248 randomization approaches, 307 response variation models experimental factor effects, 307 first-order linear models, 309 second-order interaction models, 309 second-order quadratic models, 309 Taylor expansion approximation, 307 significant variables, 306 symbols, 503 terminology experimental design, 502 experimental domain, 503 experimental domain of interest, 503 experimental plan, 503 experimental response, 502 experiments, 502 factors, 502 validation approaches, 306f, 306

F factor analysis PARAFAC (parallel factor analysis), 511 principal component analysis (PCA) multivariate statistical process control (MSPC) exploratory data analysis, 116 illustrative example, 122f, 121 offline process monitoring, 116 online process monitoring, 119 numerical optimization, 511 F distribution, 69t, 69, 71 Fibonacci numbers, 553t, 552f, 551, 554f finite-difference Newton’s method of optimization, 513 Fisher’s information matrix, 232 flow injection analysis (FIA), 567t, 568f, 567 fluorobenzene, 460t fluticasone propionate, 567t, 568f, 567 Fourier transform infrared (FTIR) microscopy near-infrared (NIR) spectral analysis general discussion, 647 polymer film data set additive B, 649f, 648 additive C, 650f, 648 general discussion, 648 Frobenius norm, 523 F-tests significance tests, 134t, 134 validation approaches, 56, 72 fundamental sampling error (FSE), 4

G gas chromatography (GC) gas chromatography–flame ionization detection (GC–FID) method, 567 simplex optimization methods, 566 Gauss–Markov optimality, 155 generalized additive model (GAM) bootstrap validation methods air pollution study, 177f, 178f, 176 confidence interval estimation, 176 fitting procedures, 175

659

660

Index to Volume 1

generalized additive model (GAM) (continued ) generalized linear models (GLMs), 174 interactions basic concepts, 179 environmental applications, 182f, 183f, 181 estimation procedures, 180 multivariate adaptive regression splines (MARS), 179 generalized least squares method, 154 generalized linear models (GLMs), 174 generalized standard deviation (GSD) optimality, 351 genetic algorithms (GAs), 631–653 applications molecular models, 641 polymer film data set analysis additive B, 649f, 648 additive C, 650f, 648 general discussion, 648 spectral analysis autocorrelation processes, 646 chromosome insertion, 644 final model architecture, 647 general discussion, 642 initial population, 644 mutation probability, 644 overfitting controls, 646 parameter influences, 645 polymer film data set, 648 reproduction process, 644 subset analysis, 645 background information, 633 basic guidelines general discussion, 635 mutation process, 638 original population, 635 reproduction process, 635 coding processes, 633 complex systems, 631 evolution theory, 632 global maximum, 641 hybrid algorithms, 640 parameters general discussion, 639 mutation probability, 640 population size, 639 reproduction process, 640 screening strategies, 270 Gini index, 204 goal programming, 625 G-optimality, 352 Grubbs’ test, 59t, 60t, 59–60

H Hadamard matrices multifactor screening study D-optimality criterion, 467t experimental design, 469t experimental matrices, 468t general discussion, 466 type I criteria, 468f saturated designs, 255t, 254 supersaturated designs Bandurek supersaturated matrix, 267t, 266 basic concepts, 264 Nguyen supersaturated matrix, 266t, 265 Tang–Wu supersaturated matrix, 266t, 265 hammer cross-belt samplers, 9f, 7 Harrington desirability scale, 604t, 603

Hasse diagram technique (HDT) basic concepts, 619 dissimilar-acting substances toxicity values, 623f, 621t phenylureas toxicity values, 622f, 621t sorted alternatives, 619f typical diagram, 620f heptanol, 460t Hessian matrix, 518 heteroskedastic components, 137f, 137 hexanol, 460t high-performance liquid chromatography (HPLC), 566, 121 homoskedastic components, 137f, 137 Hotelling T2 control charts general discussion, 113 individual observations, 113 subgroup data, 113 Huber estimator, 61f, 61, 191t, 193 hydropropylmethylcellulose (HPMC) component proportions, 432t crushing strength, 436f, 442f, 443f, 430 design coordinates/checkpoints, 432f design point values, 433t disintegration time, 437f, 442f, 443f, 430 drug release, 438f, 442f, 443f, 430 global desirability function, 439 mean diameter, 437f, 442f, 443f, 430 multicomponent systems, 430 porosity, 436f, 442f, 443f, 430 reliability study, 441f, 440 sensitivity case study, 442f, 443f, 441 variance prediction, 433f hypothesis testing Bayesian statistical analysis, 239 bootstrap validation methods basic concepts, 180 environmental applications, 182f, 183f, 181 validation approaches Bartlett test, 62t, 42 basic concepts general discussion, 31 parametric tests, 33t simultaneous behavior analysis, 34f, 33 Type I/Type II errors, 32t, 31–32 Cochran test, 42t, 62t, 41 difference in two means, 37t, 36 general discussion, 31 goodness-of-fit tests chi-square ( 2) test, 44t, 43 D’Agostino normality test, 45t, 44 independent variances, 42f, 41t, 41 interval-based tests, 38 Levene test, 62t, 43 mean of a normal distribution, 33t, 35t, 34 ratio of variances of two normal distributions, 40t, 39 two one-sided test (TOST) procedure, 39 variance of a normal distribution, 33t, 36

I increment delimitation error (DE), 5 increment extraction error (EE), 5 independent and identically-distributed (i.i.d.) random variables, 200 inter-criteria information (weights) basic concepts, 602t, 601 dominance functions, 605t, 604 sensitivity/robustness, 602 utility/desirability functions, 604t, 603 interquartile range (IQR), 192

Index to Volume 1 intracriteria information (scores) basic concepts, 599 direct rating approaches, 601 partial value functions, 601f, 600 qualitative value scales, 600 iodobenzene, 460t, 465t Ishikawa diagrams, 304f, 303, 304f

J Jacobian (J) computation, 221 Jeffreys’ prior, 230

K Karush–Kuhn–Tucker (KKT) conditions, 534, 532 Kendall’s tau association measure, 204, 195–196 KKT (Karush–Kuhn–Tucker) conditions, 534, 532 Kolmogorov–Smirnof test, 363 Kolmogorov’s test, 363, 138

L lactose component proportions, 432t crushing strength, 436f, 442f, 443f, 430 design coordinates/checkpoints, 432f design point values, 433t disintegration time, 437f, 442f, 443f, 430 drug release, 438f, 442f, 443f, 430 global desirability function, 439 lower and upper bounds, 445t mean diameter, 437f, 442f, 443f, 430 median diameter plots, 447f multicomponent systems, 430 percent (%) of modal fraction plots, 448f porosity, 436f, 442f, 443f, 430 reliability study, 441f, 440 sensitivity case study, 442f, 443f, 441 variance prediction, 433f Lagrangian function, 531 Lambert–Beer law, 642 least squares (LS) method calibration models, 130f, 129 generalized least squares method, 154 least median of squares (LMS) calibration models, 365, 161t, 162f, 163f, 160 response surface methodology (RSM), 365 least trimmed squares (LTS) method robust linear regression basic concepts, 197 outlier map, 200f residual plots, 199f scatter plots, 198f univariate regression, 161 response surface methodology (RSM), 349 robust linear regression basic concepts, 197 outlier map, 200f residual plots, 199f scatter plots, 198f total least squares (TLS), 164 weighted least squares method, 157t, 158f, 159f, 154 least trimmed squares (LTS) method robust linear regression basic concepts, 197 outlier map, 200f residual plots, 199f scatter plots, 198f univariate regression, 161

661

Levene test, 62t, 43 limited-memory BFGS (L–BFGS) method, 524 linear independence constraint qualification (LICQ), 531 linear regression Bayes’ theorem, 226f, 225 calibration models basic concepts, 129 calibration design, 151t, 150f, 148 correlation coefficients, 133 least squares (LS) method, 130f, 129 maximum likelihood estimation (MLE), 132f, 131 random errors, 131 robust linear regression basic concepts, 197 outlier map, 200f residual plots, 199f scatter plots, 198f liquid chromatography (LC), 566, 121 locally weighted scatter plot smoothing (LOWESS) method, 203f, 202, 176 logarithmic barrier method, 534

M Mahalanobis distance Hotelling T2 control charts, 113 robust statistical analysis, 196–197 MATLAB multivariate statistical process control (MSPC), 123 statistical computations, 70 maximum likelihood estimation (MLE) calibration models, 132f, 131 validation approaches, 61 mean absolute deviation (MAD), 191t, 192 median of absolute deviation (MAD), 61–62 mesitylene, 460t M estimators, 193 methanol, 460t, 465t methoxy ethanol, 460t methyl acetate, 460t methyl ketone, 460t MEWMA (multivariate EWMA) control charts, 114 microcrystalline cellulose, 445t, 447f, 448f Microsoft Excel (software), 123 milk, 295f, 296 minimum covariance determinant (MCD), 197f, 196 mining industry See theory of sampling (TOS) Minitab (software), 123 missing data, 511–512 mixture studies experimental design and optimization, 391–452 basic concepts, 393f, 391 constrained mixtures constraint C1, 421 constraint D1, 422 design point candidates, 425t D-optimality criterion, 426t, 427f, 428f, 429f, 423 edge centroid composition, 424f, 417, 422 extreme vertices designs, 414 G-optimality criterion, 425–426 L- and U-simplexes, 395, 396f lower and upper bounds, 401f, 399 lower bounds, 398f, 397 multicomponent constraints, 418 overall centroid composition, 419f, 424f, 418, 423 referential mixture designs, 431t, 430f, 429 single-component constraints, 415 three-component system, 396f

662

Index to Volume 1

mixture studies (continued ) two-dimensional (2D) face centroid composition, 424f, 418, 423 upper bounds, 400f, 398 desirability function basic concepts, 437 elementary desirability function, 439f, 438 global desirability function, 440f, 439 reliability study, 441f, 440 ellipsoidal regions basic concepts, 444 boundaries, 444t checkpoint values, 447t design point coordinates, 444t global desirability function, 448f hybrid design point coordinates, 446t lower and upper bounds, 445t median diameter plots, 447f model coefficient measurements, 447t percent (%) of modal fraction plots, 448f transformation steps, 445f general discussion, 449 methodology, 248 mixture domain constrained mixtures, 395 polyhedral mixture region, 402 response surface model (RSM), 409t, 407 simplex regions, 394f, 393, 395f pharmaceutical industry component proportions, 432t crushing strength, 436f, 442f, 443f, 430 design coordinates/checkpoints, 432f design point values, 433t disintegration time, 437f, 442f, 443f, 430 drug release, 438f, 442f, 443f, 430 mean diameter, 437f, 442f, 443f, 430 multicomponent systems, 430 porosity, 436f, 442f, 443f, 430 variance prediction, 433f polyhedral mixture region basic concepts, 402 core point determination, 405 lower and upper bounds, 404f, 405f, 403 upper bounds, 403f, 402 reliability study, 441f, 440 Scheffe´ simplex-centroid designs basic concepts, 412 design coordinates/checkpoints, 414t, 432f space configurations, 413f Scheffe´ simplex-lattice designs {3, 2} design coordinates/checkpoints, 410t {3, 3} design coordinates/checkpoints, 411t {4, 2} design coordinates/checkpoints, 411t {4, 3} design coordinates/checkpoints, 412t basic concepts, 409 internal checkpoints, 412t space configurations, 410f sensitivity case study, 442f, 443f, 441 simplex regions basic concepts, 393 four-component simplex, 395f Scheffe´ simplex-centroid designs, 412 Scheffe´ simplex-lattice designs, 409 three-component simplex, 394f monoglycerides, 445t, 447f, 448f multiattribute value theory general discussion, 599 inter-criteria information (weights) basic concepts, 602t, 601

dominance functions, 605t, 604 sensitivity/robustness, 602 utility/desirability functions, 604t, 603 intracriteria information (scores) basic concepts, 599 direct rating approaches, 601 partial value functions, 601f, 600 qualitative value scales, 600 multicriteria decision-making (MCDM) strategies, 591–629 background information, 591 basic concepts, 596 criteria values, 595t, 595 goal programming, 625 Hasse diagram technique (HDT) basic concepts, 619 dissimilar-acting substances toxicity values, 623f, 621t phenylureas toxicity values, 622f, 621t sorted alternatives, 619f typical diagram, 620f multiattribute value theory general discussion, 599 inter-criteria information (weights) basic concepts, 602t, 601 dominance functions, 605t, 604 sensitivity/robustness, 602 utility/desirability functions, 604t, 603 intracriteria information (scores) basic concepts, 599 direct rating approaches, 601 partial value functions, 601f, 600 qualitative value scales, 600 notation guidelines, 594 outranking relations basic concepts, 605 ELECTRE function ranking method concordance matrix, 610t discordance matrix, 611t ELECTRE I, 611f, 610t, 611t, 609 ELECTRE II, 612t, 611 ELECTRE III, 612 ELECTRE IV, 614 ELECTRE TRI, 614 general discussion, 609 normalized data, 610t ORESTE, 615f, 618f, 615t, 616t, 617t, 614 PROMETHEE function ranking method, 607f, 606 Pareto optimality methods, 562–566, 597f, 598f, 597 process structure, 593f sequential optimization methods, 551, 562–566 simple additive ranking (SAR) method, 599t, 598 multivariate adaptive regression splines (MARS), 179 multivariate analysis multivariate control charts basic concepts, 112 original variables-based schemes general discussion, 112 Hotelling T2 control charts, 113 multivariate statistical process control (MSPC) latent variables-based schemes general discussion, 115 illustrative example, 122f, 121 industrial potential, 120 laboratory applications, 121 monitoring measurement systems, 121 principal component analysis (PCA)-based MSPC, 116 original variables-based schemes fault diagnosis, 115 MEWMA (multivariate EWMA) control charts, 114 Shewhart-type control charts, 114

Index to Volume 1

N Nadaraya–Watson estimator, 202f, 203f, 200–201, 175 near-infrared (NIR) spectral analysis general discussion, 647 polymer film data set additive B, 649f, 648 additive C, 650f, 648 general discussion, 648 Newman–Keuls procedure, 51t, 52t, 51 Newton methods of optimization (Newton–Raphson algorithm) location–scale estimation models, 193 modified Newton’s method, 522 multivariate minimization, 520 nonlinear equations in one variable, 512 univariate minimization, 516t Nguyen supersaturated matrix, 266t, 265 nitrobenzene, 460t, 467t nitroethane, 460t nitromethane, 460t non-classical experimental designs, 453–499 combined designs continuous factor–discrete factor designs basic concepts, 471 candidate design points, 473f, 475f, 477f coefficients estimation, 478t, 480t construction procedures, 472 desirability function, 481f, 482f, 482, 482t Doehlert designs, 473f equiradial designs, 475f experimental conditions, 472t, 473t, 474t, 475t, 476t experimental results, 476, 478t multicriteria optimization, 480 objective selection, 480 sensitivity study, 484f, 483, 483t tablet hardness, 479f, 481f, 476, 478t variance function variations, 474f continuous factor–mixture factor designs basic concepts, 484 candidate design points, 485f checkpoint values, 489t coefficients estimation, 491t complete design, 487f construction procedures, 485 desirability function, 495f, 496f, 492, 497f, 495t, 496t Doehlert designs, 486f D-optimality criterion, 487t experimental conditions, 488f, 485t, 486t, 488t experimental results, 492f, 489, 490t multicriteria optimization, 491 partial desirabilities, 494f residuals table, 490t response variations, 494f three-dimensional (3D) response surfaces, 493f general discussion, 471 criteria type I criteria, 465f, 468f, 457 type II criteria, 458f, 465f, 457 type III criteria, 459 type IV criteria, 459 general discussion, 453, 494 illustrative example Box–Behnken designs, 470, 470t constraint-based experimental design study, 470f, 469t, 469, 470t multifactor screening study D-optimality criterion, 467t experimental design, 469t experimental matrices, 468t

663

general discussion, 466 type I criteria, 468f solvent effects study checkpoint values, 467t D-optimality criterion, 464t, 465t, 466t experimental matrices, 462t general discussion, 460 optimal experimental design, 466f, 465t properties, 460t type I criteria, 465f type II criteria, 465f variable transformations, 462t methodology, 249 information of sufficient quality, 455 most economical set, 455 nonparametric models nonparametric regression, 202f, 200 robust statistical analysis, 195 normal distribution basic concepts, 67t, 67 computational aspects, 71 validation approaches basic concepts, 19 confidence intervals mean measures, 23 ratio of variances of two normal distributions, 27 variance measures, 25 distribution characteristics, 21t histograms, 21f hypothesis testing mean measures, 35t, 34 mean of a normal distribution, 33t ratio of variances of two normal distributions, 40t, 39 variance measures, 33t, 36 inferential methods, 139f, 138 probability density function (PDF), 19f, 20t numerical optimization, 507–545 chemometric analysis, 511 classifications, 510t constrained optimization methods active set methods general discussion, 534 Karush–Kuhn–Tucker (KKT) conditions, 534 linear programming, 535f, 534 quadratic programming, 537 sequential quadratic programming (SQP), 542t, 539 simplex methods, 536 augmented Lagrangian method, 533 basic concepts general discussion, 531 Karush–Kuhn–Tucker (KKT) conditions, 532 Lagrangian function, 531 linear independence constraint qualification (LICQ), 531 general discussion, 542 logarithmic barrier method, 534 quadratic penalty function, 532 minimization/maximization calculations, 510f, 509 steepest ascent method, 585t, 586t, 587t, 586f, 587f, 588f, 585 unconstrained optimization methods general discussion, 542 globalization strategies backtracking strategies, 527 line search algorithms, 526 local–global convergence, 524 steepest descent algorithms, 525 trust region algorithms, 529f, 530f, 528 Wolfe conditions, 527f, 526 multivariate minimization basic concepts, 518

664

Index to Volume 1

numerical optimization (continued ) Cholesky factorization, 521 Hessian matrix, 518 modified Newton’s method, 522 necessary/sufficient conditions, 519t Newton methods of optimization, 520 quasi-Newton method of optimization, 522 Taylor expansion approximation, 520f, 520 two-dimensional functions, 521f nonlinear equations in one variable, 512 trust region algorithms basic concepts, 528 dogleg step, 530f double dogleg step, 530f optimal (hook) step, 529f univariate minimization basic concepts, 514f, 514 Newton methods of optimization, 516t, 516 quadratic model, 517f, 517 second-order conditions, 515f

O octanol, 460t one-tail tests, 32 one-variable-at-a-time (OVAT) procedure, 550f, 549, 550f optimization methods, 507–545 constrained optimization methods active set methods general discussion, 534 Karush–Kuhn–Tucker (KKT) conditions, 534 linear programming, 535f, 534 quadratic programming, 537 sequential quadratic programming (SQP), 542t, 539 simplex methods, 536 augmented Lagrangian method, 533 basic concepts general discussion, 531 Karush–Kuhn–Tucker (KKT) conditions, 532 Lagrangian function, 531 linear independence constraint qualification (LICQ), 531 general discussion, 542 logarithmic barrier method, 534 quadratic penalty function, 532 general discussion, 508 Newton methods of optimization (Newton–Raphson algorithm) location–scale estimation models, 193 modified Newton’s method, 522 multivariate minimization, 520 nonlinear equations in one variable, 512 univariate minimization, 516t one-variable-at-a-time (OVAT) procedure, 550f, 549, 550f quasi-Newton method of optimization, 522, 513–514 sequential optimization methods, 547–575 Fibonacci numbers, 553t, 552f, 551, 554f general discussion, 551 mixed sequential–simultaneous optimization methods basic concepts, 569 Doehlert designs, 572 evolutionary operation (EVOP), 572f, 572 factorial design approach, 570f steepest ascent method, 570f, 571f one-variable-at-a-time (OVAT) procedure, 550f, 549, 550f simplex methods basic concepts, 555 case studies, 567t, 568t, 568f, 566 general discussion, 561 supermodified simplex, 565f three- or more-dimensional simplex method, 557

two-dimensional (2D) simplex method, 555f, 556f, 558f, 555 variable-size/modified simplex method, 559f, 560f, 562f, 564f, 558 weighted centroid methods, 565f steepest ascent method, 577–590 applications algebraic optimization, 584 numerical optimization, 585t, 586t, 587t, 586f, 587f, 588f, 585 three-variable half-factorial design, 588t, 587 basic concepts, 578f, 578 computational optimization, 579 experimental optimization, 579 general discussion, 589 historical background, 577 literature research background, 578 methodology influence of parameter a, 584t, 583f, 584f, 582 limitations, 584 main adjustable parameters, 581t multifactor optimization, 582f, 581 single-parameter optimization, 580f, 580 mixed sequential–simultaneous optimization methods, 570f, 571f notation, 579 unconstrained optimization methods general discussion, 542 globalization strategies backtracking strategies, 527 line search algorithms, 526 local–global convergence, 524 steepest descent algorithms, 525 trust region algorithms, 529f, 530f, 528 Wolfe conditions, 527f, 526 multivariate minimization basic concepts, 518 Cholesky factorization, 521 Hessian matrix, 518 modified Newton’s method, 522 necessary/sufficient conditions, 519t Newton methods of optimization, 520 quasi-Newton method of optimization, 522 Taylor expansion approximation, 520f, 520 two-dimensional functions, 521f nonlinear equations in one variable, 512 trust region algorithms basic concepts, 528 dogleg step, 530f double dogleg step, 530f optimal (hook) step, 529f univariate minimization basic concepts, 514f, 514 Newton methods of optimization, 516t, 516 quadratic model, 517f, 517 second-order conditions, 515f ORESTE (Organisation, Rangement Et Synthe`se de donneés relaTionElles) average ranks, 615t, 616t basic concepts, 614 decision-making process, 618f distance derivations, 616t dominance graph, 618f linear orthogonal projections, 616t normalized preference intensities, 617t position matrix, 615f outranking relations basic concepts, 605 ELECTRE function ranking method concordance matrix, 610t discordance matrix, 611t

Index to Volume 1 ELECTRE I, 611f, 610t, 611t, 609 ELECTRE II, 612t, 611 ELECTRE III, 612 ELECTRE IV, 614 ELECTRE TRI, 614 general discussion, 609 normalized data, 610t ORESTE average ranks, 615t, 616t basic concepts, 614 decision-making process, 618f distance derivations, 616t dominance graph, 618f linear orthogonal projections, 616t normalized preference intensities, 617t position matrix, 615f PROMETHEE function ranking method, 607f, 606

P paired t-tests, 35 PARAFAC (parallel factor analysis) confidence intervals, 142f, 142 numerical optimization, 511 parametric models, 195 Pareto optimality methods basic concepts, 597f, 598f, 597 quadratic surface response analysis, 383f, 382 screening strategies, 275f, 274 sequential optimization methods, 562–566 Pareto’s principle, 252 partial least squares (PLS) method, 642 Pearson correlation coefficient, 195 pentanol, 460t peppermint oil screening strategies, 288t, 291t pharmaceutical industry multicomponent mixture system case studies, 430 component proportions, 432t crushing strength, 436f, 442f, 443f, 430 design coordinates/checkpoints, 432f design point values, 433t disintegration time, 437f, 442f, 443f, 430 drug release, 438f, 442f, 443f, 430 mean diameter, 437f, 442f, 443f, 430 porosity, 436f, 442f, 443f, 430 variance prediction, 433f multivariate statistical process control (MSPC), 120 simplex optimization methods, 566 phenetole, 460t phenyl ethanol, 460t phenyl propanol, 460t phenylureas toxicity values, 622f, 621t Plackett–Burman exploratory design screening strategies alias matrix, 263f, 262t, 261 basic matrix, 262t biasing potential, 263t building procedures, 255t, 256t, 255 fractional factorial designs, 256f, 257f, 255 general discussion, 253 Hadamard matrices, 255t, 255 two-factor interaction matrix, 262t, 263t validation approaches, 64t, 65t, 63 pollution studies air pollution study, 177f, 178f, 176 As Pontes, Corunã, Spain, 182f, 183f, 181 polyethylene glycol, 444t polysorbate 80, 445t, 447f, 448f precision measures, 21f, 21t, 21, 55

prediction error sum of squares (PRESS), 365 principal component analysis (PCA) multivariate statistical process control (MSPC) exploratory data analysis, 116 illustrative example, 122f, 121 offline process monitoring, 116 online process monitoring, 119 numerical optimization, 511 principal component regression (PCR), 642 principle of effect heredity, 253 principle of effect hierarchy, 253 principle of factor sparsity, 252 probability density function (PDF), 19f, 20t, 21f, 21t, 18–19 proficiency testing, 77–96 applications methodology analysis, 94 scheme reviews, 93 background information, 79 purpose, 78 quality assurance, 79 test material validation general discussion, 90 homogeneity tests, 91f, 92f, 90 stability tests, 93 z-scoring advantages/disadvantages, 88 assigned values, 80 interpretation and use, 88f, 87 kernel densities, 82f, 83f, 81 lognormal data sets, 90 mixture models, 85f, 84 quantile–quantile (Q–Q) plots, 86f requirements, 79 robust statistics, 81f, 80 Shewhart control chart, 88f skewed data sets, 89f, 89 target value (p) calculation, 87f, 85 uncertainties, 80 zone chart, 88f PROMETHEE function ranking method, 607f, 606 propanol, 460t propylene glycol, 444t

Q quadratic penalty function, 532 quadratic surface response analysis canonical analysis, 376f, 375 multiple-response analysis basic concepts, 379 confidence regions, 387f, 386 global desirability function, 381f individual desirability functions, 380f Pareto optimality methods, 383f, 382 transformation calculations, 385f, 385t, 383 ridge analysis, 378f, 377t, 377 qualitative value scales, 600 quantile–quantile (Q–Q) plots, 86f quasi-Newton method of optimization, 522, 513–514

R random effects model confidence interval estimation, 54 general discussion, 53 laboratory applications, 57f power of the ANOVA, 53 robust analysis, 62t, 62 random variables convergence, 69

665

666

Index to Volume 1

regression analysis Bayes’ theorem, 225 bootstrap validation methods, 173 least median of squares (LMS) calibration models, 365, 161t, 162f, 163f, 160 response surface methodology (RSM), 365 least squares multilinear regression, 349 linear regression Bayes’ theorem, 226f, 225 calibration models basic concepts, 129 calibration design, 151t, 150f, 148 correlation coefficients, 133 least squares (LS) method, 130f, 129 maximum likelihood estimation (MLE), 132f, 131 random errors, 131 robust linear regression basic concepts, 197 outlier map, 200f residual plots, 199f scatter plots, 198f nonparametric regression, 202f, 200 Poisson regression, 177f, 178f, 176 principal component regression (PCR), 642 probability and conditional uncertainty, 217 robust statistical analysis general discussion, 197 locally weighted scatter plot smoothing (LOWESS) method, 203f, 202 Nadaraya–Watson estimator, 202f, 203f, 200–201 nonparametric regression, 202f, 200 robust linear regression basic concepts, 197 outlier map, 200f residual plots, 199f scatter plots, 198f robust locally weighted regression, 203f, 202 repeatability, 55 reproducibility, 55 residual variance, 142f, 140 response surface design (RSD) methodology, 551 response surface methodology (RSM), 345–390 basic concepts codification, 347 empirical model, 348 estimation/validation approaches, 348 experimental domain, 347t, 346 experimental run, 348 least squares multilinear regression, 349 response comcept, 347 theoretical model, 347 experimental designs blocked designs, 353t, 373 equiradial designs, 367f first-order models, 350t, 367f, 366 general discussion, 366 methodology, 248 randomization approaches, 373 second-order models analysis of variance (ANOVA), 385t Box–Behnken designs, 373t, 371 central composite designs, 350t, 368f, 369f, 358t, 368t, 370t, 367 Doehlert designs, 371f, 371t, 372t, 370 hybrid designs, 372 level curves, 376f, 385f transformation calculations, 385f general discussion, 346, 387

quadratic surface response analysis canonical analysis, 376f, 375 multiple-response analysis basic concepts, 379 confidence regions, 387f, 386 global desirability function, 381f individual desirability functions, 380f Pareto optimality methods, 383f, 382 transformation calculations, 385f, 385t, 383 ridge analysis, 378f, 377t, 377 validation approaches analysis of variance (ANOVA) lack of fit test, 356t, 357t, 358t, 359t, 360t, 385t regression significance tests, 355t, 356t coefficient significance, 362 coefficients of determination, 363 confidence intervals, 362 joint confidence region, 362 lack of fit test additional codified variables, 360t analysis of variance (ANOVA), 356t, 357t, 358t, 359t, 360t, 385t basic concepts, 355 design and experimental response, 357t design matrix and response, 358t, 359t test point applications, 361t, 360 least median of squares (LMS), 365 normally and independently distributed (NID) errors, 354 prediction error sum of squares (PRESS), 365 prediction models, 362 regression significance tests analysis of variance (ANOVA), 355t basic concepts, 354 residual analysis, 364f, 363 T-optimal designs, 361 variance optimality estimated coefficient precision A-optimality, 351 D-optimality, 351 E-optimality, 351 experimental design and optimization matrices, 350t, 350 generalized standard deviation (GSD) optimality, 351 G-optimality, 352 orthogonal designs, 350 rotatability, 353 uniform precision, 354f, 353t, 353 variance inflation factor (VIF), 351 Rhizopus spp., 567 Rhodotorula glutinis, 567 robust statistical analysis, 189–211 correlation measures minimum covariance determinant (MCD), 197f, 196 parametric and nonparametric models, 195f, 195 robust correlation and covariance, 197f, 196 dependence structures investigations association measures, 204 copula functions empirical distribution function, 207f, 206 general discussion, 204 kernel density estimators, 206f, 207f, 206 nonparametric estimations, 205 general discussion, 203 energy of the lowest unoccupied molecular orbital (ELUMO) correlation measures parametric and nonparametric models, 195f, 195 robust correlation and covariance, 197f, 195 empirical distribution function, 207f, 206 kernel density estimators, 207f

Index to Volume 1 locally weighted scatter plot smoothing (LOWESS) method, 203f, 202 location–scale estimation models, 191t, 191–192 robust linear regression, 198f, 197 general discussion, 190 location–scale estimation models energy of the lowest unoccupied molecular orbital (ELUMO), 191t, 191–192 mean and standard deviation, 191t, 190 robust estimators basic concepts, 192 Huber estimator, 191t, 193 interquartile range (IQR), 192 mean absolute deviation (MAD), 191t, 192 M estimators, 193 trimmed means, 193 skewness–adjusted boxplots, 194f, 194 regression methods general discussion, 197 locally weighted scatter plot smoothing (LOWESS) method, 203f, 202 Nadaraya–Watson estimator, 202f, 203f, 200–201 nonparametric regression, 202f, 200 robust linear regression basic concepts, 197 outlier map, 200f residual plots, 199f scatter plots, 198f robust locally weighted regression, 203f, 202 software availability, 208 validation approaches basic concepts, 61 Huber estimator, 61f, 61 parameter estimation, 60t, 62t root mean squared error in cross-validation (RMSECV), 646 root mean squared error of prediction (RMSEP), 648 rotating Vezin cross-stream sampling systems, 8f, 7 ruggedness testing, 303, 63

S SAS (software), 123 Scheffe´ polynomials, 409t, 407 screening strategies, 251–300 basic concepts additive models, 254 experimental designs, 251 exploratory studies, 252 factor effects, 254f factorial designs, 252 Plackett–Burman exploratory design, 253 principle of effect heredity, 253 principle of effect hierarchy, 253 principle of factor sparsity, 252 qualitative variable models, 253 experimental conditions, 303, 248 multilevel designs asymmetrical designs 2m3n matrices, 280, 280t, 281t 2m4n matrices, 279 general discussion, 279 factor analysis, 276, 277f symmetrical designs, 277, 278t, 279t multilevel supersaturated designs applications, 291 construction procedures 2735//12 matrix, 282t 2735//9 matrix, 283t, 284t 311//12 matrix, 285t

667

311//9 matrix, 286t 312//12 matrix, 285t 312//9 matrix, 284t comparison studies, 284t general discussion, 281 matrix design, 282t matrix success rate plot, 286f squared mean error simulation, 287f resolution and analysis 2735//27 matrix, 288t active factor identification, 291t critical resolution values, 290t effect estimate differences, 289f experimental domain, 288t general discussion, 287 R2 value changes, 290f regression coefficients, 288t subset regression results, 291t Plackett–Burman exploratory design alias matrix, 263f, 262t, 261 basic matrix, 262t biasing potential, 263t building procedures, 255t, 256t, 255 fractional factorial designs, 256f, 257f, 255 general discussion, 253 Hadamard matrices, 255t, 255 two-factor interaction matrix, 262t, 263t sample composition binary response characteristics, 293f, 292, 293t group screening, 292 metric response characteristics, 294 multiple grouping strategies, 294t processing options, 293f strategic sample composition, 295f, 294 saturated designs alias matrix, 263f, 262t, 261 folding-over procedure, 259f, 256, 257t, 258t Hadamard matrices, 255t, 254 Plackett–Burman exploratory design alias matrix, 263f, 262t, 261 basic matrix, 262t biasing potential, 263t building procedures, 255t, 256t general discussion, 255 Hadamard matrices, 255t matrix results, 256f, 257f two-factor interaction matrix, 262t, 263t projective properties, 260f, 259 saturated fractions of factorial designs, 255t, 254 two-factor interaction effects, 259f sequential bifurcation, 297f, 296 supersaturated designs Bandurek supersaturated matrix, 267t, 266 basic concepts, 264 multilevel supersaturated designs applications, 291 construction procedures, 281 resolution and analysis, 287 Nguyen supersaturated matrix, 266t, 265 resolution and analysis 228//18 supersaturated matrix, 268t, 269t, 268 factor screen maps, 273f, 274f, 271 general discussion, 267 genetic algorithm-driven subset regression, 270 island plots, 271f, 272f, 270 noise situations, 275f, 269t, 275t Pareto analysis, 275f, 274 simulation sets, 269t stepwise–subset conventional regression strategies, 272

668

Index to Volume 1

screening strategies (continued ) Tang–Wu supersaturated matrix, 266t, 265 secant methods, 513–514 sequential injection analysis (SIA) method, 566–567 sequential optimization methods, 547–575 Fibonacci numbers, 553t, 552f, 551, 554f general discussion, 551 mixed sequential–simultaneous optimization methods basic concepts, 569 Doehlert designs, 572 evolutionary operation (EVOP), 572f, 572 factorial design approach, 570f steepest ascent method, 570f, 571f one-variable-at-a-time (OVAT) procedure, 550f, 549, 550f simplex methods basic concepts, 555 case studies, 567t, 568t, 568f, 566 general discussion, 561 supermodified simplex, 565f three- or more-dimensional simplex method, 557 two-dimensional (2D) simplex method basic concepts, 555 initial simplex, 555f, 556f, 555 successive simplexes, 558f, 556 variable-size/modified simplex method basic concepts, 559f, 558 calculations, 560 case studies, 567t, 568t, 568f, 566 contractions, 564f initial simplex, 560f, 562f, 559 successive simplexes, 559 weighted centroid methods, 565f sequential quadratic programming (SQP), 542t, 539 Shapiro–Wilks test, 363 Shewhart control charts basic concepts, 102 formulas, 104t illustrative example, 106 individuals control chart, 107f proficiency testing, 88f significance levels confidence intervals, 140 hypothesis testing, 32 simple additive ranking (SAR) method, 599t, 598 simplex methods constrained optimization methods, 536 sequential optimization methods basic concepts, 555 case studies, 567t, 568t, 568f, 566 general discussion, 561 supermodified simplex, 565f three- or more-dimensional simplex method, 557 two-dimensional (2D) simplex method basic concepts, 555 initial simplex, 555f, 556f, 555 successive simplexes, 558f, 556 variable-size/modified simplex method basic concepts, 559f, 558 calculations, 560 case studies, 567t, 568t, 568f, 566 contractions, 564f initial simplex, 560f, 562f, 559 successive simplexes, 559 weighted centroid methods, 565f Sklar’s theorem, 204 smoothing splines, 175 solid-phase microextraction (SPME) analysis, 37t, 36 Spearman’s rho association measure, 204, 196

spectral analysis genetic algorithms (GAs) autocorrelation processes, 646 chromosome insertion, 644 final model architecture, 647 general discussion, 642 initial population, 644 mutation probability, 644 overfitting controls, 646 parameter influences, 645 polymer film data set additive B, 649f, 648 additive C, 650f, 648 general discussion, 648 reproduction process, 644 subset analysis, 645 near-infrared (NIR) spectral analysis general discussion, 647 polymer film data set additive B, 649f, 648 additive C, 650f, 648 general discussion, 648 splines multivariate adaptive regression splines (MARS), 179 smoothing splines, 175 SPSS (software), 123 standard additions method, 153f, 153 Statgraphics Centurion (software), 123 statistical elements box and whisker plots, 67f, 66 central limit theorem, 70 chi-square ( 2) fitting basic concepts, 68t, 68 computational aspects, 71, 72 computational aspects, 70 distribution function, 65 F distribution, 69t, 69, 71 F-tests, 72 normal distribution basic concepts, 67t, 67 computational aspects, 71 random variables convergence, 69 Student–Fisher’s t distribution basic concepts, 68t, 67 computational aspects, 71 t-test, 71 weak law of large numbers, 70 z-test, 71 statistical process control (SPC), 97–126 basic concepts, 100 control charts autocorrelation processes autocorrelation function (ACF), 109f, 110f, 109 autoregressive (AR) models, 109f, 110f, 111f, 109 autoregressive integrated moving average (ARIMA) models, 109 basic concepts, 108 illustrative example, 108 individuals control chart, 108f, 110f average run length (ARL), 106 basic concepts, 100–102 CUSUM (cumulative sum) control charts basic concepts, 104 illustrative example, 106 individuals control chart, 107f EWMA (exponentially weighted moving average) control charts basic concepts, 105 illustrative example, 106 individuals control chart, 107f

Index to Volume 1 Hotelling T2 control charts general discussion, 113 individual observations, 113 subgroup data, 113 multivariate control charts, 112 performance assessments, 106 Shewhart control charts basic concepts, 102 formulas, 104t illustrative example, 106 individuals control chart, 107f proficiency testing, 88f engineering process control (EPC), 111 model building (Phase I concepts), 102 model exploitation (Phase II concepts), 102 multivariate control charts basic concepts, 112 original variables-based schemes general discussion, 112 Hotelling T2 control charts, 113 multivariate statistical process control (MSPC) latent variables-based schemes general discussion, 115 illustrative example, 122f, 121 industrial potential, 120 laboratory applications, 121 monitoring measurement systems, 121 principal component analysis (PCA)-based MSPC, 116 original variables-based schemes fault diagnosis, 115 MEWMA (multivariate EWMA) control charts, 114 Shewhart-type control charts, 114 out-of-control samples, 102 software availability, 123 STATISTICA (software), 123 steepest ascent method, 577–590 basic concepts, 578f, 578 computational optimization, 579 experimental optimization, 579 general discussion, 589 historical background, 577 literature research background, 578 methodology algebraic optimization, 584 influence of parameter a, 584t, 583f, 584f, 582 limitations, 584 main adjustable parameters, 581t multifactor optimization, 582f, 581 numerical optimization, 585t, 586t, 587t, 586f, 587f, 588f, 585 single-parameter optimization, 580f, 580 three-variable half-factorial design, 588t, 587 mixed sequential–simultaneous optimization methods, 570f, 571f notation, 579 Stein’s paradox, 233 straight-path cross stream sampling systems, 8f, 7 Student–Fisher’s t distribution basic concepts, 68t, 67 computational aspects, 71 mixture studies, 431–437 robust statistical analysis, 52t, 61

T Tang–Wu supersaturated matrix, 266t, 265 Taylor expansion approximation, 520f, 520, 307 tetrahydrofurane, 460t theory of sampling (TOS), 1–16 applications blasthole sampling, 7f, 6

hammer cross-belt samplers, 9f, 7 rotating Vezin cross-stream sampling systems, 8f, 7 straight-path cross stream sampling systems, 8f, 7 basic concepts, 2f, 2 continuous improvement guidelines, 14f, 13 general discussion, 13 geostatistical analyses, 10 importance, 2 large-scale variability importance customer relations, 13 general discussion, 12 geological exploration, 12 mining operations, 13 processing plants, 13 variograms definitions, 10 heterogeneity, 11 parameter selection, 11 time–distance extrapolations, 12 variability measurements, 11 sampling biases blasthole sampling, 9 flotation plant sampling, 9 general discussion, 9 small-scale variability characteristics, 3 sample preparation errors (PE), 6f, 5 sampling protocol implementation general discussion, 5 increment delimitation error (DE), 5 increment extraction error (EE), 5 weighting error (WE), 5 sampling protocol optimization fundamental sampling error (FSE), 4 grouping and segregation error (GSE), 4 in situ Nugget effect (NE), 3 variances, 10f, 10 tolerance intervals basic concepts, 29 -content tolerance intervals, 29 -expectation tolerance intervals, 29 general discussion, 23 independent tolerance intervals, 30 toluene, 460t total least squares (TLS), 164 total quality management (TQM) theory of sampling (TOS), 1–16 applications blasthole sampling, 7f, 6 hammer cross-belt samplers, 9f, 7 rotating Vezin cross-stream sampling systems, 8f, 7 straight-path cross stream sampling systems, 8f, 7 basic concepts, 2f, 2 continuous improvement guidelines, 14f, 13 general discussion, 13 geostatistical analyses, 10 importance, 2 large-scale variability importance, 12 time–distance extrapolations, 12 variograms, 10 sampling biases blasthole sampling, 9 flotation plant sampling, 9 general discussion, 9 small-scale variability characteristics, 3 sample preparation errors (PE), 6f, 5

669

670

Index to Volume 1

total quality management (TQM) (continued ) sampling protocol implementation, 5 sampling protocol optimization, 3 variances, 10f, 10 t-paired tests, 35 triangular distributions, 19f, 20t, 21f, 21t, 19 trimmed means, 193 trueness measures, 21f, 21t, 21, 54 t-tests, 71 two one-sided test (TOST) procedure, 39 Type I/Type II errors control charts, 106 hypothesis testing, 32t, 31–32

U unconstrained optimization methods general discussion, 542 globalization strategies backtracking strategies, 527 line search algorithms, 526 local–global convergence, 524 steepest descent algorithms, 525 trust region algorithms basic concepts, 528 dogleg step, 530f double dogleg step, 530f optimal (hook) step, 529f Wolfe conditions, 527f, 526 multivariate minimization basic concepts, 518 Cholesky factorization, 521 Hessian matrix, 518 modified Newton’s method, 522 necessary/sufficient conditions, 519t Newton methods of optimization, 520 quasi-Newton method of optimization, 522 Taylor expansion approximation, 520f, 520 two-dimensional functions, 521f nonlinear equations in one variable, 512 univariate minimization basic concepts, 514f, 514 Newton methods of optimization, 516t, 516 quadratic model, 517f, 517 second-order conditions, 515f uniform distributions, 19f, 20t, 21f, 21t, 19 univariate analysis, 550f, 549, 550f univariate minimization See unconstrained optimization methods

V validation approaches, 17–76 analysis of variance (ANOVA) basic concepts, 45 data arrangement, 46t fixed effects model basic concepts, 47t, 48t, 46 means comparison data, 52t, 51 Newman–Keuls procedure, 51t, 52t, 51 orthogonal contrasts, 50t, 50 power of the ANOVA, 48 results analysis, 48t uncertainty tests, 49t, 49 random effects model confidence interval estimation, 54 general discussion, 53 laboratory applications, 57f

power of the ANOVA, 53 robust analysis, 62t, 62 basic concepts general discussion, 18 mean and variance measure, 19f, 20t, 21f, 21t, 18 precision, 21f, 21t, 21 probability density function (PDF), 19f, 20t, 21f, 21t, 18–19 trueness, 21f, 21t, 21 confidence intervals basic concepts, 23 difference in two means, 26 general discussion, 23 joint confidence intervals, 28 mean of a normal distribution, 23 median measures, 28 ratio of variances of two normal distributions, 27 variance of a normal distribution, 25 experimental conditions, 306f, 306 hypothesis testing Bartlett test, 62t, 42 basic concepts general discussion, 31 parametric tests, 33t simultaneous behavior analysis, 34f, 33 Type I/Type II errors, 32t, 31–32 Cochran test, 42t, 62t, 41 difference in two means, 37t, 36 general discussion, 31 goodness-of-fit tests chi-square ( 2) test, 44t, 43 D’Agostino normality test, 45t, 44 independent variances, 42f, 41t, 41 interval-based tests, 38 Levene test, 62t, 43 mean of a normal distribution, 33t, 35t, 34 ratio of variances of two normal distributions, 40t, 39 two one-sided test (TOST) procedure, 39 variance of a normal distribution, 33t, 36 proficiency testing general discussion, 90 homogeneity tests, 91f, 92f, 90 stability tests, 93 statistical elements box and whisker plots, 67f, 66 central limit theorem, 70 chi-square ( 2) fitting basic concepts, 68t, 68 computational aspects, 71, 72 computational aspects, 70 distribution function, 65 F distribution, 69t, 69, 71 F-tests, 72 normal distribution basic concepts, 67t, 67 computational aspects, 71 random variables convergence, 69 Student–Fisher’s t distribution basic concepts, 68t, 67 computational aspects, 71 t-test, 71 weak law of large numbers, 70 z-test, 71 statistical inference accuracy measures, 63 consistency analysis, 58 data analysis, 58 data elimination, 58–59 Grubbs’ test, 59t, 60t, 59–60 Plackett–Burman exploratory design, 64t, 65t, 63

Index to Volume 1 precision, 55 robust analysis basic concepts, 61 Huber estimator, 61f, 61 parameter estimation, 60t, 62t ruggedness testing, 63 trueness, 54 tolerance intervals basic concepts, 29 -content tolerance intervals, 29 -expectation tolerance intervals, 29 general discussion, 23 independent tolerance intervals, 30 variance inflation factor (VIF) calibration models, 151t, 150f, 148–149 response surface methodology (RSM), 351

W water (H2O) analysis, 444t, 445t, 447f, 448f weak law of large numbers, 70

Weibull distribution, 19f, 20t, 21f, 21t, 19–20 weighted least squares method, 157t, 158f, 159f, 154 Willgerodt–Kindler reaction, 339f, 337f, 338t, 337 Wolfe conditions, 527f, 526

X xylene, 460t

Y Yates’ algorithm, 316t, 316

Z z score for skewness, 363 z-test, 71

671


Comprehensive Chemometrics Volume 1: Chemical and Biochemical Data Analysis

Chemometrics: data analysis for the laboratory and chemical plant

Chemometrics Data Analysis for the Laboratory and Chemical Plant

Chemical, Biochemical, and Engineering Thermodynamics

Nonlinear Computer Modeling of Chemical and Biochemical Data

Nonlinear Computer Modeling of Chemical and Biochemical Data

Chemometrics in Environmental Analysis

Comprehensive Chemical Kinetics, Volume 33: Catastrophe Theory

Chemical Engineering: Solutions Volume 1

Chemical And Biochemical Physics: New Frontiers

Chemical Resistance, Volume 1: Thermoplastics

Analysis I (Volume 1)

Combinatory Analysis -Volume 1

Low-temperature Combustion and Autoignition, Volume 35 (Comprehensive Chemical Kinetics)

Comprehensive Chemical Kinetics, Volume 36: Photokinetics : Theoretical Fundamentals and Applications

Comprehensive Semiconductor Science and Technology, Six-Volume Set, Volume 1

Guidelines for consequence analysis of chemical releases, Volume 1

Analysis of Messy Data Volume 1: Designed Experiments, Second Edition

Data collection and analysis

Comprehensive Series in Photosciences, Volume 1: Photomovement

Comprehensive Medicinal Chemistry II, Volume 1

Scholarship mathematics, Volume 1: Analysis

Mathematical Physics, Analysis and Geometry - Volume 1

Introduction to Multivariate Statistical Analysis in Chemometrics

Classical and Multilinear Harmonic Analysis, Volume 1

I: Functional Analysis, Volume 1

Comprehensive Volume and Capacity Measurements

Spatial Economic Analysis - Volume 1, Issue 1

Chemistry & Chemical Reactivity, Volume 1

Biochemical Thermodynamics: Applications of Mathematica (Methods of Biochemical Analysis)

Chemometrics: A Textbook (Data Handling in Science and Technology, 2)

Comprehensive Chemometrics Volume 1: Chemical and Biochemical Data Analysis

Chemometrics: data analysis for the laboratory and chemical plant

Chemometrics Data Analysis for the Laboratory and Chemical Plant

Chemical, Biochemical, and Engineering Thermodynamics

Nonlinear Computer Modeling of Chemical and Biochemical Data

Nonlinear Computer Modeling of Chemical and Biochemical Data

Chemometrics in Environmental Analysis

Comprehensive Chemical Kinetics, Volume 33: Catastrophe Theory

Chemical Engineering: Solutions Volume 1

Chemical And Biochemical Physics: New Frontiers

Chemical Resistance, Volume 1: Thermoplastics

Analysis I (Volume 1)

Combinatory Analysis -Volume 1

Low-temperature Combustion and Autoignition, Volume 35 (Comprehensive Chemical Kinetics)

Comprehensive Chemical Kinetics, Volume 36: Photokinetics : Theoretical Fundamentals and Applications

Comprehensive Semiconductor Science and Technology, Six-Volume Set, Volume 1

Guidelines for consequence analysis of chemical releases, Volume 1

Analysis of Messy Data Volume 1: Designed Experiments, Second Edition

Data collection and analysis

Comprehensive Series in Photosciences, Volume 1: Photomovement

Comprehensive Medicinal Chemistry II, Volume 1

Scholarship mathematics, Volume 1: Analysis

Mathematical Physics, Analysis and Geometry - Volume 1

Introduction to Multivariate Statistical Analysis in Chemometrics

Classical and Multilinear Harmonic Analysis, Volume 1

I: Functional Analysis, Volume 1

Comprehensive Volume and Capacity Measurements

Spatial Economic Analysis - Volume 1, Issue 1

Chemistry & Chemical Reactivity, Volume 1

Biochemical Thermodynamics: Applications of Mathematica (Methods of Biochemical Analysis)

Chemometrics: A Textbook (Data Handling in Science and Technology, 2)

Recommend Documents