Maintaining Mission Critical Systems in a 24 7 Environment

This page intentionally left blank

MAINTAINING MISSION CRITICAL SYSTEMS IN A 24/7 ENVIRONMENT

IEEE Press 445 Hoes Lane Piscataway, NJ 08855 IEEE Press Editorial Board Lajos Hanzo, Editor in Chief R. Abhari J. Anderson G. W. Arnold F. Cañavero

M. El-Hawary B-M. Haemmerli M. Lanzerotti D. Jacobson

O. P. Malik S. Nahavandi T. Samad G. Zobrist

Kenneth Moore, Director of IEEE Book and Information Services (BIS)

MAINTAINING MISSION CRITICAL SYSTEMS IN A 24/7 ENVIRONMENT Second Edition

Peter M. Curtis

IEEE PRESS| SERIES WÚ0S5m SERI

° " POWER ENGINEERING

IEEE PRESS

JWILEY A JOHN WILEY & SONS, INC., PUBLICATION

Copyright © 2011 by the Institute of Electrical and Electronics Engineers, Inc. Published by John Wiley & Sons, Inc., Hoboken, New Jersey. All rights reserved. Published simultaneously in Canada. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permission. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representation or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 5724002. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print, however, may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.

Library of Congress Cataloging-in-Publication Data: Curtis, Peter M. author. Maintaining Mission Critical Systems in a 24/7 Environment / Peter M. Curtis. — 2 p. cm. — (IEEE Press Series on Power Engineering) ISBN 978-0-470-65042-4 (hardback) 1. Reliability (Engineering) I. Title. TA169.C87 2011 620'.00452—dc22 2010049606 Printed in Singapore. oBook ISBN: 978-1-118-04164-2 ePDF ISBN: 978-1-118-04162-8 ePub ISBN: 978-1-118-04163-5 10 9 8 7 6 5 4 3 2 1

CONTENTS

Foreword

xvii

Preface

xix

Acknowlegments

xxi

1 An Overview of Reliability and Resiliency in Today's Mission Critical Environment 1.1 Introduction 1.2 Risk Assessment 1.2.1 Levels of Risk 1.3 Capital Cost Versus Operation Cost 1.4 Critical Environment Workflow and Change Management 1.4.1 Change Management 1.4.2 Escalation Procedures 1.5 Testing and Commissioning 1.6 Documentation and the Human Factor 1.7 Education and Training 1.8 Operation and Maintenance 1.9 Employee Certification 1.10 Standards and Benchmarking 1.11 Conclusion 1.12 Risk Analysis and Improvement 2 Energy Security and Its Effect on Business Resiliency 2.1 Introduction 2.2 Risks Related to Information Security 2.3 How Risks are Addressed 2.4 Use of Distributed Generation 2.5 Documentation and Its Relation to Information Security 2.6 Smart Grid 2.7 Conclusion 2.8 Risk Analysis and Improvement

1 2 4 6 6 8 9 10 10 14 18 19 20 21 22 22 25 25 29 34 37 40 42 44 45 v

VI

CONTENTS

Mission Critical Engineering with an Overview of Green Technologies 3.1 Introduction 3.2 Companies'Expectations: Risk Tolerance and Reliability 3.3 Identifying the Appropriate Redundancy in a Mission Critical Facility 3.3.1 Load Classifications 3.4 Improving Reliability, Maintainability, and Proactive Preventative Maintenance 3.5 The Mission Critical Facilities Manager and the Importance of the Boardroom 3.6 Quantifying Reliability and Availability 3.6.1 Review of Reliability Terminology 3.7 Design Considerations for the Mission Critical Data Center 3.7.1 Data Center Certification 3.8 The Evolution of Mission Critical Facility Design 3.9 Human Factors and the Commissioning Process 3.10 Short-Circuit and Coordination Studies 3.10.1 Short-Circuit Study 3.10.2 Coordination Study 3.11 Introduction to Direct Current in the Data Center 3.11.1 Advantages of DC Distribution 3.11.2 DC Lighting 3.11.3 DC Storage Options 3.11.4 Renewable Energy Integration 3.11.5 DC and Combined Cooling, Heat, and Power 3.11.6 Current State of the Art 3.11.7 Safety Issues 3.11.8 Maintenance 3.11.9 Education and Training 3.11.10 Future Vision 3.12 Containerized Systems Overview 3.13 Conclusion

47

4 Mission Critical Electrical System Maintenance and Safety 4.1 Introduction 4.2 The History of the Maintenance Supervisor and the Evolution of the Mission Critical Facilities Engineer 4.3 Internal Building Deficiencies and Analysis 4.4 Evaluating Your System 4.5 Choosing a Maintenance Approach 4.5.1 Annual Preventive Maintenance 4.6 Safe Electrical Maintenance 4.6.1 Standards and Regulations 4.6.2 Electrical Safety: Arc Flash 4.6.3 Personal Protective Equipment (PPE)

81 81 83

47 48 50 51 52 53 54 55 56 57 58 59 60 60 63 65 65 67 67 68 68 68 70 71 71 71 72 73

85 86 87 88 89 89 90 92

CONTENTS

4.7

4.8 4.9

Vil

4.6.4 Lockout/Tagout Maintenance of Typical Electrical Distribution Equipment 4.7.1 Thermal Scanning and Thermal Monitoring 4.7.2 15 kV Class Equipment 4.7.3 480 Volt Switchgear 4.7.4 Motor Control Centers and Panel Boards 4.7.5 Automatic Transfer Switches 4.7.6 Automatic Static Transfer Switches (ASTS) 4.7.7 Power Distribution Units 4.7.8 277/480 Volt Transformers 4.7.9 Uninterruptible Power Systems Being Proactive in Evaluating Test Reports Conclusion

5 Standby Generators: Operations and Maintenance 5.1 Introduction 5.2 The Necessity for Standby Power 5.3 Emergency, Legally Required, and Optional Systems 5.4 Standby Systems That Are Legally Required 5.5 Optional Standby Systems 5.6 Understanding Your Power Requirements 5.7 Management Commitment and Training 5.7.1 Lockout/Tagout 5.7.2 Training 5.8 Standby Generator Systems Maintenance Procedures 5.8.1 Maintenance Record Keeping and Data Trending 5.8.2 Engine 5.8.3 Coolant System 5.8.4 Control System 5.8.5 Generator Mechanics 5.8.6 Automatic and Manual Switchgear 5.8.7 Load Bank Testing 5.9 Documentation Plan 5.9.1 Proper Documentation and Forms 5.9.2 Record Keeping 5.10 Emergency Procedures 5.11 Cold Start and Load Acceptance 5.12 Nonlinear Load Problems 5.12.1 Line Notches and Harmonic Current 5.12.2 Step Loading 5.12.3 Voltage Rise 5.12.4 Frequency Fluctuation 5.12.5 Synchronizing to Bypass 5.12.6 Automatic Transfer Switch 5.13 Conclusion

98 99 100 102 102 103 103 104 105 105 105 107 107 109 109 110 111 112 113 113 113 114 115 115 116 116 117 117 117 117 118 118 118 119 119 120 121 121 121 122 122 122 123 123

VIII

CONTENTS

6 Fuel Systems Design and Maintenance 6.1 Introduction 6.2 Brief Discussion on Diesel Engines 6.3 Bulk Storage Tank Selection 6.3.1 Aboveground Tanks 6.3.2 Modern Underground Tanks and Piping Systems 6.4 Codes and Standards 6.5 Recommended Practices for All Tanks 6.6 Fuel Distribution System Configuration 6.7 Day Tank Control System 6.8 Diesel Fuel and A Fuel Quality Assurance Program 6.8.1 Fuel Needs and Procurement Guidelines 6.8.2 New Fuel Shipment Prereceipt Inspection 6.8.3 Analysis of New Fuel Prior to Transfer to On-Site Storage 6.8.4 Monthly Fuel System Maintenance 6.8.5 Quarterly or Semiannual Monitoring of On-Site Bulk Fuel 6.8.6 Remediation 6.9 Conclusion

125 125 126 126 127 128 128 129 133 135 139 141 141 144

7 Power Transfer Switch Technology, Applications, and Maintenance 7.1 Introduction 7.2 Transfer Switch Technology and Applications 7.3 Types of Power Transfer Switches 7.3.1 Manual Transfer Switches 7.3.2 Automatic Transfer Switches 7.4 Control Devices 7.4.1 Time Delays 7.4.2 In-Phase Monitor 7.4.3 Test Switches 7.4.4 Exercise Clock 7.4.5 Voltage and Frequency Sensing Controls 7.5 Design Features 7.5.1 Close Against High In-Rush Currents 7.5.2 Withstand and Closing Rating (WCR) 7.5.3 Carry Full Rated Current Continuously 7.5.4 Interrupt Current 7.6 Additional Characteristics and Ratings of ATS 7.6.1 NEMA Classification 7.6.2 System Voltage Ratings 7.6.3 ATS Sizing 7.6.4 Seismic Requirement 7.7 Installation and Commissioning, Maintenance, and Safety 7.7.1 Installation and Commissioning

149

145 146 146 148

149 151 152 152 153 163 163 164 165 165 166 166 166 167 167 167 167 167 168 168 168 168 168

CONTENTS

7.8 7.9

7.7.2 Maintenance and Safety 7.7.3 Maintenance Tasks 7.7.4 Drawing and Manuals 7.7.5 Testing and Training General Recommendations Conclusion

170 173 173 173 176 176

8 Static Transfer Switch 8.1 Introduction 8.2 Overview 8.2.1 Major Components 8.3 Typical Static Switch, One-Line Diagram 8.3.1 Normal Operation 8.3.2 Bypass Operation 8.3.3 STS and STS/Transformers Configurations 8.4 STS Technology and Application 8.4.1 General Parameters 8.4.2 STS Location and Type 8.4.3 Advantages and Disadvantages of the Primary and Secondary STS/Transformer Systems 8.4.4 Monitoring, Data Logging, and Data Management 8.4.5 Downstream Device Monitoring 8.4.6 STS Remote Communication 8.4.7 Security 8.4.8 Human Engineering and Eliminating Human Errors 8.4.9 Reliability and Availability 8.4.10 Repairability and Maintainability 8.4.11 Fault Tolerance and Abnormal Operation 8.5 Testing 8.6 Conclusion

179 179 180 180 181 181 182 183 183 184 184 184

9 Fundamentals of Power Quality 9.1 Introduction 9.2 Electricity Basics 9.2.1 Basic Circuit 9.2.2 Power Factor 9.3 Transmission of Power 9.3.1 Life Cycle of Electricity 9.3.2 Single-Phase and Three-Phase Power Basics 9.3.3 Unreliable Power versus Reliable Power 9.4 Understanding Power Problems 9.4.1 Power Quality Transients 9.4.2 RMS Variations 9.4.3 Causes of Power Line Disturbances 9.4.4 Power Line Disturbance Levels 9.5 Tolerances of Computer Equipment

193 193 195 196 196 197 198 199 201 202 202 204 207 212 212

184 185 185 186 187 187 189 189 190 190

CONTENTS

X

9.6 9.7 9.8

9.5.1 CBEMA Curve 9.5.2 ITIC Curve 9.5.3 Purpose of Curves Power Monitoring The Deregulation Wildcard Conclusion

10 UPS Systems: Applications and Maintenance with an Overview of Green Technologies 10.1 Introduction 10.1.1 Green Technologies and Reliability Overview 10.2 Purpose of UPS Systems 10.3 General Description of UPS Systems 10.3.1 What is a UPS System? 10.3.2 How Does a UPS System Work? 10.3.3 Static UPS Systems 10.3.4 Online 10.3.5 Double Conversion 10.3.6 Double Conversion UPS Power Path 10.4 Components of a Static UPS System 10.4.1 Power Control Devices 10.5 Online Line Interactive UPS Systems 10.6 Offline (Standby) 10.7 The Evolution of Static UPS Technology 10.7.1 Emergence of the IGBT 10.7.2 Two-and Three-Level Rectifier/Inverter Topology 10.8 Rotary UPS Systems 10.8.1 UPSs Using Diesel 10.8.2 Hybrid UPS Systems 10.9 Redundancy, Configurations, and Topology 10.9.1 N Configuration 10.9.2 N+l Configuration 10.9.3 Isolated Redundant Configuration 10.9.4 N + 2 Configuration 10.9.5 2N Configuration 10.9.6 2N+ 1 Configuration 10.9.7 Distributed Redundant/Catcher UPS 10.9.8 Eco-Mode for Static UPS 10.9.9 Availability Calculations 10.10 Energy Storage Devices 10.10.1 Battery 10.10.2 Flywheel Energy 10.11 UPS Maintenance and Testing 10.11.1 Physical Preventive Maintenance (PM) 10.11.2 Protection Settings, Calibration, and Guidelines

214 215 215 215 217 221 223 223 223 225 228 228 228 229 230 230 231 232 232 238 239 240 240 241 242 243 244 245 245 246 246 246 248 248 249 249 250 251 251 255 256 257 258

CONTENTS

10.11.3 Functional Load Testing 10.11.4 Steady-State Load Test 10.11.5 Steady-State Load Test at 0%, 50%, and 100% of Load 10.11.6 Harmonic Analysis and Testing 10.11.7 Filter Integrity and Testing 10.11.8 Transient Response Load Test 10.11.9 Module Fault Test 10.11.10 Battery Rundown Test 10.12 Static UPS and Maintenance 10.13 UPS Management 10.14 Conclusion Center Cooling Systems Introduction Background Information Cooling within Datacom Rooms Cooling Systems 11.4.1 Air Side 11.4.2 Cooling-Medium Side 11.5 Components Outside the Datacom Room 11.5.1 Refrigeration Equipment—Chillers 11.5.2 Heat-Rejection Equipment 11.5.3 Energy-Recovery Equipment 11.5.4 Heat Exchangers 11.6 Components Inside the Datacom Rooms 11.6.1 CRAC Units 11.7 Conclusion

XI

258 259 259 259 260 261 261 261 262 262 263

11 Data 11.1 11.2 11.3 11.4

265 265 266 266 267 267 267 269 269 273 282 287 290 290 295

12 Data Center Cooling Efficiency: Concepts and Advanced Technologies 12.1 Introduction 12.1.1 Data Center Efficiency Measurement 12.2 Heat Transfer Inside Data Centers 12.2.1 Heat Generation 12.2.2 Heat Return 12.2.3 Cooling Air 12.3 Cooling and Other Airflow Topics 12.3.1 Leakage 12.3.2 Mixing and Its Relationship to Efficiency 12.3.3 Recirculation 12.3.4 Venturi Effect 12.3.5 Vortex Effect 12.3.6 CRAC/CRAH Types 12.3.7 Potential CRAC Operation Issues 12.3.8 Sensible Versus Latent Cooling

297 297 299 300 301 302 302 303 303 303 303 304 304 304 305 305

CONTENTS

12.3.9 Humidity Control 12.3.10 CRAC Fighting—Too Many CRACs 12.4 Design Approaches for Data Center Cooling 12.4.1 Hot Aisle/Cold Aisle 12.4.2 Cold-Aisle Containment 12.4.3 In-Row Cooling with Hot-Aisle Containment 12.4.4 Overhead Supplemental Cooling 12.4.5 Chimney or Ducted Returns 12.4.6 Advanced Active Airflow Management for Server Cabinets 12.5 Additional Considerations 12.5.1 Active Air Movement 12.5.2 Adaptive Capacity 12.5.3 Liquid Cooling 12.5.4 Cold Storage 12.6 Hardware and Associated Efficiencies 12.6.1 Server Efficiency 12.6.2 Server Virtualization 12.6.3 Multicore Processors 12.6.4 Blade Servers 12.6.5 Energy-Efficient Servers 12.6.6 Power Managed Servers 12.6.7 Effects of Dynamic Server Loads on Cooling 12.7 Best Practices 12.8 Efficiency Problem Solving 12.9 Conclusion 12.10 Conversions, Formulas, Guidelines 13 Raised Access Floors 13.1 Introduction 13.1.1 What is an Access Floor? 13.1.2 What are Typical Applications for Access Floors? 13.1.3 Why Use an Access Floor? 13.2 Design Considerations 13.2.1 Determine the Structural Performance Required 13.2.2 Determine the Required Finished Floor Height 13.2.3 Determine the Understructure Support Design Type Required 13.2.4 Determine the Appropriate Floor Finish 13.2.5 Airflow Requirements 13.3 Safety Concerns 13.3.1 Removal and Reinstallation of Panels 13.3.2 Removing Panels 13.3.3 Stringer Systems 13.3.4 Protection of the Floor from Heavy Loads

307 308 308 308 309 309 309 310 310 310 310 311 311 312 312 312 313 313 313 313 313 313 314 314 316 316 317 317 317 318 319 319 320 322 323 325 326 328 328 328 330 331

CONTENTS

13.4

13.5

13.6

13.7

13.8

XIII

13.3.5 Grounding the Access Floor 13.3.6 Fire Protection 13.3.7 Zinc Whiskers Panel Cutting 13.4.1 Safety Requirements for Cutting Panels 13.4.2 Guidelines for Cutting Panels 13.4.3 Cutout Locations in Panel—Supplemental Support for Cut Panels 13.4.4 Saws and Blades for Panel Cutting 13.4.5 Interior Cutout Procedure 13.4.6 Round Cutout Procedure 13.4.7 Installing Protective Trim Around Cut Edges 13.4.8 Cutting and Installing the Trim Access Floor Maintenance 13.5.1 Best Practices for Standard High-Pressure Laminate Floor Tile (HPL) and for Vinyl Conductive and Static Dissipative Tile 13.5.2 Damp Mopping Procedure for HPL and Conductive and Static Dissipative Vinyl Tile 13.5.3 Cleaning the Floor Cavity Troubleshooting 13.6.1 Making Pedestal Height Adjustments 13.6.2 Rocking Panel Condition 13.6.3 Panel Lipping Condition (Panel Sitting High) 13.6.4 Out-of-Square Stringer Grid (Twisted Grid) 13.6.5 Tipping at Perimeter Panels 13.6.6 Tight Floor or Loose Floor—Floor Systems Laminated with HPL Tile Additional Design Considerations 13.7.1 LEED Certification 13.7.2 Energy Efficiency—Hot and Cold Air Containment 13.7.3 Airflow Distribution and CFD Analysis Conclusion

14 Fire Protection in Mission Critical Infrastructures 14.1 Introduction 14.2 Philosophy 14.3 Alarm and Notification 14.4 Early Detection 14.5 Fire Suppression 14.6 System Designs 14.6.1 Stages of a Fire 14.6.2 Fire and Building Codes 14.7 Fire Detection 14.8 Fire Suppression Systems

336 337 337 328 328 328 338 339 339 339 340 340 340 341 342 342 343 343 343 343 344 345 345 346 346 346 347 354 357 357 358 359 361 362 364 364 365 366 374

CONTENTS

XIV

14.8.1 14.8.2 14.8.3 14.8.4 14.8.5 14.8.6 14.8.7 14.8.8 14.8.9 14.8.10 Appendix A

Water Mist Systems Carbon Dioxide Systems Clean Agent Systems Inert Gas Agents IG-541 IG-55 Chemical Clean Agents Portable Fire Extinguishers Clean Agents and the Environment Conclusion

Policies and Regulations A.l Introduction A.2 Industry Policies and Regulations A.2.1 USA PATRIOT Act A.2.2 Sarbanes-Oxley Act (SOX) A.2.3 Comprehensive Environmental Response, Compensation, and Liability Act of 1980 A.2.4 Executive Order 13423—Strengthening Federal Environmental, Energy, and Transportation Management A.2.5 ISO27000 Information Security Management Systems (ISMS) A.2.6 The National Strategy for the Physical Protection of Critical Infrastructure and Key Assets A.2.7 2009 National Infrastructure Protection Plan A.2.8 North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection Program A.2.9 U.S. Security and Exchange Commission (SEC) A.2.10 Sound Practices to Strengthen the Resilience of the U.S. Financial System A.2.11 C4I—Command, Control, Communications, Computers, and Intelligence A.2.12 Basel II Accord A.2.13 National Institute of Standards and Technology (NIST) A.2.14 Business Continuity Management Agencies and Regulating Organizations A.2.15 FFIEC—Federal Financial Institutions Examination Council A.2.16 National Fire Prevention Association 1600 Standard on Disaster/Emergency Management and Business Continuity Programs

379 382 384 384 385 385 386 390 390 391 393 393 395 396 397 399 399 400 403 404 405 405 405 407 408 408 410 412 412

CONTENTS CONTENTS

XV

A.2.17 Private Sector Preparedness Act .3 Data Protection .4 Encryption A.4.1 Protecting Critical Data through Security and Vaulting .5 Business Continuity Plan (BCP) .6 Conclusion

414 414 416 417 417 419

Appendix B

Consolidated List of Key Questions

421

Appendix C

Airflow Management: A Systems Approach C.l Introduction C.2 Control is the Key C.2.1 Benefits of Control C.3 Obtaining Control C.3.1 Lower Return Air AT Versus Higher AT C.4 Air Management Technologies C.4.1 In-Row Cooling C.4.2 Overhead Cooling C.4.3 Containment Strategies C.4.4 Active-Air Management C.4.5 A Benchmark Study for Comparison C.5 Conclusion

441 441 442 444 445 445 451 452 452 452 453 454 456

Glossary

459

Bibliography

473

Index

479

IEEE Press Series on Power Engineering

This page intentionally left blank

FOREWORD

Our lives, livelihoods, and way of life are increasingly dependent on computers and data communication, and this dependence increasingly relies on critical facilities or data centers where servers, mainframes, storage devices, and communication gear are brought together. In short, we are becoming a datacentric or data center society. We are all witnessing the extraordinary expansion of the Internet, from social media to search engines, games, content distribution, and e-commerce. The advent of cloud computing, the "everything as a service" model, will further amplify the importance of the data center as a hub for the new wired world. Consequently, there is an everincreasing demand on our information infrastructure, especially our data centers, changing the way we design, build, use, and maintain these facilities. However, the industry experts have been very slow to document and communicate the vital processes, tools, and techniques needed to do this. Not only is ours a dynamic environment, but it also is complex and requires an understanding of electrical, mechanical, fire protection, and security systems, and reliability concepts, operating processes, and much more. I realized the great benefit Peter Curtis' book will bring to our mission critical community soon after I started reviewing the manuscript. I believe this is the first attempt to provide a comprehensive overview of all of the interrelated systems, components, and processes that define the data center space, and the results are remarkable. Data center facilities are shaped by a paradox: critical infrastructure support systems and the facilities housing them are designed to last 15 years or more, whereas the IT equipment typically has a life of about three years. Thus, every few years we are faced with major IT changes that dramatically alter the computer technology and invariably impact the demand for power, heat dissipation, and the physical characteristics of the facility design and operation. In addition, the last few years have seen a growing focus on energy efficiency and sustainability, reflecting society's effort to reduce its carbon footprint and reverse global warming. Data centers are particularly targeted by these efforts because they are such huge users of power. It is no secret that one of the most difficult challenges facing our industry is our ability to objectively assess risk and critical facility robustness. In general, we lack the metrics needed to quantify reliability and availability—the ability to identify and align the function or business mission of each building with its performance expectation. XVII

XVIII

FOREWORD

Other industries, particularly aircraft maintenance and nuclear power plants, have spent years developing analytical tools to assess systems resiliency, and the work has yielded substantial performance improvements. In addition, the concept of reliability is sometimes misunderstood by professionals serving the data center industry. Curtis' efforts to define and explain reliability concepts will help improve performance in the mission critical space. Further, the process of integrating all of the interrelated components—programming space allocation, design, redundancy level planning, engineered systems quality, construction, commissioning, operation, documentation, contingency planning, personnel training, and so on—to achieve reliability objectives is clear and well reasoned. The book plainly demonstrates how and why each element must be addressed to achieve reliability goals. Although this concept appears obvious, it often is not fully understood or accepted, especially in our changing IT world, where we are facing the challenge of finding the right balance between complexity, automation, and human intervention. The comprehensive review of essential electrical and mechanical systems populating these facilities, from uninterruptible power supplies to chillers and generators, yields great benefits, not only from a functional standpoint, but also because it provides the necessary maintenance and testing data needed for effective system operation. And, maybe most importantly, Curtis recognizes and deals with the vital human factor, " . . . perhaps the most poorly understood aspect of process safety and reliability management." I am confident that time will validate the approach and ideas covered here. Meanwhile, we will all profit from this admirable effort to bring a better understanding to this complex and fast-changing environment. PETER GROSS

November 2010

PREFACE

The evolution of our digital society has sped up our lives so significantly that a critical event unfolds with extreme rapidness. Because of this societal transformation, our critical infrastructure is susceptible and vulnerable to catastrophic and escalating failures more than ever. Consequently, we now need to incorporate a new mindset and decision-making structure, process, and tools in order to manage unpredictable events such as power outages, natural disasters, or manmade incidences. This book facilitates bridging the gap between a critical event and the mission critical infrastructure that needs to be in place and working according to detailed specifications in order to manage the event with safety, confidence, and situational awareness. The intent of this text is to provide the foundations of mission critical infrastructure from an engineering and operations perspective. It should be noted that this book is a work in progress and does not include every detail. It does, however, provide the foundation topics as well as more advanced subjects of critical infrastructure that are relevant to society today. Topics such as reliability; resiliency; energy security; UPS systems; standby generators; automatic and static transfer switches; power quality; data center cooling, efficiency, and air-flow management; fire systems; raised floors; mission critical engineering; safety, and green practices are discussed in layman's terms. It is imperative that anyone wishing to enter into the mission critical industry have the necessary foundation to operate critical systems, with the goal of reducing operational risk, improving safety, and decreasing greenhouse gases. Not having the proper standardized training in place will lead to lost lives and cascading failures of every type due poor operational decisions. The purpose of this book is to transition today's facilities and IT engineers into the growing mission critical industry and properly equip them to deal with the hazards and challenges of a field that is such a lynchpin for the reliability and resiliency of our ever-evolving digital society. XIX

XX

PREFACE

Accompanying this book is a complete accredited online course for professional engineers, union personnel, corporate facilities engineers, property managers, IT managers, and policymakers. To learn more about the professional development hours (PDHs) offered for online training, go to www.missioncritical.powermanage.com. If you would like to make a comment regarding new material for the next edition please feel free to contact me at 510 Grumman Road West, Morrelly Homeland Security Center, Bethpage, NY 11714, [email protected]. PETER M. CURTIS

Bethpage, New York May 2011

ACKNOWLEDGMENTS

Creating this book could not be possible through the efforts of only one person. I have attended various conferences throughout my career, including 7/24 Exchange, IFMA, AFE, Data Center Dynamics, AFCOM, and BOMI, and harvested insights offered by many mission critical professionals from all walks of the industry. I am grateful for the professional relationships that were built at these conferences, which allow the sharing of the knowledge, know-how, information, and experiences upon which this book is based. I am also grateful to IEEE/Wiley for taking on this project almost 12 years ago. The format that initially began as material for online educational classes transcended into an entire manuscript passionately assembled by all who came across it. Professionals in the mission critical field have witnessed its evolution from a fledgling 40-hour-a-week operation into the 24/7 environment that our digital society demands today. The people responsible for the growth and maintenance of the industry have amassed an invaluable cache of knowledge and experience along the way. Compiling this information into a book provides a way for those new to this industry to tap into the years of experience that have emerged since the industry's humble beginnings. This book's intended audience includes every business that understands the consequences of downtime and seeks to improve its own business resiliency. Reviewed by senior management, technicians, vendors, manufacturers, and contractors alike, this book gives a comprehensive, 360 degree perspective on the mission critical industry as it stands today. Its importance lies in its use as a foundation toward a seamless transition to the next stages of education and training for the mission critical industry. I am grateful to many people and organizations for their help, support, and contributions that have enabled this information to be shared with the next generation of mission critical engineers and business continuity professionals.

Chapter Contributors • Don Beaty, P.E., DLB Associates (Chapter 11—Data Center Cooling Systems) • Charles Berry, Power Management Concepts, LLC (Chapter 12—Data Center Cooling Efficiency, and Chapter 10—UPS Systems) • Tom Bronack, CBCP (Appendix A—Policies and Regulations) • Dan Catalfu, Täte Access Floors (Chapter 13—Raised Access Floors) xxi

ACKNOWLEDGMENTS

XXII

• Howard L. Chesneau, Fuel Quality Services (Chapter 6—Fuel Systems and Design) • George E. Ello, Long Island Power Authority (Chapter 2—Energy Security) • Edward English III, Fuel Quality Services (Chapter 6—Fuel Systems and Design) • Brian K. Fabel, P.E., ORR Protection Systems (Chapter 14—Fire Protection) • James P. Fulton, Ph.D., Suffolk Community College (Appendix C—Airflow Management) • John Golde, P.E., Golde Engineering, PC (Chapter 3—Mission Critical Electrical System Maintenance and Safety) • John Kammetor (Chapter 8—Static Transfer Switches) • Walter Phelps, Degree Controls, Inc. (Chapter 12—Data Center Cooling Efficiency) • Dean Richards, Mitsubishi Electric Power Products (Chapter 10—UPS Systems) • Ron Ritorto, P.E., Mission Critical Fuel Systems (Chapter 6—Fuel Systems and Design)

Technical Reviewers and Editors • • • •

• • • • • • • • • • •

Scott Alwine, Täte Access Floors, Inc. (Chapter 13—Raised Access Floors) Bill Campbell, Emerson Network Power (Chapter 10—UPS Systems) Steve Carter, Orr Corporation (Chapter 14—Fire Protection) Thomas Corona, Jones Lang LaSalle (Chapter 4—Mission Critical Electrical System Maintenance and Safety, and Chapter 12—Data Center Cooling Efficiency) John C. Day, PDI Corp. (Chapter 8—Static Transfer Switches) John De Angelo, Power Service Concepts, Inc. (Chapter 10—UPS Systems) John Diamond, DAS Associates (Chapter 5—Standby Generators, and Chapter 10—An Overview of UPS Systems) Doug Dittmas, East Penn Manufacturing Company (Chapter 10—UPS Systems) Michael Fluegeman, P.E., PlanNet Consulting (Chapter 7—Power Transfer Switch Technology, and Chapter 10—UPS Systems) Steve Guzzardo, Hewlett-Packard (Chapter 1—Reliability and Resiliency) Richard Greco, P.E., California Data Center Design Group (Chapter 3—Mission Critical Engineering) Ross M. Ignall, Dranetz-BMI (Chapter 9—Fundamentals of Power Quality) Ellen Leinfuss, Dranetz-BMI (Chapter 9—Fundamentals of Power Quality) Teresa Lindsey, BITS (Chapter Questions) Wai-Lin Litzke, Brookhaven National Labs (Appendix A—Policies and Regulations)

ACKNOWLEDGMENTS

XXIII

• Michael Mallia, AFCO Systems (Appendix C —Air Flow Management) • Joseph McPartland III, American Power Conversion (Chapter 10—UPS Systems) • Stefan Miesbach, Siemens (Chapter 2—Energy Security) • Kevin McCarthy, EDG2 Inc. (Chapter 10—UPS Systems) • Mark Mills, Digital Power Group • David P. Mulholland, PDI (Chapter 8—Static Transfer Switches) • Gary Olsen, P.E., Cummins (Chapter 5—Standby Generators) • Ted Pappas, Keyspan Engineering (Chapter 3—Mission Critical Engineering) • Anthony Pinkey, Layer Zero Power Systems, Inc. (Chapter 8—Static Transfer Switches) • Richard Rotanz, Applied Science Foundation for Homeland Security (Appendix A—Policies and Regulations) • Dan Sabino, Power Management Concepts, LLC (Chapter 7—Power Transfer Switch Technology) • Douglas H. Sandberg, GHI Group (Chapter 7—Power Transfer Switch Technology) • Ron Shapiro, P.E., Cosentini Mission Critical, (Chapter 4—Mission Critical Electrical Systems Maintenance and Safety) • Terri Sinski, Strategic Planning Partners (Appendix A—Policies and Regulations) • Kenneth Uhlman, P.E., Eaton/Cutler Hammer (Technical Discussions) • Steve Vechy, Enersys (Chapter 10—UPS Systems) I am grateful to Dr. Robert Amundsen, Director of the Energy Management Graduate Program at New York Institute of Technology, who gave me my first teaching opportunity in 1994. It allowed me to continually develop professionally, learn, and pollinate many groups with the information presented in this book. I would like to thank two early pioneers of this industry for defining what mission critical really means to me and the industry, and I appreciate the knowledge they have imparted to me: Borio Gatto for sharing his engineering wisdom, guidance, and advice and Peter Gross, P.E. for his special message in the Foreword and for expanding my view of the mission critical world. Thank you to my good friends and colleagues for their continued support, technical dialogue, feedback, and advice over the years: Thomas Weingarten, P.E. of Power Management Concepts, LLC and Vecas Gray, P.E. and Mark Keller, Esq. of Abramson & Keller. I would also like to thank Lois Hutchinson for assisting in organizing and editing the initial material, acting as my sounding board, and supporting me while I was assimilating my ideas, and also for keeping me on track and focused when necessary. I would like to express gratitude to all the early contributors, students, mentors, in-

XXIV

ACKNOWLEDGMENTS

terns, and organizations that I have been working with and learning from over the years for their assistance, guidance, advice, research, and organization of the information presented in this book: John Altherr, P.E., Nada Anid, Ph.D., Elijan "Al" Avdic, Tala Awad, Anna Benson, Charles Berry, Nancy Camacho, Joseph Cappiello, Ralph Ciardulli, Charles Cottitta, Guy Davi, Kenneth Davis, Brad Dennis, David Donatov, Stephen Worn, David Dunatov, Andres Fortino, Ph.D., Ralph Günther, P.E., Kevin Heslin, Patrick Hoehn, Hartley Jean-Aimee, Al Law, John Nadon, John Mezic, John Montana, David Nowak, Jay O'Neill, Daniel Ortiz, Shawn Paul, Arnold Peterson, P.E., Victoria Pierre-Louis, Richard Realmuto, P.E, Michael Recher, Adam Redler, Edward Rosavitch, P.E, Christie Rotanz, Brad Weingast, Jack Willis, P.E., Anthony Wilson, and my special friends at 7x24 Exchange, AFE Region 4, Data Center Dynamics, Long Island Forum for Technology (www.lift.org), and Mission Critical Magazine. Special thanks to my friends and partners Bill Leone, CPA and Eduardo Browne for sharing my initial vision and building upon it. They have been pillars of strength for me. Most of all I would also like to express gratitude to my loving and understanding wife Elizabeth for her continual support, encouragement, and acceptance of my creative process, and leaving the house when I needed extreme quiet to focus and concentrate. I am also grateful to Elizabeth for her editorial assistance and feedback when my ideas were flowing 24/7, Thank you, Love! This book is dedicated to Brian K. Fabel, Al Baker, Bill Mann, Kenneth Morrelly, and John Kammeter—five inspirational men who contributed vastly to the mission critical and homeland security industries and enriched all of our lives. You will be missed. I would also like to dedicate this book to the men, women, and children who lost their lives on 9/11 and to the emergency personnel who responded to this horrific event. "We all die. The goal isn 't to live forever; the goal is to create something that will. " —Chuck Palahniuk Lastly, my deepest apologies go to anyone I have forgotten. P.M.C.

1 AN OVERVIEW OF RELIABILITY AND RESILIENCY IN TODAY'S MISSION CRITICAL ENVIRONMENT

1.1

INTRODUCTION

Continuous, clean, and uninterrupted power and cooling is the lifeblood of any data center, especially one that operates 24 hours a day, 7 days a week. Critical enterprise power is the power without which an organization would quickly be unable to achieve its business objectives. Today, more than ever, enterprises of all types and sizes are demanding 24-hour system availability. This means enterprises must have 24-hour power and cooling day after day, year after year. One such example is the banking and financial services industry. Business practices mandate continuous uptime for all computer and network equipment to facilitate round-the-clock trading and banking activities anywhere and everywhere in the world. Banking and financial service firms are completely intolerant of unscheduled downtime, given the guaranteed loss of business that invariably results. However, providing the best equipment is not enough to ensure 24-hour operation throughout the year. The goal is to achieve reliable 24-hour power and cooling at all times, regardless of the technological sophistication of the equipment or the demands placed upon that equipment by the end user, be it a business or municipality. Today, all industries are constantly expanding to meet the needs of the growing global digital economy. Industry as a whole has been innovative in the design and use of the latest technologies, driving its businesses to become increasingly digitized in Maintaining Mission Critical Systems in a 24/7 Environment, Second Edition. Peter M. Curtis © 2011 the Institute of Electrical and Electronics Engineers, Inc. Published 2011 by John Wiley & Sons, Inc.

1

2

RELIABILITY AND RESILIENCY IN TODAY'S MISSION CRITICAL ENVIRONMENT

this highly competitive business environment. Industry is progressively more dependent on continuous operation of its data centers in reaction to the competitive realities of a global economy. To achieve optimum reliability when the supply and availability of power are becoming less certain is challenging to say the least. The data center of the past required only the installation of stand-alone protective electrical and mechanical equipment, mainly for computer rooms. Data centers today operate on a much larger scale, 24/7. The proliferation of distributed systems using hundreds of desktop PCs and workstations connected through LANs and WANs, simultaneously using dozens of software business applications and reporting tools, makes each building a computer room. As we add the total number of locations utilized by each bank all over the world utilizing the Internet, we now realize the necessity of a critical infrastructure and the associated benefits of uptime and reliability. The face of corporate America was severely scarred in the last decade by a number of historically significant events: the collapse of the dot.com bubble and the high-profile corporate scandals. These events have taken a significant toll on financial markets and have served to deflate the faith and confidence of investors. In response, governments and other global organizations enacted new or revised existing laws, policies, and regulations. In the United States, laws such as the Sarbanes-Oxley Act of 2002 (SOX), Basel II, and the U.S. PATRIOT Act were created. In addition to management accountability, another imbedded component of SOX makes it imperative that companies not risk losing data or even risk downtime that could jeopardize accessing information in a timely fashion. These laws can actually improve business productivity and processes. Many companies thoughtlessly fail to consider installing backup equipment or the proper redundancy based on their risk profile. Then, when the lights go out due to a major power outage, these same companies suddenly wake up, with the outcome of taking a huge hit operationally and financially. During the months following the Northeast blackout of 2003, there was a marked increase in the installation of uninterruptible power supply (UPS) systems and standby generators. Small and large businesses alike learned how susceptible they are to power disturbances and the associated costs of not being prepared. Some businesses that were not typically considered mission critical learned that they could not afford to be unprotected during a power outage. The Northeast blackout of 2003 emphasized the interdependencies across the critical infrastructure and the cascading impacts that occur when one component falters. Most automated teller machines (ATMs) in the affected areas stopped working, although several had backup systems that enabled them to function for a short period. Soon after the power went out, the Comptroller of the Currency signed an order authorizing national banks to close at their discretion. Governors in a number of affected states made similar proclamations for state-charted depository institutions. The end result was a loss of revenue, profits, and almost the loss of confidence in our financial system. More prudent planning and the proper level of investment in mission critical infrastructure for electric, water, and telecommunications utilities, coupled with proactive building infrastructure preparation and operations, could have saved the banking and financial services industry millions of dollars. At the present time, the risks associated with cascading power supply interruptions from the public electrical grid in the United States have increased due to the ever in-

1.1

INTRODUCTION

3

creasing reliance on computer and related technologies. Today, there are close to one trillion devices and one billion people connected to the Worldwide Web. As the number of computers and related technologies continue to multiply in this increasingly digital world, the demand for reliable quality power increases as well. Businesses are not only competing in the marketplace to deliver whatever goods and services are produced for consumption, but now they must compete to hire the best engineers from a dwindling pool of talent who can design the best infrastructures needed to obtain and deliver reliable power and cooling. This keeps the mission critical manufacturing and technology centers up and running with the ability to produce the very goods and services that sustain them. The idea that businesses today must compete for the best talent to obtain reliable power is not new, and neither are the consequences of failing to meet this challenge. Without reliable power, there are no goods and services for sale, no revenues, and no profits—only losses. Hiring and keeping the best-trained engineers employing the very best analyses, making the best strategic choices, and following the best operational plans to keep ahead of the power supply curve is essential for any technologically sophisticated business to thrive and prosper. A key to success is to provide proper training and educational resources to engineers so they may increase their knowledge and keep current on the latest mission critical technologies available all over the world, which is one of the purposes of this book. In addition, all companies need to develop an educational system and certification programs for young mission critical engineers to help combat the decreasing workforce necessary to sustain the growing mission critical industry. It is also essential for critical industries to constantly and systematically evaluate their mission critical systems, assess and reassess their level of risk tolerance versus the cost of downtime, and plan for future upgrades in equipment and services that are designed to meet business needs and ensure uninterrupted power and cooling supplies in the years ahead. Simply put, minimizing unplanned downtime reduces risk. Unfortunately, the most common approach is reactive, that is, spending time and resources to repair a faulty piece of equipment after it fails as opposed to identifying when the equipment is likely to fail and repairing or replacing it without interruption. If the utility goes down, install a generator. If a ground-fault trips critical loads, redesign the distribution system. If a lightning strike burns power supplies, install a new lightning protection system. Such measures certainly make sense, as they address real risks associated with the critical infrastructure; however they are always performed after the harm has occurred. Strategic planning can identify internal risks and provide a prioritized plan for reliability improvements that identify the root causes of failure before they occur. In the world of high-powered business, owners of real estate have come to learn that they, too, must meet the demands for reliable power supply to their tenants. As more and more buildings are required to deliver service guarantees, management must decide what performance is required from each facility in the building. Availability levels of 99.999% (5.25 minutes of downtime per year) allow virtually no facility downtime for maintenance or other planned or unplanned events. Moving toward high reliability is imperative. Moreover, avoiding the problems that can cause outages and unscheduled downtime never ends. Even planning and impact assessments are tasks that are never completed; they should be reviewed at least once every budget cycle.

4

RELIABILITY AND RESILIENCY IN TODAY'S MISSION CRITICAL ENVIRONMENT

The evolution of data center design and function has been driven by the need for uninterrupted power. Data centers now employ many unique designs developed specifically to achieve the goal of uninterrupted power within defined project constraints based on technological need, budget limitations, and the specific tasks each center must achieve to function usefully and efficiently. Providing continuous operation under all foreseeable risks of failure, such as power outages, equipment breakdown, internal fires, and so on, requires use of modern design and modeling techniques to enhance reliability. These include redundant systems and components, standby power generation, fuel systems, automatic transfer and static switches, pure power quality, UPS systems, cooling systems, raised access floors, and fire protection, as well as the use of probability risk analysis modeling software (each will be discussed in detail later) to predict potential future outages, develop maintenance, and upgrade action plans for all major systems. Also vital to the facilities life cycle is two-way communication between upper management and facilities management. Only when both ends fully understand the three pillars of infrastructure reliability—design, maintenance, and operation of critical environments (including the potential risk of downtime and recovery time)—can they fund and implement an effective plan. Because the costs associated with reliability enhancements are significant, sound decisions can only be made by quantifying performance benefits against downtime cost estimates for each upgrade option to determine the best course of action. Planning and careful implementation will minimize disruptions while making the business case to fund necessary capital improvements and implement comprehensive maintenance strategies. When the business case for additional redundancy, specialized consultants, documentation, and ongoing training reaches the boardroom, the entire organization can be galvanized to prevent catastrophic data losses, damage to capital equipment, and danger to life and limb.

1.2

RISK ASSESSMENT

Critical industries require an extraordinary degree of planning and assessing. It is important to identify the best strategies to reach the targeted level of reliability. In order to design a critical building with the appropriate level of reliability, the cost of downtime and the associated risks need to be assessed. It is important to understand that downtime occurs due to more than one type of failure: design failures, catastrophic failures, equipment failures, or failures due to human error. Each type of failure will require a different approach to prevention. A solid and realistic approach to business resiliency must be a priority, especially because the present critical infrastructure is inevitably designed with all the eggs located in one basket. Within the banking and financial services industries, planning the critical area places considerable pressure on designing an infrastructure that evolves in an effort to support continuous business growth. Routine maintenance and upgrading of equipment alone does not ensure continuous availability. The 24/7 operation of such services means an absence of scheduled interruptions for any reason, including routine maintenance, modifications, and upgrades. The main question is how and why infrastructure failures occur. Employing new methods of distributing critical power, under-

1.2

RISK ASSESSMENT

5

standing capital constraints, and developing processes that minimize human error are some key factors in improving recovery time in the event critical systems are impacted by base-building failures. The infrastructure reliability can be enhanced by conducting a formal risk management assessment (RMA), gap analysis, and by following the guidelines of the critical area program (CAP). The RMA and the CAP are used in other industries and customized specifically for needs of data center environments. The RMA is an exercise that produces a system of detailed, documented processes, procedures, and checks and balances designed to minimize operator and service-provider errors. The CAP ensures that only trained and qualified people are associated with and authorized to have access to critical sites. These programs coupled with probability risk assessment (PRA) address the hazards of data center uptime. The PRA looks at the probability of failure of each type of electrical power equipment. Performing a PRA can be used to predict availability, number of failures per year, and annual downtime. The PRA, RMA, and CAP are facilitating agents when assessing each step listed below: • • • • • • • • •

Engineering and design Project management Testing and commissioning Documentation Education and training Operation and maintenance Employee certification Risk indicators related to ignoring facility life cycle process Standard and benchmarking

Industry regulations and policies are more stringent than ever. They are heavily influenced by Basel II, the Sarbanes-Oxley Act (SOX), NFPA 1600, and the U.S. Securities and Exchange Commission (SEC). Basel II recommends "three pillars"—risk appraisal and control, supervision of assets, and monitoring of financial markets—to bring stability to the financial system and other critical industries. Basel II implementation involves identifying operational risk, then allocating adequate capital to cover potential loss. As a response to corporate scandals in the last decade, SOX came into force in 2002 and contains the following sections: The financial statement published by issuers is required to be accurate (Sec 401) Issuers are required to publish information in their annual reports (Sec 404) Issuers are required to disclose to the public, on an urgent basis, information on material changes in their financial condition or operations (Sec 409) Penalties of fines and/or imprisonment are imposed for not complying (Sec 802) The purpose of the NFPA 1600 Standard is to help the disaster management, emergency management, and business continuity communities to cope with critical events.

6

RELIABILITY AND RESILIENCY IN TODAY'S MISSION CRITICAL ENVIRONMENT

Keeping up with the rapid changes in technology has been a longstanding priority. The constant dilemma of meeting the required changes within an already constrained budget can become a limiting factor in achieving optimum reliability.

1.2.1

Levels of Risk

Risk can be described as the worst possible scenario that might occur while performing a task within a facility. Risk assesses how much we know or can predict about unforeseen circumstances. As we review risk, it is essential that the facility IT team has the proper change management processes and procedures in place for planned events, so that downtime can be minimized. Reducing the frequency of these events and understanding their impact is the key to proper critical environment management. Table 1.1 shows the three typical levels of impact—high, medium, and low—that result from event occurrence.

1.3

CAPITAL COSTS VERSUS OPERATION COSTS

Businesses are at the mercy of the mission critical facilities sustaining them. Each year, billions of capital dollars are spent on the electrical and mechanical infrastruc-

Table 1.1. Levels of risk impact on facilities Risk impact

Effects of system failure

High

It will cause an immediate interruption to the clients' critical operations such as: • Activity requiring a planned major utility service outage, or temporary elimination of system redundancy in the critical environment • Activity that would disrupt critical production operations • Activity that would likely result in an unplanned outage or disruption of operations if unsuccessful

Medium

There is time to recover without impacting the clients' critical operations, including any: • Activity requiring a planned service outage that does not affect systems but may impact noncritical operations • Activity that involves a significant reduction in system redundancy • Activity that is not likely to result in an unplanned outage in the critical environment or disruption of operations if unsuccessful

Low

It will not interrupt operations and will have minimum potential of affecting the clients' critical operations including: • Activity involving systems directly supporting operations but the execution of which will be transparent to operations • Activity that cannot result in an unplanned outage of the critical environment or impact operations if unsuccessful

None

Activity not associated with the critical environment

1.3

CAPITAL COSTS VERSUS OPERATION COSTS

7

ture that supports IT around the globe. It is important to keep in mind that downtime can cost companies millions of dollars per hour or more. An estimated 94% of all businesses that suffer a large data loss go out of business within two years, regardless of the size of the business. The daily operations of our economic system and our way of life depend on critical infrastructure being available 100% of the time with no exceptions. Critical industries are operating continuously, 365 days a year. Because conducting daily operations necessitates the use of new technology, more and more servers are being packed into a single rack. The growing number of servers operating 24/7 increases the need for power, cooling, and airflow. When a disaster causes the facility to experience lengthy downtime, a prepared organization is able to quickly resume normal business operations by using a predetermined recovery strategy. Strategy selection involves focusing on key risk areas and selecting a strategy for each one. Also, in an effort to boost reliability and security, the potential impacts and probabilities of these risks, as well as the costs to prevent or mitigate damages and the time needed to recover, should be established. Many organizations associate disaster recovery and business continuity only with IT and communication functions and miss other critical areas that can seriously impact their business. Within these areas may be a multitude of critical systems that require maintenance, the development of procedures, and appropriate documentation. Some of these systems are listed later in Table 1.3. One major area that necessitates strategy development is the banking and financial services industry. The absence of strategy that guarantees recovery has an impact on employees, facilities, power, customer service, billing, and customer and public relations. All areas require a clear, well-thought-out strategy based on recovery time objectives, cost, profitability impact, and safety. The strategic decision is based on some of the following factors: • The maximum allowable delay time prior to the initiation of the recovery process • The time frame required to execute the recovery process once it begins • The minimum computer configurations required to process critical applications • The minimum communication device and backup circuits required for critical applications • The minimum space requirements for essential staff members and equipment • The total cost involved in the recovery process and the total loss as a result of downtime Developing strategies with implementation steps means no time is wasted in a recovery scenario. The focus is to implement the plan quickly and successfully, and in order to accomplish this people must be properly trained. Is the person you hired 3 months ago up to this task? The right strategies implemented will effectively mitigate damages, minimize disruptions, reduce the cost of downtime, and remove the threat to life and safety.

8

RELIABILITY AND RESILIENCY IN TODAY'S MISSION CRITICAL ENVIRONMENT

1.4 CRITICAL ENVIRONMENT WORKFLOW AND CHANGE MANAGEMENT To assure reliable operation, a critical environment workflow and change management process must be established and followed. Commensurate roles and responsibilities of the engineering, technology, and security groups must be developed, implemented, and adhered to in order to manage both planned and unplanned events and associated risks. The critical environment (CE) is defined as the physical space and the systems within a facility that are uniquely configured, sized and dedicated to supporting specific critical business operations as defined by the user. There are many specific rooms and areas within facilities in today's ever-changing environment. Some are located within the buildings structure whereas others are located outside. Regardless of where a CE may be located, these locations have immediate impact on the client's ability to maintain business operations and continuity. Examples of some of these CE areas can be seen in Table 1.2.

Table 1.2. Critical Areas Data centers Operations center Electrical switchgear rooms Network equipment rooms (NER) Intermediate distribution frames (IDF) Main distribution frames (MDF) Main equipment rooms (MER) Telecom rooms (TR) Switching and hub rooms Voice telephone and data closets Server rooms Business continuity and technology recovery rooms Tape silo and storagetek rooms Local area network (LAN) rooms Business operations control rooms Uninterruptible power supply (UPS) rooms Command centers Chiller rooms and thermal energy storage spaces Building management, monitoring, and automation centers Mechanical equipment rooms Standby emergency power (SEP) generator and switchgear rooms

1.4

CRITICAL ENVIRONMENT WORKFLOW AND CHANGE MANAGEMENT

9

Critical infrastructure systems are prevalent throughout a facility. Depending on the facility size, there could be many redundant systems supporting the same critical environment. Knowing which systems could impact the clients' critical functions and operations is paramount. Some of these systems are listed in Table 1.3.

1.4.1

Change Management

Change management is a process for managing and communicating change across relevant functions and business units to ensure and deliver integration of procedures and processes. Note that during emergency situations, established emergency response and escalation procedures must be followed. When work is must be done within the critical environment, ranging from certain simple or routine cleaning and inspection tasks to very complex and detailed preventive maintenance, corrective maintenance, or construction efforts, it is essential that an orderly and thorough approach to work planning and execution be undertaken. In every instance where work is planned in the critical environment, all departments must ensure that risk to company operations is thoroughly assessed and that appropriate risk mitigation is in place while the work is performed. The level of detail required in a method of procedure (MOP) must be correlated to the complexity of the work and magnitude of the potential risk. Relatively complex or high-risk work must be meticulously detailed in the MOP. The detail required for less complex work would not necessarily be as extensive. The bottom line is that a proper-

Table 1.3. Critical systems Compressed air systems Utility power feeder systems Diesel engine fuel systems Fire and life safety systems Natural gas supply systems Electrical distribution and grounding systems Condenser water systems Telephone andfiberoptic communications systems Standby emergency power (SEP) systems Glycol systems Environmental control systems (chillers, CRACs, etc.) Water service systems Building management systems (BMS) Boilers Uninterruptible power supply (UPS) systems

10

RELIABILITY AND RESILIENCY IN TODAY'S MISSION CRITICAL ENVIRONMENT

ly developed, reviewed, and approved MOP will result in reduced risk to business operations. Required change request information includes: • • • • • • • •

Who is doing the work? What systems will be affected? Which areas of the building will be affected? Is there redundancy for the system being disrupted? Are there detailed procedures for the proposed task Is assistance needed from other lines of business? What hardware will be moved, added, or changed? How long will the task last?

If an outage is required: • • • • • • •

1.4.2

How long will power be out? Are there any critical points during the process (high risk) that can be identified? Will those systems be protected by UPS, generators, or other redundancy? What kind of backup systems are available if a problem arises? Will utility power be taken down? Are other feeds to the building affected? If redundancy is to be reduced, what redundancy will be lost and for how long?

Escalation Procedures

The purpose of the critical escalation procedures is to allow for the successful response to a critical site event. Following the escalation process assures proper notification and timely response. By assessing the event first, critical information will be available early on. It is important that a chain of command be followed because when events arise, teams need to ensure that communication and reactions are escalated in the proper fashion.

1.5

TESTING AND COMMISSIONING

The definition of commissioning (Cx) has developed over the past 12 years from being nothing more than vendor start-up to the full quality-control process it is today. Some still cling to the idea that commissioning is something that starts at the end of construction or after the design is complete. In some circles, they talk about levels of commissioning and level one starts after the design is complete at factory acceptance testing (FAT). The use of levels of commissioning goes out the window as soon as leadership in energy-efficient design (LEED) certification is injected into the mix, as LEED requires design reviews and other requirements that start earlier in the project. The best

1.5 TESTING AND COMMISSIONING

11

definition of commissioning can be found in ASHRAE's Guideline 0.* ASHRAE's Commissioning Guideline 0-2005 is a recognized model and good resource that explains commissioning as a quality-control process in detail and can be applied to critical facilities with some embellishment when it comes to verifying critical system performance. The quality-control process given by ASHRAE is in phases, starting with the predesign phase and continuing through the occupancy and operations phase. A summary of the phases given in the ASHRAE Guideline are given below, with some additions for mission critical facilities, and these phases should be included in all mission critical facility projects. • Predesign phase —Document owner's project requirements and basis of design • Design phase —Commission-focused design review —Writing Cx specifications • Construction phase —Factory acceptance testing (FAT) —Construction-check listing —Start-up (prefunctional) testing • Acceptance phase —Site acceptance (functional) —Acceptance testing to verify performance of critical equipment —Integrated testing —O&M document review —Staff training oversight —Develop and prove out EOPs, SOPs, and MOPs • Occupancy and operations phase —Continual review and updating of materials —Continual training of O&M staff —Reliability assurance testing (continual commissioning) A quality-control process would never be overlooked in any valuable production project. Why would we neglect quality control for critical facility projects? In the case of critical facility projects, quality control or commissioning starts at the predesign phase so we can make sure the owner's project requirements (OPRs) are fully developed and track all relevant documents such as the basis of design (BOD). We do this so that during value engineering we can evaluate any impact to the BOD and OPR, and if the proposed change does not meet the documented requirements, then the team must sign off on it. It should be clear that these documents are the foundation of any *www.wbdg.org/pdfs/comm_def.pdf

12

RELIABILITY AND RESILIENCY IN TODAY'S MISSION CRITICAL ENVIRONMENT

project and part of the quality control process required for the commissioning of a critical facility. In the design phase, we have a commissioning-focused design review that should not be confused with a peer review. In a commissioning-focused design review, the commissioning authority (CxA) should provide input on making the building and systems easier to commission and comment on equipment layout as it pertains to operational symmetry to help prevent operational staff from making errors during crisis situations. The CxA should also verify that bid documents adequately specify building commissioning as this will help reduce vendor change orders. It many cases, it is better to have the CxA provide commissioning specifications and have them included in the prepurchase and other bid sets. The focused review also needs to verify that there are adequate monitoring and control points specified to facilitate commissioning and O&M (trending capabilities, test ports, control points, gages, and thermometers). A review needs to include a review of design as it pertains to the reliability and redundancy standards of the owner and industry standards and verify that building O&M plan and documentation requirements specified are adequate. During the construction phase, much of the rudimentary testing is accomplished. During this time, the factory acceptance testing is being conducted and it is important to have the CxA involved to verify that the controls and interlocks will work with the complete system. During this time, equipment is being delivered and installed. During this installation, the vendors and GC should be verifying the installation using construction checklists provided be the CxA. These checklists basically track the construction process and verify that the vendor delivered what was paid for in good condition, and that it was installed properly and has the proper clearance. The vendor start-up will follow and, if performed in accordance with the agreed procedures, all the functions including all the alarms will be verified. It is important to track all these documents and have them signed off by the vendors as proof of proper start-up. In some cases, the vendors will sign off the documents and not perform all the requirements, and that will slow down the acceptance and integrated testing. In this case of improper start-up, the delays can be back charged to the responsible vendor. In the acceptance phase, the CxA will first operate all the equipment in all configurations and verify proper start-up by the vendors. Once this is completed, the CxA should verify performance of certain equipment without using vendor-provided test gear. This is done to keep the quality control process in the hands of the CxA and make sure that only calibrated equipment is used and the calculations are performed without bias. The equipment listed below is recommended to be subjected to this extra acceptance test phase: 1. 2. 3. 4.

Emergency power systems and controls Uninterruptible power supply (UPS) systems and batteries Flywheel energy storage systems Static transfer switches (STS) and all associated controls

If the acceptance testing is done properly, it as a minimum will verify that the equipment is worthy of critical load. In some cases, deficiencies found during this

1.5 TESTING AND COMMISSIONING

13

process have forced the vendors to meet their own specifications and improve product quality. We are now ready for integrated testing, and the intent of this test is to verify that the building and all the systems work together to meet the client's design requirements. Some hints for having a successful integrated test that proves proper operation and no unwanted system interaction are: 1. Perform a full data center heat-load test, including any enclosed cooling systems 2. Perform integrated testing at 25%, 50%, 75%, and 100% of the design load 3. Use data loggers on the data center floor to verify measured data and BMS controls 4. Check all operating modes, including maintenance configurations Staff training and operations documents need to be provided before we can start operations. Proper training must be given to the staff for systems and integrated operations. I would suggest that the vendors provide system training and the CxA provide overall operations documents. The operations documents should include maintenance operation procedures (MOP), emergency operation procedures (EOP), standard operation procedures (SOP), and alarm response procedures (ARP). With properly trained staff and proper operations documents, the human error faults can be minimized. As we stated earlier, the commissioning process continues into the occupancy phase to maintain operational continuity. A yearly review and update of training as required due to system upgrade or operational requirements maintains staff and procedural quality. For mission critical facilities, a yearly verification of performance for critical electrical systems prevents loss of productivity due to system degradation; this is referred to as reliability assurance testing. These tests should be similar to those performed in the system acceptance-test procedures used during the acceptance phase and should use the original data for trending any changes in the system. The reliability assurance testing should be performed after the vendor has provided preventive maintenance (PM). The reason we perform these tests after the vendor preventive maintenance routine is that the vendor will have interacted with a commissioned system and disassembled some portions. In some cases, they provide updated software or control boards. The system now needs to be certified through reliability assurance testing to be worthy of critical load. Remember that the vendor-provided PM does not measure performance or track system degradation, so without a reliability assurance testing program the quality control process will have been compromised. Before the facility goes online, it is crucial to resolve all potential equipment problems (technology, operations, etc.). This is the construction team's sole opportunity to integrate and commission all the systems, due to the facility's 24/7 mission critical status. At this point in the project, all systems installed have been tested at the factory and witnessed by a competent commissioning authority (CxA) familiar with the equipment processes and procedures. Once the equipment is delivered, set in place, and wired, it is time for the second phase of certified testing and integration. The importance of this phase is to verify and

14

RELIABILITY AND RESILIENCY IN TODAY'S MISSION CRITICAL ENVIRONMENT

certify that all components work together and to fine-tune, calibrate, and integrate the systems. There is a tremendous amount of preparation for this phase. The facilities engineer must work with the factory, field engineers, and independent test consultants to coordinate testing and calibration. Critical circuit breakers must be tested and calibrated prior to placing any critical electrical load on them. When all the tests are completed, the facilities engineer must compile the certified test reports, which will establish a benchmark for all future testing. The last phase is to train the staff on each major piece of equipment and prepare for the transition to operations. Many decisions regarding how and when to service a facility's mission critical electrical and mechanical equipment are going to be subjective. The objective is easy: a high level of safety and reliability from the equipment, components, and systems. But discovering the most cost-effective and practical methods required to accomplish this can be challenging. Network with colleagues, consult knowledgeable sources, and review industry and professional standards and best practices before choosing the approach best suited to your maintenance goals. Also, keep in mind that the individuals performing the testing and service should have the best training and experience available. You depend on their conscientiousness and decision-making ability to avoid potential problems with perhaps the most crucial equipment in your building. Most importantly, learn from your experiences and those of others. Maintenance programs should be continuously improving. If a scheduled procedure has not previously identified a problem, consider adjusting the schedule respectively. Examine your maintenance programs on a regular basis and make appropriate adjustments to constantly improve. Acceptance and maintenance testing are pointless unless the test results are evaluated and compared to standards and to previous test reports that have established benchmarks. It is imperative to recognize failing equipment and to take appropriate action as soon as possible. Common practice in this industry is for technicians to perform maintenance without reviewing prior work tickets and records. This approach defeats the value of benchmarking and trending, and must be improved. The mission critical facility engineers can then keep objectives in perspective and depend upon their options when faced with a real emergency. The importance of taking every opportunity to perform preventive maintenance thoroughly and completely, especially in mission critical facilities, cannot be stressed enough. If not, the next opportunity will come at a much higher price: downtime, lost business, and lost potential clients, not to mention the safety issues that arise when technicians rush to fix a maintenance problem. So do it correctly ahead of time and avoid shortcuts because it will be very difficult to do it again.

1.6

DOCUMENTATION AND THE HUMAN FACTOR

The mission critical industry's focus on physical infrastructure enhancements descends from the early stages of the trade, when all efforts were directed solely toward design and construction techniques to enhance mission critical equipment. Twenty-five years ago, the technology supporting mission critical loads was simple. There was little sophistication in the electrical load profile; at that time the indus-

1.6

DOCUMENTATION AND THE HUMAN FACTOR

15

try was in its infancy. Over time, the data centers have grown from a few mainframes supporting minimal software applications to server farms that can occupy 100,000 ft2 or more, with Google and Microsoft being prime examples. As more processing power is required to sustain our global economy, the electrical and mechanical systems supporting the critical load became increasingly complex. With businesses relying on this infrastructure, more capital dollars were invested to improve the uptime of the business's lines. Today, billions of dollars are invested on an enterprise level in the infrastructure that supports the business 24/7 applications; the major investments are normally in design, equipment procurement, and project management. Few capital dollars are invested in documentation, change management, education and training, or operations and maintenance. The initial capital investment was just the tip of the iceberg (Figure 1.1). Years ago, most organizations relied heavily on their workforce to retain much of the information regarding the mission critical systems. A large body of personnel had a similar level of expertise. They remained with their company for decades. Therefore, little emphasis was placed on maintaining a living document for a critical infrastructure. Tables 1.4 to 1.6 identify questions with regard to managing loss of personnel, documentation, and managing during a critical event. The mission critical industry can no longer manage critical systems as they did 25 years ago. The requirements are very different today in that the sophisticated nature of the data center infrastructure requires the constant refreshing and updating of documentation. One way to achieve this is to include a living document system that provides the level of granularity necessary to operate a mission critical infrastructure in a capital project. This will assist in keeping the living document current each time a project is completed or a milestone is reached. Accurate information is the first level of

Figure 1.1

Hidden costs of operations.

RELIABILITY AND RESILIENCY IN TODAY'S MISSION CRITICAL ENVIRONMENT

Table 1.4

Managing loss of critical personnel

The issues: employee turnover, retirement, sick leave, or vacation Was knowledge lost? Where is the existing documentation? How are new employees trained? What risks are faced during the transition?

Table 1.5

Documentation issues

The issue: traditional documentation systems are inconsistent, inaccessible, and unstructured. How is information shared? Is system data readily available? Where is the documentation? How are revisions approved and made available to all users?

Table 1.6

Managing during critical events

The threats:fires,natural disasters, blackouts, and intentional disruption Who should be contacted? Is your critical system data defined? Where are the procedures? Will you be able to respond in time?

support that provides first responders the intelligence they need to make informed decisions during critical events. It also acts like a succession plan as employees retire and new employees are hired, thus reducing risk and improving their learning curve. Remember that greater than 50% of all downtime can be tracked to human error. Human error as a cause of hazard scenarios must be identified and the factors that influence human errors must be considered. Human error is a given and will arise in all stages of the process. It is vital that the factors influencing the likelihood of errors be identified and assessed to determine if improvements in the human factors design of a process are needed. Surprisingly, human factors are perhaps the most poorly understood aspect of process safety and reliability management. Balancing system design and training operating staff in a cost-effective manner is essential to critical infrastructure planning. When designing a mission critical facility, the level of complexity and ease of maintainability is a major concern (Figures 1.2 and 1.3). When there is a problem, the facilities manager (FM) is under enormous pressure to isolate the faulty system while maintaining data center loads and other critical loads.

1.6

DOCUMENTATION AND THE HUMAN FACTOR

17

Figure 1.2 Typical Screenshot of mission critical access. (Courtesy of Power Management Concepts, LLC.)

Figure 1.3 Mission critical access Screenshot. (Courtesy of Power Management Concepts, LLC.)

18

RELIABILITY AND RESILIENCY IN TODAY'S MISSION CRITICAL ENVIRONMENT

The FM does not have the time to go through complex switching procedures during a critical event. A recipe for human error exists when systems are complex, especially if key system operators and documentation of emergency action procedures (EAP) and standard operating procedures (SOP) are not immediately available or have not been reviewed or updated periodically. A rather simplistic electrical system design will allow for quicker and easier troubleshooting during this critical time. To further complicate the problem, equipment manufacturers and service providers are challenged to find and retain the industry's top technicians within their own company. As 24/7 operations become more prevalent, the talent pool available will continue to diminish. This would indicate that response times could increase from the current standard of four hours to a much higher and less tolerable timeframe. The need for a simplified, easily accessible, and well-documented design is only further substantiated by the growing imbalance of supply and demand of highly qualified mission critical technicians. When designing a mission critical facility, a budgeting and auditing plan should be established. Each year, substantial amounts of money are spent on building infrastructure, but inadequate capital is allocated to sustain that critical environment through the use of proper documentation, education, and training.

1.7

EDUCATION AND TRAINING

Technology has been progressing faster than Moore's Law. Despite attaining high levels of technological standards in the mission critical industry, most of today's financial resources remain allocated for planning, engineering, equipment procurement, project management, and continued research and development. Unfortunately, little attention is given to the actual management of these systems. As equipment reliability increases, a larger percentage of downtime results from actions by personnel who were not properly trained or did not have access to accurate data during crisis events. The diversity among mission critical systems severely hinders people's ability to fully understand and master all necessary equipment and relevant information. In the past, a greater percentage of people were hands-on, and it was natural for many families to make their own home and auto repairs just out of necessity. In doing so, they became mechanically inclined and attained an understanding of how systems operate. This experience gave a number of today's mission critical professionals a set of skills to build upon. Today's "Nintendo generation" is gaining a slightly different set of skills through computers, software, and video games. They are gaining valuable experience with IT systems, and will have a solid foundation to continue to develop more advanced IT skills. The next step is to create a strong succession plan that teaches them how critical infrastructure operates and connects their already abundant IT knowledge to engineering. Then, existing professionals can show them how to apply that knowledge in the field. The best strategy may be to start training successors as early as possible so, upon retirement of current staff, someone is trained with the necessary experience to take on operational responsibilities. Such training may be online (Figure 1.4). New college programs that include internships should be developed and made attractive for young

1.8 OPERATION AND MAINTENANCE

19

Figure 1.4 Screenshot of an online training program. (Courtesy of Power Management Concepts, LLC, and Mission Critical Magazine.) engineers. These programs need to show real career path options and align with corporate needs. It is time to invest in our future, so that the people who will be running the critical infrastructure of our country will have the necessary skill sets needed to meet and exceed our current standards. We need to constantly evolve and improve as professionals or risk becoming extinct. If not addressed in a timely and proper manor, we jeopardize the foundation of how our everyday business is run and our e-commerce generated. Imagine what would happen if, due to inadequate training, no one fully understands how to operate and maintain our critical infrastructure before all the experience-hardened experts retire. With that being said, certified training programs should be developed by industry and instituted so there are established standards and best practices. It is only through education and training that we can guarantee facility employees are knowledgeable about all equipment and processes.

1.8

OPERATION AND MAINTENANCE

What can facility managers do to ensure that their critical system is as reliable as possible? The seven steps to improved reliability and maintainability are: 1. Planning and impact assessment 2. Engineering and design

20

RELIABILITY AND RESILIENCY IN TODAY'S MISSION CRITICAL ENVIRONMENT

3. 4. 5. 6. 7.

Project management Testing and commissioning Documentation Continuing education and training programs Operations and maintenance

Hire competent professionals to advise at each step of the way. When building a data processing center in an existing building, you do not have the luxury of designing the electrical and mechanical systems from scratch. A competent engineer will design a system that makes the most of the existing building design. However, before investing precious capital be sure you understand your business requirements for the next 3-5 years, as well as the reliability levels you must sustain. Use contractors who are experienced in data processing installations. Have an experienced firm inspect all systems, such as performing tests on circuit breakers and using thermal-scan equipment to find hot spots due to high-resistance connections or faulty equipment. Finally, you should plan for routine shutdowns of your facility so that you can perform preventive maintenance on critical equipment. Facility managers as well as senior management must not underestimate the cost-effectiveness of a thorough preventive maintenance program. Maintenance is not a luxury; it is a necessity. Do you want electrical outages to be scheduled or unscheduled? Or better yet, can you afford to deal with the consequences of an unscheduled outage? Integrating the ideal infrastructure is just about impossible. Therefore, seek out the best possible industry authorities to solve your problems. Competent consultants will have the knowledge, tools, testing equipment, training, and experience necessary to understand the risk tolerance of your company, as well as recommend and implement the proper and most-advanced proven designs. Whichever firms you choose, always ask for sample reports, testing procedures, and references. Your decisions will determine the system's ultimate reliability, as well as the ease of system maintenance. Seek experienced professionals from within your own company as well as outside professionals: information systems, property and operations managers, space planners, and the best consultants in the industry for all engineering disciplines. The bottom line is to have proven organizations working on your project.

1.9

EMPLOYEE CERTIFICATION

Empowering employees to function effectively and efficiently can be achieved through a well-planned certification program. Employees have a vested interest in working with management to reduce risk. Empowering employees to take charge in times of crisis creates valuable communication allies who not only reinforce core messages internally, but also carry them into daily operations. The internal crisis communication should be conducted using established communication channels and venues in addition to those that may have been developed to manage specific crisis scenarios.

1.10 STANDARDS AND BENCHMARKING

21

Whichever method of internal crisis communication a company may choose, the more upfront management is about what is happening, the better informed and more confident employees feel. In this way, security can be placed on an operation or a task requiring that an employee be certified to perform that action. Certification terms should be defined by industry best practices. Furthermore, the company's risk profile should include training and periodic recertification. Should these evaluations fall below standard over a period of time, the system could recommend decertification. Technology is driving itself faster than ever. Large investments are made in new technologies to keep up to date with advancements, yet industries are still faced with operational challenges. One possible reason for this is the limited training provided to employees operating the mission critical equipment. Employee certification is crucial not only to keep up with advanced technology, but also to promote quick emergency response and situational awareness. In the last few years, technologies have been developed to solve the technical problem of linkage and interaction of equipment but without well-trained personnel. How can we confirm that the employee meets the complex requirements of the facility to insure high levels of reliability?

1.10

STANDARDS AND BENCHMARKING

The past decade has seen wrenching change for many organizations. As firms and institutions have looked for ways to survive and remain profitable, a simple but powerful change strategy called benchmarking has become popular. The underlying rationale for the benchmarking process is that learning by example and from best-practice cases is the most effective means of understanding the principles and the specifics of effective practices. Recovery and redundancy together cannot provide sufficient resiliency if they can be disrupted by a single unpredictable event. A mission critical data center must be able to endure hazards of nature, such as earthquakes, tornados, floods, and other natural disasters, as well as human-made events. Great care should be taken to ensure those critical functions that will minimize downtime. Standards should be established with guidelines and mandatory requirements for continuity of business applications. Procedures should be developed for the systematic sharing of safety- and performance-related material, best practices, and standards. The key is to benchmark the facility on a routine basis with the goal of identifying performance deviations from the original design specifications. Done properly, this will provide an early warning mechanism to allow a potential failure to be addressed and corrected before it occurs. Once deficiencies are identified, and before any corrective action can be taken, a method of operation (MOP) must be written. The MOP will clearly stipulate step-by-step procedures and conditions, including who is to be present, the documentation required, phasing of work, and the state in which the system is to be placed after the work is completed. The MOP will greatly minimize errors and potential system downtime by identifying responsibility of vendors, contractors, the owner, the testing entity, and anyone else involved. In addition, a program of ongoing

22

RELIABILITY AND RESILIENCY IN TODAY'S MISSION CRITICAL ENVIRONMENT

operational staff training and procedures is important to deal with emergencies outside of the regular maintenance program. The most important aspect of benchmarking is that it is a process driven by the participants whose goal is to improve their organization. It is a process through which participants learn about successful practices in other organizations and then draw on those cases to develop solutions most suitable for their own organizations. True process benchmarking identifies the hows and whys for performance gaps and helps organizations learn and understand how to perform with higher standards of practice. Keep in mind that you cannot improve if you do not measure and benchmark.

1.11

CONCLUSION

Everyday industries are becoming increasingly dependent on continuous business operations. As a result, companies need to understand the level of reliability that they can supply to their customers and evaluate how this can either be improved or maintained. The following chapters will reinforce the concept that reliability and resiliency is dependent on an array of variables such as education and training, operation and maintenance, documentation, and testing and commissioning. It is the responsibility of employees at all levels of a hierarchy to communicate and develop best practices that will strengthen their business.

1.12

RISK ANALYSIS AND IMPROVEMENT

Below is a list of questions that you may wish to ask yourself about the needs analysis and risk assessment of the mission critical infrastructure you are supporting with regard to reliability and resiliency. Your answers to these questions should help to shed some light on areas where you can improve your operations. 1. How much does each minute, hour, or day of operational downtime cost your company if a specific facility is lost? 2. Have you determined your recovery time objectives for each of your business processes? 3. Does your financial institution conduct comprehensive business impact analyses (BIAs) and risk assessments? 4. Have you considered disruption scenarios and the likelihood of disruption affecting information services, technology, personnel, facilities, and service providers in your risk assessments? 5. Have your disruption scenarios included both internal and external sources, such as natural events (e.g., fires, floods, severe weather), technical events (e.g., communication failure, power outages, equipment and software failure), and malicious activity (e.g., network security attacks, fraud, terrorism)?

1.12

RISK ANALYSIS AND IMPROVEMENT

23

6. Does this BIA identify and prioritize business functions and state the maximum allowable downtime for critical business functions? 7. Does the BIA estimate data loss and transaction backlog that may result from critical business function downtime? 8. Have you prepared a list of "critical facilities" to include any location where a critical operation is performed, including all work area environments such as branch backroom operations facilities, headquarters, or data centers? 9. Have you classified each critical facility using a critical facility ranking/rating system such as the Tier I, II, III, and IV rating categories? 10. Has a condition assessment been performed on each critical facility? 11. Has a facility risk assessment been conducted for each of your key critical facilities? 12. Do you know the critical, essential, and discretionary loads in each critical facility? 13. Must you comply with the regulatory requirements and guidelines discussed in this chapter? 14. Are any internal corporate risk and compliance policies applicable? 15. Have you identified business continuity requirements and expectations? 16. Has a gap analysis been performed between the capabilities of each company facility and the corresponding business process recovery time objectives residing in that facility? 17. Based on the gap analysis, have you determined the infrastructure needs for your critical facilities? 18. Have you considered fault tolerance and maintainability in your facility infrastructure requirements? 19. Given your new design requirements, have you applied reliability modeling to optimize a cost-effective solution? 20. Have you planned for rapid recovery and timely resumption of critical operations following a wide-scale disruption? 21. Following the loss of accessibility of staff in at least one major operating location, how will you recover in a timely manor and resume critical operations? 22. Are you highly confident, through ongoing use or robust testing, that critical internal and external continuity arrangements are effective and compatible? 23. Have you identified clearing and settlement activities in support of critical financial markets? 24. Do you employ and maintain sufficient geographically dispersed resources to meet recovery and resumption activities? 25. Is your organization sure that there is diversity in the labor pool of the primary and backup sites, such that a wide-scale event would not simultaneously affect the labor pool of both sites?

24

RELIABILITY AND RESILIENCY IN TODAY'S MISSION CRITICAL ENVIRONMENT

26. Do you routinely use or test recovery and resumption arrangements? 27. Are you familiar with National Fire Protection Association (NFPA) 1600— Standard on Disaster/Emergency Management and Business Continuity Programs, which provides a standardized basis for disaster/emergency management planning and business continuity programs in private and public sectors by providing common program elements, techniques, and processes?

2 ENERGY SECURITY AND ITS EFFECT ON BUSINESS RESILIENCY

2.1

INTRODUCTION

Our nation's antiquated energy infrastructure and dependence on cheap fuel presents significant risks to our security. The transportation sector alone keeps the economy moving, and accounts for about two-thirds of all U.S. oil consumption. Electricity, on the other hand, plays a uniquely important role in the operations of all industries and public services. The loss of electricity for any length of time compromises data and communication networks, as digital electrical loads provide physical and operational security for all mission critical infrastructures. Without oil and electricity, our economy and security would come to a complete halt. Of concern to those considering energy security is the dependence of the United States on foreign energy sources. Fossil fuels make up the bulk of our energy sources, as seen in Figure 2.1, leaving us vulnerable to disruptions in their supply as well as their contribution to the release of greenhouse gases and other pollutants. Developing a more diverse energy portfolio is the solution to this issue; we must take a phased approach to introducing renewable energy sources into the power grid. Mission critical facilities may take a similar path, albeit on a smaller scale. By introducing renewable energy sources on-site, facilities may reduce their dependency on the grid and improve resiliency in the event of an outage by complementing conventional energy with alternatives. Maintaining Mission Critical Systems in a 24/7 Environment, Second Edition. Peter M. Curtis © 2011 the Institute of Electrical and Electronics Engineers, Inc. Published 2011 by John Wiley & Sons, Inc.

25

26

ENERGY SECURITY AND ITS EFFECT ON BUSINESS RESILIENCY

Figure 2.1. U.S. energy sources. (Courtesy ofCleanPower.org.) The electrical grid is so important that it was addressed in a late-2002 White House briefing involving the President's Critical Infrastructure Protection Board. The briefing specifically noted that the electric power grid now stands "in the crosshairs of knowledgeable enemies who understand that all other critical-infrastructure components depend on uninterrupted electricity for their day-to-day operations." In October 2009, the federal government announced plans to inject $8 billion into grid modernization efforts nationwide. The funding will be used to upgrade the existing grid, augment manufacturing of infrastructure components, and install 18 million smart meters. Computer hackers pose a significant threat to our information systems. There have been instances in which hackers have gained access to an electric power plant's computers and could have triggered major power interruptions. These events indicate how vulnerable and fragile our critical infrastructure really is. The electric grid is not the only area we need to be concerned with; the government, military, and business networks are all at risk, and officials are not acting swiftly enough to address these vulnerabilities. Steps need to be taken to improve information security and mitigate the threat of cyber attacks. The government is a major target for cyber attacks. From 2005 to 2007 alone, the Homeland Security Department, responsible for protecting civilian computer systems, suffered 850 cyber attacks, and they are becoming more frequent, targeted, and sophisticated. A distinction should be made between the actions of discontented teenagers and those of foreign or domestic operatives with the backing of a large organization or nation, who pose a far greater threat. Whereas the former group may launch an isolated attack, it is the latter group and their resources who are more likely to damage large computer networks and remotely interrupt power production or delivery. Military networks need to be safeguarded as well. Research shows that cyber threats are forming and are searching for ways to remotely disrupt operations. Denial of-service attacks are a major threat accomplished by bombarding a computer system

2.1

INTRODUCTION

27

with automated message traffic, causing bandwidth overload, the effects of which could approach the magnitude of a weapon of mass destruction. Analyses of the vulnerabilities of other critical infrastructure sectors have reached similar conclusions: The loss of electric power quickly brings down communications and financial networks; cripples the movement of oil, gas, water, and traffic; and paralyzes emergency response services. Conversely, a disruption in the transportation of coal, oil, and gas can also bring down central power plants along with the power grid. The stark reality is that a sustained interruption of any energy delivery system would cripple our country. The national electric grid is inherently vulnerable since a small number of large central power plants are linked to millions of customers by hundreds of thousands of miles of exposed transmission and distribution lines. Nearly all high-voltage electric lines run above ground throughout the country, with only a handful of high-voltage lines serving major metropolitan areas. The national electric grid is a vast, sprawling, multitiered structure that reaches everywhere and is used by everyone. The North American electric grid and the Internet are the largest networks on the planet. When one key transmission line fails, the load is spread to other lines, which may become overloaded and also fail, causing a domino effect and cascading outages. Most accidental grid interruptions last less than two seconds, and many powerquality issues involve problems that persist for only a few cycles or milliseconds. In most areas of the country, electric outages of less than a couple of hours occur only a few times per year, with longer outages even less common. Unless deliberate, there is a low risk that several high-voltage lines feeding a metropolitan area from several different points could fail simultaneously, and when one high-voltage transmission line does fail, resources are dispatched quickly to isolate the problem and make appropriate repairs and any necessary improvements. Deliberate assaults, by contrast, are much more likely to disable multiple nodes on the network simultaneously. A 2002 National Academy of Sciences report drove this reality home, observing: "A coordinated attack on a selected set of key points in the [electrical] system could result in a long-term, multi-state blackout. While power might be restored in parts of the region within a matter of days or weeks, acute shortages could mandate rolling blackouts for as long as several years."* Operations that can afford to simply shut down and wait out short blackouts may not be able to take that approach in response to the mounting threats of longer outages. Future plans for implementing a "smart grid" will reduce the effects a deliberate attack on the system. Over 90% of the top tier of the grid is typically fueled by coal, uranium, water, or gas, and the remainder by oil and nonhydro renewables such as solar photovoltaic, geothermal, biomass, and wind. The relative amount of each fuel used for electricity generation is shown in Figure 2.2. The 1% "other" category is composed of fuels such as propane, batteries, tire-derived fuels, and hydrogen. Technologies such as batteries and pumped storage can be used to store power produced off-peak and release it at peak demand. Each lower tier is typically fueled initially by the electric power delivered from the tier above. Power plants in the top tier deliver electrical power via miles of *National Research Council, Making the Nation Safer: The Role of Science and Technology in Countering Terrorism, National Academy Press, 2002.

28

ENERGY SECURITY AND ITS EFFECT ON BUSINESS RESILIENCY

Figure 2.2. Fuel sources for electricity generation in the United States in 2009. (Source: eia.doe.gov.)

high-voltage, long-haul transmission lines, which feed power into substations. The substations dispatch power, in turn, through miles of local distribution lines. At the same time, a few large power plants can provide all the power required by a large city. Many communities are served by just a handful of smaller power plants, or fractional shares of a few larger power plants. Since 2000, production from coal-fired plants has increased slightly, but electrical production by natural gas has increased by over 50% and wind power has increased almost tenfold. Nuclear power also saw a slight increase through higher efficiencies at existing plants. No new nuclear power plants have come online in the United States since 1996. Many different power plants operate in tandem to maintain power flows over regions spanning thousands of miles. In principle, segments of the grid can be cut off when transformers fail or lines go down, so that failures can be isolated before they cascade and disrupt power supplies over much larger regions. The effectiveness of such failure isolation depends on the level of spending on the electric grid, which has been in decline for years as a consequence of electric industry deregulation. Where investments are made, monitoring is improved to adequately transfer loads through the distribution system, thereby bringing the system up quicker after an outage. Identical strategies of isolation and redundancy are used on private premises to make the supplies of power to critical loads absolutely assured, insulating those loads from problems that may affect the grid. Switches control the flow of power throughout the grid, from the power plant down to the ultimate load. Interties between high-voltage transmission lines in the top tiers allow even the very largest plants to supplement and back up each other. When power stops flowing through the bottom tiers of the public grid, on-premise generators are designed to start up automatically. In defining priorities and designing new transmission and distribution, collaboration between utilities and critical power customers is becoming increasingly important,

2.2

RISKS RELATED TO INFORMATION SECURITY

29

most notably because power is essential for maintaining critical services for first responders, 911 call centers, air traffic control, wireline and wireless carriers, emergency response crews, hospitals, and data centers, among others. Critical facilities often have their own on-site backup generators and some, such as hospitals, are required to by code, so that utility power loss at any given time can be remedied rather promptly. However, owners and/or users of these critical facilities must provide adequate maintenance for the local generators and periodically exercise them under load to assure they operate properly and reliably when called upon. In addition, the facility must either tolerate or mitigate the initial power interruption from its onset until the backup generator successfully starts and is able to pick up the loads. From even a cursory review of the challenges faced by today's electrical grid, it is clear that a long-term solution for our production, distribution, and security needs will require a synthesis of our modern digital, renewable, and power distribution technologies. The smart grid is one solution proposed to fit our needs. An initial step will be the introduction of smart meters, which will allow end users to closely monitor and control their energy usage, as well as sell back excess energy produced by on-site renewable resources and generators. Such a digital system, when integrated on a large scale, will allow utilities to more efficiently produce power and provide it where it is needed, as well as help decrease the frequency and severity of outages. Energy security has serious repercussions for mission critical facilities. If the power is not flowing, business comes to a screeching halt. Whereas improving our energy security is a national/global imperative, facility owners and managers also have the option of taking steps to ensure the continued operation and success of their businesses. This may manifest itself in many different ways, through improving physical and cyber security, decreasing reliance on the electrical grid, improving employee training to decrease the occurrence of preventable service outages, and developing an effective disaster recovery plan.

2.2

RISKS RELATED TO INFORMATION SECURITY

The security of all of these networks is the subject of urgent, ongoing assessment. Much of the analysis has been focused on physical and cyber security—protecting the physical structures themselves or the computers that are used to control them. But their greatest vulnerability is the loss of power upon which every aspect of their control and operation ultimately depends. Although the multiple layers of the utility's critical infrastructure are highly interdependent, electric power is, more often than not, the prime mover—the key enabler of all the others. However, in the past, the energy industry has not typically been focused on information security risks, and has been even less concerned about privacy. Equipment failures due to information security vulnerabilities are not usually anticipated, and except for an acknowledgement of damage caused by data theft, the exploitation of those vulnerabilities is not usually seen as a likely cause of catastrophic events. The root cause of the August 2003 Northeast blackout was attributed to "Human decisions by various organizations, corporate and industry policy deficiencies, and inadequate manage-

30

ENERGY SECURITY AND ITS EFFECT ON BUSINESS RESILIENCY

ment."* Proper policies backed by strong information security measures are part of the solution, as well as solid training programs that include refresher courses in emergency action, alarm response, and standard operating procedures. According to the Federal Energy Regulatory Commission, both domestic and foreign hackers are now devoting considerable time and capital to mapping the technology infrastructures of companies. The network exploitation done to explore a network and map it has to be done whether the intruder is going to steal information, bring the network down, or corrupt data. Information security experts believe that this may be the cause of a few recent major blackouts. Hackers are like digital spies with the ability to steal information or disrupt networks remotely. Officials need to be more aware of security breaches, as they are a national/global security issue. Intellectual capital and industrial secrets are at risk, and keeping the risks quiet only makes the situation worse. The private sector, which owns most information networks that operate power plants, dams, and other critical infrastructure, needs to do more to improve security and protect critical data. A cyber attack could disrupt critical operations and impact customers. The smart grid, being a digital system, would be vulnerable to cyber attacks. To address this hazard, recommendations have been made to build the smart grid from the ground up with security in mind. An intelligent system would be able to detect intrusions and bypass affected nodes to keep electricity flowing to consumers. This capacity to "heal" through the use of installed "smart" switches throughout the network would create a grid that is more resilient to deliberate attacks and natural disasters. How do power outages relate to the level of reliability your company requires from an energy standpoint? Facilities can generally be classified by tiers, with Tier I being the most basic and Tier IV being the most reliable facility. The reason for having different tiers is due in large part to maintainability—maintaining the facility without shutting it down. Tiers I and II must be shut down to perform maintenance; Tiers III and IV are deemed concurrently maintainable. Critical functions will usually require a facility in the Tier III to Tier IV range or utilize other strategies such as colocation. Although rare, it is possible that critical business functions will be located in a Tier II or even a Tier I facility configuration, despite the fact that both lack full backup and redundancy support. This practice is not encouraged. Figure 2.3 identifies types of electric load interruptions associated with the recent significant power outages shown in Table 2.1. In fact, the energy industry is just beginning to utilize the latest operation technology. Some organizations lack even accurate and up-to-date information to provide first responders to grid outages with the intelligence and support necessary to make informed decisions during critical events. Keeping personnel motivated, trained, and ready to respond to emergencies is a challenge, made even greater without an appropriate records retrieval program in place. Augmenting security for utilities is seeing some progress. The U.S. Federal Government is taking steps to enhance physical and cyber security for utilities. The Critical Infrastructure Protection Cyber Security Standards, mandated by the Federal Energy *Bullock, J. A. et al., Introduction to Homeland Security, 2nd ed., Elsevier Butterworth-Heinemann, 2005; www.electripedia.info/reliability.asp.

2.2 RISKS RELATED TO INFORMATION SECURITY

31

Figure 2.3. Potential causes of load interruption or downtime. Regulatory Commission (FERC), are designed to reduce the risk to the reliability of the electric utility system and enhance security by protecting critical cyber assets (CCA). The Cyber Security Standards requires utilities to implement and document a program to identify, classify, and protect information associated with CCAs. Some facilities, control centers, and substations must undergo security assessment and augmentation when identified as critical assets. Access to these critical assets, whether in person or through cyber and electronic means, has to be authorized and will be controlled, monitored (with an immediate response to all unauthorized access attempts), and logged. Physical access will likely be controlled by the use of card reader systems. To be authorized for access, affected employees, contractors, and vendors are required to have an appropriate level of personnel risk assessment consisting of identity verification, seven-year criminal record search, and terrorist watch list search. In addition, they are also required to attend annual cyber security training and regular security awareness training. Most utilities are required to be compliant with the North American Electric Reliability Corporation (NERC) Cyber Security Standards CIP-002 through CIP-009. In order to be compliant, there are a number of physical security access control requirements that must be met at bulk power electric substations. These are substations handling large power transmission capabilities, not solely local electric distribution to local areas. The requirements are to control, monitor, and log access to critical cyber assets contained within the control houses at these substations. There is also a noncompliance self-reporting requirement that mandates utilities to self-report to NERC any known violation of the CIP standards. Another growing threat to be wary of is an EMP or electromagnetic pulse attack. An EMP can be generated via a nuclear warhead that is detonated above the Earth's at-

32

ENERGY SECURITY AND ITS EFFECT ON BUSINESS RESILIENCY

Table 2.1. Recent significant power outages Location

Cause

Effect

Denver, CO

Power rerouting for maintenance caused a system trip (7/1/2010)

• 7 substations brought down • 100,000 with no power for 15 minutes and 20,000 without power for over an hour

Chicago, IL

Severe storms (6/22/2010)

• 550,000 customers lost power for approximately 4 days

Kentucky

Winter storm (12/25/2009)

• 607,000 customers went without power

Siberia

Long-term maintenance negligence (8/17/09)

• 75 workers killed, 2 day blackout, oil spill

Florida

Significant equipment failure (2/26/08)

• 4,400,000 are left without power

New England

Lightning storms cause debris to damage power transmission lines (1/14/08)

• 20,000 people report power loss over the span of a week-long storm

San Francisco, CA Data center backup power generators failed (7/24/07)

• 40,000 customers directly affected. Internet users worldwide couldn't access Internet sites

Los Angeles, CA

Massive power outage, utility worker wiring error (9/12/05)

• Traffic and public transportation problems • Fear of a terrorist attack

Indonesia

Transmission line failure between Java and Bali (8/18/05)

• 100 million without power

Gulf Coast (Florida/ New Orleans)

2004/05 Hurricanes: Iban, Charley, Frances, Katrina, etc.

• Millions of customers without power, water, food and shelter; government records lost due to flooding

China

20 million kilowatt power shortage, • Multiple sporadic brownouts equivalent to the typical demand in • Government shuts down least the entire state of New York energy-efficient consumers (Summer 2005)

Greece

Temperatures near 104°F, mismanagement of electric grid (7/12/04)

• Over half of the country left without power

O'Hare Airport, Chicago, IL

Electrical explosion (7/12/04)

• Lost power to two terminals • Flight delays over course of a day

Logan Airport, Boston, MA

Electrical substation malfunction (7/5/04)

• Flight delays and security screening shutdown for 4 hours

Italy

Power line failures, bad weather (9/29/03)

• Nationwide power outage, 57 million people effected

2.2 RISKS RELATED TO INFORMATION SECURITY

33

Table 2.1. Continued Location

Cause

Effect

London

National grid failure (8/29/03)

• Over 250,000 commuters stranded

Northeast, Midwest, and Canada

Human decisions by various organizations, corporate and industry policy deficiencies, inadequate management (8/14/03)

• 50 million people effected due to 61,800 MW of capacity not being available

Brazil

Lightning strike (3/11/99)

• 75 million without power

Source: Google Alert—Major Power Outages. mosphere or by a "regular" explosion with the correct combination of an electrically sourced magnetic field. Essentially, this type of attack causes a massive electric surge that can potentially be over 10,000 volts per meter. An EMP attack would damage computers, electronics, electrical networks, and control systems. In an era that has become completely reliant on digital technology, such an assault would not only cause disorder, but completely shatter the ability of a country to operate normally. The United States has developed the Electromagnetic Pulse Commission to analyze the growing EMP threat. The main areas that the commission recommends to decrease susceptibility are deterrence, defense, and protection and recovery. Alongside other international organizations, the EMP Commission is aiding in developing a framework for infrastructure protection. But what organizations have been developed within the mission critical industry to educate and set standards for protecting vital infrastructures against such attacks? Today, EMP is not really an area of concern within critical industries and perhaps an EMP protection engineering field needs to be developed to start educating facility managers of this concern. Planning rationally for infrequent but grave contingencies is inherently difficult. Organizations that have prepared properly for yesterday's risk profiles may be unprepared for tomorrow's. The risk-of-failure profiles of the past reflect relatively benign threats, including routine equipment failures, lightning strikes on power lines, and such small-scale hazards as squirrels chewing through insulators or cars colliding into utility poles. Now, in addition to concerns about weather-related outages (hurricanes and ice storms in particular), as well as recent experiences underscoring the possibility of widespread operational outages, there is also the heightened concern of deliberate attacks on the grid. The latter changes the risk profile fundamentally; it poses the risk of outages that last a long time and extend over wide areas. The possibility that any parent company of any energy provider may have the ability to link their own network to those that control the power grid makes the threat of attack increasingly likely. The planning challenge now shifts from issues of power quality or reliability to issues of business sustainability. Planning must now take into account outages that last not for seconds or a single hour, but for days as a result of deliberate actions. Mission critical facilities do not have the luxury of being able to shut down or run at a reduced capacity during outages, whether they last minutes, hours, or days. Disaster

34

ENERGY SECURITY AND ITS EFFECT ON BUSINESS RESILIENCY

recovery plans are a necessity for mission critical facilities, involving the proper training of business continuity personnel to enact enterprise-level plans for business resiliency. In the past a company may have had a single employee responsible for emergency procedures at all of its locations, but the reality of today's threats necessitate a larger organization to support mission critical facilities. Those in charge of data centers must be familiar with the plans for their facility and able to work with local utilities and emergency personnel to ensure uptime continuity.

2.3

HOW RISKS ARE ADDRESSED

The need to provide continuous operation under all foreseeable risks of failure, such as power outages, equipment breakdown, natural phenomena, and terrorist attacks, requires the use of many techniques to enhance reliability and resiliency. These techniques include redundant systems and components such as standby power generation, UPS systems, automatic transfer switches, static transfer switches, and the use of probability-risk-analysis modeling software. This software identifies potential weaknesses of critical infrastructure, develops maintenance programs, and upgrades action plans for all major systems. Electric utility transmission and distribution system planners attempt to predict future load growth, design capital projects to construct the necessary additional capacity, and attempt to design adequate redundancy if the main supply line ever fails. This design concept identifies preferred and alternate or contingency supplies. It is used by most utilities, and is commonly referred to as an n + 1 or n + 2 design. However, the alternate supplies are not infinite in number. Electric system supply redundancy can be constructed in a number of ways. One method is to construct a power plant in an area (a load pocket) that needs the electricity. Power plants are huge capital investments, however, and must be offline periodically for extended maintenance and upgrades. Another way of bringing power as well as redundancy is extending transmission facilities from an adjacent area. This, too, can be a costly and time-consuming process, especially taking into account the permitting processes in many states. The upshot is that, ultimately, either strategy will likely be used under any given circumstances. However, state governmental utility regulators, permit grantors, and even the Federal Energy Regulatory Commission (FERC) all have a voice in how generation and transmission systems are constructed and reinforced in the future. Overall electric reliability is also dependent upon how the utility transmission and distribution facilities are constructed. Whereas lightning strikes affect overhead and underground facilities alike, storms affect overhead constructed facilities to a much greater extent than they affect underground facilities. In spite of this added disadvantage, the vast majority of utility facilities are overhead, constructed on poles. Underground construction, though more insulated from mechanical storm damage, is more expensive to construct, needs more redundancy built into the system, and is more labor intensive to troubleshoot and repair. Damage to overhead poles and wires is immediately visible and repaired, and end-of-life replacement is also much less costly. Pure economics clearly favors such overhead construction. As a result, owners of critical fa-

2.3 HOW RISKS ARE ADDRESSED

35

cilities will be predominantly supplied by overhead utilities that are subject to more, shorter interruptions, but are easier to repair. Even with the implementation of increased utility transmission, distribution system monitoring, and data communications effectively building toward the smart grid, outages may become less frequent, and corrected faster, but they will still occur. Factors such as overhead versus underground construction, alternate feeder supplies, and susceptibility to weather and other mechanical damages all play into a utility's overall reliability record. Electric utility reliability metrics employ several measurements, but the most commonly used are: • The Sustained Average Interruption Frequency Index (SAIFI). This is a measurement of the months between interruptions for the utility's electric customers. For example, a SAIFI of 0.9 indicates that the utility's average customer experiences a sustained electric interruption every 0.9 * 12 months = 10.8 months. • The Customer Average Interruption Duration Index (CAIDI). This is an average of outage minutes experienced by each customer that experiences a sustained interruption. For example, if a utility has a CAIDI of 120 minutes, this means that on average a power outage will be restored within 120 minutes. However, since utilities generally follow a prioritized restoration practice whereby outages affecting many customers are addressed prior to smaller and single-customer outages, the larger outages may be restored in 10 minutes (via automated switching), and smaller outages may take as long as 300 minutes to be restored. These long and short outages result in the overall CAIDI average of 120 minutes. • The Momentary Average Interruption Frequency Index (MAIFI). This measures the average number of momentary interruptions experienced by utility customers. Depending upon state regulations, momentary interruptions are defined as any interruption lasting less than 2 to 5 minutes. If the criterion is less than 5 minutes, an interruption of 4 minutes and 59 seconds will not count toward the utility SAIFI metric, but is considered momentary. It should be noted that in the mission critical industry, an outage of eight milliseconds can be catastrophic if the facility is not properly protected by a UPS or the critical systems do not operate according to their design specifications. Each metric measures a different statistic, and any organization can consult their local utility for information about the actual historical reliability metrics as they pertain to their facility's specific feeder supply. For an organization that is forward-thinking, these utility reliability indices will drive the level of redundancy and business resiliency as it pertains to the critical infrastructure. The organization should, in conjunction with the local utility's input, assess the utility's SAIFI, CAIDI, and MAIFI for both the utility's service territory as well as for the local distribution circuit supplying power to that business. Once these historical reliability metrics are known, the organization can plan for the likeliest and most feasible outage scenarios (many sustained interruptions but few momentary outages, long utility repair times, etc.) As mentioned previously, to

36

ENERGY SECURITY AND ITS EFFECT ON BUSINESS RESILIENCY

address human risk factors, SOPs, EAPs, and ARPs need to be available at a moment's notice so trained personnel can respond with situational awareness and confidence. Many companies use Web-based information management systems to address human risk factors. A living Web-based document system can produce a database of perpetually refreshed knowledge, providing the level of granularity necessary to operate, maintain, and repair mission critical infrastructure. Keeping the ever-changing documents current and secure can then be easily addressed each time a capital project is completed or an infrastructure change is made. One such program is M.C. Access, a Web-based document portal shown in Figure 2.4. It is important to secure this critical infrastructure knowledge and also leverage this asset for employee training and succession planning. Events such as the terrorist attacks of September 11th, the Northeast blackout of 2003, the 2006 hurricane season, and the outages in Italy and Greece in 2003 and 2004, respectively, which left many millions without power, have emphasized our interdependencies with other critical infrastructures, most notably telecommunications. There are numerous strategies and sector-specific plans such as Basel II, U.S. PATRIOT Act, SOX and, NFPA 1600, all of which highlight the responsibility of the private sector for increasing resiliency and redundancy in business processes and systems. These events have also prompted the revision of laws, regulations, and policies governing reliability and resiliency of the power industry. Some of these measures also

Figure 2.4. Mission critical access—a Web portal database program. (Courtesy of Power Management Concepts, LLC.)

2.4 USE OF DISTRIBUTED GENERATION

37

delineate controls required of some critical infrastructure sectors to maintain businesscritical operations during a critical event (see Appendix A for further information). The unintended consequence of identifying vulnerabilities is the fact that such diligence can actually invite attacks tailored to take advantage of them. In order to avoid this, one must anticipate the vulnerabilities created by responses to the existing ones. New and better technologies for energy supply and efficient end use will clearly be required if the daunting challenges of the decades ahead are to be adequately addressed. In 2000, the Electric Power Research Institute (EPRI) launched a consortium dedicated to improving electric power reliability for the new digital economy. Participants in this endeavor, known as the Consortium for Electric Infrastructure to Support a Digital Society (CEIDS), include power providers and a broad spectrum of electric reliability stakeholders. Participation in CEIDS is also open to digital equipment manufacturers, companies whose productivity depends on a highly reliable electricity supply, and industry trade associations. According to EPRI, CEIDS (now known as IntelliGrid) represents the second phase of a bold, two-phase national effort to improve overall power system reliability. The first phase of the plan, called the Power Delivery Reliability Initiative, launched in early 2000, brought together more than twenty North American electric utilities as well as several trade associations to make immediate and clearly necessary improvements to utility transmission and distribution systems. In the second phase, CEIDS addresses more specifically the growing demand for "digital quality" electricity. "Unless the needs of diverse market segments are met through a combination of power delivery and end-use technologies, U.S. productivity growth and prosperity will increasingly be constrained," explains Karl Stahlkopf, a former Vice President of Power Delivery at EPRI. "It's important that CEIDS study the impact of reliability on a wide spectrum of industries and determine the level of reliability each requires." Specifically, CEIDS focuses on three reliability goals: 1. Preparing high-voltage transmission networks for the increased capacity and enhanced reliability needed to support a stable wholesale power market. 2. Determining how distribution systems can best integrate low-cost power from the transmission system with an increasing number of distributed generation and storage options. 3. Analyzing ways to provide digital equipment, such as computers and network interfaces, with an appropriate level of built-in protection. It is only through these wide-reaching efforts to involve all industry constituencies that the industry can raise the bar with respect to protective measures and knowledge sharing.

2.4

USE OF DISTRIBUTED GENERATION

The way electricity was produced before the advent of our modern electric grid has received renewed interest over the past couple of decades. Unlike the large centralized

38

ENERGY SECURITY AND ITS EFFECT ON BUSINESS RESILIENCY

power plants located at the top tier of our electric grid, much smaller distributed generation (DG) systems are now being deployed at the bottom tier, typically installed on-site by the end electric user. This trend is spurred on by growing concern over our aging electric infrastructure, widespread outages like those that occurred in 2003, customer desire to have an alternative to their grid electric supply, and environmental impact. Some new technologies such as fuel cells and microturbines that have much lower emissions than fossil-fuel central power plants are being used on-site for a wide variety of applications. Other "green" renewable technologies, including solar and wind, are also becoming more widely applied as distributed resources (DR). They are helped financially by an assortment of federal, state, and utility incentives that provide various grants, rebates, or tax benefits to help justify the installation. Some generalizations that favor DG include: 1. Since most DG is of relatively small scale, it is more modular and can be sized to match a facility's base load, sized to just supplement the grid supply, or, in cogeneration applications, sized to just meet the thermal requirements. 2. In general, DG is a cleaner alternative to grid electricity. Solar and wind power produce no emissions and completely offset the emissions and greenhouse gas production of central power plants. 3. Even with fossil-fuel-based DG, if waste-heat recovery offsets the use of some separate fuel combustion for thermal needs, a net reduction in greenhouse gas emissions can be realized. 4. Many DG systems are designed to run grid-parallel, and only operate when the grid supply is available, such as systems that employ induction generators or line commutated inverters. DG systems that use synchronous generators or selfcommutated inverters can also be designed to run grid-independent to provide standby power. 5. Of the various DG benefits, some may be less tangible than others, but, in general, most DG is installed with the objective of reducing the owner's energy costs while also providing an acceptable rate of return on the capital investment. For cogeneration systems, it is the spark spread—the difference in cost between the DG fuel and the cost of grid electricity—along with the amount of waste heat that can be recovered that determines energy savings. For renewable-energy systems, renewable-energy credits and/or carbon credits are another revenue stream that can improve the economics. 6. To the extent that distributed generation uses renewable energy (e.g., solar or wind), or uses fossil fuels more efficiently (i.e., cogeneration units in which power is produced and waste heat is also recaptured and used), then overall greenhouse gas production is reduced. These are some typical barriers that discourage DG: 1. Grid interconnect requirements vary from state to state and utility to utility. The time and costs associated with the approval process can be a deterrent for wouldbe self-generators.

2.4 USE OF DISTRIBUTED GENERATION

39

2. The economics of combustion-based DG are very dependent on spark spread. The volatility of energy prices today makes it difficult to project long-term cost of ownership. For renewable technologies like wind and solar, which typically have a high initial cost, the availability of incentives like rebates and tax credits greatly impact system payback. The high rate-of-return requirements of many businesses typically pose a challenge for prospective DG projects. 3. Distributed generation must disconnect from the utility whenever there is a momentary or sustained interruption. This requirement is intended to prevent distributed generation units that are outside of the utility's direct control from backfeeding into the utility distribution system and jeopardizing the safety of linemen who may be working to restore the grid supply. With induction generators and line-commutated inverters, this causes the distributed generation units to drop offline and not supply backup power to their host building. Synchronous generators and self-commutated inverters that can operate in island mode must employ an intertie breaker to disconnect the host site from the grid while the DG continues to provide backup power. In this case, synchronizing equipment is required so that the DG can be reparalleled when the grid supply returns. The cost of these controls and relay protection can impact DG economics, especially for smaller DG systems, since the cost of these controls is not directly proportional to DG system size. In addition, most interconnection requirements require that after a relay protection trip, the DG must wait at least 5 minutes before reconnecting to the grid. Since grid disturbances that trip protective relays are fairly common, some percentage of an owner's demand savings is lost each time the DG system must disconnect from the grid. 4. Utility distribution feeders have a limited capacity to accept distributed generation. For example, in New York State, if the DG capacity exceeds 50 kW on a single-phase branch of a radial distribution circuit, or 150 kW on a single distribution feeder, the interconnect applicant may have to pay for a "coordinated electric system interconnection review." 5. The fundamental issue is that the electric utility is both the supplier of last resort and is responsible for the quality of power (such as it is today) delivered to every metered customer. This means that utilities will endeavor to live up to these responsibilities and design a system robust enough to minimize liabilities due to poor power quality. Therefore, the technical requirements placed upon distributed generation designs will be very stringent in an effort to make such generation units utility grade in quality. The above factors tend to both drive and discourage renewable energy and distributed generation economies at the same time. A rational plan that fosters a diverse energy supply, advances easy-to-permit distributed resources for a redundant supply, and cultivates distributed generation into a dependable source of capacity must be developed. Given the technical and institutional realities, there is still much room within which any critical facility owner can design a power supply system that includes at least these

40

ENERGY SECURITY AND ITS EFFECT ON BUSINESS RESILIENCY

three key components: reasonably reliable utility electric service, a reliable and wellmaintained backup power generation system, and a distributed generation unit using high-efficiency or renewable energy sources to drive down energy costs while reducing air pollution and greenhouse gas emissions. In certain installations, the distributed generation unit can be engineered to work in concert with the backup generation system. Fuel cell technology has seen uses in mission critical environments since its inclusion in NASA's Gemini space missions in the mid-1960s using hydrogen and oxygen to produce power. Modern fuel cells, which convert natural gas to electricity without combustion, are deployed as a combined heat and power (CHP) system to provide reliable, high-quality power and recoverable waste heat, while reducing the carbon footprint of a facility. To make the best use of a fuel cell installation, it should be sized at or below the base load of the data center or other critical facility that it will serve. In areas with reliable grid service, the financial viability of fuel cells is dependent on low natural gas prices, but fuel cells may also be deployed as the primary power source in areas with poor grid reliability due to a lack of utility investment or extreme weather conditions. There are also many examples of data centers incorporating photovoltaic technology in their power systems. Due to federal and state subsidies and incentives, PV technology is often a cost-effective way to use green technology and lower costs with a return on investment as low as a few years. One proposed larger-scale solution is the use of virtual power plants, whereby multiple distributed generation resources are linked together via the Internet so they can be managed as a single entity. This model allows for a mix of resources to work together to negate some of the disadvantages and power quality issues traditionally associated with small energy sources.

2.5 DOCUMENTATION AND ITS RELATION TO INFORMATION SECURITY In recent years, there have been critical infrastructure drawings found on unsecure laptop computers, in garbage pails, and blowing around the streets of major cities. These security leaks provide an opportunity for cyber threats to occur, and make our national infrastructure vulnerable to people who want to disrupt the electrical grid or specific critical buildings vital to our national and economic security. Examples of these security leaks include a major banking and finance company's laptop computer that was found in India with critical infrastructure drawings on it, transportation drawings found in a trash can outside a major transportation hub, and most recently, the New York City Freedom Tower drawings found in the trash. The occurrence of these situations can compromise corporate and national safety and security if these documents fall into the wrong hands. Business officials travelling abroad are also a major target for information theft. Spyware installed on electronic devices and laptops can open communications with outside networks, exposing information stored on them. In the environment we live in today, we need a steadfast plan to secure invaluable informa-

2.5

DOCUMENTATION AND ITS RELATION TO INFORMATION SECURITY

41

tion such as critical drawings, procedures, and business processes. The following items should be considered when you are evaluating your internal security. Security Questions: 1. Have you addressed physical security concerns? 2. Have all infrastructures been evaluated for the type of security protection needed (e.g., card control, camera recording, key control)? 3. If remote dial-in or Internet access is provided to any infrastructure system, have you safeguarded against hacking, or do you permit read-only functionality? 4. How frequently do you review and update access-permission authorization lists? 5. Are critical locations included in security inspection rounds? Network and Access: 1. Do you have a secured network between your facility's IT installations? 2. Do you have an individual on your IT staff responsible for managing the security infrastructure of your data? 3. Do you have an online file repository? If so, how is use of the repository monitored, logged, and audited? 4. How is data retrieved from the repository and then kept secure once it leaves the repository? 5. Is your file repository available through the public Internet? Techniques for Addressing Information Security: 1. Enforce strong password management for properly identifying and authenticating users. 2. Authorize user access to only permit access needed to perform job functions. 3. Encrypt sensitive data. 4. Effectively monitor changes on mainframe computers. 5. Physically identify and protect computer resources. Enhancements that Can Improve Security and Reliability: 1. Periodic assessments of the risk and magnitude of harm that could result from the unauthorized access, use, disclosure, disruption, modification, or destruction of information and information systems. 2. Policies and procedures that: Are based on risk assessments Cost-effectively reduce risks Ensure that information security is addressed throughout the life cycle of each system. Ensure compliance with applicable requirements 3. Plans for providing adequate information security for networks, facilities, and systems.

42

ENERGY SECURITY AND ITS EFFECT ON BUSINESS RESILIENCY

4. Security awareness training to inform personnel of information security risks and of their responsibilities in complying with agency policies, procedures, and practices performed. 5. A process for planning, implementing, evaluating, and documenting remedial action to address deficiencies in information security policies, procedures, or practices. 6. Plans and procedures to ensure continuity of operations for information systems. Recommendations for Executive Action: 1. Update policies and procedures for configuring mainframe operations to ensure that they provide the necessary detail for controlling and documenting changes. 2. Identify individuals with significant security responsibilities and ensure that they receive specialized training. 3. Expand the scope for testing and evaluating controls to ensure more comprehensive testing. 4. Enhance contractor oversight to better ensure that contractors' noncompliance with information security policies is detected. 5. Update remedial action plans to ensure that they include what, if any, resources are required to implement corrective actions. 6. Identify and prioritize critical business processes as part of contingency planning. 7. Test contingency plans at least annually.

2.6

SMART GRID

The smart grid (Figure 2.5) is the convergence of electric distribution systems and modern digital technology. Whereas our current electric grid was designed for the oneway flow of energy, the smart grid will allow bidirectional energy flow and two-way digital communication over the same distribution system. Such communication potential would allow utilities to overhaul their pricing plans and provide time-of-day metering, charging more for electricity consumed during peak hours. This would encourage consumers to program their appliances to operate during off-peak periods that have lower electric rates, thereby saving money. While the consumer is using lower priced electricity, the utility is encouraging load shifting into periods of lower electric demand, thereby reducing peak period electric usage. This will defer major utility capital improvements such as distribution system upgrades and the construction of additional generation capacity. Smart meters should also provide greater grid accessibility for distributed generation equipment through the net metering capabilities built into these meters. And instead of relying entirely on costly peaking plants to handle peak loads, the smart grid will facilitate the participation of customer-owned load-shedding equipment and on-site generation in demand-response programs. Security is a key component for the development of the smart grid. In March 2007, the DHS Aurora Generator Test demonstrated the dangers of a hacker attack on the grid by remotely causing a diesel-powered generator to overheat and fail. Such a see-

43

2.6 SMART GRID

Figure 2.5. The smart grid network and its features. nario is becoming increasingly more likely as utilities and system operators become more dependent on digital technology and the Internet to control their assets. With new solutions come new challenges; for example, should data from smart meters be leaked, it could give would-be criminals an indication that a home is unoccupied, making utility customers more susceptible to break-ins. The smart grid must be designed with inherent security, robust enough to prevent unauthorized access, or at least capable of providing early warning of tampering attempts so that damage can be minimized. In September 2009, the National Institute of Standards and Technology (NIST) issued a road map for developing smart grid deployment standards, with security being a top priority. It remains to be seen what action the federal government will take with regard to this issue, but many expect military-grade security to be a necessity for critical portions of the smart grid. In summary, the generalized conception of the smart grid includes the installation of communication links, high-voltage switches, and smart electric meters that would enable: • Automatic switches that detect system faults and open to isolate just the faulted areas, keeping the major portion of the grid intact • Real-time load-flow information to identify system load pockets for local generation dispatch

44

ENERGY SECURITY AND ITS EFFECT ON BUSINESS RESILIENCY

• Creation of pricing mechanisms and rate structures based upon actual power supply costs, updated by the hour or even by the minute, to encourage customers to shift power use to system off-peak times • Dispatch on or off of customer-owned generation units to better match system needs • IP-addressable electric meters for customers, allowing two-way communication between the electric meter and the utility. This will enable the interchange of unique pricing, voltage, and power-quality information on a meter-by-meter basis. • The utility will be able to remotely turn electric service on or off, assuming that smart meters have this capability. Turning off a meter for nonpayment or if an electrical fault is sensed within the building electric service would be possible. • The smart meter may be able to generate alerts to customers or real-time electric load information that will enable building load-management systems to automatically schedule noncritical equipment operations such as shutting down or delayed starting of air conditioning compressors, resulting in more effective peak shaving. There are major cost hurdles to be overcome in the design, planning, and implementation of any smart grid. To the extent that the federal government has endorsed its development and provided some funding, it is starting to take shape. Exactly what this will mean for mission critical facilities remains to be seen, but it is likely that it will at a minimum allow greater flexibility in installing on-site power-generating devices and facilitate the shifting of discretionary loads from peak periods to off-peak periods that have lower electric rates.

2.7

CONCLUSION

It is important to address the physical and cyber security needs of critical infrastructures including systems, facilities, and assets. Security requirements may include capabilities to prevent and protect against both physical and digital intrusion, hazards, threats, and incidents, and to expeditiously recover and reconstitute critical services. Personnel should be trained and made aware of security risks and consequences, ensuring that sensitive information will not be leaked and lead to a security breach. A sensible and cost-effective security approach can provide a protection level achieved through design, construction, and operation that mitigates adverse impact to systems, facilities, and assets. This can include vulnerability and risk assessment methodologies that identify prevention, protection, monitoring, detection, and sensor systems to be deployed in the design. Also, less frequent but greater consequence risks must be taken into consideration. No longer are accidents the only risks to be expected; deliberate attacks must be anticipated as well. The increased use of advanced information technology, coupled with the prevalence of hacking and unauthorized access to electronic networks, requires physical security to be complemented by cyber security considerations. Hacking techniques are

2.8 RISK ANALYSIS AND IMPROVEMENT

45

becoming more sophisticated, and before enabling remote access for monitoring and/or control of critical infrastructure systems, cyber security protection must be assured. Major damage can be done remotely, and the greatest effort should be made to prevent illicit access to critical networks. For mission critical facilities, this cements the need for the design of backup power systems, including UPS, generators, transfer switches, and so on, to be on par with the challenges facing us as today's digital society evolves and necessitates the digitalization of the power grid. The importance of ongoing and technical maintenance programs coupled with strong training and education should also be stressed. Without these elements, no business recovery plan will be successful.

2.8

RISK ANALYSIS AND IMPROVEMENT

Below is a list of questions about power utilities that you may wish to ask yourself about the mission critical infrastructure you are supporting. Your answers to these questions should help to shed some light on areas where you can improve your operations. 1. Do you have a working and ongoing relationship with your electric power utility? 2. Do you know who in your organization currently has a relationship with your electric power utility, such as facilities management or accounts payable? 3. Do you understand your electric power utility's electric service priority (ESP) protocols? 4. Do you understand your electric power utility's restoration plan? 5. Are you involved with your electric power utility's crisis management/disaster recovery tests? 6. Have you identified regulatory guidelines or business continuity requirements that necessitate planning with your electric power utility? 7. What is the relationship between the regional source power grid and the local distribution systems? 8. What are the redundancies and the related recovery capacity for both the source grid and local distribution networks? 9. What is the process of restoration for source grid outages? 10. What is the process of restoration for local network distribution outages? 11. How many network areas are there in your city? 12. What are the interrelationships between each network segment and the source feeds? 13. Does your infrastructure meet basic standard contingency requirements for route grid design? 14. What are the recovery-time objectives for restoring impacted operations in any given area?

46

ENERGY SECURITY AND ITS EFFECT ON BUSINESS RESILIENCY

15. What are recovery-time objectives for restoring impacted operations in any given network? 16. What are the restoration priorities to customers, both business and residential? 17. What are the criteria for rating in terms of service restoration? 18. Where does your industry rank in the priority-restoration scheme? 19. How do you currently inform clients of a service interruption and the estimated time for restoration? 20. What are the types of service disruptions, planned or unplanned, that your location could possibly experience? 21. Could you provide a list of outages, types of outages, and length of disruptions that have affected your location during the last 12 months? 22. What are the reliability indices and who uses them? 23. During an outage, would you be willing to pass along information regarding the scope of interruptions to a central industry source, for example, an industry business-continuity command center? 24. Are the local and regional power utilities cooperating in terms of providing emergency service? If so, in what way? If not, what are the concerns surrounding the lack of cooperation? 25. Would you be willing to provide schematics to select individuals and/or organizations on a nondisclosure basis? 26. Could you share your lessons learned from the events of 9/11 and the Northeast outage of 8/14/03? 27. Are you familiar with the Critical Infrastructure Assurance Guidelines for Municipal Governments document written by the Washington Military Department Emergency Management Division? Is so, would you describe where your city stands in regard to the guidelines set forth in that document? 28. Independent of the utility's capability to restore power to its customers, can you summarize your internal business continuity plans, including preparedness for natural and manmade disasters (including but not limited to weather-related events, pandemics, and terrorism)?

3 MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

3.1

INTRODUCTION

Businesses that are motivated to plug into the information age require reliability and flexibility regardless of whether the companies are large Fortune 1000 corporations or small companies serving global customers. This is the reality of conducting business today. Whatever type of business they are in, many organizations have realized that a 24/7 operation is imperative. An hour of downtime can wreak havoc on project schedules, resulting in lost hours rekeying electronic data, not to mention the potential for losing millions of dollars. Twenty-five years ago, the facilities manager (FM) was responsible for the integrity of the building. As long as the electrical equipment worked 95% of the time, the FM was doing a good job. When there was a problem with downtime, it was usually a computer fault. As technology improved on both the hardware and software fronts, producers of information technology began to design their hardware and software systems with redundancy. As a result of IT's efforts, computer systems have become so reliable that they are only down during scheduled upgrades. Today, the major reasons for downtime are human-error, utility failures, poor power quality, power distribution failures, and environmental system failures (although the percentage remains small). When a problem does occur, the facilities manager is usually the one in the hot seat. Problems are not limited just to power quality, but also that the staff has not been properly trained in certain situations. Further complicating matMaintaining Mission Critical Systems in a 24/7 Environment, Second Edition. Peter M. Curtis © 2011 the Institute of Electrical and Electronics Engineers, Inc. Published 2011 by John Wiley & Sons, Inc.

47

48

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

ters, recruiting qualified inside staff and outside consultants can be difficult, as facilities management, protection equipment manufacturers, and consulting firms are all competing for the same talent pool. Minimizing unplanned downtime reduces risk but, unfortunately, the most common approach is reactive, that is, spending time and resources to repair a faulty piece of equipment after it has failed. However, strategic planning can identify internal risks and provide a prioritized plan for reliability improvements. Also, only when everyone fully understands the potential risk of outages, including recovery time, can they fund and implement an effective plan. Because the costs associated with reliability enhancement are significant, sound decisions can only be made by quantifying the performance benefits and weighing the options against their respective risks. Planning and careful implementation will minimize disruptions while making the business case to fund capital improvements and maintenance strategies. When the business case for additional redundancies, consultants, and ongoing training reaches the boardroom, the entire organization can be galvanized to prevent catastrophic data losses, damage to capital equipment, and even danger to life safety.

3.2 COMPANIES' EXPECTATIONS: RISK TOLERANCE AND RELIABILITY In order to design a building with the appropriate level of reliability, a company must first assess the cost of downtime and determine its associated risk tolerance. Because recovery time is now a significant component of downtime, downtime can no longer be equated to simple power availability, measured in terms of one nine (90%) or six nines (99.9999%). Today, recovery times are typically many times longer than outages, since operations have become much more complex. Is a 32-second outage really only 32 seconds? Is it perhaps 2 hours or 2 days? The real question is, How long does it take to fully recover from the 32-second outage and return to normal operational status? Although measuring in terms of nines has its limitations, it remains a useful measurement we need to identify. Table 3.1 shows the law of nines for a 24/7 facility. In new 24/7 facilities, it is imperative to not only design and integrate the most reliable systems, but also to keep them simple. When there is a problem, the facilities manager is under enormous pressure to isolate the faulty system without disrupting any critical electrical loads, and does not have the luxury of time for complex switching procedures during a critical event. An overly complex system can be a quick recipe for failure via human error if key personnel who understand the system functionality are unavailable. When designing a critical facility, it is important that the building design does not outsmart the facilities manager. Companies can also maximize profits and minimize cost by using the simplest design approach possible. In older buildings, facility engineers and senior management need to evaluate the cost of operating with obsolete electrical distribution systems and the associated risk of an outage. Where a high potential for losses exists, serious capital expenditures to upgrade the electrical distribution system are monetarily justified by senior management. The cost of downtime across a spectrum of industries exploded in recent years, as busi-

3.2 COMPANIES' EXPECTATIONS: RISK TOLERANCE AND RELIABILITY

49

Table 3.1. Law of nines % Uptime/reliability level 99% 99.9% 99.99% 99.999% 99.9999%

Downtime per year 87.6 hours 8.76 hours 52 minutes 5.25 minutes 32 seconds

ness has become completely computer dependent and systems have become increasingly complex (Table 3.2). Imagine that you are the manager responsible for a major data center that provides approval of checks and other online electronic transactions for American Express, MasterCard, and Visa. On the biggest shopping day of the year, the day after Thanksgiving, you find out that the data center has lost its utility service. Your first reaction is that the data center has a UPS and standby generator, so there is no problem, right? However, the standby generator is not starting due to a fuel problem and the data center will shut down in 15 minutes, the amount of time the UPS system batteries can supply power at full load. The penalty for not being proactive is loss of revenue, potential loss of major clients, and, if the problem is large enough, your business could be at risk of financial collapse. You, the manager, could have avoided this nightmare scenario by exercising the standby generator every week for 30 minutes—the proverbial ounce of prevention. There are about three times as many UPS systems in use today than there were 10 years ago, and many more companies are still discovering their worth after losing data during a power line disturbance. Do you want electrical outages to be scheduled or unscheduled? Serious facilities engineers use comprehensive preventative maintenance procedures to avoid being caught off-guard.

Table 3.2. The cost of downtime* Industry Brokerage Energy Credit card operations Telecommunications Manufacturing Retail Health care Media Human life

Average cost per hour $6,400,000 $2,800,000 $2,600,000 $2,000,000 $1,600,000 $1,100,000 $640,000 $340,000 Priceless

"■Prepared by a disaster-planning consultant of Contingency Planning Research.

50

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

Many companies do not consider installing backup equipment until after an incident has already occurred. During the months following the U.S. Northeast blackout of 2003, the industry experienced a boom in the installation of UPS systems and standby generators. Small and large businesses alike learned how susceptible they are to power disturbances and the associated costs of not being prepared. Some businesses that are not typically considered mission critical learned that they could not afford to be unprotected during a power outage. For example, the blackout of 2003 destroyed $250 million of perishable food in New York City alone.* Businesses everywhere, and of every type, are reassessing their level of risk tolerance and cost of downtime.

3.3 IDENTIFYING THE APPROPRIATE REDUNDANCY IN A MISSION CRITICAL FACILITY Mission critical facilities cannot be susceptible at any time to an outage, including during maintenance of the subsystems. Therefore, careful consideration must be given in evaluating and implementing redundancy in systems design. Examples of redundancy are classified as N + 1 and N+2 configurations, and are normally applied to these systems: • • • • • • •

Utilities service Power distribution UPS Emergency generator Fuel system supplying emergency generator Mechanical systems Fire-protection systems

A standard N+ 1 system is a combination of two basic schemes that meet the criteria of furnishing an essential component plus one additional component for backup. This design provides the best of both worlds at a steep price with no economies of scale considered. A standard system protects critical equipment and provides long-term protection to critical operations. In a true N + 1 design, each subsystem is configured in a parallel redundant arrangement such that full load may be served even if one system is offline due to scheduled maintenance or a system failure. The next level of reliability is a premium system. The premium system meets the criteria of an N + 2 design by providing the essential component plus two components for backup. It also utilizes dual electric service from two different utility substations. Under this configuration, any one of the backup components can be taken offline for maintenance and still retain N + 1 reliability. It is also recommended that the electric services be installed underground as opposed to aerially. *Source: New York City Comptroller William Thompson.

3.3

IDENTIFYING THE APPROPRIATE REDUNDANCY IN A MISSION CRITICAL FACILITY

51

Facilities can also be classified into different tiers based on their required level of reliability and maintainability. Tiers for mission critical facilities range from I to IV, with Tier IV being the most maintainable (Table 3.3).

3.3.1.

Load Classifications

• Critical Load. Requires 100% uptime. Must have uninterrupted power input to safeguard against facility damage or losses, prevent danger and injury to personnel, or keep critical business functions online. • Essential Load. Supports routine site operations. Able to tolerate power failures without data loss or affecting overall business continuity. • Discretionary Load. Load that indirectly supports operation of the facility such as administrative and office functions. Can be shed without affecting overall business continuity in order to keep critical loads online.

Table 3.3. Uptime tiers Tier I—basic nonredundant

Tier II—basic redundant

• No redundancy • Susceptible to interruptions from planned and unplanned activities • Equipment configurations minimum required for equipment to operate • Operation errors or failures will cause an interruption in service

• Limited backup and redundancy

Source: Uptime Institute.

Tier III—concurrently Tier IV—fault maintainable and failure tolerant

• Full single system backup and redundancy (N + 1 ) • Susceptible to • Planned disruptions from preventative and planned and programmable unplanned activities maintenance activities, repairs, • May contain limited testing, etc. can be criticality functions conducted without that can be shut down interruption of properly without service adverse effects • Errors in operation on business or spontaneous • UPS and/or failures of generator backup infrastructure may may be installed for cause disruption parts of the building of power to the • Failures may cause load a disruption in facility service

• Facility functions cannot tolerate any downtime • No single points of failure, and multiple system backup with automated recovery (2N) • Capable of withstanding one or more component failures, errors, or other events without disrupting power to the load • Full load can be supported on one path without disruption while maintenance/testing is performed on the other

52

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

3.4 IMPROVING RELIABILITY, MAINTAINABILITY, AND PROACTIVE PREVENTATIVE MAINTENANCE The average human heart beats approximately 70 times a minute, or a bit more than once per second. Imagine if a heart missed three beats in a minute. This would be considered a major power-line disturbance if we were to compare it to an electrical distribution system. Take the electrical distribution system that feeds your facility or, better yet, the output of the UPS system, and interrupt it for 3 seconds. This is an eternity for computer hardware. The critical load is disrupted, your computers crash, and your business loses two days worth of labor or, worse, is fined $10 to $20 million by the federal government because they did not receive the quota of $500 billion dollars of transaction reports by the allocated time. All this could have been prevented if electrical maintenance and testing were performed on a routine basis and the failed electrical connections were detected and repaired. Repairs could have been quickly implemented during the biannual infrared scanning program that takes place before building maintenance shutdowns. What can the data processing or facility manager do to ensure that their electrical system is as reliable as possible? The seven steps to improved reliability and maintainability are (Figure 3.1): 1. 2. 3. 4.

Planning and impact assessment Engineering and design Project management Testing and commissioning

Figure 3.1. Seven steps is a continuous cycle of evaluation, implementation, preparation, and maintenance. (Courtesy of Power Management Concepts, LLC.)

3.5 THE MISSION CRITICAL FACILITIES MANAGER

53

5. Documentation 6. Education/training and certifications with annual recertification 7. Operations and maintenance When designing a data processing center, it is important to hire competent professionals to advise at each step of the way. If the data processing center is being installed in an existing building, you do not have the luxury of designing the electrical system from scratch. A proficient electrical engineer will design a system that makes the most of the existing electrical distribution. Use electrical contractors who are experienced in data processing installations. Do not attempt to save money using the full 40% capacity for a conduit, because as quickly as new, state-of-art equipment is installed, it is deinstalled. Those same number 12 wires will need to come out of the conduit without disturbing the working computer hardware. Have an experienced electrical testing firm inspect the electrical system, perform tests on circuit breakers, and use thermal-scan equipment to find hot spots due to improper connections or faulty equipment. Finally, plan for routine facility shutdowns to perform preventative maintenance on all critical equipment. Facility managers must not underestimate the cost-effectiveness of a thorough preventative maintenance program, nor must they allow senior management to do so. Critical system maintenance is not a luxury; it is a necessity. Again, do you want electrical outages to be scheduled or unscheduled? Integrating the ideal critical infrastructure is just about impossible. Therefore, seek out the best possible industry authorities to solve your problems. Competent consultants will have the knowledge, tools, testing equipment, training, and experience necessary to understand the risk tolerance of your company, as well as recommend and implement the proper and most advanced proven designs. Equipment manufacturers and service providers are challenged to find and retain the industry's top technicians. As 24/7 operations become more prevalent, the available talent pool will diminish. This could cause response times to increase from the current industry standard of 4 hours. Therefore, the human element has a significant impact in risk and reliability. No matter which firms you choose, always ask for sample reports, testing procedures, and references. Your decisions will determine the system's ultimate reliability, as well as how easy the system is to maintain. Seek experienced professionals from both your own company and third parties for information systems, property and operations managers, and space planners, and the best consultants in the industry for all engineering disciplines. The bottom line is to have proven organizations working on your project.

3.5 THE MISSION CRITICAL FACILITIES MANAGER AND THE IMPORTANCE OF THE BOARDROOM To date, mission critical facilities managers have not achieved high levels of prestige within the corporate world. This means that if the requirements are 24/7, forever, the mission critical facilities manager must work hard to have a voice in the boardroom. The board can then become a powerful voice that supports the facilities manager and

54

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

establish a standard for managing the risks associated with older equipment or maintenance cuts. For instance, relying on a UPS system that has reached the end of its useful life but is still deployed due to cost constraints increases the risk of failure. The facilities manager is in a unique position to advise and paint vivid scenarios for the board. Imagine incurring losses due to downtime, plus damage to the capital equipment that is keeping the Fortune 1000 company in business. Board members understand this language; it is comparable to managing and analyzing other types of risk, such as whether to invest in emerging markets in unstable economies. The risk is one and the same; the loss is measured in the bottom line. The facilities engineering department should be run and evaluated just like any other business line; it should show a profit. But instead of increased revenue, the business line shows increased uptime, which can be equated monetarily, plus far less risk. It is imperative that the facilities engineering department be given the tools and the human resources necessary to implement the correct preventative maintenance training, and document management requirements, with the support of all company business lines.

3.6

QUANTIFYING RELIABILITY AND AVAILABILITY

Data center reliability ultimately depends on the organization as a whole weighing the dangers of outages against available enhancement measures. Reliability modeling is an essential tool for designing and evaluating mission critical facilities. The conceptual phase, or programming, of the design should include full probabilistic risk assessment (PRA) methodology. The design team must quantify performance (reliability and availability) against cost in order to push fundamental design decisions through the approval process. Reliability predictions are only as good as the ability to model the actual system. In past reliability studies, major insight was gained into various electrical distribution configurations using IEEE Standard 493-1997, Recommended Practice for the Design of Reliable Industrial and Commercial Power Systems, or the IEEE Gold Book. The latter is also the major source of data on failure and repair rates for electrical equipment. There are, however, aspects of the electrical distribution system for a critical facility that differ from other industrial and commercial facilities. Therefore, internal data accumulated from the engineer's practical experience is needed to complement the Gold Book information. Reliability analysis with PRA software provides a number of significant improvements over earlier, conventional reliability methods. The software incorporates reliability models to evaluate and calculate reliability, availability, unreliability, and unavailability. The results are compared to a cost analysis to help reach a decision about facility design. The process of evaluating reliability includes: • Analyzing any existing system and calculating the reliability of the facility as it is currently configured • Developing solutions that will increase the reliability of the facility

3.6 QUANTIFYING RELIABILITY AND AVAILABILITY

55

• Calculate the reliability with the solutions applied to the existing systems • Evaluate the cost of applying the solutions

3.6.1

Review of Reliability Terminology

Reliability (R) is the probability that a product or service will operate properly for a specified period of time under design operating conditions without failure. The failure rate (A) is defined as the probability that a failure per unit time will occur in the interval, given that no failure has occurred prior to the beginning of the interval. For a constant failure rate À, reliability as a function of time is R(t) = e-kt Mean time between failures (MTBF), as its name implies, is the mean of probability distribution function of failure. For a statistically large sample, it is the average time the equipment performed its intended function between failures. For the example of a constant failure rate, MTBF = 1/A Mean time to repair (MTTR) is the average time it takes to repair the failure and get the equipment back into service. Availability (A) is the long-term average fraction of time that a component or system is in service and satisfactorily performing its intended function. This is also called steady-state availability. Availability is defined as the mean time between failures divided by the mean time between failures plus the mean time to repair: A = MTBF/(MTBF + MTTR) High reliability means that there is a high probability of good performance in a given time interval. High availability is a function of failure frequency and repair times, and is a more accurate indication of data center performance. As more and more buildings are required to deliver service guarantees, management must decide what performance is required from the facility. Availability levels of 99.999% (5.25 minutes of downtime per year) allow virtually no facility downtime for maintenance or other planned or unplanned events. Therefore, moving toward high reliability is imperative. Since the 1980s, the overall percentage of downtime events caused by facilities has grown as computers become more reliable. And although this percentage remains small, total availability is dramatically affected because repair times for facility events are so high. A further analysis of downtime caused by facility failures indicates that utility outages have actually declined, primarily due to installation of standby generators. The most common response to these trends is reactive, that is, spending time and resources to repair the offender. If a utility goes down, install a generator. If a ground

56

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

fault trips critical loads, redesign the distribution system. If a lightning strike burns power supplies, install a new lightning protection system. Such measures certainly make sense, as they address real risks in the data center. However, strategic planning can identify internal risks and provide a prioritized plan for reliability improvements. Planning and careful implementation will minimize disruptions while making the business case to fund these projects. As technological advances find their way onto the data center's raised floor, the facility will be impacted in unexpected ways. As equipment footprints shrink, free floor area is populated with more hardware. However, because the smaller equipment emits the same amount of heat, power and cooling densities grow dramatically and floor space for cooling equipment increases. The large footprint now required for reliable power without planned downtime, such as switchgear, generators, UPS modules, and batteries, also affects the planning and maintenance of data center facilities. Over the last two decades, the cost of the facility relative to the computer hardware it houses has not grown proportionately. Budget priorities that favor computer hardware over facilities improvement can lead to insufficient performance. The best way to ensure a balanced allocation of capital is to prepare a business analysis that includes costs associated with the risk of downtime. This cost depends on the consequences of an unplanned service outage in that facility and the probability that an outage will occur.

3.7 DESIGN CONSIDERATIONS FOR THE MISSION CRITICAL DATA CENTER In most mission critical facilities, the data center constitutes the critical load in the daily operations of a company or institution. The costs of hardware and software could run to $3000-$4000 per square foot, often resulting in an investment of millions and millions of dollars depending on size and risk profile. In a data center of only 5000 square feet, you could be responsible for a $20 million capital investment, without even considering the cost of downtime. Combined, the cost of downtime and damage to equipment could be catastrophic. Proper data center design and operations will protect the investment and minimize downtime. Early in the planning process, an array of experienced professionals must review all the factors that affect operations. This is no time to be a "jack of all trades, master of none." Here are basic steps critical to designing and developing a successful mission critical data center: • Determine the needs of the client and the reliability of the mission critical data center. • Develop the configuration for the hardware. • Calculate the air, water, and power requirements. • Determine your total space requirements. • Validate the specific site. Be sure that the site is well located and away from natural disasters and that electric, telecommunications, and water utilities can provide the high level of reliability your company requires.

3.7 DESIGN CONSIDERATIONS FOR THE MISSION CRITICAL DATA CENTER

57

• Develop a layout after all parties agree. • Design the mission critical infrastructure to N + 1, TV + 2, or higher redundancy level, depending on the risk profile and reliability requirements. • Once a design is agreed upon, prepare a budgetary estimate for the project. • Have a competent consulting engineer prepare specifications and bid packages for equipment purchases and construction contracts. Use only vendors that are familiar with and experienced in the mission critical industry. • After bids are opened, select and interview vendors. Take time to carefully choose the right vendor. Make sure you see their work, ask many questions, verify references, and be sure that everybody is on the same page.

3.7.1

Data Center Certification

In addition to the previously mentioned design considerations, The Leadership in Energy and Environmental Design (LEED) Certification is also important to consider. LEED is an internationally recognized green building certification system developed by the U.S. Green Building Council and should not be overlooked when designing a data center. LEED Certification will impress customers as well as the boardroom, since its implementation requires energy efficiency improvements. Although initial costs for LEED certification are 2-A% higher than standard building construction, it theoretically results in an energy savings annuity over the facility's lifecycle. LEED has yet to develop specific standards that apply to data centers in the accreditation process in its current version 2.2. However, a LEED data center draft proposal aimed at remediating this situation through specific standards associated with data centers is under development. In addition, LEED 2009 introduced a new method of weighting credits and systems, called LEED Bookshelf, that will allow the creation of new credits, which is very promising for data centers' energy challenges. Certification involves categories such as sustainable sites, water efficiency, energy and atmosphere, materials and resources, indoor environmental quality, innovation and design, and regional priority. A 100 point scale with 10 bonus points is assigned to these categories, and there are four levels of certification: certified, silver, gold, and platinum. The most points can be earned in the energy and atmosphere category. They are given for building designs that are capable of tracking building performance, managing refrigerants in order to reduce CFCs, and renewable energy use. Many data centers have attained LEED certifications in an effort to curb energy consumption and reduce GHGs. One of the first data centers in the world to achieve LEED Platinum was the Citigroup Data Center in Frankfurt, Germany in April 2009. Some features implemented to achieve certification were the use of fresh-air free cooling, reverse osmosis water treatment in cooling towers to reduce water use, a vegetated roof, a vegetated green wall irrigated using harvested rain water, extensive use of server virtualization, and a data center layout that reduces required cabling by 250 km. New data center designs should take these practices and expand on them in order to achieve LEED certification. Another internationally used rating system similar to LEED is the Building Research Establishment Environmental Assessment Method (BREEAM), which is based

58

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

in the United Kingdom. As you can see from its title, BREEAM is an environmental assessment method for all types of new and existing buildings that provides best practices for sustainable design. BREEAM has a set of standard categories called schemes by which a building can be assessed. The array of schemes they provide cover a wide range of facilities including prisons, schools, offices, and industrial factories. There is incredible flexibility with this system because if there is a building that does not fall under standard categories, BREEAM will develop a customized criterion to evaluate the building. When trying to attain BREEAM certification, the ratings that can be attained include Pass, Good, Very Good, and Excellent. In 2010, BREEAM launched its new scheme, called BREEAM Data Centres, to evaluate the environmental performance of data centers. Data centers are unique environments as they have limited employees but still use huge amounts of energy. In addition, the intensive energy use will only increase with rising computing needs and demands. According to BREEAM, this new framework could reduce energy use in data centers by more than 50%.* Over the next year, it is hoped that this new scheme will be refined even further to provide design, construction, and operation of data centers with the most efficient and effective standards possible.

3.8

THE EVOLUTION OF MISSION CRITICAL FACILITY DESIGN

To avoid downtime, facilities managers also must understand the trends that affect the utilization of data centers, given the rapid evolution of technology and its requirements on power distribution systems. The degree of power reliability in a data center will impact the design of the facility infrastructure, the technology plant, system architecture, and end-user connectivity. Today, data centers are pushed to the limit. Servers are crammed into racks, and their high-performance processors all add up to outrageous power consumption. In the early 2000s, data center power consumption increased about 25% each year according to Hewlett-Packard. At the same time, processor performance has gone up 500%, and as equipment footprints shrink, free floor area is populated with more hardware. However, because the smaller equipment is still emitting the same amount of heat per unit, cooling densities are growing dramatically. Traditional design using watts per square foot has grown enormously and can also be calculated as transactions per watt. All this increased processing power generates heat, but if the data center gets too hot, all applications grind to a halt. Many data center designers (and their clients) would like to build for a 20-year life cycle, yet the reality is that most cannot realistically look beyond 2 to 5 years. As companies push to wring more data-crunching ability from the same real estate, the lynchpin technology of future data centers will not necessarily involve greater processing power or more servers, but improved heat dissipation and better air flow management. To combat high temperatures and maintain the current trend toward more powerful processors, engineers are reintroducing an old technology—liquid cooling—which was used to cool mainframe computers a decade ago. To successfully reintroduce liq*http://www.breeam.org/newsdetails.jsp?id=672.

3.9

HUMAN FACTORS AND THE COMMISSIONING PROCESS

59

uid cooling into computer rooms, standards will need to be developed, another arena in which standardization can promote reliable solutions that mitigate risk for industry. The large footprint now required for reliable power without planned downtime also affects the planning and maintenance of data center facilities. Over the past two decades, the cost of the facility relative to the computer hardware it houses has not grown proportionately. Budget priorities that favor computer hardware over facilities improvement can lead to insufficient performance. The best way to ensure a balanced allocation of capital is to prepare a business analysis that shows the costs associated with the risk of downtime.

3.9

HUMAN FACTORS AND THE COMMISSIONING PROCESS

There is no such thing as plug and play when critical infrastructure is deployed or existing systems are overhauled to support a company's changing business mission. Reliability is not guaranteed simply by installing new equipment, or even building an entirely new data center. An aggressive and rigorous design, failure mode analysis, testing/commissioning process, and operations plan proportional to the facility criticall y level are necessities and not options. Of particular importance are the actual commissioning process and developing a detailed operations plan. More budget dollars should be allocated to testing and commissioning, documentation, education and training, and operations and maintenance because more than 50% of data center downtime can be traced to human error. Due to the facility's 24/7 mission critical status, this will be the construction team's sole opportunity to integrate and commission all of the systems. At this point in the project, a competent, independent test engineer familiar with the equipment has witnessed testing of all installed systems at the factory. Commissioning is a systematic process of ensuring, through documented verification, that all building systems perform according to the design intent and to the future owner's operational needs. The goal is to provide the owner with a safe and reliable installation. A commissioning agent who serves as the owner's representative usually manages the commissioning process. The commissioning agent's role is to facilitate a highly interactive process of verifying that the project is installed correctly and operating as designed. This is achieved through coordination with the owner, design team, construction team, equipment vendors, and third-party commissioning provider during the various phases of the project. As previously mentioned, ASHRAE's Commissioning Guideline 0-2500 is a recognized model and a good resource that explains this process in detail and can be applied to critical systems. Prior to installation at the site, all equipment should undergo factory acceptance testing that is witnessed by an independent test engineer familiar with the equipment and the testing procedures. However, relying on the factory acceptance test is not sufficient. Once the equipment is delivered, set in place, wired, and functional testing completed, integrated system testing begins. The integrated system test verifies and certifies that all components work together as a fully integrated system. This is the time to resolve all potential equipment problems. There is no one-size-fits-all formula. Before a new data center or renovation within an existing building goes online, it is

60

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

crucial to ensure that the systems are burned-in and failure scenarios are tested no matter the schedule, milestones, and pressures. You will not have a chance to do this phase over, so get it right the first time. A tremendous amount of coordination is required to fine-tune and calibrate each component. For example, critical circuit breakers must be tested and calibrated prior to exposing them to any critical electrical load. After all tests are complete, results must be compiled for all equipment and the certified test reports prepared, establishing a benchmark for all future testing. Scheduling time to educate staff during systems integration testing is not considered part of the commissioning process but is extremely important in order to reduce human error. This activity can be considered part of the transitions-to-operations process. Hands-on training is invaluable because it improves situational awareness and operator confidence, which in turn reduces human error. The training can also break through misplaced confidence. Sometimes, we deem ourselves ready for a task, but we are really not. Handing off the new infrastructure or facility to fully trained and prepared operations teams improves success and uptime throughout the facility lifecycle. When you couple the right design with the right operations plan (training programs, documentation/MOP (method of procedures), and preventative maintenance), the entire organization will be much better prepared to manage through critical events and the unexpected. If proper training and preparation is not done during the commissioning stage, building engineers will not become familiar with various process and procedures. Learning on the job increases operational risk. Knowing this up front, there is absolutely no reason not to commit the necessary budget for training, technical maintenance programs, accurate documentation and storage, and, finally, some type of credible certification that is continually revisited. If done correctly, potential mishaps and near misses will be avoided and the reduced risk will be like an annuity paying reliability/availability dividends.

3.10 3.10.1

SHORT-CIRCUIT AND COORDINATION STUDIES Short-Circuit Study

Whenever a fault occurs in an electrical power system, relatively high currents flow, producing large amounts of destructive energy in the forms of heat and magnetic forces. When an electrical fault exceeds the interrupting rating of the protective device or the fault exceeds the rating of equipment such as switchgear or panel boards, the consequences can be devastating, including injury, damaged electrical equipment, and costly downtime. A short-circuit study (SCS) is required to establish minimum equipment ratings for power system components to withstand these mechanical and thermal stresses that occur during a fault, so SCSs are mandated by Article 110.9 of NEC 2008. Also required by NEC are markings and nameplates of the SCS ratings on equipment. These include industrial control panels (409.110), industrial machinery (670.3 [A]), HVAC equipment (440.4[B]), meter disconnect switches (230.82[3]), and motor controllers (430.8). The short-circuit calculations that are the basis of an SPS involve the reduction of

3.10 SHORT-CIRCUIT AND COORDINATION STUDIES

61

the electric power distribution system supplying current to each fault location to a Thévenin equivalent circuit of system voltage sources and impedances at each bus. [Refer to ANSI/IEEE Standard 399, Recommended Practice for Power System Analysis (Color Book Series—the Brown Book).] These calculations can be done by hand, but since they can be very tedious for mid-size to large distribution systems, they are usually performed using the same specialized software that models the power system under normal load flow conditions to determine system power flows and voltage drop. When the software is used for a SCS, the model is used to calculate the resultant fault current at each bus in the system. Typically, results are presented for both three-phase and line-to-ground faults. (Software used by electric utility companies also calculates line-to-line fault current values.) In order for the electrical distribution system analysis software to produce reliable results, the electrical distribution system being analyzed must be modeled accurately. All current sources must be investigated and input into the model, including: • The Electric Utility. As a minimum, the service voltage and available threephase and single line-to-ground short-circuit current (usually given in amps or MVA) and the three-phase and line-to-ground equivalent circuit reactance/resistance ratio, referred to as the X/R ratio, are required. Many utilities also provide the equivalent system impedances at the point of service connection. The system short-circuit contribution data must be requested from the electric utility. • On-site Generation. If the worst-case short-circuit levels are to be calculated, all generation that can operate in parallel with the utility supply must be included in the model. Rated voltage, kVA, power factor, and generator subtransient and transient impedances should be obtained from the equipment nameplate or manufacturer for input into the model. • In-Service Motors. Rotating electric motors have stored kinetic energy and underfault conditions can act as generators for a short period and return some of this stored energy into the fault in the form of short-circuit current. All across the line, induction motors and synchronous motors should be modeled. If induction motors are supplied by variable-frequency drives, only motors that are fed from drives that are regenerative should be included. Typically, lumping contributing motors together on each bus provides reasonably accurate results. Software packages have default impedances and power factors that cover typical motors. Actual impedances should be used where known, especially for very large motors. • UPS Systems. UPS systems often are limited as to the amount of short-circuit current they can contribute to a downstream fault. Consult with the equipment supplier for characteristics of any such equipment on the system being analyzed. All impedances in the system must be defined and input: • Transformers. Correct modeling of transformers is critical to an accurate model. Winding voltage ratings, kVA ratings, and impedances should be obtained from transformer nameplates or manufacturers. Winding connections must be input, as well as any ground impedances.

62

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

• Cables. Wire and cable size, length, and routing type must be included in the model. Where existing cables sizes are not known, use the NEC as a reference and include any assumptions that you made in the report as a reference for future investigations. Cable lengths can be estimated based on a site visit and inspection, or by making reasonable assumptions from building and site plans. Whether or not cable is routed in a steel conduit or raceway can also make a difference in cable impedance. • Reactors. Where line reactors are included to limit short-circuit current or minimize harmonics, they should be included in the model. Obtain ratings from the equipment nameplate or the manufacturer. Once the electrical distribution system model is created, the system can be analyzed for all operating scenarios. These might include generators operating or not, bus-tie breakers closed or open, and so on (Figure 3.2). The maximum short-circuit fault current calculated from any of the scenarios should be recorded at each bus in the distribution system. Protective device manufacturers assign short-circuit interrupting and

480 Disc PP-B1

PP-B1 600AS/400AF

CBL-0105

480 Main PP-B1

PP-B1 MDS 400AS/400AF

PP-B1

CBL-0396

CBL-0395

a

MTRI-0065

a

MTRI-0066

Figure 3.2. Sample SCS Screenshot. (Courtesy of Power Management Concepts LLC.)

3.10 SHORT-CIRCUIT AND COORDINATION STUDIES

63

fault-withstand ratings to their equipment, signifying the maximum fault condition under which the device may be safely applied. The currents calculated in a SCS are used to specify the required ratings for new equipment and to evaluate the adequacy of existing system components to withstand and interrupt these high-magnitude currents. Once the electrical distribution system is modeled, proposed changes in system configuration can be analyzed in order to determine what, if any, equipment must be upgraded to support the proposed changes. An up-to-date SCS is also a requirement for protective device coordination studies as well as arc-flash hazard analysis.

3.10.2

Coordination Study

The goal of a protective-device coordination study is to determine overcurrent device settings and selections that maximize selectivity and power system reliability. In a well-coordinated protective device scheme, faults are cleared by the nearest upstream protective device. This minimizes the portion of the electrical distribution system interrupted as a result of a fault or other disturbances. At the main distribution panel level, feeder breakers and fuses should trip before the main one. Likewise, panel board branch breakers should trip before the feeder breaker or fuse supplying the panel. As with short-circuit studies, a protective-device coordination is usually performed using the same specialized integrated software that is used for the short-circuit, loadflow, and arc-flash calculations. Protective device types, ratings, and settings can be incorporated while the model is built, or added after initial studies are completed. The use of computerized calculations allows the system protection engineer to evaluate a number of setting options in a short period of time, thereby allowing him or her to finetune settings to achieve the best possible coordination. Using the computer model, a time current curve (TCC) is developed for each circuit fed from service switchboards and other critical buses. The curve includes the overcurrent devices for the largest loads in series on the circuit, the worst case from a coordination aspect. All fuses, breakers, and electromechanical or electronic relays and trip devices are entered into the computer model, if not entered previously. Transformer and cable damage curves should be selected for inclusion on the TCC to verify that critical equipment is being protected (Figure 3.3). Transformer inrush points must be included to verify that the protective device feeding the unit will not operate when the transformer is energized. Device selection and settings from the database are reviewed to determine if changes are required to improve coordination. If so, settings or selections are modified and the resulting TCCs are printed for inclusion in the report. Protective device settings and selections are also summarized in tabular form. In many cases, protective device coordination is a matter of compromise. Rather than being a choice of black or white, device coordination requires making selections that result in the best coordination that can be achieved with the devices that are installed: fuse characteristics are not as varied as electronic trip devices, instantaneous elements in series cannot be reliably coordinated, and transformer protection must be selected to provide protection from damage from a fault while passing inrush current when the unit is energized. Luckily, where power system reliability must be maximized and, therefore, strict

64

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

Figure 3.3. Sample TCC curve analysis. (Courtesy of Power Management Concepts, LLC.)

coordination is required, electronic relays and trip devices offer a variety of settings, curve shapes, and other functions that allow the system protection engineer to achieve this goal. Electronic relays come with a variety of trip characteristic curves which, along with time-delay and pick-up settings, allow a great deal of flexibility when programming the device. The zone-selective interlocking feature available in many static trip devices allows the arming of instantaneous settings on breakers in series without losing coordination. The upstream breaker trip device (such as the main device on a bus) communicates with downstream breakers. If the downstream device sees a fault current event, it sends a signal to the main device to block the tripping of the main breaker, thus allowing the downstream device to operate and minimizing the extent of the electrical system affected by the fault. As discussed, state-of-the-art protective devices can make a significant contribution to protective-device coordination, minimizing or eliminating unnecessary outages due to compromised coordination. If project requirements demand strict coordination, electrical equipment selection may be affected. It is, therefore, important to consider how these requirements affect equipment selection, specification, and layout as early

3.11

INTRODUCTION TO DIRECT CURRENT IN THE DATA CENTER

65

in the project as possible. Selection of the wrong type of equipment may negate the ability to take advantage of the technological advances discussed above.

3.11 INTRODUCTION TO DIRECT CURRENT IN THE DATA CENTER All of our modern electronic equipment today relies on solid-state semiconductor technology, which will only operate on direct current, or DC. According to the Green Buildings Forum, 72% of the energy used in the United States is consumed in commercial real estate buildings. A study by the University of Virginia shows that 80% of this energy is used by semiconductor technology, which means that this much AC power must be converted to DC. The 2007 EPA "Report to Congress on Server and Data Center Energy Efficiency" shows that data centers in the United States have the potential to save up to $4 billion in annual electricity costs through more energy-efficient equipment and operations. One movement to accomplish this that is currently gaining acceptance among data center designers, and garnering support from the Electric Power Research Institute (EPRI), is the use of a direct current distribution system. When we step back and look at the end-to-end power distribution in a conventional data center, we see several power conversions taking place. Incoming AC utility power is first rectified to DC at the UPS for the purpose of connecting to DC battery storage. It is then inverted back to AC for distribution to the server racks. At the racks, the AC is then rectified back to DC again in the power supplies for each server. We may ask ourselves, Are all these back-and-forth conversions really necessary? The answer is no.

3.11.1

Advantages of DC Distribution

Suppose we made only one conversion of the incoming AC to DC at the UPS, and then distributed the DC throughout the facility without any further conversions, thereby eliminating two power conversions along with the attendant losses. Not only would this make for a more efficient system, it would also reduce the number of components, thereby eliminating points of failure, making for a more reliable system. As an example, a 380 V DC distribution system will result in a 200% increase in reliability, a 33% reduction in required floor space, and a 28% improvement in efficiency over conventional UPS systems, or a 9% improvement over best-in-class AC UPS architectures. A DC distribution system would also facilitate the integration of on-site DC-generating power sources, such as solar PV arrays, wind power, and fuel cells, which all can provide DC power without a single power conversion. For example, the best published efficiency for a fuel cell is 50%. According to UTC Power, by eliminating the AC power conditioning subassembly and utilizing waste heat in a combined cooling, heating, and power application, efficiencies can exceed 85% with a high load factor. Figure 3.4 shows the various power conversions that are required in a conventional AC distribution system, along with those that are required to connect alternate energy

66

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

Solar PV

:~u

Fuel cell

1

300-400VDC

DC/AC

Lighting

ELECTRONIC BALLAST

UPS Utility grid

AC/DC - DC/AC POWER SUPPLIES

AC/DC - DC/AC

— AC/DC - DC/AC

Electronic loads

Motor loads

Figure 3.4. Traditional AC distribution. sources. Figure 3.5 shows the fewer conversions that are required with a DC distribution system and the simpler integration of alternate energy sources. This kind of DC distribution system can provide up to a 25% decrease in energy usage when compared to a traditional AC power distribution system by eliminating the losses associated with the inefficiencies of power conversion equipment. It also reduces the initial cost for electrical distribution equipment by about half. And since

Figure 3.5. DC distribution.

3.11

INTRODUCTION TO DIRECT CURRENT IN THE DATA CENTER

67

there are fewer conversions from AC to DC and DC to AC, which translates into less equipment, the distribution system occupies significantly less space. Proponents of DC power in data centers tout fewer single points of failure in DC systems due to the reduction in components. There is also the accompanying decrease in heat production. This reduces cooling capacity requirements and provides further reductions in operating costs.

3.11.2

DC Lighting

Besides all of the electronic equipment that is found in the data center, another significant consumer of electricity is the lighting system. Fluorescent lighting is typically fed with 120 V AC, or 277 V AC in larger commercial buildings. Since fluorescent lamps vary by type and length, they operate on varying voltages and currents. A fluorescent ballast is used to adjust the supplied voltage and regulate current accordingly. In an effort to improve fluorescent lighting efficiency, the electronic ballast (Figure 3.6) was invented in the 1970s. Besides adjusting the supplied voltage, electronic ballasts also changed the frequency from 60 Hz to 20,000 Hz or higher, which substantially eliminates flicker while also providing higher system efficiency. By supplying the lighting system with DC, we eliminate another power conversion and improve data center efficiency even further.

3.11.3

DC Storage Options

The fact that electrical energy is most easily stored in batteries as DC is the primary reason power conversions are necessary when AC distribution is used in the data center. The two types of batteries that are most commonly used in conjunction with a UPS are lead-acid wet cell (flooded) and valve-regulated lead-acid (VRLA). Another storage technology that is gaining popularity is flywheels. Conventional batteries and flywheels are covered in more detail in Chapter 10. Also worth mentioning here are some new battery storage technologies called zinc bromide flow batteries and a megawatt-class of batteries that use a sodium sulfur electrolyte, which are both, by nature, DC sources. Both of these batteries have the potential to be used in grid-storage-class applications and may undergo many deep discharges without suffering ill effects.

Figure 3.6. Electronic ballast. (Courtesy of Antron Electronics Co., Ltd.)

68

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

As of this writing, sodium sulfur batteries have already been installed at several locations globally, including a wind farm in Japan and a bus depot in Garden City, NY. Zinc bromide batteries have yet to prove themselves in anything more than a handful of demonstration installations, but the technology seems promising for the storage of electricity from intermittent sources and load-leveling applications, similar to sodium sulfur batteries.

3.11.4

Renewable Energy Integration

Solar power is the most common renewable resource that can be used for the on-site generation of electrical power for a data center. Since photovoltaic (PV) arrays produce DC electricity, they are easily integrated with a DC distribution system. Only a voltage regulator is required, or a charge controller if the PV array is used to charge battery storage. Wind power is another option. Inverter-based wind turbines already produce DC power. The power conversion that would normally be needed to convert the DC to AC is not required, eliminating the inverter and any need for synchronization.

3.11.5

DC and Combined Cooling, Heat, and Power

Combined cooling, heat, and power (CCHP), also known as cogeneration, is an ideal strategy for improving data center energy efficiency. This is accomplished by using some form of power generation equipment that also produces thermal energy as a byproduct. Since data centers typically require 24/7 cooling, the recovered thermal energy can be used to activate absorption chillers that turn the waste heat into free cooling (Figure 3.7). There are two types of power generating equipment that can be easily integrated with a DC distribution system. One is a fuel cell. The fuel cell is an electrochemical device that produces DC electricity, much like a battery (Figure 3.8). Only a voltage regulator is needed to match the DC voltage of the fuel cell to the DC distribution voltage. Since the chemical reaction within the fuel cell produces heat, this thermal energy can be recovered and used to drive a water-based absorption chiller. The microturbine is another prime mover that can produce DC power (Figure 3.9). Microturbines operate on the Brayton cycle to rotate a small permanent-magnet alternator at very high rpm to generate high-frequency AC, which is rectified to DC. A voltage regulator can be used to match this DC to the DC distribution voltage. The turbine exhaust is run through a heat recovery absorption chiller to make chilled water.

3.11.6

Current State of the Art

In general, operating costs for DC distribution are lower than AC distribution. However, since DC distribution is a relatively new concept, early adopters will no doubt be faced with higher installation costs, even though less equipment is needed, because specialized DC equipment must be developed. Product development is already underway. In the case of servers and storage hardware, some manufacturers may not offer

3.11

INTRODUCTION TO DIRECT CURRENT IN THE DATA CENTER

69

Figure 3.7. Absorption chiller. (Courtesy of Yazaki Energy Systems, Inc.)

DC solutions, but OEMs are beginning to prepare offerings with DC input directly to servers and storage systems. Delta Electronics has announced the first commercial offering of 380 V DC fans, servers, UPS units, and racks with DC distribution power supplies. Commercial proof-of-concept demonstrations have been installed in Sweden, Japan, and California. Direct Power Technologies, Inc. (DPTI) is working with Satcon

Figure 3.8. Typical fuel cell. (Courtesy UTC Power.)

70

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

Figure 3.9. Microturbine CCHP system. (Courtesy UTC Power. The pictured system is the 360M, which has six microturbines. There are alsofive-and six-microturbine versions.)

Power Systems to develop high-efficiency rectifiers and DC-DC converters specifically for DC system architecture. Satcon is currently providing power conversion products for the Navy's new DD(X) destroyer program for a DC-based integrated power system. DPTI is also working with EPRI to develop high-efficiency rectifiers and DC-DC converters specific for DC system architecture. DPTI is working with Anderson Power to develop a 380/400 V DC connector and also with multiple companies to develop a 380/400 V DC plug strip.

3.11.7

Safety Issues

One of the first steps in accomplishing the paradigm shift to DC is developing some consensus as to what DC voltage level is best for distribution, given the need to keep current levels and conductor ampacity within reason. The schools of thought range from 48 to 550 V DC. These voltage levels that are under consideration fall under the National Electrical Code, ANSI/NFPA 70, which defines low voltage as any voltage up to 600 V. The grounding of DC systems is also covered in the NEC in Sections 2503 and 250-22. Once the preferred voltage level is established, more specific codes and standards will have to be developed. Codes and standards development is in progress overseas. The European Standard ETSI EN 300 132-3 v 1.2.1 (2003-08) covers DC systems up to 400 V.

3.11

INTRODUCTION TO DIRECT CURRENT IN THE DATA CENTER

3.11.8

71

Maintenance

One of the most valuable data center maintenance tools is the power quality monitor. Whereas the monitoring of AC power quality has been done for many years, doing the same thing on DC distribution systems will require different equipment. Fortunately, there are already some instruments available (Figure 3.10) that are up to the task, like the PSL PQube® AC/DC Power Monitor and the Dranetz-BMI Encore Series Model 61000.

3.11.9

Education and Training

Initially, there may be some difficulty in locating personnel with the appropriate experience to install and maintain these DC systems. With some aggressive training programs, existing personnel will develop the necessary skills. This is the same training and education that is currently underway in the photovoltaic industry, where electrical distribution systems rated up to 600 V DC are key ingredients. It should also be noted that there will remain a need for AC expertise, since data center HVAC systems will no doubt continue to be powered by AC, and, consequently, still need the conventional AC distribution.

3.11.10

Future Vision

Looking to the future, global work on fusion reactors has demonstrated that future power generation should be DC. For long-distance distribution, high-voltage DC systems are less expensive and suffer lower electrical losses. High-temperature superconductors promise to revolutionize power distribution by providing near-lossless transmission of electrical power. The development of superconductors with transition temperatures higher than the boiling point of liquid nitrogen has made the concept of superconducting power lines commercially feasible, at least for high-load applications. It has been estimated that the waste would be halved using this method, since the necessary refrigeration equipment would consume about half the power saved by the elimination of the majority of resistive losses. Some companies such as Consolidated Edi-

Fiqure 3.10. DC monitoring equipment.

72

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

son and American Superconductor in the United States have already begun commercial production of such systems. Many people are unaware that superconductors are already being used commercially to construct the ultrastrong magnets used in magnetic resonance imaging (MRI) scanners. All the advantages and benefits of the DC alternative will have to be proven over time. The industry must continue the dialogue and take steps to ensure that the concept is thoroughly vetted. Small-scale demonstrations are a good first step. Pentadyne, a leader in flywheel energy storage, assembled the first bench test at their Chatsworth, CA headquarters in 2005. An industry group sponsored by the California Energy Commission through Lawrence Berkeley National Labs and headed by EPRI Solutions and Ecos Consulting is planning more demonstrations of data center applications. Commercial proof-of-concept demonstrations are also being conducted in Sweden and Japan. Although retrofitting existing facilities for DC may not be cost-effective in the near term, the DC alternative should be thoroughly investigated when designing a new data center. The increase in efficiency and the associated reduction in operating costs will be the primary driving force behind a DC revolution. And the fact that DC promises higher tier ratings with lower capital costs makes improved reliability another big attraction. As with any change that upsets the status quo, the conversion to DC will no doubt meet with opposition, but the benefits seem too irresistible to pass up.

3.12

CONTAINERIZED SYSTEMS OVERVIEW

When you hear the term containerized systems, modular data centers are probably the first technology to come to mind. Initially, the primary purpose of this technology was aimed at disaster recovery operations. However, with the increasing challenges of high-density computing, including cooling, power distribution, and the continual expansion of facilities to meet customer needs, containerized systems and modular infrastructure have emerged as an innovative solution to these problems. Several vendors offer fully containerized data centers housed in trailer-like enclosures (Figure 3.11), marketed as both a rapid expansion solution and, for some companies, a complete data center architecture. The units are fitted with specially designed racks and chilled-water cooling units. Electrical power, chilled water, and network connections are all that is required to commission a complete data center. These modules can be connected to a centralized power/cooling plant or each can be individually equipped with their own dedicated power and cooling systems. This is proving to be a quintessential plug-and-play solution because it allows companies to forecast accurately and efficiently expand their data centers to meet computing needs. Energy efficiency is reported to be higher than traditional data centers, due in part to the compact, high-density design. As a result, companies such as Google and Microsoft are finding this technology to be invaluable. It is important to note that modular systems can include an array of technologies not solely restricted to a shipping container. They may incorporate modularized infrastructure such as prefabricated UPS rooms and chiller and generator containers. The benefit

3.13 CONCLUSION

73

Figure 3.11. Open rear door of containerized data center. (Courtesy Sun Corporation.) of these prefabricated solutions is that the quality of construction of these products is higher and more controlled than on a chaotic construction site, therefore ensuring more reliable equipment. Additional benefits include easy integration with existing sites, reduced costs of construction, and portability and flexibility of relocation.

3.13

CONCLUSION

Designing or managing a mission critical facility is not a role to be taken lightly. As technology continues to develop and advance, critical data centers, hospitals, disaster recovery sites, and other mission critical facilities are only going to become more complex. I have included some questions in this section of the chapter to help you assess the status of your facility. Take care when answering these questions and use them as a guide to improving your operations.

Installation 1. Is your organization sure that there is diversity in the labor pool of the primary and backup sites, such that a wide-scale event would not simultaneously affect the labor pool of both sites? 2. Do you routinely use or test recovery and resumption arrangements? 3. Are you familiar with National Fire Protection Association (NFPA) 1600, Standard on Disaster/Emergency Management and Business Continuity Pro-

74

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

4. 5.

6. 7. 8. 9. 10. 11.

12.

13. 14. 15. 16. 17. 18. 19. 20. 21.

grams, which provides a standardized basis for disaster/emergency management planning and business continuity programs in private and public sectors by providing common program elements, techniques, and processes? Has the owner, working with an engineering professional, developed a design intent document to clearly identify quantifiable requirements? Have you prepared a basis of design document that memorializes, in a narrative form, the project intent, future expansion options, types of infrastructure systems to be utilized, applicable codes and standards to be followed, design assumptions, and project team decisions and understandings? Will you be provided the opportunity to update the basis of design document to reflect changes made during the construction and commissioning process? Are the criteria for testing all systems and outlines of the commissioning process identified and incorporated into the design documents? Have you identified a qualified engineering and design firm to conduct a peer project-design review? Have you considered directly hiring the commissioning agent to provide true independence? Have you discussed and agreed to a division of responsibilities between the construction manager and the commissioning agent? Do you plan to hire the ultimate operating staff ahead of the actual turnover to operations so they will benefit from participation in the design, construction, and commissioning of the facility? Have you made a decision on the commissioning agent early enough in the process to allow participation and input on commissioning issues and design review by the selected agent? Is the proper level of fire protection in place? Is the equipment (UPS or generator) being placed in a location prone to flooding or other water damage? Do the generator day tanks or underground fuel cells meet local environmental rules? Does the battery room have proper ventilation? Has adequate cooling or heating been specified for the UPS, switchgear, or generator room? Are the heating and cooling for the mechanical rooms on the power protection system? Have local noise ordinances been reviewed and does all the equipment comply with the ordinances? Are the posting and enforcement of no-smoking bans adequate, specifying, for example, no smoking within 100 feet? Are water detection devices used to alert building management of flooding issues?

3.13 CONCLUSION

75

Procurement 22. Is there a benefit to using an existing vendor or supplier for standardization of process, common spare parts, or confidence in service response? 23. Have the commissioning, factory, and site-testing requirement specifications been included in the bid documentation? 24. If the project is bid, have you conducted a technical compliance review that identifies exceptions, alternatives, substitutions, or noncompliance to the specifications? 25. Are the procurement team members versed in the technical nuances and terminology ofthe job? 26. If delivery time is critical to the project, have you considered adding latepenalty clauses to the installation or equipment contracts? 27. Have you included a bonus for early completion of project? 28. Have you obtained unit rates for potential change orders? 29. Have you obtained a GMP (guaranteed maximum price) from contractors? 30. Have you discussed preferential pricing discounts that may be available if your institution or your engineer and contractors have other similar large purchases occurring?

Construction 31. Do you intend to create and maintain a list of observations and concerns that will serve as a checklist during the acceptance process to ensure that these items are not overlooked? 32. Will members of the design, construction, commissioning agent, and operations teams attend the factory-acceptance tests for major components and systems such as UPS, generators, batteries, switchgear, and chillers? 33. During the construction phase, do you expect to develop and circulate for comment the start-up plans, documentation formats, and prefunctional checklists that will be used during start-up and acceptance testing? 34. Since interaction between the construction manager and the commissioning agent is key, will you encourage attendance at the weekly construction-status meetings by the commissioning team? 35. Will an independent commissioning and acceptance meeting be run by the commissioning agent, ensuring that everything needed for that process is on target? 36. Will you encourage the construction, commissioning, and operations staff to walk the job site regularly to identify access and maintainability issues? 37. If the job site is an operating critical site, do you have a risk assessment and change-control mechanism in place to ensure reliability? 38. Have you established a process to have independent verification that labeling on equipment and power circuits is correct?

76

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

Commissioning and Acceptance 39. Do testing data result sheets identify expected acceptable result ranges? 40. Are control sequences, checklists, and procedures written in plain language, not technical jargon that is easily misunderstood? 41. Have all instrumentation, test equipment, actuators, and sensing devices been checked and calibrated? 42. Is system acceptance testing scheduled after balancing of mechanical systems and are electrical cable and breaker testing complete? 43. Have you listed the systems and components to be commissioned? 44. Has a detailed script sequencing all activities been developed? 45. Are all participants aware of their responsibilities and the protocols to be followed? 46. Does a team directory with all contact information exist and is it available to all involved parties? 47. Have you planned an "all hands on deck" meeting to walk through and finalize the commissioning schedule and scripted activities? 48. Have the format and content of the final report been determined in advance to ensure that all needed data is recorded and activities are scheduled? 49. Have you arranged for the future facility operations staff to witness and participate in the commissioning and testing efforts? 50. Who is responsible for ensuring that all appropriate safety methods and procedures are deployed during the testing process? 51. Is there a process in place that ensures that training records are maintained and updated? 52. Who is coordinating training and ensuring that all prescribed training takes place? 53. Will you videotape training sessions to capture key points and for use as refresh training? 54. Is the training you provide both general systems training as well as specifically targeted to types of infrastructure within the facility? 55. Have all vendors performed component-level verification and completed prefunctional checklists prior to system-level testing? 56. Has all system-level acceptance testing been completed prior to commencing the full system integration testing and "pull the plug" power failure scenario? 57. Is a process developed to capture all changes made and to ensure that these changes are captured on the appropriate as-built drawings, procedures, and design documents? 58. Do you plan to reperform acceptance testing if a failure or anomalies occur during commissioning and testing? 59. Who will maintain the running punch list of incomplete items and track resolution status?

3.13 CONCLUSION

77

Transition to Operations 60. Have you established specific operations planning meetings to discuss logistics of transferring newly constructed systems to the facility operations staff? 61. Is all as-built documentation, such as drawings, specifications, and technical manuals, complete and has it been turned over to operations staff? 62. Have position descriptions been prepared that clearly define roles and responsibilities of the facility staff? 63. Are standard operating procedures (SOP), emergency action procedures (EAP), updated policies, and change-control processes in place to govern the newly installed systems? 64. Has the facility operations staff been provided with warranty, maintenance, repair, and supplier-contact information? 65. Have spare parts lists, setpoint schedules after Cx is complete, TAB report, and recommissioning manuals been given to operations staff? 66. Are the warranty start and expiration dates identified? 67. Have maintenance and repair contracts been executed and put into place for the equipment? 68. Have minimum response times for service, distance to travel, and emergency 24/7 spare stock locations been identified?

Security Considerations 69. Have you addressed physical security concerns? 70. Have all infrastructures been evaluated for type of security protection needed (e.g., card control, camera recording, and key control)? 71. Are the diesel oil tank and oil fill pipe in a secure location? 72. If remote dial in or Internet access is provided to any infrastructure system, have you safeguarded them against hacking or do you permit read-only functionality? 73. How frequently do you review and update access-permission authorization lists? 74. Are critical locations included in security inspection rounds?

Documentation 75. What emergency plans, if any, exist for the facility? 76. Where are emergency plans documented (including the relevant internal and external contacts for taking action)? 77. How are contacts reached in the event of an emergency? 78. How are plans audited and changed over time? 79. Do you have complete drawings, documentation, and technical specifications of your mission critical infrastructure including electrical utility, in-facility

78

MISSION CRITICAL ENGINEERING WITH AN OVERVIEW OF GREEN TECHNOLOGIES

80. 81. 82. 83. 84. 85. 86. 87. 88. 89. 90. 91. 92. 93. 94. 95. 96. 97. 98. 99. 100. 101. 102. 103. 104. 105. 106.

electrical systems (including power distribution and ATS), gas and steam utilities, UPS/battery/generator, HVAC, security, and fire suppression? What documentation, if any, exists to describe the layout, design, and equipment used in these systems? How many forms does this documentation require? How is the documentation stored? Who has access to this documentation and how do you control access? How often does the infrastructure change? Who is responsible for documenting change? How is the information audited? Can usage of facility documentation be audited and tracked? Do you keep a historical record of changes to documentation? Is a formal technical training program in place? Is there a process in place that ensures personnel have proper instrumentation and that the instrumentation is calibrated as recommended? Are accidents and near-miss incidents documented? Is there a process in place that ensures action will be taken to update procedures following accidents or near-miss events? How much space does your physical documentation occupy today? How quickly can you access the existing documentation? If a consultant is used to make changes to documentation, how are consultant deliverables tracked? Is your organization able to quickly provide information to legal authorities, including emergency response staff (e.g., fire and police)? How are designs or other configuration changes to infrastructure approved or disapproved? How are these approvals communicated to responsible staff? Does workflow documentation exist for answering staff questions about what to do at each stage of documentation development? In the case of multiple facilities, how is documentation from one facility transferred or made available to another? What kind of reporting on facility infrastructure is required for management? What kind of financial reporting is required in terms of facility infrastructure assets? How are costs tracked for facility infrastructure assets? Is facility infrastructure documentation duplicated in multiple locations for restoration in the event of loss? How much time would it take to replace the documentation in the event of loss? How do you track space utilization (including cable management) within the facility?

3.13 CONCLUSION

79

107. Do you use any change management methodology (i.e., ITIL) in the day-today configuration management of the facility?

Staff and Training 108. How many operations and maintenance staff do you have within the building? 109. How many of these staff do you consider to be facility's "subject matter experts?" 110. How many staff members manage the operations of the building? 111. Are specific staff members responsible for specific portions of the building infrastructure? 112. How long has each of your operations and maintenance staff, on average, been in his or her position? 113. What kind of ongoing training, if any, do you provide for your operations and maintenance staff? 114. Do training records exist? 115. Is there a process in place to ensure that training records are maintained and updated? 116. Is there a process in place that identifies an arrangement for training? 117. Is there a process in place that ensures the training program is periodically reviewed and identifies changes required? 118. Is the training you provide general training, or is it specific to an area of infrastructure within the facility? 119. How do you design changes to your facility systems? 120. Do you handle documentation management with separate staff, or do you consider it to be the responsibility of the staff making the change?

This page intentionally left blank

4 MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

4.1

INTRODUCTION

Before a corporation or institution embarks on a design to upgrade an existing building or erect a new one, key personnel must work together to evaluate how much protection is required. Although the up-front costs for protection can be somewhat startling, the cost and impact of providing inadequate protection for mission critical systems can lead to millions in lost revenue, risks to life and safety, and perhaps a tarnished corporate reputation. The challenge is finding the right level of protection for your business requirements. However, investing a significant amount of time and money in systems design and equipment to safeguard a building from failure is just the beginning. An effective maintenance and testing program for your mission critical systems is key to protecting the investment. Maintenance procedures and schedules must be developed, staff properly trained, spare parts provisioned, and mission critical electrical equipment performance tested and evaluated regularly. Predictive maintenance, preventive maintenance, and reliability centered maintenance (RCM) programs play an important role in the reliability of the electrical distribution systems. How often should electrical maintenance be performed? Every 6 months or every 6 years? Part of the answer lies in what level of reliability your company can live with. More accurately, what are your company's expectations in terms of risk tolerance or Maintaining Mission Critical Systems in a 24/7 Environment, Second Edition. Peter M. Curtis © 2011 the Institute of Electrical and Electronics Engineers, Inc. Published 2011 by John Wiley & Sons, Inc.

81

82

MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

goals with regard to uptime? As previously discussed, if your company can live with 99% reliability, or 87.6 hours of downtime per year, then the answer would be to run a maintenance program every 3 to 5 years. However, if 99.999% reliability, or 5.25 minutes of downtime per year is mandatory, then you need to perform an aggressive preventive maintenance program every 6 months. The cost of this hard-line maintenance program could range between $250 and $500 annually per kilowatt (kW), not including the staff to manage the program. The human resource cost will vary depending on the location and complexity of your facility. There are several excellent resources available for developing the basis of an electrical testing and maintenance program. One is the InterNational Electric Testing Association (NETA), which publishes the Maintenance Testing Specifications that recommends frequencies of maintenance tests. The recommendations are based on equipment condition and reliability requirements. Another is the National Fire Protection Association's (NFPA) 70B Recommended Practice for Electrical Equipment Maintenance. These publications, along with manufacturer's recommendation, give guidance for testing and maintenance tasks and schedules to incorporate in your maintenance program. Over the last ten years, there have been significant changes in the design and application of infrastructure support systems for mission critical facilities. This has been driven by the desire for higher reliability levels and has been fueled by continuing technological innovations. The resulting system configurations have yielded equipment that is more fault tolerant. Herein lies an opportunity. The response to increases in equipment has been to increase preventive maintenance tasks and schedules. However, since these systems are more fault tolerant, the effect(s) of a failure may be insignificant. Consequently, different approaches to maintenance can be taken. In addition, significant numbers of mission critical failures are directly attributable to human error. Therefore, reducing human intervention during various maintenance tasks can have a substantial impact on improving availability. Reliability centered maintenance (RCM) was developed in the 1960s by the aircraft industry when it recognized that the cost of aircraft maintenance was becoming prohibitively expensive while the reliability results were well below acceptable levels. Since then, RCM strategies have been developed and adapted by many industries. Traditionally, the goal of a maintenance program has been to reduce and avoid equipment failures. Facilities management goes to great lengths to prevent the failure of a device or system, regardless of the failure's consequence on the facility's mission. RCM shifts the focus from failure avoidance to understanding and mitigating the failure effect upon the process it protects. This is a major shift from a calendar-based program. For many facilities, the net effect will be a change from the present labor-intensive process driven by the operating group, as well as by the equipment suppliers, to a more selective method wherein the understanding of the system responses to component failures plays a critical role. Although the benefits are proven, the process needs to be supported by thorough analysis. Reliability centered maintenance analyzes how each component and/or system can functionally fail. The effects of each failure are analyzed and ranked according to their impact on safety, environment, business mission, and cost. Failures that are

4.2

THE HISTORY OF THE MAINTENANCE SUPERVISOR

83

deemed to have a significant impact are further analyzed to determine the root causes. Finally, preventative or predictive maintenance is assigned based on findings of the analysis, with emphasis on condition-based procedures to help ensure the optimal performance level.

4.2 THE HISTORY OF THE MAINTENANCE SUPERVISOR AND THE EVOLUTION OF THE MISSION CRITICAL FACILITIES ENGINEER Managing a facilities engineering department in a corporation or institution has changed significantly in the past three decades. Thirty years ago, the most important qualification for a facilities manager was exposure to different trades and hands-on experience gained from working up through the ranks of maintenance, beginning at the lowest level and progressing through a craft by way of on-the-job training or various apprentice programs. Upon mastering all the trades, and gaining knowledge of the facility from on-the-job experience, upper management would promote this craftsperson to maintenance supervisor. The maintenance supervisor was an important player in getting equipment and facility up and running quickly, in spite of day-to-day problems, procedures, and policies. This was the key to advancement. That supervisor was regarded as someone who would do whatever it takes to get the down equipment operating quickly, whatever situation would occur. Operations managers would expect the maintenance supervisor to take $200 out of his own pocket in order to buy that critical part. Coming into the facility at all hours of the night, weekends, and holidays further projected the image that this maintenance supervisor was giving his all for the company. After many years of performing at this level of responsibility, the maintenance supervisor advanced to a facilities management position with responsibility for maintenance supervisors, craftspeople, budgeting, and overall operations. This promotion was pivotal in his career. Because he had worked hard and made many sacrifices, he reached the highest levels in the facilities engineering industry. The facilities manager continued to get involved with the details of how equipment is repaired and facilities are maintained. He also continued to ensure that he was called at any hour of the day or night to personally address problems in the facilities. The secret to his success was reacting to problems and resolving them immediately. Many facilities managers have tried to adapt from reacting to problems to anticipating and planning for them, but fall short in achieving a true 24/7 mission critical operations department. The most advanced technologies and programs are researched and deployed, and personnel are assigned to utilize them. Yet, with all of these proactive approaches, the majority of facilities organizations fall short of adequate levels of functionality, in some cases due to limited human or monetary resources. The dilemma is that these tools overwhelm many facilities managers; day-to-day management of the facilities becomes a Sisyphean task. Today, the facilities manager continues with the tenacious effort and determination to resolve problems, conflicts, and issues. Additionally, the facilities manager must

84

MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

know, understand, and implement the programs, policies, and procedures dictated by state, local, and federal agencies. He must also support corporate governances of his employer. Today's manager must formulate capital and operational budgets and report departmental activities to senior management. He must be aware of safety and environmental issues while all the while his role continues to be very critical. However, to be successful when operating mission critical 24/7 operations, a proactive approach (not reactive) is imperative. The days of brute force and reacting to problems are gone. The real key to success is planning, anticipation, and consistency. Today's facility manager is a manager in every sense of the word. Most facilities organizations appear to run smoothly on the surface: critical equipment is up and running; craftspeople have more projects than ever; and facilities managers and supervisors are juggling the responsibilities of administrative, engineering, technical, and operational duties. Yet, when we look beyond the façade, we can see that the fundamentals of maintenance engineering are changing. In fact, this change is occurring along with developments in technology. The drive to move to state-of-the-art technologies, practices, and programs cannot blind facilities management's ability to ensure that the building blocks of good maintenance are applied. Technologies such as database-driven preventative maintenance programs are another tool in the facility manager's toolbox. Like any other tool, they are most effective when used in the manner in which they were designed to be used. In order to keep up and build a successful mission critical facilities operations department, the facilities manager must develop and sustain: • A clear mission, vision, or strategy for the facilities organization • Narrative descriptions of the design intent and basis of design documents • Written control sequences of operation that clearly explain to anyone how the facility is supposed to operate • Standard and emergency operating procedures • Detailed, up-to-date drawings • Ongoing employee skills assessment and training programs • Effective document management systems • Maintenance planning and scheduling programs • Maintenance inventory control • Competent consultants in all engineering disciplines • The most advanced communications, hardware, and software • Adequate capital and operating budgets • Senior management support The facility maintenance manager must keep his team focused on their primary responsibilities—maintenance of the mission critical electrical distribution system. After reacting to equipment failures when they do occur and getting the facility back in operation, preventative maintenance must be the department's top priority. Many maintenance electricians would much prefer to be working on the project-related tasks of in-

4.3 INTERNAL BUILDING DEFICIENCIES AND ANALYSIS

85

stalling new systems and equipment. Such project tasks should not take priority over preventative maintenance. Equipment and system downtime is a direct result of a loss of priority in maintenance responsibilities. With the increasing complexity of mission critical facilities and the technologies involved, the emphasis has shifted away from the resourceful craftsman who made repairs on the fly. The focus has transitioned to the mission critical facilities engineer charged with ensuring 24/7 operation, whereas repairs and preventive maintenance take place through thoughtful design and operation of the facility. Indeed, the fundamental of maintenance engineering and the role of the mission critical facilities engineer continues to evolve and grow in importance with each new technology applied.

4.3

INTERNAL BUILDING DEFICIENCIES AND ANALYSIS

In addition to having a robust UPS system, building design must eliminate the probability of single points of failure occurring simultaneously in several locations throughout the electrical distribution system. Mission critical facilities engineers have exclusive accountability for identifying and eliminating problems that can lead to points of failure in the power distribution infrastructure. Failures can occur anywhere in an electrical distribution system: utility service entrance, main circuit breakers, transformers, disconnect switches, standby generators, automatic transfer switches, branch circuits, and receptacles to name a few. A reliability block diagram must be constructed and analyzed to quantify these single points of failure. In a mission critical power distribution system, there are two basic architectures for power distribution to critical loads: 1. Electrical power from the UPS system is connected directly through transformers or redundantly from two sources through static transfer switches to remotely mounted circuit breaker panels with branch circuits running to the critical loads. 2. Power from the UPS system is fed to power distribution units (PDUs), which house transformers and panel boards, with branch circuits running to the critical loads. Typically, in data or operations centers with raised floors, liquid-tight flexible conduit runs the branch circuits from the PDUs to the critical loads. This allows for easy removal, replacement, and relocation of the circuit when computer equipment is upgraded or changed. In an office application, the typical setup involves a wall-mounted panel board hard wired to a dedicated receptacle that is feeding computer equipment. Where the UPS system output voltage is 480 volts, transformers in the PDUs are required to step down voltage to 120/208 volts. Electrical loads equipped with two power cords are common. This equipment will accept two sources of power, and may either operate on a single cord at a time or split the load between the two cords. In the event that one power sources fails, the load draws on the second source and the application remains online. This is a tremendous step toward computer equipment that is tolerant of power quality problems or internal

86

MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

electrical interruptions. However, it does challenge the facility to provide two different sources of power supplied to the two cords on each piece of equipment. This can mean the complete duplication of the electrical distribution system from the service entrance to the load, and the elimination of scheduled downtime for maintenance. Two utility switchboards, along with redundant utility substations, power dual UPS systems that feed dual PDUs. This system topography constitutes what is known as a 2N system. Although dual-corded loads may be serviced by a N or N + 1 system, far greater reliability is realized when power is brought to the load from separate substations.

4.4

EVALUATING YOUR SYSTEM

Read the operation and maintenance manuals for all electrical distribution equipment and you will likely find a recommendation for preventive maintenance. However, can you justify preventive maintenance service for your mission critical equipment? Is comprehensive annual maintenance cost-effective for the user? Should all of a mission critical facility's power distribution equipment be serviced on the same schedule? If not, what factors should be considered in establishing a realistic and effective preventive maintenance program for a given facility? To answer these questions, first evaluate the impact to your business if there were an electrical equipment failure. Then examine the environment in which the electrical equipment operates. Electrical distribution equipment is most reliable when kept dustfree, dry, cool, clean, tight, and, where appropriate, lubricated. Electrical switchgear, for example, if kept in an air-conditioned space, regardless of maintenance practices, will be considerably more reliable than identical equipment installed in the middle of a hot, dirty, and dusty operating floor. In such uncontrolled environmental locations, it would be difficult to pick an appropriate maintenance interval. The amount of load on the electrical equipment is also critical to its life. Loading governs how much internal heat is produced, which will expedite the aging process of nearly all insulating materials and dry out lubricants in devices such as circuit breakers. High current levels can lead to overheating at cable connectors, circuit breaker lugs, interconnections between bus and draw-out devices, contact surfaces, and so on. An important exception is equipment located outdoors or in a humid environment. In those instances (especially prevalent in 15 kV-class equipment), if the electrical load is too light, the heat created by the flow of current will be insufficient to prevent condensation on insulating surfaces. If the proper number and size of equipment heaters are installed and operating in the 15 kV-class equipment, condensation is less of a concern. Operating voltage levels should influence the maintenance interval. Tracking across contaminated insulating surfaces can proceed much more rapidly on 15 kVclass equipment than at 480 V. Also, higher voltage equipment is often more critical for the successful operation of a larger portion of the building or plant; therefore, more frequent preventive maintenance can be justified. Consider the age of electrical equipment. It is important to perform preventive maintenance on older equipment more frequently. Test new equipment before placing it in service, then implement a preventive maintenance schedule based on manufactur-

4.5 CHOOSING A MAINTENANCE APPROACH

87

ers' and experienced consultant recommendations. Certified testing will identify defects that can cause catastrophic failures or other problems. Additionally, it will document problems and repair/replacement, as well as the need for additional testing while still under warranty. Another consideration is the type and quality of equipment initially installed. Was lowest first cost a primary consideration? If the engineer's design specified high-quality, conservatively rated equipment and it was installed correctly, you are at a good preventive maintenance starting point. It will be less difficult and less expensive to achieve continued reliable operation from a properly designed project. The benefits of quality design and well-thought-out specifications will aid preventive maintenance, because the higher quality equipment typically has the features that enable service. On the other hand, do not allow simple minor service checks to become the building standard. It is easy to test only the protective relays (because it can be accomplished while in normal operation) and let important shutdown services slide. If minimal service becomes the norm, most of the benefits of effective preventive maintenance will not be realized. Therefore, demand regular, major preventive maintenance on your mission critical equipment. Then you can expect continued, reliable operation.

4.5

CHOOSING A MAINTENANCE APPROACH

NFPA 70B says, "Electrical equipment deterioration is normal, but equipment failure is not inevitable. As soon as new equipment is installed, a process of normal deterioration begins. Unchecked, the deterioration process can cause malfunction or an electrical failure. Effective maintenance programs lower costs and improve reliability. A maintenance program will not eliminate failures but can reduce failures to acceptable levels." There is little doubt that establishing an electrical maintenance program is of paramount importance. There are various approaches for establishing a maintenance program. In most cases, a program will include a blend of the strategies listed below: • Preventive Maintenance (PM) is the completion of tasks performed on a defined schedule. The purpose of PM is to extend the life of equipment and detect wear as an indicator of pending failure. Maintenance procedures describing the tasks to be performed are fundamental to a PM program. They instruct the technician on what to do, what tools and equipment to use, what to look for, how to do it, and when to do it. Tasks can be created for routine maintenance items or for breakdown repairs. • Predictive Maintenance uses instrumentation to detect the condition of equipment and identify pending failures (Figure 4.1). A predictive maintenance program uses these equipment condition indices for scheduling maintenance tasks. Successful predictive maintenance requires a higher level of instrumentation, monitoring, and analysis than does preventative maintenance. • Reliability Centered Maintenance (RCM) is the analytical approach to optimize reliability and maintenance tasks with respect to the operational requirements of the business. Reliability, as it relates to business goals, is the funda-

88

MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

Figure 4.1. Theory of predictive maintenance. mental objective of the RCM process. RCM is not equipment centric, but business centric. Reliability centered maintenance analyzes each system and how it can functionally fail. The effects of each failure are analyzed and ranked according to their impact on safety, mission, and cost. Those failures that are deemed to have a significant impact are further explored to determine the root causes. Finally, maintenance is assigned based on effectiveness, with a focus on conditionbased tasks. RCM requires a very high level of information gathering and analysis, and will likely take several years to establish and significant resources to maintain.

4.5.1

Annual Preventive Maintenance

The objective of annual preventive maintenance is to reach the optimum point at which the benefits of reliability are maximized while the cost of maintenance is minimized. Essentially, it relies upon the use of a checklist of tests and adjustments to "tune up" the apparatus or system. Through this process, a facility can develop an additional list of problem areas that need to be addressed immediately or within the coming months. It also allows facilities to develop documentation of persistent issues and keep a log of past issues. Also, the testing of items such as transformer oil and medium-voltage (MV) cable on a regular basis can be very helpful in indicating trends that can be used to diagnose equipment aging and deterioration before it results in failure. Transformeroil gas content, for example, can indicate a "hot" connection inside the transformer, coronal discharge in the unit, or deterioration of the insulating capabilities. Several conditions can be addressed and the deterioration halted before the unit fails and results in downtime for the mission critical facility. In most cases, it is impossible to perform complete preventive maintenance on energized gear due to the requirements of NFPA 70E and inherent limitations of personal protective equipment (PPE). Therefore, relying on PPE only should be used in limited

4.6 SAFE ELECTRICAL MAINTENANCE

89

situations, such as to take measurements, operate switches or breakers, lock out/tag out gear, and draw out or rerack breakers for maintenance. In order to provide a safe environment for maintenance personnel, system design must include "concurrent maintainability" (the flexibility that allows sections to be isolated and secured without interruption of critical loads or impact to system functionality). Unfortunately, many legacy systems do not contain this design feature and cannot be fully maintained. In this case, your options would include: • Do not perform complete maintenance. • Provide temporary hard-wired, wrap-around circuits. • Place personnel in harm's way and hope for the best (a violation of OSHA requirements). Another essential element of annual maintenance is to ensure that a formal electrical safety program is established at facilities and that it requires vendor/contractor compliance. Equipping and training personnel in comprehensive safety procedures and the use of personal protective equipment (PPE) is important to both employee safety and equipment safety. Strict adherence to an electrical safety program and maintenance of proper documentation safeguards both workers and the company and is not to be considered optional under present regulations.

4.6 4.6.1

SAFE ELECTRICAL MAINTENANCE Standards and Regulations

Electrical construction hazards are plentiful. There are many types, including electrical shock, short circuits, arc flash, and trauma. Electrical standards and regulations are necessary to ensure a safe working environment. The associated codes and standards discussed include National Electric Code (NFPA 70), Standardfor Electrical Safety in the Workplace (NFPA 70E); Occupational Safety and Health Administration (OSHA); and IEEE Guide for Performing Arc-Flash Hazard Calculations (IEEE 1584). The following are critical to a successful, safe electrical maintenance program: 1. Prior to contemplating electrical maintenance, it is critical to have up-todate one-line diagrams. A facility's one-line diagram is a baseline document that indicates how power flows from source(s) to loads. Having a correct and up-to-date one-line diagram reduces the chance of inadvertent power transmission from an alternate source (back-feed) and unexpected downstream energized parts, and increases overall familiarity with the facility's electrical distribution system. These drawings should be accompanied by a current short-circuit calculation and protective-device coordination study. These will ensure that equipment is properly rated and that protective devices will have been programmed/set to trip in sequence to reduce mass outages due to misapplication and minimize the portion of the electrical distribution system that must be

90

MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

interrupted in order to clear a fault. The requirements found in NFPA 70E can be helpful in securing funds to perform these needed engineering studies. 2. The National Electric Code, which typically applies to new or renovation work, does not have specific requirements regarding when maintenance may be completed. However, the NEC does require the application of a label to electrical equipment, such as switchboards, panelboards, industrial control panels, meter socket enclosures, and motor control centers that are in other than dwelling occupancies and are likely to require examination, adjustment, servicing, or maintenance while energized, and should be field marked to warn qualified persons of potential electric arc flash hazards. Keep in mind that this is a field-applied warning label only and no application or site-specific information is required. 3. OSHA Regulation 29 CFR Subpart S.1910.333 states that "Live parts to which an employee may be exposed shall be de-energized before the employee works on or near them...." It goes on to state that working on or near energized electrical equipment is permissible if"... the employer can demonstrate that deenergizing introduces additional or increased hazards or is infeasible due to equipment design or operational limitations." The determination of whether work may occur on energized equipment is clearly based on one's definition of "infeasible." This is a discussion that must be documented among the individual decision makers regarding maintenance in your facility. This OSHA statement establishes a policy that energized electrical work is not typical or ordinary and requires that an energized electrical work permit be generated. 4. Once it is determined that work will occur on or near live parts to which an employee may be exposed, the investigation to determine whether a hazard is present, and what to do, is the responsibility of the employer of the electrical personnel or the employers' delegate. Compliance with the OSHA regulations lead the investigator in part to NFPA 70E-2009 Standard for Electrical Safety in the Workplace and IEEE 1584-2002 Guide for Performing Arc Flash Hazard Calculations for the hazard calculation. 5. NFPA 70E contains many requirements for maintaining safe electrical working practices, including protection from shocks and traumatic injury to a worker.

4.6.2

Electrical Safety: Arc Flash

Two important terms used when discussing electrical faults are bolted faults and arc faults. Traditionally, electrical equipment has been designed and tested to standards identifying its short-circuit rating, whether it be the equipment's ability to function as an overcurrent protective device to open and clear a fault, or for equipment to simply withstand or survive that amount of bolted fault current. Bolted faults are conditions that occur in a test lab and in the field on very rare occasions. A bolted fault, as the name implies, includes the solid connection of any combination of energized electrical phases (ungrounded) or an electrical phase and ground. Most equipment currently does not carry ratings related to the equipment's ability to withstand or survive arc faults.

4.6 SAFE ELECTRICAL MAINTENANCE

91

IEEE Guide for Testing Metal-Enclosed Switchgear Rated Up to 38 kV for Internal Arcing Faults (C37.20.7-2007) provides guidance for design and selection of arc-resistant switchgear. Arc faults include the inadvertent connection of energized electrical phases (ungrounded) or an electrical phase and ground with the air acting as an electrical path. A bolted fault will typically result in the rapid operation of an upstream overcurrent-protective device, resulting in relatively low equipment damage. An arcing fault will cause lower fault current to travel as compared to a bolted fault, and result in longer clearing times of an upstream overcurrent-protective device. This is due to the air, which would typically act as an insulator between phases or phase and ground, acting as a conductor. Arc faults may result in high-temperature radiation, molten metal material, shrapnel, sound waves, pressure waves, metallic vapor, and rapid expansion of plasma (ionized air). Much of what we do while maintaining mission critical electrical infrastructure is intended to avoid unplanned downtime as a result of this problem, so the last thing we want is an arcing fault. An arc flash is literally a ball of fire resulting from an electrical fault. Ralph Lee's technical paper entitled "The Other Electrical Hazard; Electric Arc Blast Burns"* provides the following graphic definition: "current passing through a vapor of the arc terminal conductive metal." Recognition of this hazard has increased emphasis on safety standards and changed the traditional approach to maintenance and repair. However, serious workplace injuries and fatalities from electrical arc flash incidents continue to occur each year. Here are a few arc flash facts for consideration: • An arc flash reaches temperatures of 35,000°F (the face of the sun is believed to be approximately 10,000°F). • The material vaporized by an arc flash expands 70,000 times in volume. • The pressure blast from an arc flash moves at 5000 to 6000 feet per second with a physical force of 500 to 600 pounds per square inch. • Specify and install fault-tolerant switchgear. Manufactures are racing to develop designs that incorporate a variety of solutions and features in order to create market differentiation and mitigate the danger of arc flashes. Designs may include guides or shutters designed to direct an arc up and out, safely away from personnel. New arc flash detectors trip upstream breakers or trigger downstream devices to create a bolted fault. Some breaker manufacturers are incorporating a service switch on circuit breakers that temporarily overrides normal trip settings so that, should an event occur during maintenance or testing, the subject breaker will trip as quickly as possible. Installation of special view ports enable IR scans to be done without opening covers and doors. However, the best viewport lens material is fragile and may not stand up well over time. A more robust material may not offer the most desirable transmission rate of infrared energy and can degrade over time. View ports may also not allow for a thorough scan of all bus *Lee, R. H., The Other Electrical Hazard: Electric Arc Blast Burns, IEEE Transactions on Industry Applications, IA-18, 246-261, 1982.

92

MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

and cable joints or connections due to a limited field of view. This is especially true when bus or cables are "stacked" behind one another.

4.6.3

Personal Protective Equipment (PPE)

When it is determined that work will be performed on energized equipment, an energized-work permit is required. A sample of this document can be found in NFPA 70E, Annex J. It includes information documenting why and how the energized work will be performed. NFPA 70E presents a table [130.7(C)(9)(a)] for simplified personal protective equipment selection, but its use is contingent on compliance with certain criteria (notes) at the end of the table (see Table 4.1) related to bolted fault current values and protective device clearing times, information that is not normally available to electricians assigned to work on the equipment. Any deviation from the criteria makes the simplified table unusable. Where the table is deemed inappropriate for the system due to compliance issues with the table notes, a detailed calculation/analysis must be performed. Procedures and equations for detailed analysis can be found in NFPA 70E and IEEE 1584. The detailed calculation/analysis may be recommended even if the NFPA 70E tables are used, as sometimes it results in lower personal protective equipment (PPE) requirements since the detailed calculations can address work-specific conditions, such as increased distance to the energized parts, resulting in lower incident energy and thus lower PPE level selection. It is important to understand that the use of the prescribed level of PPE does not eliminate the danger to the electrician performing energized electrical work; it merely helps to manage the risk. Arc flash PPE only limits the damage to the skin resulting from an arc flash incident to a second degree burn, which does not permanently damage skin cells. Also, PPE does not protect one from the physical trauma resulting from such an explosive event. In summary, energized work puts an electrician at an elevated level of risk, and is not to be taken lightly. PPE is also required while performing tasks that do not require an energized electrical work permit, such as troubleshooting and taking measurements. Figure 4.2 from NFPA 70E indicates boundaries which, if crossed, require PPE and/or an energized electrical work permit. Classification of Hazard/Risk Category. Table 4.2 shows the minimum recommended PPE required to protect personnel at various maximum energy levels in accordance with specified classification of hazard/risk categories (NFPA 70E lists other PPE required for each category, such as hearing protection and face and eye protection). PPE Description • PPE should cover all clothing that can be ignited and should not restrict visibility and movement. • Nonconductive protective headwear is required when in contact with live parts. The face, neck, and chin should be protected.

4.6 SAFE ELECTRICAL MAINTENANCE

93

Table 4.1. Types of personal protection Task (assumes equipment is energized and work is done within the flash-protection boundary)

Hazard/risk V-rated V-rated category gloves tools

Panelboards rated 240 V and below Circuit breaker (CB) or fused switch operation with covers on CB or fused switch operation with covers off Work on energized parts, including voltage testing Remove/Install CBs or fused switches Removal of bolted covers (to expose bare, energized parts) Opening hinged covers (to expose bare, energized parts)

0 0 1 1 1 0

N N Y Y N N

N N Y Y N N

Panelboard of switchboards rated >240 V and up to 600 V CB or fused switch operation with covers on CB or fused switch operation with covers off Work on energized parts, including voltage sensing

0 1 2

N N Y

N N Y

0 0 1 0

N N N Y

N N N Y

2 3 2 2 3 2

Y N N Y N N

Y N N N N N

2 1 2 2 2 1 1

N N Y Y Y N N

N N Y N N N N

600 V class switchgear (with power circuit breakers or fused switches) CB or fused switch operation with enclosure doors closed Reading a panel meter while operating meter switch CB or fused switch operation with enclosure doors open Work on control circuits with energized parts 120 V or below, exposed Work on control circuits with energized parts >120 V, exposed Insertion or removal of CBs from cubicles, doors open Insertion or removal of CBs from cubicles, doors closed Application of safety grounds, after voltage test Removal of bolted covers (to expose bare, energized parts) Opening hinged covers (to expose bare, energized parts) Other 600 V class (277 V-600 V nominal) Removal of bolted covers (to expose bare, energized parts) Opening hinged covers (to expose bare, energized parts) Work on energized parts, including voltage testing Application of safety grounds after voltage test Insertion or removal Cable trough or tray cover removal or installation Miscellaneous equipment cover removal or installation Source: NFPA 70E, 130.7(c)(9)(a).

Safety glasses should be worn underneath the headgear. Multiple layers of clothing provide greater thermal insulation. A bib overall worn with a shirt provides higher protection to the chest area. A coverall is a garment combining shirt and pants. Jackets are usually multilayered, similar to multilayered shirts. The hood, part of the headgear, has face protection and fire-resistant fabric over the head, ears, neck, and shoulders.

94

MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

Figure 4.2. Arcflashboundaries. (From NFPA 70E-2009, used with permission.) Table 4.3 shows glove and boot classification and the corresponding maximum voltage. Electrical PPE Inspection • Insulating PPE must be inspected for damage before each day's use and immediately following any incident that can reasonably be suspected of having caused damage. Insulating gloves must be given an air test when inspected. • PPE must be stored in a location and in such a manner as to protect it from light, temperature extremes, excessive humidity, ozone, and other substances and harmful conditions. • If PPE has a hole, tear, an embedded foreign object, or texture changes such as swelling, softening, or hardening, or becomes sticky or elastic it should not be used. PPE should be tested on a regular basis. Table 4.4 shows the frequency of certain types of PPE testing. Working Near Generators • Authorized employees who conduct the servicing/inspections must comply with the lockout/tagout requirements previously mentioned.

95

4.6 SAFE ELECTRICAL MAINTENANCE

Table 4.2. Minimum recommended PPE Category

Maximum energy level N/A 5 cal/cm2 8 cal/cm2 25 cal/cm2

Nonmelting, flammable materials Fire-resistant shirt and pants Cotton underwear plus fire-resistant shirt and pants Cotton underwear plus fire-resistant shirt and pants plus fire-resistant coverall Cotton underwear plus fire-resistant shirt and pants plus double-layer switching coat and pants

40 cal/cm2

4

Typical PPE examples

Source: NFPA 70E. Notes: V-rated gloves are gloves rated and tested for the maximum line-to-line voltage upon which work will be done. V-rated Tools are tools rated and tested for the maximum line-to-line voltage upon which work will be done. For systems that are 600 V or less, theflashprotection boundary shall be a minimum of 4 ft in front of "live" electrical equipment.

Table 4.3. Glove and boot classification Glove/boot voltage classification

Maximum working voltage

Proof test voltage, kV

500 1,000 7,500 17,000 26,500 36,000

2.5 5.0 10 20 30 40

Class 00 Class 0 Class 1 Class 2 Class 3 Class 4

Source: OSHA Regulations 29 CFR—1910.137 Table 1-5—Rubber Insulating Equipment Voltage Requirements, http://www.osha.gov/pls/oshaweb/owadisp.show_document7p_table =STANDARDS&p_id=9787.

Table 4.4. Frequency of PPE testing Type of equipment

When to test

Rubber Rubber Rubber Rubber Rubber

Upon indication that insulating value is suspect Upon indication that insulating value is suspect Before first issue and every 12 months Before first issue and every 6 months Before first issue and every 12 months

insulating insulating insulating insulating insulating

line hose covers blankets gloves sleeves

Source: OSHA Regulations 29 CFR- 1910.137 Table 1-6—Rubber Insulating Equipment Test Intervals http://www.osha.gov/pls/oshaweb/owadisp.show_document?p_table=STANDARDS&p_id=9787.

96

MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

• Control devices should be mechanically locked out to prevent the release of stored energy, for example, the unexpected start of a fan belt. Engine emergency stop button(s) are not acceptable to prevent inadvertent operation during engine work. • When working near a running generator, proper ear protection must be worn. • If working near moving parts or rotating machinery, avoid wearing loose clothing and jewelry, and tie back long hair. • Guards, barriers, and access plates must be maintained to prevent employees from contacting moving parts. Working with Fuel Oil • All petroleum products, including fuel oil, are potentially dangerous. Heated fuel oil may generate vapors that are flammable, explosive, and dangerous if you inhale them. • Personnel must have thorough knowledge of these hazards. • The following list covers the most important precautions: 1. Do not allow anyone to smoke or to carry matches or lighters while handling fuel oil. 2. Use only an approved type of protected lights when working near fuel oil. 3. Do not allow oil to accumulate in bilges, voids, and so forth. The vapor from even a small pool of heated fuel oil can cause an explosion. 4. Never raise the temperature of fuel oil above 120°F in fuel oil tanks. Note: Recent environmental regulations have resulted in changes to fuel formulations that will result in a reduced shelf life for diesel fuels. These changes require that facilities work closely with their fuel suppliers to ensure that a reliable fuel supply is available on-site when it is required for generator operation. Working with Batteries 1. Operate all battery-related switches and breakers and provide lockout/tag-out devices and personal protective equipment (PPE) at all times as required by National Electrical Code (NEC) and Occupational Safety and Health Administration (OSHA) for required safety. 2. Provide appropriate personal protective equipment in accordance with NEC, OSHA, and IEEE guidelines, including but not limited to: • Safety glasses with side shields, goggles, or face shields, as appropriate • Acid-resistant gloves • Protective aprons and safety shoes • Portable or stationary water facilities in the battery vicinity for rinsing eyes and skin in case of contact with acid electrolyte • Bicarbonate of soda solution, mixed as 100 grams bicarbonate of soda to 1 liter of water, to neutralize lead-acid battery spillage. Note: The removal and/or

4.6 SAFE ELECTRICAL MAINTENANCE

• • • •

97

neutralization of an acid spill may result in production of hazardous waste. The service contractor must comply with appropriate governmental regulations. Class C fire extinguisher. Note: Some manufacturers do not recommend the use of C0 2 fire extinguishers due to the potential for thermal shock. Acid- (or alkali-) neutralizing solution and spill-containment kit Lifting devices of adequate capacity, when required Adequately insulated tools and instrumentation

In addition to the safety guidelines mentioned above, the following IEEE recommended precautions are to be observed: • Use caution when working on batteries because they present shock and arcing hazards. • Neutralize static buildup just before working on the battery by contacting the nearest effectively grounded surface. • Check the voltage to ground (AC and DC) before working around the battery. If the voltage is other than anticipated or is considered to be in an unsafe range, do not work on the battery until the situation is understood and/or corrected. Wear protective equipment suitable for the voltage. • Prohibit smoking and open flame, and avoid arcing in the immediate vicinity of the battery. • Provide adequate ventilation, and follow the manufacturer's recommendations during charging. • Ensure unobstructed egress from the battery work area. • Avoid the wearing of metallic objects such as jewelry while working on the battery. • Ensure that the work area is suitably illuminated. • Follow the manufacturer's recommendations regarding cell orientation. • Follow the manufacturer's instructions regarding lifting and handling of cells. • An uninterruptible power system (UPS) or other systems might not be equipped with an isolation transformer. In addition to DC voltage, an AC voltage may also be present. Lack of an isolation transformer may provide a direct path to ground of the DC supply to the UPS. This can substantially increase the electrocution and short-circuit hazards. Provide all required tools and rigging for required work including but not limited to: • General tools for panel cover removal and reinstallation as well as minor repairs • Multimeters for verification of power. Note: All test instruments should be checked and calibrated annually against working standards that are traceable to the National Bureau of Standards. All calibration certification must be available and provided to the manager upon request.

98

MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

• • • • • • • •

Calibrated torque wrenches (foot-pound and inch-pound increments) Socket sets Cleaning solvents (denatured alcohol) and lint-free rags Circuit tracers Ladders Hoists and cranes Battery-spill containment kit Battery manufacturer's recommended anticorrosion compound for terminals and posts

Suggested Codes and Standards • • • • • •

4.6.4

OSHA Regulations 29 CFR Safety-related work practices (1910.331—1910.335) Personal Protective Equipment (1910.132—1910.138) National Fire Protection Association, 2002 National Electric Code® NFPA 70E, Standard for Electrical Safety in the Workplace. National Fire Protection Association, 2004 Lockout/Tagout

Before any employee performs maintenance or repair on a piece of equipment subject to an unexpected energizing, start-up, or release of stored energy, proper lockout/ tagout procedures must be in place to isolate and render the equipment inoperative. All parties must be informed of the planned servicing and the possibility of an unexpected power outage. Lockout is the procedure of using keyed or combination security devices (locks) to avoid the activation of mechanical or electrical equipment accidentally. In conjunction with lockout, tags are used to visibly indicate that equipment is not being energized, that locks should not be removed without authorization, and that the disconnecting means should not be operated without permission. This is also known as tagout. The protocol and procedures for lockout/tagout need to be implemented by employees in order to protect life and equipment safety. Where all or some of the electrical distribution is scheduled to be deenergized as part of the work procedures or electrical-work permit, NFPA 70E offers detailed suggestions regarding lockout/tagout, including a sample lockout/tagout procedure (NFPA 70E Annex G). Equipment in an electrically safe work condition per the NFPA 70E standard is essential to electrical safety and work practices around electrical equipment. OSHA's standard for the control of hazardous energy sources requires employers to establish and implement procedures to disable machinery and equipment and to prevent the release of hazardous energy sources while maintenance and servicing activities are being performed.

4.7 MAINTENANCE OF TYPICAL ELECTRICAL DISTRIBUTION EQUIPMENT

99

Below is a detailed description of OSHA's lockout/tagout procedures developed by Thomas L. Bean, in collaboration with Timothy W. Butcher and Timothy Lawrence, as part of a fact sheet developed at the Ohio State University.* Phase 1—Lockout/Tagout, Deenergize Machinery or Equipment 1. The authorized employee notifies all affected people that a lockout/tagout procedure is ready to begin. 2. The machinery or equipment is deenergized. 3. The authorized employee releases or restrains all stored energy. 4. All locks and tags are checked for defects. If any are found, the lock or tag is discarded and replaced. 5. The authorized employee places a personalized lock or tag on the energy-isolating device. 6. The authorized employee holds the key throughout the duration of the service or maintenance. 7. If multiple vendors will be servicing or performing maintenance on the same equipment, an authorized employee from each vendor will place their own lockout/tagout device. 8. The authorized employee tries starting the machinery or equipment to ensure that it has been isolated from its energy source. The machinery is then deenergized again after this test. 9. The machinery or equipment is now ready for service or maintenance. Phase 2—Return the Machinery or Equipment to Production 1. The authorized employee checks the machinery or equipment to be certain no tools have been left behind. 2. All safety guards are checked to be certain that they have been replaced properly. 3. All affected people are notified that the machinery or equipment is about to go back into production. 4. The authorized employee performs a secondary check of the area to ensure that no one is exposed to danger. 5. The authorized employee removes the locks and/or tag from the energy-isolating device and restores energy to the machinery or equipment.

4.7 MAINTENANCE OF TYPICAL ELECTRICAL DISTRIBUTION EQUIPMENT In order to provide more specific recommendations, let us examine a select portion of a high-rise office building with mission critical loads. The system comprises 15 kV "OSHA's lockout/tagout standard may be found at http://ohioline.osu.edu/aex-fact/pdf/0595.pdf.

100

MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

vacuum circuit breakers with protective relays, solid dielectric-shielded feeder cable, and a fused-load break switch. On the 480 V system, there is switchgear with moldedcase circuit breakers on the main and feeders. The mission critical loads are then connected to motor control centers and panel boards with molded-case circuit breakers.

4.7.1

Thermal Scanning and Thermal Monitoring

A high-rise building of this type can easily justify the cost of an infrared scan once per year. There is no more cost-effective maintenance procedure that can positively identify potential problems quickly and easily and without a shutdown. Its principal limitation is that there are certain areas of the system and equipment where the necessary covers cannot be removed while in service. Without line-of-sight access, the scanning equipment is very limited in its capabilities. Where electrical panel covers cannot be removed during normal working hours, hinged panel covers should be installed during scheduled building shutdowns; subsequently, all mission critical board panel covers can be changed to hinged panel board covers. This will eliminate a branch circuit breaker from tripping or opening when removing panel board covers. As we know, all it takes is the right outage at the right time to create chaos in a data center. This is a small price to pay to avoid such an occurrence. For many key businesses—financial, media, petrochemical, oil and gas, telecommunications, large-scale manufacturing, and so on—consequences of sudden unexpected power failures can result in severe financial and/or safety issues. As mentioned previously, the current accepted method of noncontact thermal inspection has been through thermography via thermal imaging cameras. However, this only gives a snapshot of the day; problems could arise the next day, undetected. Another solution is a thermal monitoring system that is permanently installed within the electrical distribution system. These systems are small, low-cost, accurate, noncontact, infrared sensors that are permanently fitted where they can directly monitor key components like motors, pumps, drive bearings, gearboxes, generators, and even inside enclosures monitoring high- and low-voltage switchgear (Figure 4.3). The sensors feed back signals to

Figure 4.3. A small thermographie camera and a typical installation. (Courtesy of Exetherm.)

4.7 MAINTENANCE OF TYPICAL ELECTRICAL DISTRIBUTION EQUIPMENT

101

a PC where automatic data logging provides instant on-screen trend graphs (Figure 4.4). Two alarm levels per sensor automatically activate in the event that preset levels are exceeded, thus identifying problem components and allowing planned maintenance to be carried out. These sensors, which require no external power, measure the target temperature in relation to ambient with the signal indicating the °C rise on ambient, thus avoiding inaccuracies due to weather conditions, environment, and so on. Here are some advantages to this approach: • 7 x 24 x 365 thermal monitoring provides a constant stream of thermal data as opposed to the "snapshot in time" afforded by traditional infrared scans. Combined with comprehensive power monitoring, trend analysis becomes a new weapon in the arsenal of failure prevention. • Doors and covers need not be opened, thereby eliminating the need for PPE and reducing personal risk. • Interpretation of the information is simplified. Combined with comprehensive power monitoring, a thermal map can be developed and trends monitored so that anomalies can be addressed long before a failure occurs. Combined with comprehensive power measurement/analysis, this pinpoints inefficiencies so they may be corrected and energy demand reduced. This provides the following advantages: • This technology uses a nonproprietary, open protocol for integration with existing BMS. • Information is available globally for those responsible for operations. • No power supply is required for thermal sensors. • Devices are self-calibrating. • Direct contact devices are available where line-of-site measurement is impractical.

Figure 4.4. Sample IR scanning tracking. Left: shows the benchmark correlation between power and delta T of the bus or cable joint being monitored. Right: shows the abnormal rise of the delta T for the same power benchmark, an early indicator of a potential failure. (Courtesy of Exetherm.)

102

MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

• Depending on the equipment involved, the time periods between traditional maintenance events may be lengthened or, in some cases, events may be eliminated.

4.7.2

15 kV Class Equipment

Some procedures that should be a part of a preventive maintenance program include the following: • Insulators should be inspected for damage and/or contamination. • Cables, buses, and their connections should be visually inspected for dirt, tracking, water damage, overheating, or oxidation. • Liquid-filled transformers should be checked for temperature, liquid level, and nitrogen pressure. • Transformer oil should be sampled and tested for moisture and gases that indicate a variety of insulation failure processes. • Dry transformers should be checked for temperature and a physical inspection performed. • Switchgear encompasses enclosures, switches, interrupting devices, regulating devices, metering, controls, conductors, terminations, and other equipment that should be inspected and have recommended deenergized and energized tests performed. • Complete testing and recalibration of the relays • Vacuum integrity test on the breaker interrupters • Insulation-resistance measurements on the breaker and switch • Accurate determination of the contact resistance over the complete current paths of each device. Check for oxidation and carbon buildup. • Operational checks of the mechanical mechanisms, interlocks, auxiliary devices, and so on • Inspection and cleaning of contacts and insulators; lubrication of certain parts • Analysis of all results by someone with the proper training and experience who is on-site and participating in the work Remember that observations and analysis of experienced engineers cannot be replaced by a software program that "trends" the data obtained, or by a later comparison of the results to existing or manufacturers standards.

4.7.3

480 Volt Switchgear

The 480 V switchgear and breakers serve the same function as the higher-voltage switchgear. It is easier to justify less-frequent annual service for this equipment because insulation failure can be a slower process at the lower voltage, and if there is a problem, it will affect a smaller portion of the system. Two years is a common interval and many buildings choose 3 years between thorough maintenance checkouts on this type of equipment. A comprehensive maintenance program at this level is very similar

4.7 MAINTENANCE OF TYPICAL ELECTRICAL DISTRIBUTION EQUIPMENT

103

to the medium-voltage program above. One change results from the fact that lower voltage breakers have completely self-contained protection systems rather than separately mounted protective relays and current transformers. The best protection-confirmation test is primary current testing. A large test device injects high currents to simulate overloads and uses very low voltage to create a fault. The breaker is tested and operated as it must during a fault or overload situation. Use the breaker manufacturer's secondary test device for a less-extensive test on breakers with solidstate protective devices. This can range from a test equal to injecting high current levels, to little more than powering up the trip unit and activating an internal self-test feature. The latter test method may be appropriate at alternate service periods but cannot be considered a comprehensive evaluation and should not be relied upon as proper maintenance. Lower voltage breaker contacts should be cleaned and investigated after the arc chutes are removed. Precisely measure the resistance of the current path on each pole as well as insulation resistance between poles and ground. Verify proper operation and lubrication of the open/close mechanisms and various interlocks. Again, the inspection and analysis of the results by a knowledgeable and trusted service technician familiar with the equipment and previous history should conclude the test. Refer to manufacturer recommendations and professional society standards for specific recommendations on the equipment installed at your site.

4.7.4

Motor Control Centers and Panel Boards

Located further downstream are the motor control centers and panel boards. These components should be thoroughly examined with an infrared scanning device once a year. Testing and service beyond that level can vary considerably. To have a testing company come to scan your electrical distribution equipment can cost upwards of $1500 per day during normal working hours. Molded-case breakers are not designed for servicing and, consequently, many buildings limit them to scanning only. An exception is where molded- or insulated-case breakers are used in place of the air circuit breakers at the substation. Here they serve a more critical role, frame sizes can be quite large, and testing is more easily justified. If the breakers are of nondraw-out construction, as most circuit breakers are, primary current injection testing is more difficult, especially on the large frame sizes. Secondary testing of solid-state circuit breakers, along with an accurate determination of the contact resistances, might be justified more often than for air circuit breakers. At a minimum, exercise molded case breakers by manual opening and closing on a consistent schedule. If so equipped, push the trip button; this will exercise the same mechanism that the internal protective devices do. If periodic testing by primary current injection is available, bring the test equipment to the breaker mounting locations and test them in place wherever practical.

4.7.5

Automatic Transfer Switches

The automatic transfer switch (ATS) is a critical component of the emergency power system. Correct preventive maintenance of an ATS depends on the type of switch and where it is located within the building. There are four basic types of ATS:

104

MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

1. Break-before-make (open transition) is most common and allows the load to be interrupted during the transfer from normal to emergency power or from one source to another. 2. Make-before-break (closed transition) allows a transfer to occur without dropping the critical load. 3. Delayed transition (center off) is designed for applications in which large inrush currents exist, allowing the magnetic fields associated with inductive loads to completely collapse before reconnection. 4. Static transfer switches (STS) do not incorporate the traditional mechanical transfer switch. These transfer switches instead rely on transistor or SCR technology where subcycle transfers are possible. The following is a list of basic preventive maintenance items for automatic transfer switches. • Infrared scan the ATS while loaded to identify hot spots and high-resistance connections. • Deenergize the switchgear and isolate it electrically. Make sure the standby power generator is locked out and tagged out. • Remove arc chutes and pole covers and conduct visual inspection of the main and arcing contacts. • Test and recalibrate all trip-sensing and time-delay functions of the ATS. This step will vary depending on the manufacturer. • Vacuum the dust from the switchgear and accessory panels. Never use air to blow out dirt; you may force debris into the switch mechanism. • Inspect for signs of moisture, previous wetness, or dripping. • Clean grime with solvent approved by manufacturer. • Inspect all insulating parts for cracks and discoloration caused by excessive heat. • Lubricate all mechanical slide equipment such as rollers and cams • Exercise the switch mechanically and electrically. • After maintaining the switch, test it for full functionality.

4.7.6

Automatic Static Transfer Switches (ASTS)

• Infrared scan the ASTS while loaded to identify hot spots and high-resistance connections. • Visually inspect the cabinet and components of the unit. • Take voltage reading (phase to phase) of inputs 1 and 2 and the output. • Record the voltages and compare the measured readings to the monitor. Calibrate metering as required. • Retrieve information from the static voltage, detected from the event log. • Clear the event log.

4.7 MAINTENANCE OF TYPICAL ELECTRICAL DISTRIBUTION EQUIPMENT

105

• Test: Manually switch from preferred source to the alternate source a prescribed number of times. • Replace any covers or dead fronts and close unit doors.

4.7.7 • • • • • • • • • • •

4.7.8

Power Distribution Units Visually inspect the unit: transformer, circuit breakers, and cable connections. Check all LED/LCD displays. Measure voltages and current. Check wires and cables for discoloration, shorts, and tightness. Infrared scanning on each unit and visually inspect transformer. Check for missing screws and bolts. Check all monitors and control voltage. Check and calibrate monitors. Check high/low thresholds. Check for worn or defective parts. Check for general cleanliness of units, especially vents, and vacuum out transformer windings and enclosure if dirt and dust are present.

277/480 Volt Transformers

As long as transformers operate within their designed temperature range and specifications, they will operate reliably. However, if the transformer is not ventilated adequately and is operating hotter than manufacturer's specifications, problems will occur. The most important maintenance function for transformers is to verify proper ventilation and heat removal. Even though there are no moving parts in a transformer, except for fans in larger units, perform the following preventive maintenance: • Infrared scanning for loose connections and hot spots, then rétorque per manufacturer's specifications. • Remove power. • Inspect wire for discoloration. • Vacuum inside transformer and remove dust. • Verify that the system is properly grounded.

4.7.9

Uninterruptible Power Systems

There are many configurations of uninterruptible power supply installations, such as single-module systems, parallel redundant, and isolated redundant. But there are common tests and maintenance procedures for the major components of these systems. It is important to perform the manufacturer's recommended maintenance in addition to independent certified performance testing.

106

MISSION CRITICAL ELECTRICAL SYSTEM MAINTENANCE AND SAFETY

Visual and Mechanical Inspection 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

Inspect physical, electrical, and mechanical condition. Check for correct anchorage, required area clearances, and correct alignment. Verify that fuse sizes and types correspond to drawings. Test all electrical and mechanical interlock systems for correct operation and sequencing. Inspect all bolted electrical connections for high resistance. Verify tightness of accessible bolted electrical connections by the calibrated torque-wrench method in accordance with the manufacturer's published data. Perform a thermographie survey at full load. Thoroughly clean the unit prior to tests, unless as-found and as-left tests are required. Check operation of forced ventilation. Verify that filters are clean and in place and/or vents are clear. Replace dirty filters as necessary.

Electrical Tests 1. Perform resistance measurements through all bolted connections with a low-resistance ohmmeter, if applicable. 2. Test static transfer from inverter to bypass and back at 25%, 50%, 75%, and 100% load. 3. Set free-running frequency of oscillator. 4. Test DC undervoltage trip level on inverter input breaker. Set according to manufacturer's published data. 5. Test alarm circuits. 6. Verify sync indicators for static switch and bypass switches. 7. Perform electrical tests for UPS system breakers. 8. Perform electrical tests for UPS system automatic transfer switches. 9. Perform electrical tests for UPS system batteries for 5 minutes at full load. 10. See above for a more complete battery maintenance procedure, if applicable. Test Values 1. Compare bolted connection resistances to values of similar connections. 2. Bolt-torque levels must be in accordance with manufacturer's specifications. 3. Micro-ohm or millivolt drop values must not exceed the high levels of the normal range as indicated in the manufacturer's published data. If manufacturer's data is not available, investigate any values that deviate from similar connections by more than 25% of the lowest value.

4.9 CONCLUSION

4.8

107

BEING PROACTIVE IN EVALUATING TEST REPORTS

Acceptance and maintenance testing are pointless unless the test results are objectively evaluated and compared to standards, and to previous test reports that have established benchmarks. It is imperative to recognize failing equipment and to take appropriate action as soon as possible. However, it is common practice for maintenance personnel to perform maintenance without reviewing prior maintenance records. This approach defeats the purpose of benchmarking and trending and must be avoided. The importance of taking every opportunity to perform preventive maintenance thoroughly and completely, especially in mission critical facilities, cannot be stressed enough. If not, the next opportunity will come at a much higher price: downtime and lost business and clients, not to mention the safety issues that come up when technicians rush to fix a maintenance problem. So do it right ahead of time, and do not take shortcuts.

4.9

CONCLUSION

Many decisions regarding how and when to service a facility's mission critical electrical power distribution equipment are going to be subjective. It is easy to choose the objective: a high level of safety and reliability from the equipment, components, and systems. But discovering the most cost-effective and practical methods required to get there can be a challenge. Network with colleagues and knowledgeable sources, and review industry and professional standards before choosing the approach best suited to your maintenance goals. Also, keep in mind that the individuals performing the testing and service should have the best education, skills, training, and experience available. You depend on their conscientiousness and decision making to avoid future problems with perhaps the most crucial equipment in your building. Most importantly, learn from your experiences, and those of others. Maintenance programs should be continuously improving. If a task has historically not identified a problem at the scheduled interval, consider adjusting the schedule respectively. Examine your maintenance programs on a regular basis and make appropriate adjustments to take advantage of improvements in technology and changes in codes, regulations, and procedures.

This page intentionally left blank

5 STANDBY GENERATORS: OPERATIONS AND MAINTENANCE

5.1

INTRODUCTION

Failure of a standby diesel generator to start—which can be due to batteries that are not sufficiently charged, faulty fuel pumps, control or main circuit breaker being left in off position, or low coolant level—is inexcusable. In a facility where the diesel generator is being used for life safety or to serve mission critical loads, the cost of generator failure can be immeasurable. Standby generator reliability is contingent on appropriate equipment selection, proper system design, installation, and proper operations/maintenance. To confirm that maintenance is being done correctly, it is imperative to develop a comprehensive documentation and performance plan. Since standby generators run occasionally, the schedule for performing specific maintenance tasks is usually stated in terms of daily, weekly, monthly, semiannual, and annual time frames, rather than in hours of operation. Diesel engines are one of the most reliable sources of emergency power. Every day, thousands of truckers successfully start a diesel engine without a second thought. An emergency diesel generator should be no exception. However, because truckers operate their engines daily, they tend to be more familiar with the operation and maintenance needs of those engines. Engines being operated daily are getting the use and attention they require. Most emergency diesel generators are neglected because they are not being used on a consistent basis and, consequently, are not being maintained. Maintaining Mission Critical Systems in a 24/7 Environment, Second Edition. Peter M. Curtis 109 © 2011 the Institute of Electrical and Electronics Engineers, Inc. Published 2011 by John Wiley & Sons, Inc.

110

STANDBY GENERATORS: OPERATIONS AND MAINTENANCE

More importantly, the engines are effectively tested under real load conditions every time they are being used. Running an emergency generator on a periodic basis does not provide a significant amount of operational experience. Facilities engineers who have been running one-hour weekly plant exercises or load tests over ten years have less than 600 hours of operational history. It is not until a disaster occurs, and a long run of 100 consecutive hours or more is needed, that one sees the issues associated with power plant operations and their associated maintenance requirements. In this chapter, the reader will become familiar with developing an effective standby generator operations and maintenance program.

5.2

THE NECESSITY FOR STANDBY POWER

The necessity for standby power is an essential concern for businesses that need to continuously operate. Traditionally, standby power has been synonymous with emergency generators. With the advent of very sensitive electronic devices, an emergency generator can fall short of meeting requirements for standby power. There are a number of ways a standby power system can be provided for a load, namely, motor-generator sets, uninterruptible power systems (UPS), and emergency generators. The combinations of the last two are used most often for providing protection from all power-quality and availability problems. The main motive in the use of standby power is to increase the availability of equipment. There are a number of different root causes for why a standby power source may be required; lack of reliable service for the area, utility supply/demand problems, or weather-related problems. A key recent issue referring to the ability of a utility to supply power was the 2003 Northeast blackout, which, due to overloads and untested and malfunctioning utility distribution protection systems, left millions on the East Coast without power. Weather-related problems are normally caused by a loss of transmission service due to high winds, tornadoes, severe snowstorms, or the formation of ice on transmission lines. In general, a standby generation system can be justified for two main reasons: dealing with a chronic problem that leads to power loss or trying to protect the system from utility interruptions. The solutions to these can be very different. Electrical loads in power systems can be roughly divided into two groups. The first group consists of the traditional linear electrical loads, such as lighting and motors. The second group consists of more sensitive nonlinear electronic devices such as computers. The first group can resume normal operation after short power interruptions, provided (a) the interruption is not repeated continuously and the device has a relatively high tolerance for over- and undervoltages, voltage sags, and voltage swells; and (b) the device operates successfully under moderate levels of noise and harmonics. In other words, the normal variation and power quality problems of the utility service does not affect the loads significantly and as soon as the load is energized with auxiliary power, the equipment will resume its function immediately. For these loads, an emergency generator is sufficient to serve as standby power. After loss of utility service, an emergency generator usually begins to energize the load in about 10 seconds.

5.3

EMERGENCY, LEGALLY REQUIRED, AND OPTIONAL SYSTEMS

111

On the other hand, the second group of loads—electronic devices—will be affected by power quality problems. Moreover, they cannot sustain a momentary outage without possible hardware failure or software lockup. Therefore, an emergency generator cannot meet the standby power requirements of the equipment. As mentioned previously, a UPS will provide a reliable source of conditioned power where the load remains energized without any interruptions after the loss of utility service. A UPS's main purpose is to serve as an online power-conditioning device to protect against unwanted power line disturbances and for outages on the short term, as standby generators are there to supply power to the load through the UPS in the long term. With the proliferation of real-time computing in LAN, midrange, and mainframe environments, a standby power source is a necessity in many commercial and institutional facilities. A number of power problems can harm the operations of electrical equipment. They include blackouts, brownouts, voltage swells and sags, under- and overvoltage conditions, power surges, and spikes. Regular power received from the utility is usually stable and reliable, so its presence is almost taken for granted. There is a tendency to presume the source of power problems is external but, in reality, more than 50% of power troubles stem from in-house equipment. Although major power blackouts are uncommon, a major electrical outage in July 1996 impacted more than 2 million in the Northwest United States. In August 1996, another power outage affected more than 4 million in five states. Similarly, several areas in New England and Canada were without electric power in the winter of 1998. Localized blackouts due to weather occur more frequently. For instance, thunderstorms, tornadoes, ice storms, and strong heat waves, as we experienced in the summer of 1999, resulted in power outages. To protect the critical equipment and reduce or eliminate the risk of downtime during a long-term power interruption, organizations rely on backup generators.

5.3 EMERGENCY, LEGALLY REQUIRED, AND OPTIONAL SYSTEMS One of the most misunderstood electrical terms applied to the design of electrical distribution is the word "emergency." Although all emergency systems are standby in nature, there are numerous standby systems that are not emergency systems within the meaning of the National Electrical Code. These other systems, whether legally required or optional, are subject to considerably different prerequisites. It is not admissible to select the rules of the NEC to apply the design of an alternate power source to either supply the circuits of an emergency or standby system. Once an alternate power source is correctly classified as an emergency power source, conditions that apply to standby systems must not be used to decrease the burden of more stringent wiring. A standby power source is either required by law or required by a legally enforceable administrative agency. Emergency systems and legally required standby systems share a common thread in that both are legally required. If a standby system is not legally required, then it is not an emergency or legally required standby system. Once it is established that the law requires the standby system, it must be classified as either an emer-

112

STANDBY GENERATORS: OPERATIONS AND MAINTENANCE

gency system or simply a required standby system. In some cases, the applicable regulation will specify the precise code article. In other cases, this determination can be difficult to make, since there is some overlap in the two articles. The relative length of time that power can be interrupted without undue hazard is the most useful criterion. In a true emergency system, there are three important considerations in identifying a load as one requiring an emergency source of power, all of which state the basic question of how long an outage would be permissible. The nature of the occupancy is the first consideration, with specific reference to the numbers of people that would be congregated at any one time. A large assembly of people in any single location is ripe for hysteria in the occurrence of a fire, particularly when the area turns dark. Panic must be avoided, since it can develop with intense speed and contribute to more casualties than the fire or other problems that caused it. Therefore, buildings with high occupancy levels, such as highrise buildings and large auditoriums, usually require emergency systems. Criticality of the loads is the second consideration. Egress lighting and exit directional signs must be available at all times. Additionally, lighting, signaling, and communication systems, especially those that are imperative to public safety, must also be available with minimal interruption. Other loads, such as fire pumps or ventilation systems important to life safety, will be unable to perform their intended function if they are disconnected from the normal power source for any length of time. Danger to staff during an outage is the third consideration. Some industrial processes, although not involving high levels of occupancy, are dangerous to workers should power be unexpectedly altered. To the highest degree possible, the applicable guideline will designate the areas or loads that must be served by the emergency system. One document refered to often for this purpose is NFPA 101. It states that all industrial occupancies must have emergency lighting for designated corridors, stairs, and so on that lead to an exit. There are exceptions for uninhabited operations and those that allow adequate daylight for all egress routes during production. A note advises authorities having jurisdiction (AHJ) to review large locker rooms and laboratories using hazardous chemicals, to be sure that major egress aisles have adequate emergency illumination.

5.4

STANDBY SYSTEMS THAT ARE LEGALLY REQUIRED

Provisions of the NEC apply to the installation, operation, and maintenance of legally required systems other than those classified as emergency systems. The key to separating legally required standby systems from emergency systems is the length of time an outage can be permitted. They are not as critical in terms of time for recovery, although they may be very critical to concerns other than personnel safety. They are also directed at the performance of selected electrical loads, instead of the safe exit of personnel. For instance, there are several rules requiring standby power for large sewage treatment facilities. In this case, the facility must remain in operation in order to prevent environmental problems. Another example is the system pressure of the local water utility, which must always be maintained for fire protection as well as public health and safety. Although critical, this is a different type of concern, allowing a longer time

5.7

MANAGEMENT COMMITMENT AND TRAINING

113

delay between loss and recovery than is permitted for lighting that is crucial to emergency evacuation.

5.5

OPTIONAL STANDBY SYSTEMS

Optional standby systems are unrelated to life safety. Optional standby systems are intended to supply on-site generated power to selected loads, such as mission critical electrical and mechanical infrastructure loads, either automatically or manually. The big difference between true emergency circuits and those that are served by standby or optional sources is that care must be exercised to select the correct code articles that apply to the system in question.

5.6

UNDERSTANDING YOUR POWER REQUIREMENTS

Managers cannot evaluate the type of backup power requirements for an operation without a thorough understanding of the organization's power requirements. Managers need to evaluate whether they want to shut down the facility if a disturbance occurs, or if they need to ride through it. Since the needs of organizations vary greatly, even within the same industry, it is complicated to develop a corrective solution that can be valid for varying situations. Facilities managers can use answers to the following questions to aid in making the correct decision: 1. What would be the impact of power outages on mission-critical equipment in the organization? 2. Is the impact a nuisance, or does it have major operational and monetary consequences? 3. What is the reliability of the normal power source? 4. What are the common causes of power failures in the organization? Are they overloaded power lines, weather related, or other causes? 5. What is the common duration of power failures? In most cases, there are always far more momentary "blips" and other aberrations then true outages. Are there brief interruptions or brownouts/blackouts that can last minutes, hours, or days? 6. Are there common power quality problems, such as voltage swells, voltage sags, ripples, over- or undervoltages, or harmonics? 7. Do the power interruptions stem from secondary power-quality problems generated by other in-house equipment?

5.7

MANAGEMENT COMMITMENT AND TRAINING

The actions discussed in this chapter and book cannot occur without the support of company senior management and administration. This assistance is critical for the re-

114

STANDBY GENERATORS: OPERATIONS AND MAINTENANCE

liable operation of the critical infrastructure. It is possibly the most demanding task for the user to make management aware of and responsive to the costs involved with maintaining standby systems. It is the responsibility and mission of the user to work with management to institute reasonable operating budgets and expectations based on the level of risk of your organization. Both budgets and goals must be balanced against the ultimate cost of a power loss. Management and the user need to make sure that each is providing the other with realistic expectations and budgets. It is in this area that maintenance records are of vital importance. They form the basis of accurately projecting cost estimates for the future based upon current actual costs. Items that are consuming too much of the maintenance budget can be singled out for replacement. Records kept of power quality problems and the costs avoided because of system reliability are certain to be invaluable in this process. There needs to be a specific plan to monitor and maintain the system, a system of checks to verify that the maintenance is done properly, and tests to be sure that the equipment has not deteriorated. There also has to be a commitment to continuous training to keep the knowledge fresh in the minds of critical people who are in charge of dealing with power failures. One major issue in mission critical facilities is for managers to allow the proper testing of the system, such as an annual pull-the-plug test. In order to accomplish this, you actually need to expose the system to a power failure to know that it and all associated functions, including people, perform accordingly. That does insert a bit of risk into the power reliability, but it is far better to find a problem under controlled conditions.

5.7.1

Lockout/Tagout

Repairs made to equipment and systems using electricity require that the energy be controlled so workers are not electrocuted. The means of accomplishing this is called lockout/tagout. At points where there are receptacles, circuit breakers, fuse boxes, and switches, facilities must use a lock that prevents the energy from being applied and post a label warning personnel that the receptacle has been deenergized and that the power will not be applied until repairs are made. The lock also prevents the switch from accidentally being thrown in the closed or "on" position. The locks should be key operated, and the keys stored in a safe, secure area. It is also a good idea to have a trustworthy person handle the keys and keep them on a key ring, with each key properly labeled. This person should be in charge of repairs made to power sources. Before beginning maintenance, be aware that a generator can start without warning and has rotating parts that can cause serious injury to the unwary. Some other safety concerns are: 1. Never wear loose clothing around a generator. 2. Always stay clear of rotating and hot components. 3. Hearing protection and safety glasses must be worn any time the generator set is running.

5.8 STANDBY GENERATOR SYSTEMS MAINTENANCE PROCEDURES

115

4. Be aware of high voltages. 5. Only qualified and trained personnel should work on standby generators.

5.7.2

Training

Facilities managers should be aware of and take advantage of available resources to help companies meet the challenges of safety issues and stay informed as the regulations change from year to year. Since power failures are rare, it is nearly impossible for people to remember what to do when they occur, so training is needed to keep actions fresh and provide practice so they are properly performed. One such remedy is enhanced training. Whether the training is done in-house or performed by an outside source, a complete and thorough safety training session should be implemented. Some safety topics require more time to finish, but that should not be a factor in the amount of time spent learning about ways to keep workers safe. After all, workers are an organization's most precious assets. Always have regular safety training sessions, usually every year, and practice sessions quarterly. This schedule will instill basic safety requirements while helping the worker remember procedures in wearing and removing personal protective equipment. Practice sessions can be done in-house with ease. There should be a formal program documenting that each critical person did the training, was tested on proper operations, and performed properly. Safety products made with durable material suited for rough applications can make the difference between worker productivity and worker death. It also is wise to buy any upgrades to the safety equipment because technology changes rapidly. Up-to-date information will help ensure complete worker safety. Managers must remember that safety pays. Cutting back on worker safety is not only detrimental to productivity; it is also expensive. How much is the company willing to pay for medical expenses and lost time versus keeping in the clear with OSHA? In the 1970s, worker deaths were at an all-time high. Now, increased awareness and safety training has reduced these figures, but that does not mean that OSHA does not still target companies, especially repeat offenders. Remember that the cost of being safe is insignificant in comparison to the debilitating nature of injuries and the value of a human life.

5.8 STANDBY GENERATOR SYSTEMS MAINTENANCE PROCEDURES When a standby generator (Figure 5.1) fails to start, it is usually due to oversight of the people accountable for its maintenance. Generator sets are reliable devices, but failures often occur because of accessory failures. Managers must take a system approach to power to get more reliability. As stated before, people play a big part in this. They need to be trained and monitored and their performance documented. Standby generators are too dependable and too easily maintained for a failure to occur. If facilities engineers do their part to establish maintenance and testing programs to prove its reliability, the standby generator can be counted on to perform in an emergency situation. As

116

STANDBY GENERATORS: OPERATIONS AND MAINTENANCE

Figure 5.1. Generator. (Courtesy of ADDA.)

dependable as standby generators are, that reliability factor is only as good as their ongoing maintenance and testing programs.

5.8.1

Maintenance Record Keeping and Data Trending

One of the most important elements in any successful quality assurance program is the generation and storage of relevant information. Relevant information can include items such as procurement contracts, certificates of conformance, analysis results, system modification, operational events, operational logs, drawings, emergency procedures, and contact information. It is important to realize that by generating and maintaining the appropriate documentation, it is possible to review important trends and significant events or maintenance activities for their impact on system operation. Sections 5.8.2 through 5.8.7 cover suggested maintenance items for standby generators.

5.8.2

Engine

1. Verify and record oil pressure and water temperature. 2. Inspect the air intake system, including air filter condition, crankcase breather, and turbocharger. 3. Inspect the muffler system and drain condensation trap (if applicable) and verify rain cap operation. 4. Inspect the engine-starting system and verify cable integrity and connections. 5. Inspect exhaust flex coupling and piping for leaks and proper connection. 6. Check for abnormal vibration or noise.

5.8 STANDBY GENERATOR SYSTEMS MAINTENANCE PROCEDURES

5.8.3

117

Coolant System

1. 2. 3. 4. 5. 6.

Inspect clamps, verify condition of all hoses, and identify any visual leaks. Check temperature gauges for proper operation of engine-jacket water heater. Test coolant's freezing point and verify coolant level. Test coolant additive package for proper corrosion inhibitors. Inspect belt condition and tension; correct as required. Inspect radiator core for visual blockage or obstructions. Keep it clean and look for junk laying around that might blow into it and block it. 7. Inspect for proper operation of intake louvers, motorized or gravity type (if applicable). 8. Verify proper operation of remote radiator motor and belt condition (if applicable).

5.8.4

Control System

1. Verify and record output voltage and adjust voltage regulator if necessary. However, the only people who should be adjusting voltage regulators and governing and protecting equipment are those who are specifically trained in how to make the adjustments. 2. Calibrate control meters. 3. Verify and record output frequency and adjust governor if necessary. 4. Verify operation of all lamps on control panel. 5. Inspect for any loose connections and terminals or discoloration. Thermographie inspection would work well here.

5.8.5

Generator Mechanics

1. Inspect and lubricate generator and ball bearing. (Note: this is usually only necessary for older generators; most new models come lubricated for life. Also, when possible, this should be done by a qualified technician.) 2. Look for blocked cooling air passages around the alternator and general condition of generator. 3. Inspect for abnormal vibration. 4. Verify connections and insulation condition. 5. Verify that the ground is properly attached. 6. Verify proper operation of shunt trip on the mainline circuit breaker (if applicable).

5.8.6

Automatic and Manual Switchgear

1. Verify proper operation of exercise clock (adjust if necessary). 2. Visually inspect all contacts and connection points.

118

STANDBY GENERATORS: OPERATIONS AND MAINTENANCE

3. Perform building load test (if practical). 4. Verify operation of all lamps on control.

5.8.7

Load Bank Testing

Load bank testing (Figure 5.2) exercises a single piece of equipment. Tests verify proper operation of generators and associated equipment such as transfer switches. Exercise is part of a maintenance process to verify load-carrying capability and sometimes to remove deposits from injection systems that can occur due to light-load operation. Perform a test for approximately 4 hours that includes a detailed report on the performance of your system. The duration of the test can be shorter in warm climates and may need to be longer in colder ones. The important factor is to run the test long enough so that all temperatures (engine and generator) stabilize. (In addition to ensuring a proper test, this will help drive off condensation.) During the test, all vital areas should be monitored and recorded, including electrical power output, cooling system performance, fuel delivery system, and instrumentation. All of this is done with no interruption of normal power! Load bank testing is the only sure way to test your system. Manufacturers recommend that testing always be done at full load.

5.9 5.9.1

DOCUMENTATION PLAN Proper Documentation and Forms

When the generator is operating, whether in an exercise mode or the load test mode, it must be checked for proper performance conditions. Many facilities have chosen an automated weekly exercise timer. One of the unpleasant aspects that is found in this type

Figure 5.2. Load bank testing. (Courtesy of ADDA.)

5.10

EMERGENCY PROCEDURES

119

of system is that the generator is rarely checked when its weekly exercising is automated. Anytime the generator is being run, it must be checked and all operating parameters verified and thereafter documented. It is through the exercising and testing of the generator that deficiencies become evident. Yet, many facilities with automated weekly exercise timers on the automatic transfer switches do not know that the generator has run, except by reading the incremental run hour clock. If your facility selects this automated feature, be sure that the building engineer or maintenance technician is also available, so that the generator never runs without being checked and having the proper documentation completed. A weekly exercise is recommended but not always practical. However, manufacturers recommend that generators be exercised at least once per month.

5.9.2

Record Keeping

Keeping good records of inspections can give better insights into potential problems that might not be obvious. In other words, certain types of deterioration occur so gradually that they will be hard to detect in a single inspection. However, combined with prior inspection data, a particular trend of potential failure might become evident. For instance, measuring the insulation value of a cable without any past data might show it to be satisfactory, but when compared to a prior test, the insulation value might be seen to have dropped a certain percentage during every test. It would then be easy to estimate an approximate failure time for the cable in the future, before which the cable would need to be replaced. The success of an effective PM program is based on good planning. When deciding when to schedule a shutdown, pick a time that will have minimal impact on the operation. The continual proliferation of electronic equipment has created higher dependence on electrical power systems. As the hardware for electronic devices becomes more robust, the reliability of the electrical distribution may become the weak link in overall system capability. Many elaborate reliability considerations that were typically considered for mission critical applications exclusively are now more commonplace for other facilities. Facilities mangers play a more critical role than ever in the operational and financial success of a company. Now, when facilities managers speak about the electrical distribution system, facility executives are much more likely to listen. The Internet is also becoming a key tool in shaping the way businesses communicate with customers. A growing number of maintenance operations are setting up home pages designed, among other things, to provide information to and foster communications with customers. The most active in this regard have been colleges and universities. Additionally, a Web-based application can be set up so everybody can keep track of scheduled maintenance as well as to provide pertinent documents and spreadsheets.

5.10

EMERGENCY PROCEDURES

Even with the best of testing and maintenance procedures, it is wise to prepare for a potential generator-set failure so that its affects are minimized. The generator-set emergency stops and shut-offs should be easily identifiable and so labeled. A posted

120

STANDBY GENERATORS: OPERATIONS AND MAINTENANCE

basic trouble-shooting guide should be readily available right beside the unit, with emergency call numbers of affected areas and repair contractors. A clean copy of the generator's operation and maintenance (O&M) manuals, complete with schematics, should be kept in the generator room at all times. Never leave the original (O&M) manuals in the generator room as they might become soiled and unusable. Keep the (O&M) originals in a safe place for making additional copies or for an emergency if any of the copies get misplaced or otherwise become unusable. The emergency stop and any contingency plan also need to be tested to prove their proper operation and effectiveness. This regular testing and review is key because responsible people demonstrate their understanding of the system design and their ability to respond to the common failure modes in the system. It is recommended that the generator-set logbook contain the previously noted emergency telephone numbers, basic trouble-shooting guidelines and contingency plans. Although this may be redundant with the posted procedures, it provides an extra safeguard that is strongly recommended. With the emergency procedures detailed in the generator-set logbook, more precise information and instructions can be provided to the operating technician in case of an emergency situation.

5.11

COLD START AND LOAD ACCEPTANCE

An important concern of the standby-generator design engineer is how much time it takes for the standby or emergency generator system to sense and react to a loss of power or other power quality problems. A cold start does not mean a completely cold engine but rather an engine that is not at operating temperature. There is a stipulation in certain codes and standards that state emergency generators must be able to pick up the emergency loads within 10 seconds in the United States and 15 seconds in Canada

Figure 5.3. Generator control cabinet. (Courtesy of ADDA.)

5.12

NONLINEAR LOAD PROBLEMS

121

following a power failure. Once these loads are online, then other critical loads can be connected. In most cases, there are two different standby generator systems. The first is for emergency loads such as the life safety systems. The second is for critical systems that support the data center's critical electrical and mechanical loads. The design criteria from a 10-second cold start is to equip the generator with coolant heaters or block heaters. The coolant heaters are necessary to start the machine and enable it to pick up a full rated load in one step.

5.12

NONLINEAR LOAD PROBLEMS

If the facility's electrical loads, such as computer power supplies, variable speed drives, electronic lighting ballasts, or other similar nonlinear electrical equipment, is furnished with switch-mode power supplies, it is imperative that you advise the generator supplier of such so that proper steps can be taken to avoid equipment overheating or other problems due to harmonics. Some generator manufacturers recommend lowimpedance generators and have developed winding-design techniques to reduce the effects of the harmonic currents generated. The key issue is maintaining the ability to produce a stable voltage waveform and a stable frequency. This is mostly an issue of voltage regulation system design, but can also be impacted by the governing (fuel control) system design. The winding arrangement is not as critical as the impedance of the machine relative to the utility service. The closer it is to what the utility provides, the more like the utility it will operate and the lower the probability of problems. In some instances, the generator may have to be derated and the neutral size increased to safely supply power to nonlinear loads. When a standby generator and UPS are integrated in a system together, problems occur that typically do not exist with a UPS system or generator when they are operating alone in a system. Problems arise only when the UPS and standby generator are required to function together. Neither the UPS nor the standby generator manufacturer are at fault and both manufacturers would probably need to work together to solve the problem. The following are common problems and solutions when applying a design that incorporates a standby generator and UPS.

5.12.1

Line Notches and Harmonic Current

The UPS manufacturer using a properly designed passive filter can address the problem of both line notches and harmonic currents. Most generator manufacturers have derating information to solve harmonic heating problems. However, an input filter on the UPS that reduces the harmonics to less than 10% at full load eliminates the need for derating the generator.

5.12.2

Step Loading

When a generator turns on and the ATS switch connecting it to the UPS transfers, the instantaneous application of the load to the generator will cause sudden swings in both

122

STANDBY GENERATORS: OPERATIONS AND MAINTENANCE

voltage and frequency. This condition can generally be avoided by verifying that the UPS has a walk-in feature. This requires that the UPS rectifier have some means of controlling power flow so that the power draw of the UPS can slowly be applied to the generator in a 10-20 second time frame.

5.12.3

Voltage Rise

This is an application problem that occurs when a generator is closely designed and sized to the UPS and there is little or no other electrical load on the generator. When the UPS is first connected to the generator by the ATS, its charger has turned off so that it may begin its power walk-in routine. If the input filter is the only load on the generator, it may provide increased excitation energy for the generator. The issue is the capability of the alternator to absorb the reactive power generated by the filters. The amount that can be absorbed varies considerably between different machines from the same manufacturer. System designers should evaluate the capability in the initial design of the system to avoid problems. The outcome is that the voltage roams up without control to approximately 120% by some fundamental generator design constraint, typically magnetic saturation of the generator iron. If the value does hit 120% of the nominal, it will be damaging or disruptive to the system operation. However, a UPS that disconnects its filter when its charger is off avoids this predicament altogether.

5.12.4

Frequency Fluctuation

Generators possess inherent limitations on how closely they can manage frequency regarding their response to changing electrical loads. The function is complicated and not only involves generator features, such as rotational inertia and governor speed response, but also involves the electrical load's reaction to frequency changes. The UPS charger, conversely, also has inherent limitations on how closely it can control its power needs from a source with fluctuations in voltage and frequency. Since both the generator controls and the UPS charger controls are affected by and respond to the frequency, an otherwise small frequency fluctuation may be a nuisance. The most noticeable effect of this fluctuation is a recurring alarm that is found on the bypass of the UPS, announcing that the generator frequency is changing faster than the UPS inverter can follow. In order to minimize or eliminate frequency fluctuation problems, good control design from both the engine-generator and UPS manufacturers are required. The engine must have a responsive governor, appropriately sized and adjusted for the system. The UPS should have a control responsive to fast frequency fluctuations.

5.12.5

Synchronizing to Bypass

Some applications require the UPS to synchronize to bypass so that the critical load may be transferred to the generator. This generally places tighter demands on the generator for frequency and voltage stability. When this is the case, the system integration problem may be intensified. As described above, good control design can usually reverse this problem.

5.13 CONCLUSION

5.12.6

123

Automatic Transfer Switch

Most generator/UPS projects incorporate automatic transfer switches that switch the UPS back to utility power once it becomes available again. The speed of transfer can be an obstacle and may result in a failed transfer. This, in turn, will lead to nuisance tripping of circuit breakers or damage to loads. If the ATS switch also has motor loads, such as HVAC systems, the UPS input filter will supply excitation energy during the transfer. If the transfer occurs too fast, causing an unexpected phase change in the voltage, the consequences can be devastating for both the motors and the UPS. One of the best solutions is to simply slow the transfer switch operation speed so that the damaging condition does not exist. Rather than switching from source to source in one-tenth of a second, slowing to one-half of a second will resolve the problem. UPS manufacturers can resolve this problem by providing a fast means of detecting the transfer and disconnecting the filter.

5.13

CONCLUSION

This chapter has discussed selected ways to enhance the time factor in the terms generally used to describe reliability. This is not, and should not be, regarded as the final statement on reliable design and operation of standby generators. There are many areas of this subject matter that warrant further assessment. There is also a considerable lack of hard data on this subject, making practical recommendations difficult to substantiate. For example, there is virtually no data on the trade-offs between maintenance and forced outages. It is self-evident that some maintenance is required, but what is the optimum level? This question is easily answered if your organization's level of risk has been evaluated. However, there are certain common elements on both reliable and unreliable systems that I think are relevant and I listed them accordingly. These suggestions, when seriously considered, will help to increase the level of reliability of the standby system.

This page intentionally left blank

6 FUEL SYSTEMS DESIGN AND MAINTENANCE

6.1

INTRODUCTION

The sudden loss of electrical power has a different meaning to different people. If asked, someone may explain it as being an inconvenience, such as not being able to play a game on the computer, watch the television, or make coffee. But ask the same question to an intensive care nurse, law enforcement officer, electric utility worker, or a data center manager for the banking and financial industries, and the sudden loss of electricity may be explained in terms of loss of life, civil unrest, regional blackout, or possible financial upheaval. Diesel engines have played a pivotal role in providing electrical power in many diverse nonemergency and emergency circumstances due to their reliable start-up and their ability to operate continuously under varying load conditions. Diesel engines are a source of power and electricity; therefore, diesel fuel is the lifeblood of any emergency or backup power system. Under normal everyday use, diesel engines consume diesel fuel well within the allotted shelf life of the fuel. However, fuel for diesel engines dedicated to providing periodic emergency power is generally stored well beyond the expected shelf life. This means that emergency diesel systems can potentially run the risk of becoming inoperable due to issues associated with degradation of fuel quality. This chapter discusses the issues associated with long-term storage of diesel Maintaining Mission Critical Systems in a 24/7 Environment, Second Edition. Peter M. Curtis 125 © 2011 the Institute of Electrical and Electronics Engineers, Inc. Published 2011 by John Wiley & Sons, Inc.

126

FUEL SYSTEMS DESIGN AND MAINTENANCE

fuel, fuel quality, and how to monitor the useful life of diesel fuel dedicated to emergency power systems for mission critical applications.

6.2

BRIEF DISCUSSION OF DIESEL ENGINES

Little did Rudolph Diesel realize that his fascination with steam engine efficiency ratios would lead to the invention of the diesel engine, which quickly found its niche within applications that require power with high torque and low RPM. Modern diesel engines offer many advantages over other types of prime movers. Today, the diesel engine provides power and propulsion in a variety of applications such as agriculture, construction, transportation, heavy long-distance hauling, marine propulsion, mining, military vehicles, stationary power systems, and emergency power generators. The diesel engine does not have igniters or spark plugs to assure proper fuel ignition as do gasoline engines or jet turbines. Instead, the diesel engine depends totally on the compression stroke of the piston to create high cylinder temperatures to ignite the air-fuel mixture in the engine. This unique characteristic of a diesel engine requires the ignition to occur very close to the top of the piston stroke, such that the expanding burning fuel-air mixture produces the power stroke. The original diesel engine engineered by Rudolph Diesel was designed to operate using vegetable oil, which some have speculated to be peanut oil. But with a little convincing, Rudolph Diesel redesigned his engine to run on a crude oil fractional distillate known as "middle distillate" or what is commonly referred to today as diesel fuel. Over the years, the technology used to convert crude oil into diesel fuel had an influence on the stability and shelf life of the fuel product. However, other factors such as storage temperature, exposure to oxygen, the presence of water and other fuel contaminants can have a much greater influence on the shelf life of diesel fuel. Anecdotally, there are a number of examples of diesel fuel performing well after years of storage; however, there are a greater number of examples of diesel fuel failing after just months of storage. Typically, consumption of diesel fuel occurs within 30 to 90 days after production. However, consumption of diesel fuel dedicated to mission critical systems can take place well beyond the general shelf life of 12 months. It is not uncommon for diesel fuel in mission critical systems to be stored well in excess of 12 months before being replaced. Consequently, onsite fuel storage and a fuel quality surveillance program play critical roles in the long-term availability and continuous operation of mission critical diesel engines.

6.3

BULK STORAGE TANK SELECTION

Selecting the main bulk storage tanks is an important first step in system design. There are many factors that must be evaluated in order to establish the best configuration: aboveground or underground, single wall or double wall, steel or fiberglass. Since 1998, federal and state codes that govern the installation of underground tanks made it more attractive to look at aboveground bulk storage tanks. From an environmental

6.3 BULK STORAGE TANK SELECTION

127

standpoint, underground tanks have become a difficult management issue. In 1982, the President of the United States signed into law the reauthorization of the Resource Conservation and Recovery Act (RCRA). Federal and state codes now require very sophisticated storage and monitoring systems to be placed into operation in order to protect the nation's ground water supply. In some instances, property owners prohibit the installation of petroleum underground storage tanks on their properties. On the other hand, aboveground storage tanks occupy large amounts of real estate and may not be esthetically pleasing. Here are some factors to consider when deciding on aboveground or underground bulk storage tanks.

6.3.1

Aboveground Tanks

Aboveground tanks are environmentally safer than underground tanks. The tanks and piping systems that are exposed aboveground are more stable and easier to monitor for leaks than underground systems. Since the tanks and piping are not subject to ground movement caused by settlement and frost, there is less likelihood of damage to the system and, consequently, less of a likelihood of an undetected leak. However, if large tanks are required, the aboveground storage system can be awkward and will occupy large quantities of real estate. Code officials, especially fire departments, may have special requirements or often completely forbid the installation of large quantities of petroleum products in aboveground tanks. Day tanks and fuel headers may be located at the same elevation or occasionally lower than the bulk storage tank, making gravity drains from these devices impossible to configure unless the generators are on or above the second floor of the building. Filling the tanks may require special equipment, and in some cases codes have even required secondary containment systems for the delivery truck because it is felt that the "pumped fill" is more vulnerable to catastrophic failures that can empty a compartment of the delivery truck. Additionally, constant thermal changes to the fuel can potentially stress the fuel, causing it to become unstable. Diesel fuel that is exposed to subfreezing temperatures during the winter requires special conditioning in order to make it suitable for low-temperature operability. If the fuel is not conditioned properly, wax formation may occur, causing the fuel to clog filters and fuel lines. Issues stemming from poor fuel quality are discussed further in Section 6.8 of this chapter. Aboveground tanks are almost always fabricated from mild carbon steel. The outer surface of the tank is normally painted white in order to prevent the fuel from becoming heated due to sun exposure. The tank design should always include a manway and ladder in order to allow access to the tank for cleaning, inspection, and repair. Epoxy internal coatings compatible with diesel fuel provide extra protection to the fuel from the eventual contamination from internal rust. The epoxy liner also provides extra corrosion protection to the internal tank surface. All aboveground tanks should be fabricated to ULI42 or UL2085 standards and should bear the appropriate label. The ULI42 tank is an unprotected steel tank in either single- or double-wall configuration. The UL2085 tank is a special tank that is encased in concrete. The concrete provides a listed fire rating.

128

6.3.2

FUEL SYSTEMS DESIGN AND MAINTENANCE

Modern Underground Tanks and Piping Systems

Modern underground tanks and piping systems have become more reliable, easier to install, and in general easier to operate over the past several years. Since the tanks are usually buried under parking lots or driveways, they occupy little or no real estate. Drain and return lines from fuel headers and day tanks can drain by gravity back to their respective bulk storage tank, making for more efficient piping systems. On the down side, federal, state, and local codes usually impose construction standards that may be difficult to comply with. The underground storage tank must be designed and installed in accordance with the code of federal regulations, 40CFR280. In addition, the appropriate state and local codes must also be followed. The state and local government must follow, at a minimum the federal code; however, state and local governments may implement local codes that are more stringent than the federal code. Obtaining permits can be a long and tedious process, with many agencies involved in the approval process. Most important, since the tank and piping system are out of sight, leaks can go undetected for long periods of time if monitoring systems and inventory control are not meticulously maintained. Underground tanks are most commonly fabricated from mild carbon steel or fiberglass-reinforced epoxy. Steel tank must be specially coated with either fiberglass-reinforced epoxy or urethane coatings to protect them from external corrosion. In addition to the external coating, some steel tanks are additionally provided with cathodic protection to protect the exterior of the tank from corrosion. Fiberglass tanks are usually not coated. The most common manufacturing standard for steel tanks is UL58 and for fiberglass tanks it is ULI316. Similar to aboveground tank designs, underground tanks should always include a manway and ladder in order to allow access to the tank for cleaning, inspection, and repair. For steel tanks, epoxy internal coatings compatible with diesel fuel provide extra protection to the fuel from the eventual contamination from internal rust. It also provides extra corrosion protection to the internal tank surface.

6.4

CODES AND STANDARDS

Codes and standards should always be reviewed prior to any discussion of type and location of the bulk storage tank. A manager should understand that an improperly designed fuel system is a huge risk, not just with regard to reliability, but also with respect to environmental concerns. An improperly installed or maintained fuel tank can expose a company to millions of dollars of liability for cleanup of spilled fuel. It is a good idea to have a preliminary meeting with the governing code agencies in order to discuss the conceptual design and requirements of the system. Do not assume that merely because the state building code allows the use of aboveground storage tanks, there will not be some opposition from one of the other agencies that may have jurisdiction. Fire marshals, in particular, may be completely opposed to the installation of large aboveground diesel storage tanks, especially when they are located in metropolitan areas. The usual sequence to follow when establishing the priority of compliance is

6.5

RECOMMENDED PRACTICES FOR ALL TANKS

129

to start with the state building code. The state building code will normally reference either NFPA or the uniform fire code for the installation of flammable and combustible storage systems. NFPA 30, Flammable and Combustible Liquids Code, addresses the installation of storage tanks and piping systems. NFPA 37, Standard for the Installation and Use of Stationary Combustion Engines and Gas Turbines, addresses the piping systems, day tanks, flow control systems, and, in some instances, references NFPA 30. The uniform Fire Code, Article 79, Flammable and Combustible Liquids, addresses the installation of flammable liquid storage systems. In addition, the U.S. EPA has codes in place to address performance standards for both aboveground and underground storage systems. Underground storage tank standards are covered in 40 CFR 280. This code sets minimum standards that must be followed by all states. The individual states have the option of requiring more stringent standards. Aboveground storage tanks are covered in 40 CFR 112. This code also covers certain topics relative to threshold storage limits for both aboveground and underground tanks that require the owner to maintain a document known as a spill prevention control and countermeasure plan.

6.5

RECOMMENDED PRACTICES FOR ALL TANKS

There are some recommended practices that should be adhered to regardless of what the codes require. Bulk storage tanks should always be provided with some form of secondary containment, regardless of the requirement by the local authority. Anyone who has ever been involved in a project connected to a leaking underground storage tank understands the astronomical costs that can be incurred. In addition, data centers are usually high-profile facilities. The negative press associated with a petroleum spill contaminating underground water supplies can be horrific. The most popular system and easiest to maintain is the double-wall tank. Open containment dikes for aboveground tanks are a maintenance headache, especially in cold climates where removing snow and ice is very difficult. Similarly, for underground tanks, cutoff walls and flexible liners are not effective or efficient ways to provide secondary containment. Instead, the double-wall tank is efficient and easiest to maintain. Piping systems for underground tanks can also be furnished in a multitude of different configurations. Secondary containment should always be provided for underground piping. Similar to the underground tank, the piping is out of sight and can cause grave environmental damage if a leak goes undetected for long periods of time. Some of the choices for piping are double-wall carbon steel, double-wall stainless steel, and double-wall fiberglass. Probably the most popular system in use today is the doublewall flexible piping system. It is fairly easy to install and also UL listed for use with underground petroleum storage systems. Since underground piping runs each have only two connections—one at the tank and the other where the piping leaves the ground and attaches either to a device or an aboveground pipe—there is little opportunity for leaks at joints. There are complete systems available, including high-density polyethylene termination sumps that provide secondary containment where the piping attaches to the tank and also allow access from grade without excavation to service the

130

FUEL SYSTEMS DESIGN AND MAINTENANCE

termination fittings, as well as submersible pumps, and any other equipment such as tank monitoring equipment. Figure 6.1 describes basic installation practices for all tanks, whether aboveground or underground. Ideally, the floor of the fuel storage tank should be installed with a slight pitch to encourage the collection and removal of accumulated water from the low point in the tank. As elucidated in Section 6.8 of this chapter, water accumulation in a fuel storage tank is associated with the risk of microbial contamination, which can contribute to the degradation in the quality of the stored diesel fuel, lead to corrosion of the storage tank, and result in fuel system leaks. A spill-containment fill box should be located at the opposite end of the tank with a drop tube terminating close to the bottom of the tank. Introducing new fuel at this end of the tank will help to move any water accumulated at the bottom of the tank to the opposite end where the water pumpout connection is located. The fuel maintenance system should draw fuel from the lowest point possible in the tank and return it to the opposite end of the tank. Piping systems for aboveground tanks provide fewer choices. Carbon steel and stainless steel are the primary choices. When the piping runs are in open areas such as generator rooms and machine runs, it is generally installed as a single-wall pipe. If the piping runs are in more critical areas or where codes require secondary containment, it is usually installed as a double-wall piping system. However, there are some problems associated with the double-wall system. If a leak develops in the primary pipe, it can be difficult to locate the source of the problem since the leak tends to fill the annular space between the primary and secondary pipe. Another concern is the inability to contain valves, flexible connectors, and other vulnerable fittings. Typically, the doublewall pipe is terminated with a bulkhead at valves and flexible connectors, leaving these

Figure 6.1. Basic installation practices for all tanks, whether aboveground or underground. (Courtesy of Mission Critical Fuel Systems.)

6.5

RECOMMENDED PRACTICES FOR ALL TANKS

131

devices unprotected. Since the piping systems are under rather low pressure and the product, diesel fuel, is not very corrosive, leaks are usually minor. For this reason, wherever possible, install single-wall carbon steel pipe and either place it into trenches or provide curbs around the area where it is installed. The area is then monitored with leak-detection equipment. If a leak develops, an alarm warns the facility operations personnel of the problem. Since the piping is exposed, the source of the leak can be readily determined and corrected. The pipe trench or curb will also contain the leak, thereby preventing widespread damage. Fuel distribution piping materials are of major concern. Many of the performance problems that occur with diesel fuel systems are directly related to the materials of construction. There are two categories of materials that should be avoided. The first group is described in NFPA 30. These materials include low-melting-point materials that may soften and fail under exposure to fire. They include aluminum, copper, brass, cast iron, and plastics. There is, however, one exception to this rule. NFPA 30 allows the use of nonmetallic piping systems, including piping systems that incorporate secondary containment. These systems may be used underground if built to recognized standards and installed and used within the scope of Underwriters Laboratory Inc.'s Standard for Nonmetalic Underground Piping for Liquids, UL 971. The next group is materials that contain copper or zinc. Although these materials may be commonly found in many fuel distribution systems, their contact with diesel fuel should be avoided as described in ASTM D975, Standard Specification for Diesel Fuel Oils. When copper or zinc come in contact with diesel fuel, these materials can form gummy substances that can cause filter clogging. The most popular and practical piping material is carbon steel. The piping should comply with the applicable sections of ANSI B31, American National Standard Code for Pressure Piping. Pipe joints, wherever possible, should be either butt-welded or socket-welded. Where mechanical joints are required, flanges provide the best seal. Flange gaskets should be fiber-impregnated with Viton®. Threaded connections should be avoided wherever possible. Flexible connectors should be installed wherever movement or vibration exists between piping and machinery. The flexible connectors should be listed for the appropriate service. Careful consideration should be given to the type and placement of isolation valves. Carbon steel or stainless steel ball valves with Teflon® trim and Viton®* seals provide excellent service. Once again, the avoidance of threaded end connections should be considered. Socket-welded or flanged end connections provide excellent service. If the end connections are welded, consider the use of three-piece ball valves in order to facilitate valve service and repair without disturbing the welded joint. Fire-safe valve designs should be considered, especially where codes may mandate their use. Thermal expansion of diesel fuel in piping systems should be carefully examined during the design process. The temperature of diesel fuel supplied from underground tanks is approximately 60°F. When the fuel enters a warm building and resides in the fuel piping system when the generators are not running, it will expand due to the change in temperature. If the particular pipe run has closed control valves, check *Teflon® and Viton® are both registered trade names of Dupont Performance Elastomers.

132

FUEL SYSTEMS DESIGN AND MAINTENANCE

valves, or manual valves at both ends of the pipe run, the expanded fuel develops enormous pressures and leaks through threaded fittings, flange gaskets, valve seals or wherever it can find a weak spot. These nuisance leaks usually occur a day or two after the generators are tested. Once the fuel expands and is expelled from the system, the leak ceases until the next generator run. If this condition is present in an existing fuel system, thermal expansion is usually the culprit. Adding pressure relief valves to the section of piping associated with the problem is the ultimate fix. The blow-off from the PRV should be piped to a fuel oil return pipe that empties into the primary fuel storage tank. Fuel maintenance systems or fuel polishing systems are frequently included in new installations where diesel fuel is stored for prolonged periods of time, that is, one year or longer. These systems were not common several decades ago. However, these systems are becoming a must for anyone who stores diesel fuel for emergency standby power. Fuel quality will be discussed in greater detail later in this chapter. As shown in Figure 6.1, the tank is pitched to a low area to encourage settlement of water. The suction stub for the fuel maintenance system is terminated as close to the bottom of the tank as possible at this end of the tank. When the system operates, fuel is pumped through a series of particulate filters and water separators, and then returned to the tank at the opposite end. The operation of the system is timed in order to circulate approximately 20% to 25% of the tank volume once per week. The tank fill connection should be in a spill containment fill box or spill bucket. The spill bucket is designed to contain small spills created during connection or disconnection of the delivery hose. The capacity of the spill bucket is dictated by the authority having jurisdiction. The fill box should be placed at the high end of the tank, encouraging the movement of any sediment and water to the opposite end of the tank, where it will be removed by the fuel maintenance system. Many states now require annual leak testing of theses spill containment fill boxes. A failed test means replacement of the spill bucket. Since it is cast into the concrete top pad over the tank, replacement is a costly proposition resulting in removal and replacement of the concrete pad. Several manufacturers have developed a double-wall retractable spill bucket that can be replaced without removal of the concrete. This should be considered for new installations. A connection similar to the fill connection and spill containment fill box should be located as close as possible to the opposite or low end of the tank. From this connection, any accumulated water may be removed by manually pumping it from the tank. An access manhole and internal ladder provide convenient access to the tank for cleaning and maintenance. For underground tanks, the manhole should be accessible through an access chamber that terminates at final grade. The bulk storage tanks should be provided with an automatic gauging and monitoring system. Systems range from a simple direct-reading gauge to very sophisticated electronic monitoring systems that will provide automatic inventory control, temperature-compensated delivery reports, system leak monitoring, and underground tank precision testing. Many systems include gateways that allow them to interface with many of the popular building automation systems (BAS). These systems are especially beneficial since any unusual conditions are easily identified at the BAS console. An évalua-

6.6 FUEL DISTRIBUTION SYSTEM CONFIGURATION

133

tion should be performed to determine the appropriate system for the application. Prior to deciding on a tank monitoring system, a compliance check should be performed to establish if the local authority has special requirements for tank monitoring.

6.6

FUEL DISTRIBUTION SYSTEM CONFIGURATION

The next step is assembling the pieces into a fuel storage and distribution system. It is strongly advised to divide the bulk fuel storage into at least two tanks. This is a practical approach to providing a means to quarantine new fuel deliveries and allow a laboratory analysis to be performed on it prior to introducing it into the active fuel system. Fuel testing and maintenance will be covered later in the chapter. Additionally, if it is necessary to empty one of the tanks for maintenance purposes, or if the fuel becomes contaminated in one of the tanks, or for any other reason one of the tanks becomes unusable, switching to the alternate tank is a simple task. However, without the alternate tank, it may be necessary to install a temporary tank and piping system, which can be a major undertaking. The simplest of fuel distribution systems is a single tank associated with a single diesel generator. Common configurations are underground tank, aboveground tank, and generator subbase tank. Suction and return piping must be sized within the operating limits of the generator's fuel pump, a common error encountered when calculating the size of the piping. The fuel flow rate, used for calculating the pipe size, is approximately three times the fuel consumption rate. The additional fuel is used to cool the diesel engine injectors and then returned through the return line to the storage tank. A more common system is a combination of bulk storage tanks and smaller day tanks located closer to the generators. The day tanks can be furnished in several different configurations. Generally, the configuration is one day tank per generator. This makes the generator system autonomous for better reliability. However, day tanks can provide fuel to several generators. The size of the day tank is dependent on several factors, including a predetermined period of fuel supply, usually calculated at generator full load. Other determining factors are maximum fuel quantity threshold limits as specified by the governing code. Figure 6.2 shows a typical fuel storage and distribution system flow diagram utilizing two bulk storage tanks and three day tanks. The fuel transfer pumping system consists of two submersible petroleum transfer pumps in each of the bulk storage tanks. Systems utilizing submersible petroleum pumps are efficient and dependable, and the pumps are more easily selected. The alternative is the positive displacement fuel transfer pump, which typically is located inside the facility, usually in an equipment room. If a positive displacement pump is selected, it is vital that the pressure loss in the fuel suction line from the bulk storage tank to the pump inlet be carefully calculated within the pump performance limits. Each of the submersible fuel transfer pumps should contain a dedicated supply pipe from the pump to the manifold loop that supplies the day tanks. In particular, if the bulk storage tanks are located underground, the supply piping is also underground and vulnerable to damage from ground movement due to settling and frost. It may also be damaged by excavating. This is a good place to mention that underground piping, especially nonmetal-

134

FUEL SYSTEMS DESIGN AND MAINTENANCE

Figure 6.2. Typical fuel storage and distribution systemflowdiagram.

lie piping systems, is very fragile. Therefore it is good practice to protect the piping with concrete top slabs and early warning tape. The fuel oil supply manifold shown in Figure 6.2 is piped in a loop configuration. This allows fuel flow to the day tanks from two separate directions. By placing isolation valves at strategic locations in the manifold, sections of the manifold can be isolated for maintenance or repair without disabling the entire system. Final fuel distribution is accomplished by supplying fuel via two control valves in parallel piping circuits to each day tank. Since a failed control valve will disable the day tank that it services, providing a redundant valve in a parallel circuit eliminates the single point of failure created if a single control valve is used. Control valves should always fail closed during a power failure in order to prevent a day tank overflow. As a backup to the control valves, there should be a manual bypass valve that will allow the tank to be filled manually in case a catastrophic control failure disables both of the control valves or the day tank control panel. Finally, manual isolation valves should always be provided around the control valves. This allows isolation of the control valve for servicing without disabling the remainder of the system. A day tank overflow line should be provided as a final spill-prevention device. In the event of a control system or control valve failure, the overflow line should be adequately sized to return fuel to the bulk storage tank and prevent the eventual release of

6.7 DAY TANK CONTROL SYSTEM

135

diesel fuel into the facility or to the environment. The applicable code should be closely followed in order to comply. If a diverter valve, shown in Figure 6.2, is required to divert fuel to the selected supply tank, the ports should be drilled for transflow. This will prevent the valve from stopping flow if the actuator stalls between valve positions. An overflow line can be provided between the bulk storage tanks to prevent an overflow if the diverter valve actuator fails, leaving the valve sequenced to the wrong tank. This will prevent the overflow of one of the bulk storage tanks if fuel is supplied from one tank and inadvertently returned to the other. The overflow line should not be confused with an equalization line. An equalization line enters both bulk storage tanks through the top and terminates near the bottom. The line is then primed with fuel, causing a siphon to begin, which allows the fuel level in both tanks to equalize. It also allows contaminated fuel or water from one tank to transfer to the other tank. For this reason, the use of an equalization line should be avoided.

6.7

DAY TANK CONTROL SYSTEM

There are many commercially available day tank control systems available. Systems vary from simple packaged systems that are best suited to control a single day tank associated with a single bulk storage tank to very elaborate packaged systems that provide control of multiple pumps associated with multiple bulk storage tanks and multiple day tanks. There are several important steps that should be followed when establishing the appropriate system, beginning with the expectations of the degree of reliability. The bulk of this discussion will be directed to the highest degree of reliability. From that level, optional equipment can be eliminated or tailored to fit the budget and the degree of reliability necessary. Step 1 is to establish the day tank capacity. Assuming that the generators and day tanks are located inside of the facility, as opposed to the generators being packaged and located in a parking lot, code restrictions will dictate the threshold limit of fuel that may be placed inside the facility. There are a few techniques that may increase that threshold limit. One technique is to place the day tanks in fire-rated rooms. In addition to increasing the threshold limit, this will also provide some additional physical protection for the day tanks. There is also an exception that was added to NFPA 37 in the 1998 edition stating that "Fuel tanks of any size shall be permitted within engine rooms or mechanical spaces provided the engine or mechanical room is designed using recognized engineering practices with suitable fire detection, fire suppression, and containment means to prevent the spread of fire beyond the room of origin." Common day tank capacities range from 20 minutes to as much as 24 hours of fuel supply at full load. Between 2 and 4 hours fuel supply is a very common range. An important consideration is the amount of fuel remaining after a low-level alarm is activated. If the facility is manned 7 * 24, 1 hour of fuel supply after the low-level alarm is activated is usually adequate. The day tanks should be provided with some form of secondary containment, such as a double-wall, open-top, or closed-top dike. Open-top dikes are difficult to keep clean. On the other hand, if a leak develops at a device that is installed through the top of the tank, the open-top dike will capture it whereas closed-top dikes

136

FUEL SYSTEMS DESIGN AND MAINTENANCE

and double-wall tanks will not. The tank should be labeled either ULI42 or, in some special cases as required by the permitting authority, UL2085. Step 2 is to establish a pumping and piping system configuration. As mentioned earlier, submersible petroleum pumps are efficient and dependable, and the pumps are more easily selected than positive displacement pumps. The pumps should be labeled in accordance with the UL 79 standard. When establishing the piping system configuration, it is most efficient to place bulk storage tanks and day tanks in a configuration that allows the day tank overflow and day tank drain to flow by gravity back to the bulk storage tanks. By doing this, commissioning and periodic testing of the control system is simplified since fuel must be drained from the day tanks in order to adequately exercise and test the level control system. If the piping system does not allow for a gravity drain, a pumped overflow will be necessary. Prior to selecting a pumped overflow, discuss the concept with the permitting agency. Step 3 is to establish a control system. The control system that affords the best reliability utilizes separate control panels for each of the day tanks and pump controls. An important concept in the control system design is the ability to allow manual operation of the entire system in the event of a catastrophic failure of the control system. This includes manual operation of pumps, bypass valves that will allow fuel flow around control valves, and manual tank gauging. Keeping in mind that a day tank may contain several hours' worth of fuel for the connected generator, it is possible to start a pump, manually open a bypass valve, and fill the day tank in just a few minutes. With the day tank now full, the facility operator can more easily assess the control problem and resolve it. Figure 6.3 shows a poorly arranged system. This type of arrangement depends on the control panel to control pumps and control valves. A failure of something as simple as the power feed to the control panel will disable the entire system. Figure 6.4 describes a system using the same components, but arranges them into a configuration that allows the system to operate in a manual mode if the control panel fails. Additionally, the motor starters each have a diverse electrical feed, which in-

Fiaure 6.3. Poorly arranged system. (Courtesy of Mission Critical Fuel System.)

6.7 DAY TANK CONTROL SYSTEM

137

Figure 6.4. System using the same components as in Figure 6.3 but arranging them into a configuration that allows the system to operate in a manual mode if the control panel fails. (Courtesy of Mission Critical Fuel Systems.) creases the reliability further. This system, however, still depends on a single control panel for logic for the entire system. A further improvement would be to provide individual control panels for each of the day tanks with a separate panel to control the pumps. This will allow the day tanks to function individually, preventing a single point of failure at the fuel system control panel. With this system, if the pump controls fail, a pump could be started manually and the day tanks would continue to cycle with their independent control panels. Communication between all panels can either be hard wired or via a communication bus (see Figure 6.5).

Figure 6.5. System using individual components to prevent a single point of failure. (Courtesy of Mission Critical Fuel Systems.)

138

FUEL SYSTEMS DESIGN AND MAINTENANCE

Figure 6.6 further increases reliability by adding redundancy for critical day tank components. The first location is a redundant control valve. Control valves are notoriously prone to failure. Providing two valves in parallel piping circuits increases reliability. Motor-actuated ball valves are less troublesome than solenoid valves, but come at a higher cost. Control valves should fail closed to prevent day tank overflow during power failure. The control valves should also have separate power circuits with separate fuse protection. A common single point of failure is to provide two control valves both of which are wired to the same circuit in the control panel. A short circuit in one valve actuator will disable both control valves if they are being fed from the same circuit. When redundant devices are incorporated into the controls scheme, it is of utmost importance to provide continuous monitoring for each device. Getting back to the control valve scenario, if there are two control valves in parallel fluid circuits, as shown in Figure 6.6, a failure of one of the valves would not be apparent without valve monitoring. The day tank would appear to function normally with diesel fuel flowing through the operating valve. The first indication of a problem appears when the redundant valve also fails. Monitoring devices such as actuator end switches or flow switches should be used to monitor the devices for failure.

Figure 6.6. System that further increases reliability by adding redundancy for critical day tank components. (Courtesy of Mission Critical Fuel Systems.)

6.8

DIESEL FUEL AND A FUEL QUALITY ASSURANCE PROGRAM

139

The last item is selection of day-tank level-control devices. Float-type controls are the most common devices used for day tank control. If this type of device is used, the floats should be either stainless steel or another high-quality material compatible with diesel fuel. Avoid the use of copper or copper-containing alloys for reasons stated earlier in this chapter. Noncontact-level measuring devices such as ultrasonic transmitters are increasingly popular. Due to the decline in diesel fuel quality over the past few decades, gummy substances may begin to form on float controllers, rendering them inoperative. The noncontact devices are less likely to be affected by this condition.

6.8 DIESEL FUEL AND A FUEL QUALITY ASSURANCE PROGRAM Following the enactment of the 1970 Clean Air Act, the United States Environmental Protection Agency (USEPA) has implemented programs designed to improve ambient air quality by reducing the emissions from highway, nonroad, locomotive, and marine engines. Generally, these programs involve the reduction of sulfur in the nation's diesel fuel supply. Recently, two major changes to the nation's diesel fuel supply intended to improve ambient air quality by reducing sulfur content and including the introduction of ultralow sulfur diesel and biodiesel fuels. In 2006, the USEPA launched the ultralow sulfur diesel (ULSD) program. The ultralow concentration of sulfur in the diesel fuel (^ 15 ppm) protects the newer emission systems, such as catalytic converters, and diesel particulate filters of the newer model diesel engines, which are designed to remove harmful emissions and particulates from diesel engine exhaust to very low levels. The goal of the program is to ensure that by the year 2014 all highway, nonroad,* locomotive, and marine vehicles will be using a diesel fuel that meets the maximum specification of 15 parts per million (ppm) sulfur. After years of testing and evaluation, biomass-derived diesel fuel (i.e., biodiesel) was introduced into the nation's fuel supply. Biodiesel blended fuels such as B5 and B20 offer another way to reduce sulfur and our need for foreign crude by extending current fuel supplies, and offer individuals a "green" alternative to power their diesel engines. In 2008, under the jurisdiction of ASTM1' Committee D02 on Petroleum Products and Lubricants and Subcommittee E, ASTM D975 Standard Specification for Diesel Fuel Oils was modified to reflect the increased use of biodiesel and the practice of adding up to five percent biodiesel to certain grades of diesel fuel to allow users to comply with state mandates for the use of renewable diesel fuel. Also in 2008, ASTM D7467, Standard Specification for Diesel Fuel Oil, Biodiesel Blends, was created to address fuel quality specifications for biodiesel blends ranging from B6 up to B20. It is estimated that over 95% of the petroleum-based diesel fuel is manufactured, released to market, and consumed within a relatively short period of time of three months or less. During this period of time, the diesel fuel easily remains in specifica*http://www.epa.gov/nonroad-diesel/2004fr/industry.htm. ASTM International, 100 Barr Harbor Drive, P.O. Box C700, West Conshohocken, PA 194428-2959, www.astm.org. +

140

FUEL SYSTEMS DESIGN AND MAINTENANCE

tion and is always available for use, excluding fuel quality issues that may arise due to unsatisfactory fuel storage system conditions, cleanliness issues, or unusually low temperatures. Historically, most conventional diesel fuel has been very stable and has been stored successfully for many years with minimal degradation and limited remediation required. Emergency diesel generators play a critical role of supplying emergency power when there is a loss of power. Typically, the design function of the emergency diesel generator requires that it be able to operate for extended periods of time before refueling. In some cases, large volumes of diesel fuel must be stored on-site in order to meet the operational requirement. However, during the long periods of nonemergency these systems are operated or tested much less frequently. This generally results in a very small volume of diesel fuel consumed during a 12 month period. Therefore, the storage time of on-site diesel fuel can be in excess of 12 months, which is classified as long-term storage. In some cases, diesel fuel that is stored for longer than 12 months has the potential to undergo fuel degradation that could adversely affect fuel quality, which can affect the operability and reliability of the emergency diesel generators. It has been demonstrated that small amounts of organic sediment that have formed in diesel fuel stored over long periods of time had to be removed by filtering or "polishing" the fuel in order to keep the bulk-stored diesel fuel clean and suitable for immediate use. Another issue that affects the long-term storage of petroleum-based diesel fuels is the presence of biodiesel blended with diesel fuel. To date, the industry's experience with biodiesel blends has resulted in concerns regarding the stability of the fuel under conditions of long-term storage. Typically, for diesel fuels that have been blended with up to 5% biodiesel it is strongly advised by the industry that these fuels be consumed within six months.* Therefore, due to the short-term storage restrictions associated with biodiesel, diesel fuel allocated for use in emergency power generators should be strictly restricted to petroleum-based diesel fuel that contains no biodiesel (i.e., fatty acid methyl ester—FAME). The basis for every successful fuel quality assurance program is a clear program objective, valid performance criteria, valid quality indicators, and a willingness to make process improvements from lessons learned. In the case of mission critical systems involving emergency diesel generators, the program objective is to ensure the reliability and operability of the diesel fuel so that mission critical systems are operable when called upon for service. A fuel quality assurance program considers a variety of issues that can impact fuel quality and system operability such as: 1. Type, quality, and special consideration of diesel fuel needed to operate on-site diesel engines 2. Identifying a local supplier that can meet the fuel quality needs of the facility 3. Identifying a local and competent laboratory that can perform the necessary diesel fuel analysis *"Biodesel Myths Busted," http://www.biodiesel.org/pdf_files/ftielfactsheets/Myths_and_Facts.pdf. diesel Usage Checklist," http://www.biodiesel.org/pdf_files/Usage_Checklist.pdf.

"Bio-

6.8 DIESEL FUEL AND A FUEL QUALITY ASSURANCE PROGRAM

141

4. Factors that can affect the bulk diesel fuel, such as the physical condition and location of the on-site bulk fuel storage tank as well as the annual weather conditions 5. Identifying industry changes that can negatively impact long-term quality of the diesel fuel 6. Prereceipt inspections of routine diesel fuel deliveries 7. Monitoring the ongoing condition of the on-site fuel during long-term storage conditions 8. Remediation process(es) to restore the on-site diesel fuel supply to a stable and useful condition

6.8.1

Fuel Needs and Procurement Guidelines

The first step in a fuel quality assurance program is to determine what the fuel type and fuel quality needs are in order to establish fuel quality purchasing specifications based on accepted industry standards (Table 6.1). Additional information relevant to the long-term storage of fuel that is not available in accepted industry standards should be obtained from the diesel fuel manufacturer, the distributor, the diesel engine and/or fuel injector manufacturers, industry peer groups such as ASTM, or qualified industry consultants. Other issues that are directly related to fuel quality include purchasing diesel fuel from a reputable dealer and having access to a qualified diesel fuel laboratory to perform the necessary fuel testing. In the United States, the industry standards for diesel fuel are detailed in ASTM D975, Standard Specification for Diesel Fuel Oils. The year of issue or revision is indicated by a suffix containing two or more figures, such as D975-09, signifying that the standard was revised in 2009. A second or third subsequent issue of the standard would be indicated by a letter designation; for example, D975-09a or D975-09b. Therefore, it is always prudent and in the best interest of the fuel system operability and reliability to purchase diesel fuel using the specifications quoted from the latest edition of D975. In order to further ensure and document the type, quality, and condition of the diesel fuel being purchased, the fuel dealer should agree to provide the purchaser with a certificate of analysis (COA) for each shipment of diesel fuel delivered to the site and state on the COA that biodiesel was not added to the diesel fuel.

6.8.2

New Fuel Shipment Prereceipt Inspection

In some cases, new fuel shipments may contain residual fuel such as gasoline, E10, or biodiesel from a prior fuel shipment, or water and debris that can contaminate an existing bulk fuel supply. A simple way to protect the on-site bulk diesel fuel supply is by performing a simple prereceipt inspection of the fuel shipment prior to accepting the fuel shipment (Table 6.2). When a diesel fuel shipment arrives on-site for delivery, the paperwork accompanying the shipment should be reviewed carefully to ensure that the fuel shipment matches the fuel purchase. If possible, the delivery truck compartments should also be inspected to ensure that the fuel and compartments are clear and bright,

142

FUEL SYSTEMS DESIGN AND MAINTENANCE

Table 6.1. Fuel procurement guidelines Discussion Prior to developing fuel specifications for diesel fuel oil, the purchaser should be aware of the diesel fuel oil type, quality, and special considerations, as specified by the engine manufacturer, to ensure uninterrupted fuel supply to the diesel engine when needed, regardless of weather conditions or storage limitations. Additionally, the purchaser should interview several fuel manufacturers or their authorized representatives in the local area to ensure that the type and quality of fuel is always readily available and accessible, is not delivered to site with unnecessary contaminants, and does not contain biodiesel, which can shorten the shelf life of the diesel fuel oil or result in unnecessary system remediation. Issue(s) Determine the type of diesel fuel oil and special requirements necessary to operate the on-site diesel engine(s). The diesel fuel oil is purchased or is stipulated to be purchased to address engine operation requirements, special considerations, and winter conditions. The delivery truck providing the fuel delivery to the on-site bulk storage tank has cleared/cleaned the affected fuel compartments of any prior fuel delivery, especially if it contained a fuel product with a lower flash point, biodiesel (B100), or biodiesel blend (BXX), which is commonly called switch loading. Require that the diesel fuel oil shipment does not contain biodiesel.1' The supplier will provide a "Certificate of Analysis" (COA) with each fuel load delivered to the site and is presented to the buyer upon delivery of the fuel. The COA is to be stored in the on-site document storage for future reference or dispute resolution. That every diesel fuel oil order delivered to the site will be clear of excess free water, sediment, or suspended matter (i.e., clear and bright) visible at the bottom of each fuel compartment containing the fuel delivery. The supplier, buyer, and buyer's petroleum laboratory should agree to a reasonable period of time that the fuel delivery truck is to remain on site to undergo the fuel quality inspections (Table 6.2) and analysis of the new fuel (Table 6.3). That any fuel shipment that fails a prereceipt quality inspection (Table 6.2) or new fuel test (Table 6.3) can be returned to the supplier without recourse and replaced with in-specification diesel fuel oil.

Reference(s)/recommendation(s)* 1. Self explanatory

2. ASTM D975-10, Section 1, Section 7, Table 1 with footnotes, and application Appendices 3. ASTM D975-10, Section 3, and Section 7.3

4. ASTM D975-10, Section 3, and Section 7.3 5. Certificate of Analysis to verify fuel type and quality

6. ASTM D975-10, Section 1, Section 6

7. To perform prereceipt inspections and testing

8. ASTM D975-10, Section 1, Section 6

♦Unique references specified above may be modified or updated in later revisions to ASTM D975. Be certain to consult the latest revision of ASTM D975 regarding fuel quality issues. t"Biodesel Myths Busted," http://www.biodiesel.org/pdf_files/fuelfactsheets/Myths_and_Facts.pdf. "Biodiesel Usage Checklist," http://www.biodiesel.org/pdf_files/Usage_Checklist.pdf.

6.8

DIESEL FUEL AND A FUEL QUALITY ASSURANCE PROGRAM

143

Table 6.2. New fuel shipment prereceipt inspection Part 1 Description As a first line of defense to protect the on-site bulk diesel fuel storage, the buyer should attempt to visually inspect the fuel shipment and review the certificate of analysis (COA) to confirm that the diesel fuel shipment is correct and that the shipment is absent of any free water, sediment, and suspended matter at the bottom of each fuel compartment (i.e., clean and bright) prior to testing the new fuel shipment or transfer to on-site bulk storage. Issue(s)

Reference(s)/recommendation(s)*

1. Fuel meets the purchase specification

1. Purchase Agreement, ASTM D975-10, and COA 2. The fuel compartment(s) of the delivery truck 2. ASTM D975-10, Section 1 and Section 6 are visually absent of free water, sediment, and suspended matter (i.e., clean and bright) 3. The bulk fuel shipment is clean and bright 3. ASTM D4176 with "Free Water—Pass" and "Particulates—Pass" Sample Fuel Shipment—ASTM D975-10, Appendix X2 Part 2—Red Dyed New Fuel Shipment Prereceipt Inspection Description As a first line of defense to protect the on-site bulk diesel fuel storage, the buyer should attempt to visually inspect the fuel shipment and review the certificate of analysis (COA) to confirm that the diesel fuel shipment is correct and that the shipment is absent of any free water, sediment, and suspended matter at the bottom of each fuel compartment (i.e., clean and bright) prior to testing the new fuel shipment or transfer to on-site bulk storage. However, the presence of red dye in the fuel can make it difficult to perform an adequate visual inspection. Issue(s):

Reference(s)/recommendation(s)*

1. The red dyed fuel meets purchase specification. 2. A composite sample of fuel from the fuel delivery truck is clean and bright.

1. Purchase Agreement and ASTM D975-10, COA 2. ASTM D4176 with "Water—Pass" and "Particulates—Pass". Note: The fuel dye may interfere with the test. See discussion in the method regarding interferences.

*Unique references specified above may be modified or updated in later revisions to ASTM D975. Be certain to consult the latest revision of ASTM D975 regarding fuel quality issues.

meaning that there are no signs or indications of excess and undissolved water, sediment, or suspended matter. While there is no color or odor requirement for diesel fuel, if a shipment has an appearance or odor that is distinctly different from previous shipments, further testing can be warranted to verify that the fuel is indeed the correct fuel. For example, the density, flash point, and color should be very close to the values on the certificate of analysis to verify that the product being delivered is the certified fuel.

144

FUEL SYSTEMS DESIGN AND MAINTENANCE

The purchase agreement should contain provisions that allow the purchaser to reject a fuel shipment without penalty when there is evidence of an incorrect product shipment, free water, sediment, or suspended matter in the fuel shipment that can potentially contaminate the on-site bulk fuel storage system and degrade the existing fuel.

6.8.3

Analysis of New Fuel Prior to Transfer to On-Site Storage

Some fuel quality issues with new fuel shipment that can compromise the quality of the on-site bulk diesel fuel are not visually discernable. Therefore, it is recommended that after the fuel shipment has passed the visual inspection, found to be clear and bright, and is ready to be transferred on-site that a composite sample of the new diesel fuel shipment be collected and sent to a local laboratory selected by the purchaser to be tested for biodiesel content, flash point, and clear and bright (Table 6.3). The results from this testing will further validate that the fuel shipment is ready for transfer to onsite bulk storage. If the purchaser has contracted a qualified laboratory for service and arrangements are in place to analyze the fuel sample immediately once it has been delivered to the laboratory, test results can be available for review between the laboratory and purchaser within a reasonable period of time. Provisions must be made between the purchaser and fuel supplier to allocate time to wait for test results prior to fuel transfer. If the fuel analysis results indicate that the fuel does not correspond to the diesel fuel specifications, then provisions should exist in the purchase agreement for the buyer to reject the shipment. If the fuel supplier refuses to allocate time for the fuel

Table 6.3. Analysis of new fuel shipment prior to transfer to on-site bulk storage Description As a second line of defense to protect the on-site bulk diesel fuel supply, the purchaser should obtain a composite sample of the diesel fuel oil shipment in order that it can be analyzed for the following quality indicators: biodiesel content,flashpoint, clear and bright, and color. Issue(s)

Reference(s)/recommendation(s)*

1. Obtain a composite fuel sample. 2. Biodiesel

1. ASTM D975-10, Appendix X2 2. None acceptable—ASTM D975-10, Section 7.3.3, Standard test method ASTM D7173 or EN 14078. In case of dispute, ASTM D7173 is the referee test. 3. See ASTM D975-10, Table 1 and Footnote E. Also refer to certificate of analysis. 4. ASTM D4176, Procedure 2,—Water Absent = Pass Water Present = Fail and Particulates Absent = Pass Paniculate Present = Fail. Note: The fuel dye may interfere with the test. See discussion in the method regarding interferences. 5. Compare to description on the certificate of analysis.

3. Flash point 4. Clear and bright

5. Color

"Unique references specified above may be modified or updated in later revisions to ASTM D975. Be certain to consult the latest revision of ASTM D975 regarding fuel quality issues.

6.8 DIESEL FUEL AND A FUEL QUALITY ASSURANCE PROGRAM

145

verification, then the purchaser may need to make other arrangements regarding future fuel supply.

6.8.4

Monthly Fuel System Maintenance

Regardless if diesel fuel is being stored for retail sale, municipal use, mining operations, or mission critical applications, exposure to contaminants, storage conditions, and nature's elements can influence the long-term quality and condition of the diesel fuel stored within the tank. Ideally, all diesel fuel storage should be maintained full, dry, and cool (Table 6.4). However, as a practical matter, this may not be possible in all circumstances. For example, keeping the fuel storage tank full at all times keeps the air space (ullage) at the top of the tank to a minimum. Maintaining a minimal ullage reduces the volume of outside air that can enter the tank. This is important because as the outside air cools in the ullage during the evening hours, the moisture from the air condenses and forms free water droplets that adhere to the wall of the fuel storage tank and/or collect at the bottom of the fuel storage tank. Over time, a sufficient volume of water will accumulate on the wall and the tank floor that can promote and sustain microbial contamination and growth. It is a recommended industry practice to keep diesel fuel as dry as reasonably possible by routinely draining fuel storage tanks of accumulated water. If it is determined that the bulk storage tank has microbial contamination, then it is recommended to remediate the fuel system with an effective and efficacious biocide. Maintaining the fuel cool and at a constant temperature assists in reducing or slowing the degradation reactions within the bulk fuel. This is totally dependent upon the

Table 6.4. Monthly fuel system maintenance/surveillance for water and microbial growth Description Perform monthly tank surveillance for accumulated water and microbial growth. On-site bulk fuel storage tanks can accumulate water from fuel deliveries or as a result of the cooling of moist air in the tank ullage into water droplets. Accumulated water can initiate fuel degradation issues and, if not removed, can grow and sustain microorganisms. Microorganisms are responsible for fuel degradation,filterplugging, and operational problems, as well as tank wall corrosion. Issue(s)

Reference(s)/recommendation(s)*

1. Drain and record volume of accumulated water removed from the tank.

1. ASTM D975-10, Appendix X3 and X3.7. Record volume of water drained from the bulk storage tank. If results indicate an increasing volume of water being drained from the tank, determine and remediate the source of water. 2. ASTM D975-09a, Appendix XI. 15.1, X3.4, and X6

2. Microbial growth

*Unique references specified above may be modified or updated in later revisions to ASTM D975. Be certain to consult the latest revision of ASTM D975 regarding fuel quality issues.

146

FUEL SYSTEMS DESIGN AND MAINTENANCE

location of the fuel storage tank. This is easily accomplished with underground storage tanks because the ground temperature is fairly cool and constant relative to the diurnal fluctuations of the outside air temperature. Indoor fuel storage tanks or vaults have similar temperature controls. However, in some circumstances mechanical or auxiliary equipment located near the fuel storage tank may impart a heat load onto the fuel storage tank, which elevates the temperature of the fuel inside, resulting in accelerated fuel degradation issues. Conversely, aboveground storage tanks (AST) are generally outdoors and exposed to the elements and the diurnal cycle of heat during daytime hours and cooling during the night. Over time, continuous heating and cooling of the fuel can lead to accelerated fuel degradation issues as well as accumulation of water inside the fuel storage tank.

6.8.5

Quarterly or Semiannual Monitoring of On-Site Bulk Fuel

At least quarterly or semiannually, the on-site bulk fuel storage system and fuel inventory should be evaluated to ascertain the ongoing quality of the existing bulk fuel inventory as well as the accumulation of water, microbial growth, and evidence of fuel degradation. Just as important are the locations from where the samples are drawn to provide fuel samples that are meaningful and produce information and valid results (Table 6.5). Test results provide the owner/operator the opportunity to trend, evaluate, and assess fuel quality issues as they unfold, so corrective actions can be implemented proactively to preserve fuel quality and system operability. Fuels that are stored for periods of time that exceed the manufacturer's recommended shelf life can form sediments or gums that can overload filters or plug injectors. Although diesel fuels (i.e., middle distillates) are inherently stable, if stored for extended periods of time they can undergo degradation under specific oxidizing conditions incurred during long-term storage. Additionally, for older diesel engines that circulate diesel fuel for cooling and lubrication during operation, the diesel fuel undergoes continuous heating and cooling and runs the risk of undergoing thermal degradation. Accumulation of water in bulk fuel storage can promote uncontrolled microbial contamination in fuel systems that can also cause or contribute to a variety of problems, including increased corrosivity of the fuel, decreased stability, and filterability problems. Microbial contamination in fuel systems can also cause or contribute to system damage such as pitting corrosion and fuel system leaks. Aged petroleum products can contain acidic constituents that are present as additives or as degradation products formed during service.

6.8.6

Remediation

Diesel fuels that have quarterly or semiannual test results that indicate a trend toward oxidative instability, thermal instability, or the presence of microbial contamination need to be remediated. There are several actions that can be taken by the owner/operator of the system to address these issues. Fuel additives that are available in the market can improve the suitability of fuels for long-term storage and thermal stability, but can be unsuccessful for fuels with

6.8 DIESEL FUEL AND A FUEL QUALITY ASSURANCE PROGRAM

147

Table 6.5. Quarterly or semiannual on-site bulk fuel testing Description Sample and analyze the on-site bulk diesel fuel supply each quarter or semiannually in order to monitor fuel quality during on-site storage. Contact the approved fuel laboratory for instructions on sample volume and sample containers. Issue(s)

Reference(s)/recommendation(s)*

1. Sampling, containers, and sample handling 2. Storage and Thermal Stability of Diesel Fuels 3. Oxidation Stability (ASTM D2274 or D5304)

1. ASTM D975-10, Appendix X2. 2. ASTM D975-10, Appendix X3 3. ASTM D975-10, Appendix X3, X3.5, X3.5.1,andX3.6. 4. ASTM D975-10, Appendix X3, X3.6, andX3.6.3. 5. ASTM D975-10, Appendix X3, X3.5, X3.5.3, X3.6, X3.6.4, X3.6.5, and X3.10. 6. ASTM D975-10, Appendix X1.15.X3, X3.4.2, X3.7, X3.9.2, and X6 (Tank Bottom Sample). 7. ASTM D7173 8. ASTMD664 9. ASTM 3703

4. Particulates(ASTMD6217) 5. Thermal Stability (ASTM D6468)

6. Microbial Contamination

7. Biodiesel Content 8. Acid Number 9. Hydroperoxides Comment(s)

A. Monitor and trend test results for items 3-7 above. Test results that indicate a continuous adverse trend may indicate the need to remediate the on-site bulk fuel and/or storage tank. B. The addition of items 7-9 above (biodiesel, acid number, and hydroperoxides) provides the user with secondary quality indicators to monitor the fuel quality of long-term stored fuel. *Unique references specified above may be modified or updated in later revisions to ASTM D975. Be certain to consult the latest revision of ASTM D975 regarding fuel quality issues.

markedly poor stability properties. Fuel additives generally contain one or more of the following functionalities: antioxidant; metal deactivator, corrosion inhibitor, or dispersant. Most diesel fuels are treated with additives at the refinery, which is adequate to keep diesel fuels stable for consumption within three to six months after being refined. However, diesel fuel that is stored long term (> 12 months) may require additional additives during the storage cycle. Once the fuel degradation process begins, there are some fuel additives available on the market that can temporarily stabilize the fuel condition until the fuel is filtered, polished, and cleaned, but cannot reverse or improve the condition of the degraded fuel product. The ability of degraded and oxidized fuel to act as a catalyst to initiate and facilitate the degradation of new fuel is so strong that diluting a small amount of old, oxidized fuel with a tankload of new fuel will accelerate the degradation of the new fuel. Degraded and reactive fuel must be all used or removed, and the tank emptied and cleaned to make it suitable for long-term fuel storage again.

148

FUEL SYSTEMS DESIGN AND MAINTENANCE

Microbial contamination associated with fuel and fuel systems problems generally involves bacteria and fungi (i.e., yeast and mold) only. In some literature, algae has been improperly referenced as a microorganism that can grow in fuel system. This is not the case, since algae requires sunlight for metabolic processing. Bacteria and fungi in fuel systems have several resource requirements that need to be in place to grow and proliferate, including water, nutrients, and fuel. Microbes require water for hydration and to remain viable. Sources of water include fuel deliveries and condensation of moist air in tank ullage. Nutrients such as phosphorus, nitrogen, and sulfur are provided by the fuel or water. The diesel fuel acts as a source of food energy and carbon for growth and biomass development. Uncontrolled microbial contamination of fuel and fuel systems can result in biodégradation of the fuel, cause recurring filter plugging problems, or cause various forms of corrosion that could potentially result in a fuel system leak or costly and unnecessary system maintenance. The elimination of microbial activity can be accomplished using a biocide. A biocide should be selected based on its ability to eliminate microorganisms in the water phase, overall effectiveness, and efficacy. Note that even a thorough cleaning of a tank, which can remove all microbial growth from visible surfaces, is usually not enough to remove all microbes or their spores from piping and valves. Thus, after a contaminated fuel system has been cleaned, a biocide is often used to kill any remaining microbes in out-of-the-way piping or places to ensure a complete job.

6.9

CONCLUSION

For decades, diesel engines have been a source of power for a myriad of applications and will continue to do so for the foreseeable future. Over decades, market demands and environmental requirements have influenced the methods used to extract, refine, and blend the fractional component commonly referred to as diesel fuel, which, in turn, have influenced product long-term stability. Looking forward, environmental and market demands may require the use of more nonpetroleum components in the final blending of diesel fuels, which could affect product quality. For most applications, the consumption of diesel fuel is done within the product shelf life, but mission critical systems require diesel fuel to remain stable and perform well beyond the normal shelf life. To date, experience has shown that if bulk storage tanks that contain diesel fuel are maintained cool, under constant temperature, dry (with minimal or no accumulated water), and full (to reduce exposure to oxygen), the diesel fuel tends to remain stable for longer periods of time. Coupled with a simple but effective fuel quality assurance program to continuously monitor fuel quality, mission critical systems can be expected to perform reasonably when called upon for service.

7 POWER TRANSFER SWITCH TECHNOLOGY, APPLICATIONS, AND MAINTENANCE

7.1

INTRODUCTION

This chapter is about the power transfer switch and its role in the emergency power system of mission critical facilities. The power transfer switch comes in many varieties; however, we will concentrate on the most common—the automatic transfer switch (ATS). The term mission critical has been traditionally applied to the data center; however, with the introduction of the NEC's Article 708, Critical Operations Power Systems (COPS), we can extend this mantel to any system or, specifically, power system on which a critical operation depends. And while 708 aims at governmental, municipal, and other operations critical to society, we can also consider that any business venture big or small depends on systems critical to their mission. Mission critical operations require continuous electrical power. An interruption in the flow of power could result in significant financial losses unless emergency power comes online quickly. A prolonged power outage will affect the operation of any business. From the data centers to hospitals, to wastewater treatment facilities, critical loads must be supported. Blackouts, such as the one that plunged the northeast United States and parts of Canada into darkness on August 14, 2003, drove home the importance of emergency power systems. Emergency power systems must be understood, tested, inspected, Maintaining Mission Critical Systems in a 24/7 Environment, Second Edition. Peter M. Curtis © 2011 the Institute of Electrical and Electronics Engineers, Inc. Published 2011 by John Wiley & Sons, Inc.

149

150

POWER TRANSFER SWITCH TECHNOLOGY, APPLICATIONS, AND MAINTENANCE

maintained, and documented. Facilities that followed regular maintenance programs had few, if any, problems when the power went out, while many of those that did not follow routine maintenance practices, had inaccurate drawings, insufficiently documented procedures, or insufficiently trained operating staff experienced major problems or, in some cases, complete failures. Both the National Fire Protection Association (NFPA) and National Electrical Manufacturers Association (NEMA) recognize the need for preventive maintenance. A maintenance program and schedule should be established for each particular installation in order to detect, reduce, and eliminate impeding issues that cause unexpected downtime. A preventive maintenance program includes periodic testing, tightening of connections, removal of dust and dirt, and replacement of contacts when visual inspection reveals excessive contact erosion. The NFPA requires that an emergency power transfer system be tested for at least 30 minutes at least every 30 days, although 30 minutes per week is ideal. Although many standards stress the importance of a comprehensive maintenance program, few provide sufficient detail. Your original equipment manufacturer (OEM) is your best source for detailed recommended practice to maintain reliability and protect your warranty. Automatic transfer switches are the heart of the emergency system. If the ATS does not function properly, loads could be left high and dry, no matter how sophisticated the rest of the system may be. Any discussion of power switching and control has to include the automatic transfer switch. Actually, the term automatic transfer switch is much too specific. Not all power transfer switches are "automatic" and not all power transfer mechanisms are power transfer switches. In order to be truly classified as an automatic transfer switch in the United States, units must meet the requirements of ULI 008. All automatic transfer switching devices share some common characteristics, either in construction or application. Automatic transfer switches provide for a transfer between the normal source (usually utility) and the emergency source (the engine or turbine-generator set) on site. An ATS contains two elements: the transfer switching mechanism and the control panel. The ATS control panel will recognize loss of the normal source of power, initiate a start signal to the emergency source, monitor the quality of the emergency power according to customer preferences programmed into the control panel, and, if acceptable, initiate an automatic transfer to the emergency source. Once power has been restored, the ATS control panel will sample the quality of the normal power source again according to customer preferences programmed into the control panel, initiate a timer in order to wait until power is stable, transfer the load back to the normal source, and allow the engine or turbine-generator to run unloaded or cool down for a predetermined time period For ATSs to work automatically, quickly, and dependably, they must be properly selected, sized installed, and maintained. Different types of ATSs are available with various options. Selection of a new or replacement ATS may seem straightforward; however, a careful evaluation of the facility's mission and electrical infrastructure will help determine the specifications that sat-

7.2 TRANSFER SWITCH TECHNOLOGY AND APPLICATIONS

151

isfy the technical requirements and cost considerations for a particular location and application. NFPA 70E has drastically changed the way we perform maintenance and repairs of emergency power systems. Holding briefings, establishing safe work boundaries, and utilizing personal protective equipment (PPE) are all meant to mitigate the hazard of arc flash and electrical shock. Unfortunately, the reality is that when wearing PPE, it is impossible to perform many of the tasks required by maintenance routines. This means that there are only a few tasks that can safely be accomplished wearing PPE. Comprehensive maintenance requires equipment to be deenergized, locked out, and tagged. The issue at hand is that many emergency power systems containing ATSs do not afford the necessary flexibility to isolate and secure these circuits without affecting the connected load. One solution may be the selection of an isolation-bypass-type ATS. The bypass switch isolates the ATS while maintaining power to critical infrastructure. Some ATS manufacturers provide an integrated ATS bypass switch. Switchgear configurations can also provide an ATS maintenance bypass-isolation function, which often has advantages over an integral bypass switch. Some bypass switches allow for load transfer while on bypass; others do not. The critical nature of emergency and standby power systems applications dictates the importance of rigorously maintaining the equipment involved. A good preventive maintenance program that includes operator training, maintenance, and testing of ATSs and the larger integrated system will maximize system reliability.

7.2

TRANSFER SWITCH TECHNOLOGY AND APPLICATIONS

Power transfer switches are available in low-voltage (< 600 VAC) and medium-voltage (600 to 35 kV) versions. The most familiar power transfer switching application is automatic low voltage and that is where we will direct our discussion. Low-voltage automatic transfer switches range in size from 30 to 4000 A. When the ATS controls sense a significant drop in normal source voltage (typically 80% of nominal voltage), the ATS begins the automatic transition to the emergency source. Depending on the application, a programmed delay may be employed to ride through power glitches. Once the emergency source is ready to accept load, the ATS transfers the load automatically. Let us walk through a typical transfer scenario: 1. The ATS control panel senses a loss of the normal source. 2. The ATS control panel provides a contact closure to signal the generator set to start after an adjustable time delay to prevent unnecessary engine operation for momentary power outages. 3. The ATS control panel monitors the acceptability of the emergency source. The parameters that determine acceptability are programmable. When voltage and frequency have reached an acceptable level, transfer is made from the normal to the emergency source. When the normal power source is restored, the ATS control panel again samples the voltage and frequency to determine whether or not the source is acceptable.

152

POWER TRANSFER SWITCH TECHNOLOGY, APPLICATIONS, AND MAINTENANCE

4. After an adjustable time delay to ensure that the source is stable, the load is transferred back to the normal power source. This time delay is adjustable and intended to time out only after power has remained within acceptable limits for a certain period of time. Typically, additional interruptions will reset this timer until the full period remains uninterrupted. 5. Following the retransfer to the normal source, the ATS control panel allows the engine or turbine-generator to run unloaded or cool down before the run signal is removed and the unit shut down. The steps above constitute the basic function of every type of ATS when a failure of the normal source occurs. That is where the similarity stops. There are many different applications for the ATS. Accordingly, manufacturers have developed different types and features to deal with specific applications. Health-care applications, for example, may require closed transition operation to eliminate power disruptions during testing or a selective transfer. Data center applications, on the other hand typically require three-pole, open transition switches as the bulk of the loads are balanced three-phase UPS or HVAC loads. ATSs can also be applied for redundancy purposes, to switch between two utility sources, between two generator sources, or for small loads downstream of two larger ATSs. If an emergency source other than a diesel or turbine generator is to be used (most commonly an uninterruptible power supply), the manufacturer must understand the requirement at the time the ATS is ordered, in case special engineering is required for the application.

7.3 7.3.1

TYPES OF POWER TRANSFER SWITCHES Manual Transfer Switches

Manual power transfer switches commonly come in two types: (1) manually initiated, electrically operated and (2) manually operated, quick make, quick break. Both types require the presence of a qualified operator. The handle or manual controls must be accessible from the exterior of the switch enclosure in order to allow operation without personal protective equipment in accordance with NFPA 70E. The manually initiated, electrically operated transfer switch (TS) has two positions: normal and emergency. A qualified operator must operate controls deliberately, which acknowledges the intent to affect a transfer and causes the TS mechanism to operate. The manual ATS has two positions: normal and emergency. This type of TS is totally manual. Setting the handle to the normal position causes the normal power source to be connected to the load. Setting the handle to the emergency position causes the emergency power source to be connected to the load. If the emergency source is a generator set, it must be started and an acceptable voltage and frequency must be established before or after the transfer, depending on the rating of the transfer switch. Some of these units are designed to perform a hot-to-hot transfer under load and some re-

7.3

TYPES OF POWER TRANSFER SWITCHES

153

quire the transfer to be made first and the generator or utility breaker to be closed after the TS has been transferred. Remember that emergency systems as defined in article 517 and article 700 of the National Electrical Code and NFPA 99 must automatically supply alternate power to equipment that is vital to the protection of human life. Electrical power must be automatically restored within 10 seconds of power interruption. Manual transfer switches will not meet these code requirements and automatic transfer switches are required.

7.3.2

Automatic Transfer Switches

These switches are available in 30 to 4000 amperes for low-voltage (