Distributed Storage Networks: Architecture, Protocols and Management

Distributed Storage Networks Architecture, Protocols and Management Thomas C. Jepsen IT Consultant; Programming Languages Editor, IEEE ITPro Magazine, Computer Science Instructor, North Carolina State University, USA

Distributed Storage Networks Architecture, Protocols and Management

Distributed Storage Networks Architecture, Protocols and Management Thomas C. Jepsen IT Consultant; Programming Languages Editor, IEEE ITPro Magazine, Computer Science Instructor, North Carolina State University, USA

Copyright  2003

John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England Telephone (+44) 1243 779777

Email (for orders and customer service enquiries): [email protected] Visit our Home Page on www.wileyeurope.com or www.wiley.com All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ, England, or emailed to [email protected], or faxed to (+44) 1243 770620. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Other Wiley Editorial Offices John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA Wiley-VCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany John Wiley & Sons Australia Ltd, 33 Park Road, Milton, Queensland 4064, Australia John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop #02-01, Jin Xing Distripark, Singapore 129809 John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1 Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN 0-470-85020-5 Typeset in 11/13pt Palatino by Laserwords Private Limited, Chennai, India Printed and bound in Great Britain by Biddles Ltd, Guildford and King’s Lynn This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which at least two trees are planted for each one used for paper production.

Contents Foreword Preface Acknowledgements List of Figures List of Tables 1

Introduction to Storage Networking 1.1 Overview 1.1.1 Who Should Read this Book? 1.1.2 Overview of Contents 1.2 Evolution of Storage Networking 1.2.1 Mainframe Storage Networks 1.2.2 Storage for Small Computer Systems 1.2.3 Managing ‘Islands of Storage’ 1.3 Terminology 1.3.1 What is a Storage Network? 1.3.2 What is a Storage Area Network? 1.3.3 What is Network Attached Storage (NAS)? 1.4 Storage Concepts 1.4.1 How is Storage Shared Among Different Types of Processors? 1.4.2 What is Storage Virtualization? 1.4.3 What is a RAID? 1.4.4 How is a RAID Different from a JBOD? 1.5 SAN Applications 1.5.1 Backup 1.5.2 Disk Mirroring 1.6 Summary

xiii xv xvii xix xxiii 1 1 2 3 4 4 4 5 7 7 8 12 14 14 15 16 18 18 18 20 21

vi

2

3

4

CONTENTS

Applications for Distributed Storage Networking 2.1 Storage Integration 2.2 Remote Backup/Restoral 2.3 Disk Mirroring 2.3.1 Processor-Centric Remote Disk Mirroring 2.3.2 Storage Centric Remote Disk Mirroring 2.3.3 ‘Split Mirror’ Copy 2.4 Data Migration 2.5 Business Continuity/Disaster Recovery 2.6 Remote Operation of Peripheral Devices 2.7 Mainframe/Open Systems Connectivity 2.8 Network Attached Storage (NAS) 2.8.1 NAS File Sharing Protocol 2.8.2 Distributing NAS Applications 2.9 Summary

23 24 26 27

Distance Considerations for Storage Networks 3.1 Physical Layer 3.1.1 Parallel Bus Limitations 3.1.2 Optical Networking Considerations 3.2 Protocol Considerations 3.2.1 Command Execution 3.2.2 Data Acknowledgments 3.2.3 Remote Tape Backup over Extended Distances 3.3 Caching 3.4 Summary

37 37 37 40 45 46 47

Architectures for Distributed Storage Networking 4.1 Storage Networking in the Business Park 4.2 Storage Networking in the Metro Network 4.2.1 ESCON/Fibre Channel in the MAN Using Link Extenders 4.2.2 ESCON/Fibre Channel/GigE in the MAN Using Point-to-Point WDM 4.2.3 ESCON/Fibre Channel in the MAN Using a WDM Ring Configuration 4.3 Storage Networking in the Wide Area Network 4.4 Summary

51 52 53

27 28 29 30 31 32 32 33 34 35 35

49 49 50

53 54 55 56 57

CONTENTS

vii

5

59 59 61 61

Protocols for Distributed Storage Networking 5.1 Small Computer Systems Interface (SCSI) 5.1.1 Applications 5.1.2 Standards 5.1.3 Network Topology – SCSI Parallel Interface (SPI) 5.1.4 Addressing 5.1.5 Bus Protocol 5.1.6 Physical Layer 5.1.7 SCSI Summary 5.2 Enterprise Systems Connection (ESCON) 5.2.1 Applications 5.2.2 Standards 5.2.3 Network Topology 5.2.4 Addressing 5.2.5 Link and Device Level Functions 5.2.6 Physical Layer 5.2.7 Summary 5.3 Fiber Connection (FICON) 5.3.1 Applications 5.3.2 Standards 5.3.3 Network Topology 5.3.4 Command Protocol 5.3.5 Data Framing 5.3.6 Physical Layer 5.3.7 Summary 5.4 Fibre Channel (FC) 5.4.1 Applications 5.4.2 Standards 5.4.3 Network Topology 5.4.4 Protocol Overview 5.4.5 FC-2 Functions: Links, Frames, Sequences and Exchanges 5.4.6 FC1 Functions: Transmission Coding 5.4.7 FC0 Functions: Physical Layer 5.4.8 Fibre Channel Summary 5.5 Gigabit Ethernet (GigE) and 10 Gigabit Ethernet (10 G Ethernet) 5.5.1 Applications 5.5.2 Standards

63 65 66 72 74 75 75 76 76 78 79 96 100 101 101 102 102 103 104 104 105 106 106 106 107 110 114 126 128 128 132 132 133

viii

CONTENTS

5.6 6

5.5.3 Network Topology 5.5.4 Protocol Architecture 5.5.5 Ethernet Data Framing 5.5.6 Physical Layer – Gigabit Ethernet 5.5.7 Physical Layer – 10 G Ethernet 5.5.8 Gigabit Ethernet Summary Protocol Summary

InfiniBand 6.1 Applications 6.2 Standards 6.3 Network Topology 6.4 Addressing 6.5 Data Framing 6.5.1 Service Classes 6.5.2 IBA Packet Format 6.6 Physical Layer 6.6.1 8B/10B Transmission Coding 6.6.2 InfiniBand Optical Loss Budgets and Operating Distances 6.6.3 InfiniBand MultiMode 1X-SX Optical Output Specifications 6.6.4 InfiniBand MultiMode 1X-SX Optical Input Specifications 6.6.5 Infiniband Single Mode 1X-LX Optical Output Specifications 6.6.6 InfiniBand Single Mode 1X-LX Optical Input Specifications 6.6.7 InfiniBand MultiMode 4X-SX Optical Output Specifications 6.6.8 InfiniBand MultiMode 4X-SX Optical Input Specifications 6.6.9 InfiniBand Single Mode 4X-LX Optical Output Specifications 6.6.10 InfiniBand Single Mode 4X-LX Optical Input Specifications 6.6.11 InfiniBand MultiMode 12X-SX Optical Output Specifications 6.6.12 InfiniBand MultiMode 12X-SX Optical Input Specifications 6.7 Summary

133 136 144 147 148 153 156 159 159 159 159 163 164 164 165 167 167 168 169 170 170 171 171 172 172 173 173 173 174

CONTENTS

7 MAN/WAN Protocols for Distributed Storage Networking 7.1 Wavelength Division Multiplexing (WDM) 7.2 Time Division Multiplexing and SONET 7.2.1 SONET/SDH Standards 7.2.2 SONET Network Architecture 7.2.3 Time Division Multiplexing and the Digital Hierarchy 7.2.4 SONET and the Optical Hierarchy 7.2.5 Use of TDM/SONET for Storage Networking 7.3 Asynchronous Transfer Mode (ATM) 7.3.1 Standards 7.3.2 ATM Protocol Architecture 7.3.3 Virtual Paths and Virtual Channels 7.3.4 Service Categories 7.3.5 Traffic and QoS Parameters 7.3.6 Physical Layer 7.3.7 Interworking with Storage Protocols 7.4 Generic Framing Procedure (GFP) 7.4.1 GFP Framing 7.4.2 GFP Payload Area 7.4.3 Physical Layer Interface 7.4.4 GFP Summary 7.5 Summary – WAN/MAN Protocols 8

Storage Over IP 8.1 Overview 8.1.1 Standards 8.2 Internet SCSI (iSCSI) 8.2.1 iSCSI Protocol Architecture 8.2.2 Command, Response, and Data Sequence Numbering 8.2.3 iSCSI Login 8.2.4 Command Allegiance 8.2.5 Data Transfer 8.2.6 Connection Termination 8.2.7 iSCSI PDU Format 8.2.8 iSCSI Names 8.2.9 iSCSI Security 8.2.10 iSCSI Naming and Discovery

ix

175 175 178 178 178 181 182 185 188 189 189 192 194 195 197 199 201 201 202 203 204 204 205 205 205 206 207 207 208 208 209 209 209 212 213 213

x

CONTENTS

8.3

8.4 9

Fibre Channel and IP Networks 8.3.1 Fibre Channel Frame Encapsulation Format 8.3.2 Fibre Channel over TCP/IP (FCIP) 8.3.3 Internet Fibre Channel Protocol (iFCP) Summary

Storage Management Requirements 9.1 Overview 9.2 Standards 9.3 The SNIA Shared Storage Model 9.3.1 Shared Storage Model Architecture Overview 9.3.2 Components of the Shared Storage Model 9.3.3 Layer Definitions 9.3.4 Integrating the Block and File/Record Layers 9.3.5 The Multi-Site Model for Distributed Storage Networking 9.4 The SNIA Management Model 9.4.1 Common Information Model (CIM) 9.4.2 Web Based Enterprise Management (WBEM) 9.4.3 XML 9.5 The SNIA Prototype 9.6 Distance Considerations for Storage Management 9.7 Summary

10 Security Considerations 10.1 Overview 10.2 Physical Security 10.3 User Login and Administration 10.4 Management Interfaces 10.4.1 SSL/TLS 10.4.2 Simple Network Management Protocol (SNMP) 10.4.3 Common Management Information Protocol (CMIP) 10.4.4 Desktop Management Interface (DMI) 10.5 Firewalls 10.6 Encryption and IPSec

215 216 217 223 227 229 229 230 231 231 232 233 235 236 236 237 240 242 242 244 245 247 247 248 249 249 250 251 253 254 256 256

CONTENTS

xi

10.6.1

Authentication Header (AH) and Encapsulating Security Payload (ESP) 10.6.2 Internet Key Exchange (IKE) 10.6.3 SSL/TLS and IPSec Compared 10.7 Public Key Infrastructure and Digital Certificates 10.8 Virtual Private Networks (VPNs) 10.9 SAN Security 10.9.1 Zoning and Security 10.9.2 LUN Masking 10.9.3 iSCSI Discovery Domains 10.9.4 Switch Authentication 10.10 NAS Security 10.10.1 NFS 10.10.2 CIFS 10.11 Analyzing Network Architecture for Security Issues 10.12 Summary

257 258 258 259 260 260 260 261 262 262 262 262 263 263 264

11 Planning Distributed Storage Networks 11.1 Selecting an Architecture 11.1.1 Mainframe Architectures 11.1.2 SAN vs. NAS – Choosing an Architecture 11.1.3 SAN Architecture 11.1.4 NAS Architecture 11.2 Optical Fiber Planning 11.2.1 Redundancy Planning 11.3 Security Planning 11.4 Outsourcing Storage 11.5 Return on Investment (ROI) 11.6 Summary

267 267 267 272 273 278 279 279 280 280 281 281

12 Glossary of Terms

283

Bibliography Articles and White Papers Books Standards ANSI INCITS ATM Forum Bellcore/Telcordia Distributed Management Task Force (DMTF)

299 299 300 300 300 301 301 301

xii

CONTENTS

InfiniBand Institute of Electrical and Electronics Engineers (IEEE) International Telecommunications Union (ITU) Internet Engineering Task Force (IETF) Java Community Process Secure Sockets Layer (SSL) Storage Networking Industry Association (SNIA) Web Sites Index

302 302 302 303 305 305 305 305 307

Foreword Having been a technical reviewer for Marc Farley’s ‘‘Building Storage Networks’’ and Tom Clark’s ‘‘Designing Fibre Channel Storage Area Networks’’, as well as making my own small contribution (‘‘Storage Area Network Essentials,’’ co-authored with Richard Barker, and published by John Wiley & Sons), I feel very much a part of the growing body of storage networking literature. I was therefore particularly gratified to be asked to contribute a foreword to Tom Jepsen’s ‘‘Distributed Storage Networks.’’ Tom has made a unique contribution to the literature. Today, most large enterprises operate mixtures of mainframe, UNIX, and Windows servers in their data centers. All of these platforms are moving inexorably toward a networked storage concept, but they are taking their own technology directions and moving at their own speeds. It’s possible to find sources of information about open system storage networks and other sources that deal with ESCON and FICON, but IT designers, implementers, and managers who have to cope with the reality of mixed environments are hard put to acquire the unified body of knowledge that allows them to deal with the different capabilities and requirements of the storage networks for which they are responsible. Tom’s book does a masterful job of presenting this much-needed unified body of knowledge in four areas: • it covers both mainframe and open storage network technologies side by side so that readers can develop a broad architectural view as they learn the details • it covers both technology and implementation, explaining both the whats and whys of the networks themselves, and the hows of deploying them • it deals both with storage networking within the data center and with long-distance networks, and their unique challenges and benefits

xiv

FOREWORD

• it deals with all these topics both at the overview level (Chapters 1 and 2) and in depth (Chapters 3–11), providing real value to both executives and hands-on workers. ‘‘Distributed Storage Networks’’ is also unique in that it’s designed both to be read and to be referred to. The conceptual information, particularly in the chapters that cover applications and implementation, is presented at just the right level of detail for architects and implementers–enough to stimulate the thinking processes, but not the mind-numbing detail that only a standards engineer could love. Particularly trenchant in today’s uncertain world is Tom’s advice on security–‘‘build it in; don’t count on adding it on later.’’ The many charts and tables are a handy reference for the inevitable ‘‘speeds and feeds’’ part of storage network design. Whether you’re an architect, designing your organization’s first or tenth storage network, an application implementer who’s concerned with reliability, performance, and security, an administrator needing an overview of the bundle of technologies for which you’re responsible, or a CIO trying to make some sense of the brave new world of distributed storage, Distributed Storage Networks has something to offer you. It’s my candidate for 2003’s ‘‘If you can only have one book on storage networking, make it this one’’ award. Paul Massiglia Technical Director, VERITAS Software Corporation Director, VERITAS Publishing

Preface Stored data is the most valuable asset that any organization has, and the accuracy and reliability of that data are key factors in staying competitive in today’s world. The globalization of business creates a need to link together geographically separate databases in a manner that is transparent to the end user. New government regulations in the healthcare and financial industries create requirements to provide secure and reliable data backups at remote locations without impacting system performance, and to provide strategies for business continuity in the event of data loss. Implementing these functions requires a new mindset on the part of enterprise planners and managers. The familiar computer-centric model of data processing is rapidly being replaced by a storagecentric model in which storage is geographically distributed over a large area, perhaps even the entire world. This book is intended to provide an overview of the architectures, protocols, and management functions needed to understand this new environment. The concept behind this book emerged as a result of two presentations I gave on storage networks in 2001, at EntNet@Supercom in Atlanta, Georgia, USA, and at the Asia-Pacific Network Operations and Management Symposium in Sydney, Australia. I would like to thank the individuals who made those presentations possible, and provided me with the germ of an idea for a book: Lundy Lewis, Aprisma; Shri Goyal, Verizon; and Masayoshi Ejiri, Fujitsu Ltd. Their encouragement and support are greatly appreciated. I would also like to thank the individuals and publications who granted me permission to reprint copyright material: Monica Vago and Rose Maginniss, ANSI/INCITS; Jacqueline Hansson, IEEE; Andrea Westerinen, Distributed Management Task Force; Arnold Jones, Storage Networking Industry Association; Casper Yu, InfiniBand Trade Association; and the IBM Journal of Research and Development. And lastly, I would like to thank Paul Massiglia for his enthusiastic and supportive comments.

Acknowledgements The author would like to acknowledge the use of the following source material. Tables 5.6 to 5.13, 5.27, 5.28, 5.25 and 5.26 are reprinted with permission from, ANSI INCITS 296-1997 SBCON, ANSI INCITS 2301994 (R1999) Fibre Channel, ANSI INCITS 342-2001, Fibre Channel Back Bone (FC-BB). Figures 7.8 and 7.16 are reprinted with permission from, ANSI INCITS 342-2001, Fibre Channel Back Bone (FC-BB). From IEEE Std 802.3-2002*, Copyright  2002, by IEEE. All rights reserved. Tables 5.30 to 5.33 reprinted with permission from IEEE Std 802.3-2002*, ‘‘Carrier sense multiple access with collision detection (CSMA/CD) access method and physical layer specifications’’, Section 3, Table 38-3, Table 38-4, Table 38-7, Table 38-8 and Table 38-10. Copyright  2002, by IEEE. The IEEE disclaims any responsibility or liability resulting from the placement and use in the described manner. From IEEE Std 802.3ae-2002*, Copyright  2002, by IEEE. All rights reserved. Tables 5.34 to 5.40 reprinted with permission from IEEE Std 802.3ae-2002*, ‘‘Part 3, Amendment: Media: Access Control (MAC) Parameters, Physical Layers, and Management Parameters for 10 Gb/s Operation’’, Copyright  2002, by IEEE. The IEEE disclaims any responsibility or liability resulting from the placement and use in the described manner. Figures 6.4 and 6.5, along with Tables 6-1 to 6-9. Infiband Architecture Specification, Vol. 1, Rel 1.1, Infiband Architecture Specification, Vol. 2, Rel 1.1 The charts and tables referenced above from the Infiband Architecture Specification, Vol. 1, Rel 1.1 and Infiband Architecture

xviii

ACKNOWLEDGEMENTS

Specification, Vol. 2, Rel 1.1, is provided ‘‘AS IS’’ and without warranty of any kind, including, without limitation, any express or implied warranty of non-infringment, merchantability or fitness for a particular purpose. In no event shall IBTA or any member of IBTA be liable for any direct, indirect, special, exemplary, punitive, or consequential damages, including, without limitation, lost profits, even if advised of the possibility of such damages. Figures 8.1 to 8.4, IETF draft-ietf-ips-iscsi20, 19/01/2003, Figures 8.8 and 8.9, IETF draft-ietf-ips-fcovertcpip, 12/08/2002 and Figure 8.13, draft-ietf-ips-ifcp- 14/12/2002. Copyright  The Internet Society (2002/2003). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restrictions of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works.

List of Figures 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 1.14 1.15 1.16 1.17 1.18 1.19 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9

Mainframe Storage Network Architecture (1960s) Traditional Computer Network Architecture Computer Network Architecture with Storage Network Point to Point Topology Arbitrated Loop Topology Arbitrated Loop Topology – Hub Configuration Switched Fabric Topology Redundant Fabric Switch Configuration Two Tier Layered Architecture Core/Edge Backbone Architecture Network Attached Storage (NAS) SAN versus NAS - The User Viewpoint Types of Data Sharing Storage Virtualization RAID Architecture LAN-Based Backup SAN-Based Backup Processor-Centric Disk Mirroring Storage Centric Disk Mirroring Extending SAN Functions Across the MAN or WAN Distributed SAN for Storage Sharing and Integration Remote Backup/Restore Using Distributed Storage Network Processor-Centric Disk Mirroring Using Remote Mirror Storage Centric Remote Disk Mirroring Split Mirror Copy Using Three Mirrors Use of Storage Network for Data Migration Distributed SAN for Disaster Recovery/Business Continuity Using a Storage Network for Mainframe/Open Systems Connectivity 2.10 NAS Head Architecture 3.1 Bus/Tag Architecture 3.2 SCSI Bus Topology

xx

3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 5.11 5.12 5.13 5.14 5.15 5.16

LIST OF FIGURES

Parallel Bus Signal Impairments Computing Loss Budgets for Optical Links Differential Mode Delay (DMD) Use of Wavelength Division Multiplexing Storage Protocol Data Transfer Functions CCW Pipelining in FICON Effect of Data Droop in SAN/WAN Applications Effect of Data Droop on I/O Performance Distance Limitations for Common Storage Protocols (Native Mode) Distance Versus Protocol for Storage Protocols Using Native Mode ESCON or Fibre Channel to Provide Storage Connectivity in the Business Park (