Cisco - Establishing Best Practices for Network Management 804

804 1066_05F9_c2 1 © 1999, Cisco Systems, Inc. Establishing Best Practices for Network Management Session 804 804 10...

18 downloads 418 Views 2MB Size Report

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form

DOWNLOAD PDF

804 1066_05F9_c2

1

© 1999, Cisco Systems, Inc.

Establishing Best Practices for Network Management Session 804

804 1066_05F9_c2


Copyright © 1998, Cisco Systems, Inc. All rights reserved. Printed in USA. 1066_05F9_c2.scr

2

1

Agenda • • • • •

804 1066_05F9_c2

Introduction to Best Practices Preparing the Network for Management Managing Change Fault Management Summary

3


Introduction to Best Practices

804 1066_05F9_c2



4

2

Network Downtime is Costly • The Internet and e-commerce has significantly increased the availability stakes… 24-hour banking E-trade Global economy

8

Infonetics Cost of WAN Downtime ’98

7 6 5

Average Dollars 4 per Year 3 ($000,000)

2 1 0

$3.6M Productivity Productivity Loss Loss

$4.2M

$3.6M

Revenue Revenue Loss Loss

Costs

Enterprise Network Mgmt. Budget

*Due to hard downtime and service degradations 804 1066_05F9_c2


5

Best Practices Defined • Applying what works well for others to improve overall network availability Reduce the time required for planned outages (scheduled change) and includes changes with no associated outage Reduce network downtime during unplanned outages (unscheduled change)

804 1066_05F9_c2



6

3

Lots of Practices—Some Truths • Even the best NM products can be useless with “bad” practices

Do What Works for You!

• Tools help you to do your job, they are NOT the job • Communication and security are the “bread and butter” of best practices 804 1066_05F9_c2


7

Preparing the Network

804 1066_05F9_c2



8

4

Congratulations! You’ve just Been Promoted to Manage the Entire Network for the Western Region...

804 1066_05F9_c2

9


What They’re Really Thinking…

What am I getting into… how am I going to do this? Where do I begin?

804 1066_05F9_c2

I sure hope he lasts longer than the last guy..



What a loser! Does he have any idea what he’s in for? How come we don’t have legs?

10

5

Preparing the Network for Management Best Practices 1. 2. 3. 4. 5.

804 1066_05F9_c2

Selecting the “right” tools Preparing the devices Preparing the tools Building a baseline Maintaining “management”

11


Selecting the Right Tools

? • How do I select the “right” set of management applications? Understand the technologies and buzzwords Understand your network and end-user requirements Implement company standards Many choices evaluate and choose what’s right for your environment 804 1066_05F9_c2



12

6

Platforms and Vendor Specific Management • NMS SNMP-based, status map, and trap receiver HP Openview, Tivoli Netview, CA UniCenter, SNMPc, etc. MicroMuse, Seagate, Concord, Enterprise Pro, and MRTG

• Vendor Specific Geared towards managing a specific vendors devices only Optivity, Transcend, CiscoWorks2000

804 1066_05F9_c2

13


Integrating Enterprise Management Helpdesk, Trouble-ticket, Event MOM

Application Application

DBMS DBMS

Server Server

Service

804 1066_05F9_c2

Service

Service

Network Network

Device

Device



Device

Desktop Desktop

Device

Device

User User

Device

14

7

Understand Your Organization • Roles and responsibilities • Escalation policy • Help desk vs. operations • Planners vs. administrators 804 1066_05F9_c2


15

Preparing the Devices

• Security for Management • Notification • Baseline 804 1066_05F9_c2



16

8

Securing the Devices • Identify scope of control Who needs access to what?

• Secure and log access Physical access (badge readers) Telnet and console (AAA accounting, Syslog) SNMP communities (ACL, SNMP traps) 804 1066_05F9_c2

17


Sample Security Configuration

aaa new-model aaa authentication login test tacacs+ line aaa authentication enable default tacacs+ enable access-list 8 permit 161.44.34.157 logging 161.44.34.157 logging source-interface Loopback0 snmp-server community public RO snmp-server community bitbuck RW 8 snmp-server contact Paul L. Della Maggiora snmp-server chassis-id 071293 snmp-server system-shutdown snmp-server trap-source Loopback0 snmp-server trap-authentication snmp-server host 161.44.34.157 public frame-relay

804 1066_05F9_c2



Tacacs+ SNMP Community ACL Syslog SNMP gets and sets

SNMP traps

18

9

Security Access Changes

• Password change policy Quarterly Every time an employee leaves

• Solution Use radius or tacacs+ Script the change 804 1066_05F9_c2

19


Notification • SNMP Traps Critical for NMS notification

• Syslog Cisco-specific notification

804 1066_05F9_c2



20

10

Sample Notification Configuration

aaa new-model aaa authentication login test tacacs+ line aaa authentication enable default tacacs+ enable access-list 8 permit 161.44.34.157 logging 161.44.34.157 logging source-interface Loopback0 snmp-server community public RO snmp-server community bitbuck RW 8 snmp-server contact Paul L. Della Maggiora snmp-server chassis-id 071293 snmp-server system-shutdown snmp-server trap-source Loopback0 snmp-server trap-authentication snmp-server host 161.44.34.157 public frame-relay

804 1066_05F9_c2

Tacacs+ SNMP Community ACL Syslog SNMP gets and sets

SNMP traps

21


Building a Baseline

• Document the network Maps Spreadsheets/databases

• Track inventory Identify equipment and who owns it

• Backup configurations 804 1066_05F9_c2



22

11

Building a Baseline

• Collect performance data Snapshot of the network Provides historical data for comparison Useful for capacity planning and trending

804 1066_05F9_c2


23

Discovering the Network • Auto-discovery can make documentation easy… but the daemons must be tamed Filters Seedfiles Discovery intervals Exchange inventory among multiple autodiscovery tools 804 1066_05F9_c2



24

12

Layer 2 Autodiscovery 1. Query seed device via SNMP 2. Query CDP neighbor table (ciscoCdpMIBObjects) 3. Interrogate neighbors Caveat—CDP only sees Cisco devices c55k-26 (enable) sho cdp neigh Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge S - Switch, H - Host, I - IGMP, r - Repeater Port Device-ID Capability -------- ------------------------4/1 002261261 4/1 002274433 4/1 069004796 4/1 Router_81.130 4/1 WBU_GATEWAY

804 1066_05F9_c2

Port-ID

Platform

----------------- ------------------ ------4/1 4/1 4/1 Ethernet0 Ethernet0

WS-C5000 WS-C5000 WS-C5500 cisco 4500 cisco 4500

T B S T B S T B S R R


25

Layer 3 Autodiscovery 1. Start with default router 2. Query MIB II ifTable, ipAddrTable, ipRouteTable 3. Interrogate neighbors Special cases e.g. IP unnumbered, HSRP 4500-4>sho ip rout Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, * - candidate default U - per-user static route Gateway of last resort is not set

O C C 804 1066_05F9_c2

100.0.0.0/8 is subnetted, 1 subnets 100.100.100.0 [110/70] via 172.16.11.1, 13:35:34, Serial0 153.10.0.0/16 is subnetted, 1 subnets 153.10.1.0 is directly connected, Serial1 172.16.0.0/16 is subnetted, 1 subnets 172.16.11.0 is directly connected, Serial0



26

13

Inventory

• Typical NMS is not enough IP address, comm strings, and interfaces

• Third-party management suites and vendor specific provide richer content • MIBs are generally vendor specific, although entity MIB will change this 804 1066_05F9_c2

27


Inventory • Items of interest System information Chassis information Chassis cards Interfaces Storage and memory Serial numbers

• All information available via IETF and Cisco MIBs 804 1066_05F9_c2



28

14

Configurations • Collection repository Useful for staging new configs Version control helps with space and documentation

• How to automate Scheduled backup Watch Syslog 804 1066_05F9_c2

29


Maintaining Management

• Adding new devices • Keeping the management applications up-to-date • New management products and standards An Ongoing Process! 804 1066_05F9_c2



30

15

Change Management

804 1066_05F9_c2

31


Post Mortem Blues • Unplanned outages may be the result of many factors. How do you explain and account for what occurred?

I Didn’t Do It

Fact based vs. hearsay Who, what, and when was the change made? Your job may be at stake 804 1066_05F9_c2



32

16

Some Facts

• 80% of all outages are due to human error* When an airlines reservation system went down, thousands of travel agents had to book flights manually. Estimated loss of reservations amounted to $36,000 a minute

*Based on Carnegie-Mellon Usability Study 804 1066_05F9_c2

X


33

Common Causes of Change

• Business growth or downsizing • New applications or services • Implementing new technology • Deploying product fixes or upgrades

804 1066_05F9_c2



34

17

Change Management Defined

• Configuration, software and hardware changes • Change tasks include: Anticipating and planning for change, controlling the introduction of change, and installing and implementing changes to software and hardware 804 1066_05F9_c2

35


Best Practices for Change Best Practices 1. Implementing a change control process 2. Planning for change 3. Implementing change 4. Monitoring change

804 1066_05F9_c2



36

18

Change Control Process

Change request • End user request • New app, server • New network service

Change review board • Identify risk • Schedule change • Generate work order

Change or work order • Tracking # • Detailed change requests

Close Work Order or Resubmit If Problems

Validation • Change verification • Audit

Implementation • Net admin • Engineer/tech.

804 1066_05F9_c2

37


Examples

804 1066_05F9_c2



38

19

Planning • Hardware Pre-configure, test prior to upgrade

• Software Research release, defect support, new feature set, and device compatibility

• Configuration Test prior to deployment

• Have a back-out plan 804 1066_05F9_c2

39


Implementing

• Make different types of changes one at a time • Maker/checker model • Understand contingency plan in event of failure • Validate the change was successful 804 1066_05F9_c2



40

20

Monitoring

• Identifying change, who, what, when • Audit trail • Fault notification

804 1066_05F9_c2

41


Change Management Tools Planning SWIM—Defect, SWIM—Defect, image image analysis analysis CWSI—Layer CWSI—Layer 2/Layer 2/Layer 33 topo topo Netsys—Impact Netsys—Impact of of change change

Deployment

Monitor

SWIM—Download SWIM—Download software software images images CWConfig—Deploy CWConfig—Deploy config config changes changes CiscoView—Switch CiscoView—Switch config config changes changes

CAS—Change CAS—Change audit audit and and reporting reporting service, service, logs logs software, software, config config and and hardware hardware changes changes CWSI—Topo CWSI—Topo and and user user tracking tracking

804 1066_05F9_c2



42

21

Change Scenario 1. User telnets into device and makes a config change (shutdown int) 3. C/Agent identifies device change, notifies archive

Server

Change Agent

4. Archive gets config via transport validates change w/DIFF Archive Audit Log

5. IF VALID, Archive gets Config and logs details to ENCASE Syslog Poll

Transport Change

Network 804 1066_05F9_c2

2. Device updated Syslog generated


43

Fault Management

804 1066_05F9_c2



44

22

Scenario

• Virginia building-003 network goes down • Your boss has bad breath • Multiple people making changes • Resolution takes nine hours 804 1066_05F9_c2

45


Scenario

• Result: Network was down additional four hours due to conflicting changes No one seems to know how the problem occurred or how it was resolved

804 1066_05F9_c2



46

23

Best Practices for Fault Management Best Practices 1. 2. 3. 4. 4.

804 1066_05F9_c2

Preventive Measures Coordination Reacting to Faults Escalation Policy Become Proactive


47

Preventive Measures • Maintain accurate documentation Key to quick resolution Includes maps, closets, connections, wiring, and servers May require process/policy change. Only good if up to date, easy to maintain, and useful Dump it if you can’t maintain it! 804 1066_05F9_c2



48

24

Preventive Measures • Remove single points of failure Alternate paths for mission-critical applications Redundant equipment for critical junctures Ensure appropriate bandwidth to avoid contention and over utilization Permits network rerouting 804 1066_05F9_c2

49


Coordination • Communication is KEY... Understand roles and responsibilities Place phones in closets; use cell phones, pagers Publish policies and procedures 804 1066_05F9_c2



Say What You Do, Do What You Say 50

25

Coordination

• Establish base of operations All efforts must go through one person Prevents “who dropped the baby” and “slam management” Conduct practice “scramble”

• Train staff on devices and technology 804 1066_05F9_c2

51


Determination of Faults

• Notification via: NMS status change Trap and event logs Help desk Phone call from tech (“whoops...”) 804 1066_05F9_c2



ALARM

52

26

Determination of Faults

• Remove the “noise” factor 1. Filter 2. Prioritize 3. Appropriately notify 4. Correlate 804 1066_05F9_c2

53


Reacting to Faults • Determine fault domain Which equipment, services, and users are affected?

• Determine level of response What is the severity of the fault? Can we kill the backbone? Identify dispatch timeframe and number of people 804 1066_05F9_c2



54

27

Reacting to Faults (Severe) • Determine escalation timeline Criteria and time limits to escalate to next level Opening a case with the TAC Identifying the point of drastic action 804 1066_05F9_c2

Is It Time to Hit the Big Red Switch?


55

Reacting to Faults (severe) • Coordinate, communicate, and document • Debrief Determine source of fault Evaluate recovery efforts Document resolution for continuous improvement process In order to learn, avoid CYA environment 804 1066_05F9_c2



56

28

Moving from Reactive to Proactive

• Automate fault notification, escalation and resolution via “triggers” • React to data before it goes bad • Learn device and network behavior That doesn’t look right… 804 1066_05F9_c2


57

Active vs. Passive Polling

• Polling with thresholds vs. event-based polling RMON events and alarms

• Conservation of network traffic vs. device CPU and memory • Might be a combination of both 804 1066_05F9_c2



58

29

Fault Management Tools Planning CiscoView— CiscoView— Real-time Real-time time time monitoring monitoring RME—Availability, RME—Availability, Syslog Syslog and and CCO CCO tools tools CWSI—User CWSI—User tracking, tracking, traffic traffic director director and and topo topo

Deployment SWIM— SWIM— Defect Defect analysis analysis CCO/TAC— CCO/TAC— Case Case tracking tracking tools tools Stack Stack Decoder— Decoder— Crash Crash analysis analysis

804 1066_05F9_c2

Monitor Availability— Availability— Monitor Monitor key key resources resources Syslog—Reporting, Syslog—Reporting, automated automated recovery recovery 24-Hour 24-Hour Reports— Reports— Monitor Monitor reloads, reloads, Syslog, Syslog, and and change change Traffic Traffic Director—RMON Director—RMON config config and and report report


59

Best Practices Can Improve Network Availability • Prepare the network for management Security, notification and maintenance

• Implement a change control process Plan, deploy and monitor

• Reduce unplanned outage minutes through fault management Prepare, coordinate and be proactive 804 1066_05F9_c2



60

30

For More Information • General network management portal http://netman.cit.buffalo.edu/index.html

• Another good network management portal http://compnetworking.miningco.com/msubmanage.htm? terms=network+management&cob=home&TMog= 5006366091143m&Mint=56534342191358&FFV=1

• “The Simple Times” http://www.simple-times.org/pub/simple-times/issues/

• SNMP FAQ http://www.cis.ohio-state.edu/hypertext/faq/usenet/ snmp-faq/part1/faq.html 804 1066_05F9_c2


61

For More Information • Sample Cisco device security configs http://www.cisco.com/warp/public/700/tech_configs .html#SECURITY

• Cisco device SNMP configuration tips http://www.cisco.com/warp/public/490/index.shtml

• White paper on threshold management http://www.ccci.com/product/papers/pete/papers/thresh.htm

• Public domain performance monitoring tool (MRTG) http://ee-staff.ethz.ch/~oetiker/webtools/mrtg/mrtg.html 804 1066_05F9_c2



62

31

Please Complete Your Evaluation Form Session 804

804 1066_05F9_c2


63

804 1066_05F9_c2


64


32

Cisco - Establishing Best Practices for Network Management 804

Environmental Best Management Practices for Aquaculture

Internet Management (Best Practices Series)

Accounts Receivable Management: Best Practices

Cisco - Evolution of Network Management Technologies 801

Cisco - Network Management within Campus Networks

Best Practices

Executive Compensation Best Practices (Wiley Best Practices)

Accounting Control Best Practices (Wiley Best Practices)

Cisco Network Security

Executive Compensation Best Practices (Wiley Best Practices)

Cisco Network Professional's

APM Best Practices: Realizing Application Performance Management

Building a Cisco Network for Windows 2000

Building a CISCO Network for Windows 2000

Corporate Management, Governance, and Ethics Best Practices

Best Practices in Business Technology Management

Event Management And Best Practices (IBM Redbooks)

Irrigation Management Transfer: Strategies and Best Practices

Building a Cisco Network for WIndows 2000

Benchmarking Best Practices in Maintenance Management

Project Management: Best Practices: Achieving Global Excellence

Building a Cisco Network for WIndows 2000

Cisco Network Essentials for Educational Institutions

Software Metrics: Best Practices for Successful IT Management

WebSphere eXtreme Scale Best Practices for Operation and Management

Cisco - Establishing Best Practices for Network Management 804

Environmental Best Management Practices for Aquaculture

Internet Management (Best Practices Series)

Accounts Receivable Management: Best Practices

Cisco - Evolution of Network Management Technologies 801

Cisco - Network Management within Campus Networks

Best Practices

Executive Compensation Best Practices (Wiley Best Practices)

Accounting Control Best Practices (Wiley Best Practices)

Cisco Network Security

Executive Compensation Best Practices (Wiley Best Practices)

Cisco Network Professional's

APM Best Practices: Realizing Application Performance Management

Building a Cisco Network for Windows 2000

Building a CISCO Network for Windows 2000

Corporate Management, Governance, and Ethics Best Practices

Best Practices in Business Technology Management

Event Management And Best Practices (IBM Redbooks)

Irrigation Management Transfer: Strategies and Best Practices

Building a Cisco Network for WIndows 2000

Benchmarking Best Practices in Maintenance Management

Project Management: Best Practices: Achieving Global Excellence

Building a Cisco Network for WIndows 2000

Cisco Network Essentials for Educational Institutions

Software Metrics: Best Practices for Successful IT Management

WebSphere eXtreme Scale Best Practices for Operation and Management

Recommend Documents