High Availability in the Enterprise Campus

Abstract

In today's integrated networks where IP data and IP telephony share the same physical path, it is more critical than ever to have a highly available network. In the case of IP telephony, everyone expects to pick up the phone, get a dial tone and, after dialing a specific number, connect with the called party. A lack of high availability in the network design could interfere with that requirement. Simply put, high availability takes planning, monitoring, and expense. This white paper addresses some of the recommendations and considerations for high availability in the enterprise campus.

Sample

Physical Availability

From the access layer, there are two separate layer 2 connections from each access switch, terminating on two different distribution switches. There are then two separate routed connections from each distribution switch terminating on two core switches. From each core device, there are two routed connections; either toward the server farm or toward the Enterprise Edge.

Please note that this model shows each access switch as a single point of failure. Depending on the business model that is being supported, this may or may not be acceptable.

Access Layer Availability

For higher availability and reliability, load sharing should be enabled from the access switches to the two terminating distribution switches. This can be done at layer 2 or layer 3 (depending on the type of switches used at the access layer).

Assuming layer 2 switches, load sharing would be accomplished via Spanning Tree Protocol (SPT) and VLANs. With PVRST+ (Per VLAN Rapid Spanning Tree) each VLAN has its own instance of SPT, thereby making loadsharing possible. For example: Distribution switch A (DSWA), in the diagram above, would be configured as the Root for the even VLANs and Distribution switch B (DSWB), in the diagram above, would be configured as the Root for the odd VLANs.

This is a good beginning, but a failover plan is needed. For example: The question that should be asked is which switch will assume the role of Root Switch if DSWA becomes disabled? Additional configuration would be necessary to make sure that DSWB is the backup, or secondary, root for the odd VLANs and DSWA should be the secondary root for the even VLANs. Now there is load sharing between the VLANs and there is a failover plan should one of the distribution switches become inoperative.

Other Access layer high availability considerations are

When using STP for reconvergence, use Rapid PVST+
Prune unused VLANs from trunks
Use STP transparent mode
Turn trunks on/on with no negotiate to avoid Dynamic Trunking Protocol (DPT) protocol negotiation
When configuring Port Aggregation Protocol (PAgP) for EtherChannel, use the on/on setting to avoid PAgP negotiation when restoring a node or link
Consider using multilayer switches at the access layer for faster convergence

Distribution Layer Availability

At the Distribution Layer, there should be equal cost paths to the server farm module and out to the enterprise edge. This allows for fast failover at layer 3.

If there are two equal cost paths to all destinations and one is lost, there is no reconvergence time as the routing table already has the second path and has been sending packets over it due to equal cost load balancing.

This becomes a design question rather than a question of reconvergence time. Has enough bandwidth been provisioned in the second link to support the additional traffic load if it has to carry the entire load?

The recommendation is to design using redundant triangles and not squares; that way reconvergence of routing protocols is not required.

Summarization of routes from the Distribution layer to the Core layer is desirable. This prevents reconvergence from having to occur in remote distribution blocks when a failure occurs and creates smaller routing tables in the core. Consequently this creates smaller routing tables in other distribution switches peered with the core switches.

When configuring Hot Standby Router Protocol (HSRP), Gateway Load Balancing Protocol (GLBP) or Virtual Router Redundancy Protocol (VRRP), make sure that the active router is consistent with the SPT root. In other words, make sure that if DSWA was the root for the red VLAN, that DSWA is the Active router for HSRP.

If VLANs are local to the switches (VLANs should NOT span multiple switches), the connection between the distribution switches should be a layer 3 connection. This ensures faster reconvergence when there is a link failure from the access layer switch to the distribution layer switch because there won't be a spanning tree port that needs to go from blocking to forwarding.

Other Distribution Layer High Availability Considerations include:

If summarization is configured on the distribution switches, they must be linked or routing black holes occur.
If VLANs span multiple switches, the connection between the distribution switches must be layer 2.
Use HSRP/GLBP milliseconds timers.
Configure HSRP/GLBP preempt delay to avoid black holes.
Tune Etherchannel and CEF (Cisco Express Forwarding) load balancing to ensure optimum link utilization on equal cost links.
Consider fine tuning OSPF timers to enable millisecond convergence.
Use UDLD (Unidirectional Link Detection) between access layers switches and distribution switches. Also between distribution switches, and distribution switches and core switches.