Data Center Ethernet and the Nexus Switches

Small Computer System Interface (SCSI) hard drives have been the preferred method of hard drive storage for servers in enterprise networks for a long time. The Serial attached SCSI (SAS) standard is an electrical standard in which SCSI hard drives are chained together and identified by a SCSI ID. SCSI is an electrical standard which only allows devices to be connected up to 10 meters (32.8 feet) from the server.

Fibre Channel (FC) is an optical standard that allows SCSI hard drives to be connected at much further distance depending on the type of fibre optical cable and laser used to drive the signal. Small form factor pluggable (SFP) optical transceivers are used in director switches, storage arrays, and director switches to create a storage area network (SAN). SAN is sent over storage switches, while LAN data traffic is sent over data switches.

Fibre Channel over Ethernet (FCoE) is a technology that is aimed to replace the director switches of the SAN infrastructure by allowing servers to transfer both data and storage traffic over 10 Gigabit Ethernet interfaces using server network interface cards (NIC) that have both fibre channel and Ethernet chipsets.

Fibre Channel frames have a maximum payload size of 2112 bytes, but the FCoE encapsulation process results in a 2179 byte frame. Jumbo frames must be enabled on Cisco Nexus switches to accommodate the overhead of the Ethernet and FCoE header and trailers. Ethernet jumbo frames can accommodate frame sizes of up to 9,216 bytes (9K). The transmission of fibre channel traffic over a data center infrastructure represents some challenges that Ethernet was not originally designed around. Fibre channel does not tolerate frame loss. The Ethernet infrastructure that will transmit FCoE traffic must be designed to be lossless, resulting in zero frame loss.

Cisco Nexus switches are designed around data center bridging’s low latency and low loss requirements.  Cisco’s priority flow control (PFC) is an important technology that is based on the IEEE 802.3x link level pause frame, but the technology can now be implemented at the IEEE 802.1p class of service (CoS) level.  IEEE 802.3x pause frames are used to flow control Ethernet devices that are aggressive sending data when there is congestion on an uplink. IEEE 802.3x flow control operates at the link level causing all traffic to be delayed when this condition arises. The PFC implementation allows Cisco to flow control individual class of service (CoS) values similar to the IEEE 802.1Qbb working group. Since PFC handles congestion at the CoS level, FCoE traffic can be mapped to a CoS value that does not receive a PAUSE frame while the lower priority traffic receives a pause frame. PFC will help guarantee a lossless FCoE environment.

Cisco Nexus switches support a pre-standard version of backward congestion notification (BCN) draft IEEE 802.1Qau standard. BCN allows the Cisco Nexus switches to communicate congestion to end stations and server where PFC only allows congestion throttling at the CoS level on a link by link basis.

Data center bridging exchange (DCBX) protocol allows Cisco Nexus switches to dynamically discover the data center capabilities of other Nexus switches by leveraging negotiation parameters of the Link Layer Discovery Protocol (LLDP).

The Nexus data center switches provide a migration path to fibre channel over Ethernet at a competitive cost, while running at a lower power footprint than previous switching solutions.


Dennis Hartmann (CCIE #15651, CCVP, CCSI, CCNP, CCIP, MCSE, VCP) is a Unified Communications consultant, author of Implementing Cisco Unified Communications Manager, Part 1, and a lead instructor at Global Knowledge. Dennis was first exposed to CallManager during the CallManager 2.0 time frame when Cisco acquired Selsius. Dennis has worked for various Fortune 500 companies, including AT&T, Sprint, Merrill Lynch, KPMG, and Cabletron Systems. You can read more blog posts from him on Network World.

In this article

Join the Conversation

1 comment

  1. cisco sfp Reply

    Hey thank for your information there is lot of information to be know for the system administator.