Stepsnin

6800 Series 10 Gigabit and Gigabit Ethernet Interface Modules for Cisco 6500 Series Switches Product Overview The Cisco Catalyst ® 6500 Series Switches offer a variety of 10 Gigabit and Gigabit Ethernet modules which work in conjunction with the new Catalyst® 6500 Supervisor Engine 2T/2TXL (VS-S2T-10G & VS-S2T-10GXL) to serve.

Contents

Cisco Catalyst 6500 Series Virtual Switching System 1440: An Overview The Cisco Catalyst 6500 Series Virtual Switching System (VSS) 1440 allows for the merging of two physical Cisco Catalyst 6500 Series Switches together into a single, logically managed entity. Figure 1 graphically represents this. His immaturity was a topic of discussion in his cisco 6500 supervisor engine slot early days. How cisco 6500 supervisor engine slot evil will I become? One of the most beloved pastimes these days is poker. You will locate Roulette, slot machines, vingt-et-un, Baccarat, Craps, Keno, Pai Gow, many styles of Poker games, and much more.

Introduction

The Cisco Catalyst 6500 series switch fabric modules (SFM), including the Switch Fabric Module 2 (WS-X6500-SFM2) and the Switch Fabric Module (WS-C6500-SFM), in combination with the Supervisor Engine 2, deliver an increase in available system bandwidth from the existing 32 Gbps to 256 Gbps. SFM is not supported on Supervisor Engine I-based systems. The Switch Fabric Module 2 and the Switch Fabric Module enable an architecture that allows 30 million packets per second (Mpps) of Cisco Express Forwarding-based central forwarding performance on Supervisor Engine 2 and up to 210 Mpps of distributed forwarding performance. The Distributed Feature Daughter Card (WS-F6K-DFC) is required to be installed on the line cards to deliver up to 210 Mpps of distributed forwarding.

This document describes the different modes of operation of the SFM, the types of fabric-enabled modules, and frequently asked questions concerning the SFM.

Prerequisites

Requirements

There are no specific prerequisites for this document.

Components Used

The information in this document is based on these software and hardware versions:

• Switch Fabric Module WS-C6500-SFM

• Switch Fabric Module WS-C6500-SFM2

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

Refer to Cisco Technical Tips Conventions for more information on document conventions.

Background Information

There are two types of SFMs available for Supervisor Engine 2.

The WS-C6500-SFM can only work in a Catalyst 6506, Catalyst 6509, Cisco 7606, and Cisco 7609 chassis and is inserted in either slot 5 or slot 6. The WS-C6500-SFM is not supported on a Catalyst 6513.

The WS-C6500-SFM2 can work in a Catalyst 6506, Catalyst 6509, Catalyst 6513, Cisco 7606, and Cisco 7609 chassis. On a Catalyst 6506, Cisco 7609, Cisco 7609, or Catalyst 6509 chassis, the WS-C6500-SFM2 is inserted into slot 5 or slot 6. On a Catalyst 6513, the WS-C6500-SFM2 is inserted into slot 7 or slot 8.

Switching fabric redundancy is supported on both the WS-C6500-SFM and WS-C6500-SFM2. If two SFMs are inserted into the chassis, the SFM in the higher slot number acts as a redundant SFM. Only one SFM can be active at any one time. If the active SFM fails, the standby SFM becomes the active SFM. For redundancy, the two SFMs must have the same part number.

This table summarizes the minimum code requirements and supported chassis configuration for the SFM:

Difference Between CatOS and Cisco IOS System Software

CatOS on the Supervisor Engine and Cisco IOS Software on the MSFC (Hybrid): a CatOS image can be used as the system software to run the Supervisor Engine on Catalyst 6500/6000 switches. If the optional Multilayer Switch Feature Card (MSFC) is installed, a separate Cisco IOS Software image is used to run the MSFC.

Cisco IOS Software on both the Supervisor Engine and MSFC (Native): a single Cisco IOS Software image can be used as the system software to run both the Supervisor Engine and MSFC on Catalyst 6500/6000 switches.

Note: For more information, refer to Comparison of the Cisco Catalyst and Cisco IOS Operating Systems for the Cisco Catalyst 6500 Series Switch.

Switch Fabric Architecture

The example in this section illustrates a logical diagram of a Catalyst 6509. The diagram shows the interconnections between a Supervisor Engine in slot 1, a nonswitch fabric-enabled module in slot 2, one fabric channel switch fabric-enabled module (for example, WS-X6516=) in slot 3, a dual fabric channel switch fabric-enabled module (for example, WS-X6816=) in slot 4, and a SFM in slot 5.

The switch fabric is comprised of the SFM and connecting components located on the Catalyst 6500 chassis. The dual fabric channel switch fabric-enabled module has connections to the switch fabric only.

Switch fabric-enabled modules with one fabric channel have one connection to the Data BUS and one connection to the switch fabric.

The Supervisor Engine, nonfabric-enabled module, and switch fabric-enabled module with one fabric channel have a connection to the Data BUS.

The Data BUS has a lower data forwarding capacity (32 Gbps) than the switch fabric (256 Gbps), and all data going to and from the nonfabric-enabled modules must traverse the Data BUS.

Switch Fabric Module Modes of Operation

The SFM creates a dedicated channel between the fabric-enabled module and the SFM, and provides uninterrupted transmission of frames between these modules.

This table is a summary of the different flows:

With the SFM, the traffic is forwarded to and from the modules in the modes described in this section. The mode of operation determines the flow of data through the switch.

BUS-Only or Flow-Through Mode

A Catalyst 6500 with a SFM and nonfabric-enabled modules like the WS-X6348-RJ-45 or WS-X6416-GBIC work in flow-through mode. In flow-through mode, data flowing between nonfabric-enabled modules do not use the SFM, but the 32 Gbps Data BUS. Data flowing between a Supervisor Engine/MSFC and a nonfabric-enabled module also traverse the Data BUS and do not use the SFM. Data flowing between a module with no connections to the Data BUS, like the WS-X6816-GBIC, and a nonfabric-enabled module traverse from the switch fabric-enabled module through the SFM, then to the Supervisor Engine, and then to the nonfabric module.

Truncated Mode

When the switch contains a fabric-enabled module and a nonfabric-enabled module, the fabric-enabled line cards operate in truncated mode. In this mode, the traffic between the fabric-capable module and the nonfabric modules goes through the switch fabric channel and the Data BUS through the Supervisor Engine. In the case of traffic between fabric-enabled modules, only the truncated data (the first 64 bytes of the frame) are sent over the switch fabric channel. In the case of traffic between two nonfabric-enabled modules, it acts like the flow-through mode.

You can manually specify which switching mode the system uses to improve performance by analyzing the data packets. The default mode should work well, unless another mode is needed for specific reasons. If you have nonfabric-enabled and fabric-enabled modules in the chassis, and most of the traffic is between the fabric-enabled and nonfabric-enabled module, then there is greater advantage in using BUS-mode than truncated mode. With most of the packets flowing between fabric-enabled modules, especially the jumbo-size frames, truncated mode is preferred. You can configure the threshold for the truncated mode with this command in the Cisco IOS Software:

fabric switching-mode allow {bus-mode | {truncated [{threshold [number]}]}

In this command, the threshold number is the number of the fabric-enabled line card before the truncated mode is enabled.

Compact Mode

When a chassis contains only switch fabric-enabled modules, the fabric-enabled line cards can run in compact mode. This delivers the best possible switching rate in CatOS, depending on the line cards.

Distributed Cisco Express Forwarding Mode

This mode is only available in the Cisco IOS Software with the fabric-enabled line cards that have a DFC. This delivers the best possible Layer 3 switching rate in Cisco IOS Software.

Summary

The overall data forwarding capacity of the switch increases as more traffic uses SFM than the Data BUS. BUS-only or flow-through mode has the lowest data forwarding capacity, and compact mode has the highest data forwarding capacity when using CatOS. The dCEF mode has the highest forwarding capacity on a Catalyst 6500 using Cisco IOS Software.

In CatOS, it is possible to disable the switch capability to revert to BUS-only mode when the SFM fails using the set system cross-fallback command. If the switch reverts to BUS-only mode, modules that have a connection to the BUS continue to function while modules with no connection to the BUS are powered down by the Supervisor Engine.

The mode of operation is automatically set by the Supervisor Engine, but can be configured if needed.

In Cisco IOS Software Release 12.1.11E and later, you can configure the switching mode by using this command:

• [no] fabric switching-mode allow {bus-mode | {truncated [{threshold [number]}]}

The no fabric switching-mode allow bus-mode command removes the power to all the nonfabric modules.

If you specify truncated mode, the switch operates in the truncated mode if it has even one fabric-enabled module present in the chassis with other nonfabric modules.

In truncated mode, you can also specify the number of fabric-enabled modules that need to be present in the chassis to change to truncated mode with the threshold command. The default is two. If the threshold is not met, the mode falls back to the original mode.

The show fabric switching-mode command is used to verify the mode of operation, as shown here:

A similar command exists in CatOS, but you cannot specify the threshold value with the truncated mode at this point.

• set system switchmode allow {truncated | bus-only}

The reason to have these thresholds is to improve performance. In truncated mode, the traffic from fabric-capable to nonfabric-capable module needs to travel Fabric and Data BUS, which impacts the overall performance. When BUS-only and fabric-capable cards are mixed in the same chassis, you may want to weigh their traffic patterns and see if there is any benefit in using the truncated mode. The default mode should serve best, but overall performance may be better in truncated mode if there is a lot of traffic with big frame sizes (or jumbos) between a Supervisor Engine and a single fabric-capable card (or between ports on the same fabric-capable card).

The show fabric channel switchmode command is used to verify the operation mode, as shown here:

Switching Fabric Redundancy

Data BUS Fallback Redundancy

The first generation of the fabric-enabled line cards (for example, WS-X6516-GBIC) provides a connection to both the switching fabric as well as the existing system BUS. This allows the Catalyst 6500 system to use the switching fabric as the primary means of data transfer for fabric-enabled line cards. If the switch fabric fails, the system BUS backplane takes over to ensure that packet switching continues, although at 15 Mpps, and the switch remains on line.

Note: This change in switching performance is applicable only if the system is initially forwarding at greater than 15 Mpps. If a system is running at 15 Mpps, the fabric-to-system BUS failover does not affect performance. Active fabric-to-standby fabric and active fabric-to-32-Gbps backplane failovers recover to normal operation in under three seconds.

Switch Fabric Module Redundancy

Additionally, the Catalyst 6500 series can be configured with dual SFMs (for example, in slots 5 and 6), which provide another level of fabric redundancy. In this configuration, a failure on the primary fabric module would result in a switchover to the secondary fabric module for continued operation at 30 Mpps.

The active switch fabric module fails over to the secondary switch fabric in this scenario:

1. The active SFM failed, is disabled, or is removed from the chassis.

2. All the fabric-enabled modules at the time of boot synchronize the channel with the standby and then the active (given when both SFMs are present). If any of the SFM module fails to synchronize, that SFM module is disabled.

3. If the fabric-enabled module or the SFM experiences any kind of error, loss of synchronization, cyclic redundancy check (CRC) error, heartbeat timeout, or other problem and exceeds the threshold value, the module reports this to the Supervisor Engine. The Supervisor Engine starts initiating the recovery process by resetting the channel. If the synchronization failed with the active, but is successful with the standby, the active is disabled. It it failed with the active, the module is disabled.

4. If the Supervisor Engine itself or the SFM experiences the same kinds of errors, such as CRC or heartbeat loss on the channel, and exceeds the threshold value, the Supervisor Engine tries to synchronize with the standby. If successful, it disables the active. If unsuccessful, both SFMs are disabled, and it operates without the SFMs.

   Note: Second-generation switch fabric-enabled modules work only in the presence of a SFM. If there are no SFMs in a chassis with second-generation switch fabric-enabled modules, the modules do not function.

Types of Fabric-Enabled Modules

Connection to Both the Data BUS and Switch Fabric

These modules have a single serial channel to the switch fabric and a connection to the Data BUS. These modules can function in a chassis, with or without a SFM:

• WS-X6K-S2-MSFC2 and WS-X6K-S2-PFC2

• WS-X6516-GBIC

• WS-X6502-C10GE

• WS-X6548-RJ-45

• WS-X6548-RJ-21

• WS-X6516-GE-TX

• WS-X6524-MT-RJ

Connection to the Switch Fabric Only

This module has dual serial channels to the switch fabric and does not have a connection to the Data BUS. Without an operational SFM in the chassis, the module does not function:

• WS-X6816-GBIC

Frequently Asked Questions

Q1: The switch is producing the error message 'Invalid Feature index set for module X' when the SFM is inserted.

This message is produced by a switch running CatOS. It means that the code running on the switch does not support the installed SFM. The minimum code requirement for the WS-C6500-SFM is 6.1(1d), and the minimum code requirement for the WS-C6500-SFM2 is 6.2(2).

Q2: Do nonswitch fabric-enabled line cards work with a SFM in the chassis?

Nonswitch fabric-enabled cards do not utilize the switch fabric, but they do work and use the switching BUS for data forwarding. In this case, the SFM operates in either truncated or BUS-only mode, depending on the presence of other fabric-enabled modules.

Q3: Does the Supervisor Engine 1 (WS-X6K-SUP1-2GE) or Supervisor Engine 1A (WS-X6K-SUP1A-2GE) support a SFM?

The SFM only works with a Supervisor Engine 2. The SFM cannot function in a chassis with a Supervisor Engine 1 or 1A.

Q4: Does the SFM module work in a Catalyst 6000 chassis?

The SFM only functions in a Catalyst 6500 chassis. The Catalyst 6000 chassis does not have the hardware support for the switch fabric.

Q5: What is the difference between the WS-C6500-SFM and WS-C6500-SFM2?

The WS-C6500-SFM can only support up to eight fabric-enabled modules. For this reason, the WS-C6500-SFM can only function in a 6-slot or 9-slot 6500 chassis. The WS-C6500-SFM2 can support 11 fabric modules and functions in the 6-slot, 9-slot and 13-slot chassis.

Note: All line card slots in a 6-slot or 9-slot chassis have dual fabric channels. A 13-slot chassis, on the other hand, only has dual fabric channels in slots 9 through 13. Make sure to insert dual fabric-enabled modules into the correct dual fabric slots for each chassis.

Q6: What are the differences between a fabric-capable module and a fabric-only module?

This table provides a list of some of the differences between a fabric-capable module and a fabric-only module:

Q7: Does a SFM-capable module require a DFC daughter card to use the switch fabric?

A DFC allows a module to support dCEF. The dCEF is the ability for a module to make routing decisions independent of the Supervisor Engine or MSFC2. Similar to the Cisco 7500 Versatile Interface Processor (VIP), the DFC works by replicating Layer 2 (L2) and Layer 3 (L3) forwarding logic from the Supervisor Engine and MSFC2, thereby allowing the module to make a L2 or L3 forwarding decision locally on the module. The DFC is only supported in Cisco IOS Software. The DFC card is a further enhancement and, in combination with the SFM, can increase the data forwarding capacity to 210 Mpps.

Related Information

The Catalyst 6500 is a modular chassisnetwork switch manufactured by Cisco Systems since 1999, capable of delivering speeds of up to '400 million packets per second'.^[1]

A 6500 comprises a chassis, power supplies, one or two supervisors, line cards and service modules. A chassis can have 3, 4, 6, 9 or 13 slots each (Catalyst model 6503, 6504, 6506, 6509, or 6513, respectively) with the option of one or two modular power supplies. The supervisor engine provides centralised forwarding information and processing; up to two of these cards can be installed in a chassis to provide active/standby or stateful failover. The line cards provide port connectivity and service modules to allow for devices such as firewalls to be integrated within the switch.

• 2Operating systems
• 3Methods of operation
• 4Power supplies
  • 4.1Chassis support
  • 4.2Power redundancy options

Supervisor[edit]

The 6500 Supervisor comprises a Multilayer Switch Feature Card (MSFC) and a Policy Feature Card (PFC). The MSFC runs all software processes, such as routing protocols. The PFC makes forwarding decisions in hardware.

The supervisor has connections to the switching fabric and classic bus, as well as bootflash for the Cisco IOS software.

The latest generation supervisor is the Supervisor 2T.This supervisor was introduced at Cisco Live Las Vegas in July 2011.It provides 80 gigabits per slot on all slots of 6500-E chassis.

Operating systems[edit]

The 6500 currently supports three operating systems: CatOS, Native IOS and Modular IOS.

CatOS[edit]

CatOS is supported for layer 2 (switching) operations only. To be able to perform routing functions (e.g. Layer 3) operations, the switch must be run in hybrid mode. In this case, CatOS runs on the Switch Processor (SP) portion of the Supervisor, and IOS runs on the Route Processor (RP) also known as the MSFC. To make configuration changes, the user must then manually switch between the two environments.

CatOS does have some functionality missing and^[2] is generally considered 'obsolete' compared to running a switch in Native Mode.

Native IOS[edit]

Cisco IOS can be run on both the SP and RP. In this instance, the user is unaware of where a command is being executed on the switch, even though technically two IOS images are loaded—one on each processor. This mode is the default shipping mode for Cisco products and enjoys support of all new features and line cards.

Modular IOS[edit]

Modular IOS is a version of Cisco IOS that employs a modern UNIX-based kernel to overcome some of the limitations of IOS.^[3] Additional to this is the ability to perform patching of processes without rebooting the device and in service upgrades.

Methods of operation[edit]

The 6500 has five major modes of operation: Classic, cef256, dcef256, cef720 and dcef720.

Classic Bus[edit]

The 6500 classic architecture provides 32 Gbit/s centralised forwarding performance.^[4] The design is such that an incoming packet is first queued on the line card and then placed on to the global data bus (dBus) and is copied to all other line cards, including the supervisor. The supervisor then looks up the correct egress port, access lists, policing and any relevant rewrite information on the PFC. This is placed on the result bus (rBus) and sent to all line cards. Those line cards for whom the data is not required terminate processing. The others continue forwarding and apply relevant egress queuing.

The speed of the classic bus is 32gb half duplex (since it is a shared bus) and is the only supported way of connecting a Supervisor 32 engine (or Supervisor 1) to a 6500.

cef256[edit]

This method of forwarding was first introduced with the Supervisor 2 engine. When used in combination with a switch fabric module, each line card has an 8Gbit/s connection to the switch fabric and additionally a connection to the classic bus. In this mode, assuming all line cards have a switch fabric connection, an ingress packet is queued as before and its headers are sent along the dBus to the supervisor. They are looked up in the PFC (including ACLs etc.) and then the result is placed on the rBus. The initial egress line card takes this information and forwards the data to the correct line card along the switch fabric. The main advantage here is that there is a dedicated 8 Gbit/s connection between the line cards. The receiving line card queues the egress packet before sending it from the desired port.

The '256' is derived from a chassis using 2x8gb ports on 8 slots of a 6509 chassis: 16 * 8 = 128, 128 * 2 = 256. The number is doubled because of the switch fabric being 'full duplex'.

dcef256[edit]

dcef256 uses distributed forwarding. These line cards have 2x8gb connections to the switch fabric and no classic bus connection. Only modules that have a DFC (Distributed Forwarding Card) can use dcef.

Unlike the previous examples, the line cards hold a full copy of the supervisor's routing tables locally, as well as its own L2 adjacency table (i.e. MAC addresses). This eliminates the need for any connection to the classic bus or requirement to use the shared resource of the supervisor. In this instance, an ingress packet is queued, but its destination looked up locally. The packet is then sent across the switch fabric, queued in the egress line card before being sent.

cef720[edit]

This mode of operation acts identically to cef256, except with 2x20gb connections to the switch fabric and there is no need for a switch fabric module (this is now integrated into the supervisor). This was first introduced into the Supervisor Engine 720.

Cisco 6500 Supervisor

The '720' is derived from a chassis using 2x20gb ports on 9 slots of a 6509 chassis. 40 * 9 = 360 * 2 = 720. The number is doubled to the switch fabric being 'full duplex'. The reason 9 slots are used for the calculation instead of 8 for the cef256 is that it no longer needs to waste a slot with the switch fabric module.

dcef720[edit]

This mode of operation acts identically to dcef256, except with 2x20gb connections to the switch fabric.

Power supplies[edit]

The 6500 is able to deliver high densities of Power over Ethernet across the chassis. Because of this, power supplies are a key element of configuration.

Chassis support[edit]

The following goes through the various 6500 chassis and their supported power supplies and loads.

6503[edit]

The original chassis permits up to 2800W and uses rear-inserted power supplies different from the others in the series.

6504-E[edit]

This chassis permits up to 5000W (119A @ 42V) of power and, like the 6503, uses rear-inserted power supplies.

6506, 6509, 6506-E and 6509-E[edit]

The original chassis can support up to a maximum of 4000W (90A @ 42V) of power, because of backplane limitations. If a power supply above this is inserted, it will deliver at full power up to this limitation (i.e. a 6000W power supply is supported in these chassis, but will output a maximum of 4000W).

The 6509-NEB-A supports a maximum of 4500W (108A @ 42V).

With the introduction of the 6506-E and 6509-E series chassis, the maximum power supported has been increased to in excess of 14500 W (350A @ 42V).

6513[edit]

This chassis can support a maximum of 8000W (180A @ 42V). However, to obtain this, it must be run in combined mode. Therefore, it is suggested that it be run in redundant mode to obtain a maximum of 6000W (145A @ 42V).

Power redundancy options[edit]

The 6500 supports dual power supplies for redundancy. These may be run in one of two modes: redundant or combined mode.

Redundant mode[edit]

When running in Redundant mode, each power supply provides approximately 50% of its capacity to the chassis. In the event of a failure, the unaffected power supply will then provide 100% of its capacity and an alert will be generated. As there was enough to power the chassis ahead of time, there is no interruption to service in this configuration. This is also the default and recommended way to configure power supplies.

Combined mode[edit]

In combined mode, each power supply provides approximately 83% of its capacity to the chassis. This allows for greater utilisation of the power supplies and potentially increased PoE densities.

In systems that are equipped with two power supplies, if one power supply fails and the other power supply cannot fully power all of the installed modules, system power management will shut down devices in the following order:

• Power over Ethernet (PoE) devices— The system will power down PoE devices in descending order, starting with the highest numbered port on the module in the highest numbered slot.
• Modules—If additional power savings are needed, the system will power down modules in descending order, starting with the highest numbered slot. Slots containing supervisor engines or Switch Fabric Modules are bypassed and are not powered down.

This shut down order is fixed and cannot be changed.

Online Insertion & Removal[edit]

OIR is a feature of the 6500 which allows hot swapping most line cards without first powering down the chassis. The advantage of this is that one may perform an in-service upgrade. However, before attempting this, it is important to understand the process of OIR and how it may still require a reload.

To prevent bus errors, the chassis has three pins in each slot which correspond with the line card. Upon insertion, the longest of these makes first contact and stalls the bus (to avoid corruption). As the line card is pushed in further, the middle pin makes the data connection. Finally, the shortest pin removes the bus stall and allows the chassis to continue operation.

Cisco 6500 Switch Price

However, if any part of this operation is skipped, errors will occur (resulting in a stalled bus and ultimately a chassis reload). Common problems include:

Cisco Switch 6500

• Line cards being inserted incorrectly (and thus making contact with only the stall and data pins and thus not releasing the bus)
• Line cards being inserted too quickly (and thus the stall removal signal is not received)
• Line cards being inserted too slowly (and thus the bus is stalled for too long and forces a reload).

Cisco 6500 Supervisor Engine

See also[edit]

References[edit]

1. ^Cisco Catalyst 6500 Series Supervisor Engine 720
2. ^Comparison of the Cisco Catalyst and Cisco IOS Operating Systems for the Cisco Catalyst 6500 Series Switch
3. ^Cisco Catalyst 6500 Series with Cisco IOS Software Modularity
4. ^Cisco Catalyst 6500 Supervisor Engine 32 Architecture