Deployment of the Cisco Nexus FEX technology and analysis by Indeni

Cisco launched the Nexus 2000 Fabric Extender (FEX) series in 2009. The FEX technology is based on the IEEE 802.1BR standard. The Nexus 2000 Series Fabric Extenders behave logically like remote line cards of the parent Cisco Nexus switch. The parent switch can be a Cisco Nexus 5000 Series Switch, Nexus 6000 Series Switch, Nexus 7000 Series Switch or a Cisco Nexus 9000 Series Switch. Though the Nexus parent switch and Nexus 2000 series switch are physically separate, they are managed as a single entity from a software image, configuration, and STP perspective. FEXs are a cost-efficient way to design the access layer and they can offer simplicity to the data center access operations and architecture. Finally, the Nexus 2000 series switches offer a broad range of interface options including 40 Gigabit Ethernet, 10 Gigabit Ethernet, 1 Gigabit Ethernet, 100 MB and Fibre Channel over Ethernet (FCoE).

Here’s a list of the Nexus 2000 Series switch FEX technology topics covered in this article:

  • Cisco Nexus 2k series switch introduction
  • Fabric Extender Features Overview & Guidelines
  • Analysis of the Nexus Fabric Extender technology by Indeni
  • Alerting and proactively monitoring of the FEX by Indeni
  • Summary

Are you a Network Administrator, System Engineer, Software Engineer, Indeni Knowledge Expert (IKEs), Indeni Software Developers or Tech Geek? If yes, then read on! This article is for you!

Cisco Nexus 2k series switches FEX introduction

The Cisco Nexus 2000 platform offers over 10 models starting with a 24-port 1GE (Nexus 2224TP) all the way up to 48-port 1/10GE SFP/SFP+ (Nexus 2300). The following Nexus 2000s Series switches are described below:

Nexus 2k Model

Host interfaces

Host interface types

Fabric interfaces

Fabric Interface speed

Hardware Forwarding Performance

Oversubscription ratio

FCoE support

Cisco Nexus 2224TP

24

100BASE-T/1000BASE-T ports: RJ-45 connectors

2

20 Gbps in each direction

88 Gbps or 65 mpps1.2:1

No

Cisco Nexus 2248TP

48

100BASE-T/1000BASE-T ports: RJ-45 connectors

4

40 Gbps in each direction

176 Gbps or 131 mpps

1.2:1

No

Cisco Nexus 2248TP-E

48

100BASE-T/1000BASE-T ports: RJ-45 connectors

4

40 Gbps in each direction

176 Gbps or 131 mpps

1.2:1

No

Cisco Nexus 2332TQ

32

100BASE-T, 1/10GBASE-T ports: RJ45 connectors

6 40 GE QSFP (24 x 10 GE)

Up to 240 Gbps in each direction

960 Gbps or 1440 mpps

2:1

Yes

Cisco Nexus 2348TQ

48

100BASE-T, 1/10GBASE-T ports: RJ45 connectors

6 40 GE QSFP (24 x 10 GE)

Up to 240 Gbps in each direction

1440 Gbps or 2160 mpps

2:1

Yes

Cisco Nexus 2348TQ-E

48

100BASE-T, 1/10GBASE-T ports: RJ-45 connectors

6 40 GE QSFP (24 x 10 GE)

Up to 240 Gbps in each direction

1440 Gbps or 2160 mpps

2:1

Yes

Cisco Nexus 2232TM-E

32

1/10GBASE-T ports: RJ-45 connectors

8 SFP+

80 Gbps in each direction

800 Gbps or 595 mpps

4:1

Yes

Cisco Nexus 2232TM

32

1/10GBASE-T ports: RJ-45 connectors

Uplink module: 8 SFP+

80 Gbps in each direction

800 Gbps or 595 mpps

4:1

No

Cisco Nexus 2348UPQ

48

1/10 GE ports SFP/SFP+

2 to 6 40 GE QSFP (24 x 10 GE)

Up to 240 Gbps in each direction

1440 Gbps or 2160 mpps

2:1

Yes

Cisco Nexus 2248PQ

48

1/10 GE ports SFP/SFP+

4 QSFP+ (16 x 10 GE)

160 Gbps in each direction

1440 Gbps or 2160 mpps

3:1

Yes

Cisco Nexus 2232PP

32

1/10 GE ports SFP/SFP+

8 SFP+

160 Gbps in each direction

800 Gbps or 595 mpps

4:1

Yes

Cisco Nexus FEX Main Features Overview

Enterprises need to simplify their data center implementation, reduce operational complexity, increase scalability and flexibility, and simplify management. The Cisco Nexus 2k series switches and the fabric extender technology are a highly scalable unified solution that meets all these requirements.

Let’s have a quick overview of the NX-OS FEX technology and Nexus 2k series switches which are widely deployed from small to large scale Data Centers.

✓ Cisco Fabric Extender (FEX) ports are designed to support end host connectivity

✓ Offers a broad range of connectivity options, including 100 Mbps, 1, 10, and 40 Gigabit Ethernet, copper and fiber

✓ Cisco FEX ports do not support connectivity to any Layer 2 switch which generate spanning-tree BPDU packets. A Cisco FEX port set by default the port in err disable status in case that receives a BPDU packet

✓ Deployed with deep shared buffers that can absorb bursts of traffic from the wide variety of workloads

✓ Each parent Nexus switch can support a maximum number of Cisco FEX and this value is based on the model of the parent switch.

✓ Cisco Fabric Extender uplinks toward the parent switch can be configured as port channel connections in order to offer high availability and load sharing.

✓ If the Cisco FEX is single-homed to only one member of the switch pair, it is configured as a standard port channel. However, if the Cisco FEX is dual-homed to both members of the vPC switch pair it is configured as a vPC on the port channel

✓ FEX switches are compact 1RU and built for nonstop operation with redundant hot-swappable power supplies and hot-swappable fan tray with redundant fans

✓ Offer unified ports for flexible LAN and SAN deployment with support for Ethernet, Fibre Channel, and Fiber Channel over Ethernet (FCoE) connectivity

Cisco Nexus & Indeni Lab Environment

This section describes the Cisco Nexus switches, Indeni platform and Software versions deployed at the lab environment in order to setup and test the FEX technology. The Nexus switches are configured with the FEX technology in order to be discovered and analyzed by Indeni. The deployed CISCO Nexus lab physical topology is illustrated in the next diagram.

Cisco Nexus – Indeni Lab Physical Topology

The software versions information of the Network devices and Indeni equipped to the lab can be found in the following table.

Hostname

Model

NX-OS Version/Release

N5k-UP

N5K-C5548UP-SUP

7.0(2)N1(1)

N5k-down

N5K-C5548UP-SUP

7.0(7)N1(1)

FEX-100

N2K-C2248TP-1GE

7.0(7)N1(1)

FEX-101

N2K-C2248TP-1GE

7.0(7)N1(1)

ISR-DC-Lab

cisco ISR4331/K9

Cisco IOS XE Software, Version 03.16.02.S

Indeni

Indeni

Installed Indeni packages:

indeni-collector  6.1.4.65

indeni-ds         6.1.4.65

indeni-server     6.1.4.65

indeni-triton     6.1.4.65

indeni-vigile     6.1.4.65

indeni-walt       6.1.4.65

Nexus Hardware and Software revisions

Finally, a brief description for the used Cisco Nexus switches for deploying the FEX technology and testing the Indeni vPC analysis capabilities is provided below.

Nexus 5548UP: A powerful and widely deployed Nexus switch commonly used as “End of Row” at a Data Center with thirty-two fixed Unified ports 1/10 Gbps fixed SFP+ on the base chassis along with one expansion slot.

Nexus 2248TP-E: A popular Nexus FEX with forty-eight ports 100M/1000BaseT Enhanced Host interfaces (server interfaces) and four 10-Gigabit Ethernet Fabric uplinks.

Cisco Nexus FEX & Indeni Analysis

Indeni currently monitors and analyzes several important metrics related to the NX-OS FEX technology. Indicative metrics are described below:

✓ FEX connectivity status and service degradation metrics
✓ FEX diagnostic metrics which guarantee the optimum operation of the N2k series switch
✓ FEX environmental metrics

The “show fex” and “show fex detail” NX-OS command are executed at the Nexus 5k switch deployed at the lab to check the FEX connectivity status, Model, S/N, Uplinks type etc. The output of this NX-OS command proves that the FEX switches toward the N5k switches are operational without any issue (status: online).

N5k-DOWN# show fex
FEX FEX FEX FEX Fex 
Number Description State Model Serial 
------------------------------------------------------------------------
100 FEX0100 Online N2K-C2248TP-1GE FOX1842GGZ3
112 FEX2k - under N5kUP Online N2K-C2248TP-1GE FOX1938GDUY
N5k-DOWN# show fex detail 
FEX: 100 Description: FEX0100 state: Online
FEX version: 7.0(7)N1(1) [Switch version: 7.0(7)N1(1)]
FEX Interim version: 7.0(7)N1(1)
Switch Interim version: 7.0(7)N1(1)
Extender Serial: 1434332423432
Extender Model: N2K-C2248TP-1GE, Part No: 73-13232-02
Card Id: 99, Mac Addr: bc:16:f5:ab:db:61, Num Macs: 64
Module Sw Gen: 12594 [Switch Sw Gen: 21]
post level: complete
pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/31
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Po100 - Interface Up. State: Active
Eth1/31 - Interface Up. State: Active

Let’s continue! We are connecting to the Indeni server and review the discovered Nexus switches deployed at the lab. The dual Nexus 5k switches previously configured with the vPC and FEX technology have been successfully discovered.

We review the Indeni FEX analysis of one of the aforementioned Nexus 5k series switches. As is illustrated in the next capture, it is necessary initially to choose one of the discovered Nexus switch e.g. N5k-down and then to choose the “More Device Info” tab on the right side.

Indeni has completed the Fabrix Extender analysis which does not need in most of the cases more than a few minutes. We review the Indeni analysis by selecting the first tab of the FEX named “FEX Connectivity Status”. It is verified that the FEX connectivity “online” status between the vPC peers as was displayed also by running the relevant NX-OS command.

We are moving to the next tab named “FEX Diagnostic Status” which provides the results of the diagnostic tests of a Fabric Extender chassis collected by Indeni. The provided information includes metrics such as ASIC (application-specific integrated circuit) status, Temperature sensors, Power Supplies and SPROMs (Serial Programmable Read-only Memory) diagnostic tests results. The list of all the NX-OS diagnostic results analyzed by Indeni can be found in the next capture.

We move forward by selecting the next tabs named “FEX Fan status”, “FEX Module Power status”, “FEX Power Supply status”, “FEX Temperature status” which provide real time analysis of the Nexus with the environmental sensor status including FAN, Power-Supply, Power and Temperature status respectively.

Let’s move to the Indeni Lab and see how Indeni responds in case of failure of the vPC connectivity status. The FEX uplink is set down to one of the N5k switches installed at the lab.

N5k-UP# sh fex detail 
FEX: 112 Description: FEX2k - under N5kUP state: Offline
FEX version: 7.0(2)N1(1) [Switch version: 7.0(2)N1(1)]
FEX Interim version: 7.0(2)N1(1)
Switch Interim version: 7.0(2)N1(1)
Extender Serial: FOX1938GPUY
Extender Model: N2K-C2248TP-1GE, Part No: 73-13232-02

Indeni instantly reacts to this change, analyzes the relevant NX-OS outputs in the backend and triggers the next alert.

Let’s see the details of this alert raised by Indeni.

The FEX is in Offline state! The network administrator or NOC is notified for this critical issue and detailed remediation steps are provided to resolve it. Below can be found a list of possible connectivity FEX states where Indeni can analyze, trigger an alert and provide detailed remediation steps.

  • Online – FEX is up and interfaces are online.
  • Offline – FEX is down and interfaces are not available.
  • Online Sequence – FEX is in the process of coming online.
  • Offline Sequence – FEX is in the process of going offline.
  • Discovered – FEX has been detected. Online sequence will begin shortly.
  • FEX AA Upg Fail – Upgrade of Dual Homed FEX failed.
  • AA Version Mismatch – FEX is connected via vPC to two parent switches running different NX-OS versions. vPC can be established but dual home FEX is not possible to work.
  • Fex Type Mismatch – Configured FEX type does not match physical FEX inserted.

The next capture informs the Network Administrator that one of the FEX of the N5k switch has just been installed and is downloading the new NX-OS image from the parent switch.

Finally, Indeni analyzes all of the FEX environmental metrics and provides information for all these metrics with remediation steps as is illustrated in the next capture.

Summary

This article describes the main components and features of the Nexus Fabric Extender (FEX) Nexus technology. A pair of Nexus 5k and N2K-C2248TP-1GE switches are used at a lab environment by Indeni in order to get a dynamic report of the FEX activated features and status. In addition, it is described how to get access to the FEX Indeni Analysis tabs and captures are provided to describe the NX-OS FEX capabilities offered by Indeni via the Live Config tab. It is demonstrated how the Indeni platform continuously analyzes the relevant NX-OS outputs and triggers the relevant alerts including detailed remediation steps on how to check the status of the feature, how to apply it and finally a reference to the Cisco configuration guide or best practices design guide are provided.

This article is part of the Nexus & Indeni series of articles discussing the cool features and powerful capabilities that Indeni offers for the NX-OS FEX technology. More articles about Indeni and the Cisco Nexus technologies are coming soon. Stay tuned ☺

Learn more by joining the Indeni Community. If you liked this article please share it by clicking on one of the social media sharing icons at the top of this page. Thank you to Vasileios Bouloukos for his contributions to this article.

Vasileios Bouloukos
About the author
Vasileios Bouloukos