Cisco Nexus OSFP vPC Implementation & Verification
Cisco Nexus virtual Port Channel (vPC) is a virtualization technology launched in the mid of 2009. The majority of Cisco Nexus Series Switches (Nexus 9000, 7000, 5000 and 3000 Series) supports the Cisco Nexus virtual Port Channel (vPC), which is a virtualization technology launched in the mid of 2009. In brief, the vPC enables the deployment of a link aggregation from a downstream network device to two individual and independent Cisco NX-OS switches (vPC peers).
Last year Cisco introduced the Layer 3 over vPC feature. By using this feature, a Layer 3 device can form dynamic routing peering adjacency between both the vPC peers in a vPC network architecture. Routing adjacency between a Layer 3 device and a vPC peer can be formed now without Layer 3 inter-switch links between the vPC peers but over the vPC peer-link introducing essential advantages.
This article is part of the series of vPC Nexus-Indeni articles. This blog reviews the benefits introduced by this new feature introduced to NX-OS. The OSPF routing protocol over vPC is configured and tested in a lab environment. We will discuss the configurations, implementation and verification steps in detail.
Cisco Nexus Layer 3 routing over vPC
Before the introduction of the Layer 3 routing over vPCA a Network Architect had the following routing options:
- Use separate Layer 3 links to connect Layer 3 devices to the vPC domain.
- Only static routing for a Layer 3 device attached to a vPC domain. The static routes should point to the HSRP address configured on the vPC peer devices.
- When both Layer3 and Layer2 traffic are needed two different links are required. An individual dedicated Layer 3 link is used for routed traffic and a separate Layer 2 link for Layer2 traffic. In case of redundancy, multiple links should be grouped in portchannels increasing the required number of interconnections for the design.
By taking advantage of the new Layer3 Routing over vPC design, a Layer 3 device can form peering adjacency between both the vPC peers in a vPC setup. Routing adjacency is now formed without Layer 3 inter-switch links between the vPC peers but over the vPC peer-link. Take the following prerequisites into consideration:
- The vPC peer switches should have the same number of VLANs
- The peer-gateway feature should be enabled before configuring the Layer 3 over vPC feature. This feature allows the vPC standby peer to forward packets on behalf of the Active peer. So, it is saved bandwidth by avoiding traffic over the peer link.
- Adjacency is formed on the vPC VLAN
- Equal L3 metrics across vPC interfaces
- Requires explicit configuration of the “layer3 peer-router” NX-OS command to the vPC
The embedded vPC loop avoidance mechanism is still available for all traffic, but now the TTL of the traffic sent over a peer link is not decremented. The benefits of Dynamic Routing over vPC feature are summarized below:
- No Static routes limitations
- No extra Parallel links
- No additional design changes
- Route peering across vPC’s over existing infrastructure
- Routing between vPC DCI
- Faster convergence when a link or device fails
The following table summarizes the NX-OS versions and the Routing Protocols Adjacencies supported over vPC VLANs.
Table 1: Nexus Routing over vPC supported NX-OS versions
More details with the supported Network Topologies and NX-OS versions can be found to the next link published by Cisco:
Cisco Nexus Lab Environment
The following section depicts the Cisco Nexus switches and Software versions deployed at the lab environment in order to configure and test the Routing over vPC feature. The deployed CISCO Nexus lab physical topology is illustrated in the next diagram.
Figure 1: Layer 2 Setup for OSPF routing over vPC 10
The hostnames, models and software versions information of the Network devices to the lab can be found to the following table:
Table 2: Nexus Hardware and Software revisions
The following figure provides an easy mnemonic to elaborate the numbering convention of the deployed NX-OS versions.
Figure: 2: NX-OS Release explanation
Here’s a brief description for the used Cisco Nexus switches for deploying and testing the vPC technology:
Nexus 93180YC-EX: The Cisco Nexus 93180YC-EX Switch has 48 1/10/25G-Gbps Small Form Pluggable Plus (SFP+) ports and 6 40/100-Gbps Quad SFP+ (QSFP+) uplink ports. All ports are line rate, delivering 3.6 Tbps of throughput in a 1-rack-unit (1RU) form factor.
Nexus 9504: It provides high performance 1, 10, 25, 40, 50 and 100GE GE in a compact 7-rack-unit modular chassis. This versatile modular switch is designed for high-density, end-of-row, and high-performance aggregation-layer deployments in both traditional and Cisco Application Centric Infrastructure (ACI)–enabled data centers.
Implementation and Testing
The pair of Nexus 93180YC-EX is configured with vPC and the Layer3 routing over vPC is activated. Then the OSPF routing protocol is activated between the vPC peers 93180YC-EX and the vPC edge device Nexus 9504. The Layer 3 topology is illustrated to the next network diagram.
Figure 3: Layer 3 Setup for OSPF routing over vPC
vPC Configuration and Verification
The vPC peer configuration is applied to the pair of the Nexus 93180YC-EX switches. Details about how to configure and verify the operation of the vPC can be found in the Nexus and Indeni series:
Initially, we validate that both Nexus peer switches run an image that supports the Layer 3 over vPC feature and that the vPC basic setup is operational. Then, to activate the Layer 3 routing over vPC featue the peer-gateway feature is enabled and configured on both the peers.
The layer 3 routing over vPC is enabled by running the NX-OS command “layer3 peer-router” in the vpc domain to both vPC peers.
Note-1: If you configure the “layer3 peer-router” command without enabling the peer-gateway feature, a syslog notification message is received to enable the peer-gateway feature.
Note-2: If you configure the “layer3 peer-router” command only on one of the peers or you disable it on one peer, the operational state of layer 3 peer-router gets disabled.
Here’s the relevant configuration from both vPC peers:
feature vpc vpc domain 1 role priority 100 peer-keepalive destination 169.254.0.2 source 169.254.0.1 vrf vpc-keepalive delay restore 360 peer-gateway layer3 peer-router track 10 auto-recovery ip arp synchronize interface port-channel1 description *** VPC PEER LINK *** switchport mode trunk switchport trunk allowed vlan 1,102 spanning-tree port type network vpc peer-link
Figure: 4: N9k-A vPC Peer-link configuration& Keepalive configuration
feature vpc vpc domain 1 peer-keepalive destination 169.254.0.1 source 169.254.0.2 vrf vpc-keepalive delay restore 360 peer-gateway layer3 peer-router track 10 auto-recovery ip arp synchronize interface port-channel1 description *** VPC PEER LINK *** switchport mode trunk switchport trunk allowed vlan 1,102 spanning-tree port type network vpc peer-link interface port-channel2 description *** VPC KEEPALIVE LINK *** no switchport vrf member vpc-keepalive ip address 169.254.0.1/30
Figure 5: N9k-B vPC Peer-link & Keepalive configuration
The show vpc brief command is executed at the Nexus 93k-A switch deployed at the lab to check the vPC domain ID, the peer-link status, the keepalive message status, whether the configuration consistency is successful, and if a peer link has formed. It also states the status of the vPC Port Channels. The output of this NX-OS command proves that the l3 routing and the peer-gateway features have been activated over the vPC.
Legend: (*) - local vPC is down, forwarding via vPC peer-link vPC domain id : 1 Peer status : peer adjacency formed ok vPC keep-alive status : peer is alive Configuration consistency status : success Per-vlan consistency status : success Type-2 consistency status : success vPC role : primary Number of vPCs configured : 197 Track object : 10 Peer Gateway : Enabled Dual-active excluded VLANs : - Graceful Consistency Check : Enabled Auto-recovery status : Enabled, timer is off.(timeout = 240s) Delay-restore status : Timer is off.(timeout = 360s) Delay-restore SVI status : Timer is off.(timeout = 10s) Operational Layer3 Peer-router : Enabled vPC Peer-link status --------------------------------------------------------------------- id Port Status Active vlans -- ---- ------ ------------------------------------------------- 1 Po1 up 1,102
Figure 6: show vPC output
OSPF Configuration & Verification
The OSPFv2 is configured across the previously configured vPC pair of the Nexus N93k and the N95k. The next table illustrates the IP addressing scheme used for the Routing over the vPC.
Table 3: Nexus Transit L3 VLAN
The OSPFv2 is an IETF link-state protocol for IPv4 networks. It has the following main prerequisites, limitations and guidelines for Nexus switches:
- You have configured at least one interface for IPv4 that can communicate with a remote OSPFv2 neighbor.
- You have installed the Enterprise Services license.
- You have enabled the OSPF feature (see the “Enabling OSPFv2” section).
- You can have up to four instances of OSPFv2 in a VDC.
- Cisco NX-OS displays areas in dotted decimal notation regardless of whether you enter the area in decimal or dotted decimal notation.
- BFD with OSPF over vPC peer links is not currently supported.
The first step in configuring OSPFv2 after enabling the OSPF NX-OS feature, is to create an OSPFv2 instance. Assign a unique instance tag for this OSPFv2 instance. The instance tag can also be any string.
The OSPFv2 router identifier is configured by assigning it IP address of the loopback. This IP address identifies the OSPFv2 instance and must exist on a configured interface in the system. Below are the relevant OSPF configurations for all the Nexus switches:
feature ospf router ospf 1 router-id 10.138.255.1 passive-interface default interface Vlan102 description *** OSPF Peering with N95k *** no shutdown ip address 10.138.254.3/29 no ip ospf passive-interface ip router ospf 1 area 0.0.0.0 interface loopback0 ip address 10.138.255.1/32
Figure 7: OSPF neighbor configuration to N93k-A
feature ospf router ospf 1 router-id 10.138.255.2 passive-interface default interface loopback0 ip address 10.138.255.2/32
Figure: 8: OSPF neighbor configuration to N93k-B
feature ospf router ospf 1 router-id 10.0.254.1 passive-interface default interface Vlan102 description *** OSPF Peering with N93k over vPC *** no shutdown ip address 10.138.254.1/29 no ip ospf passive-interface ip router ospf 1 area 0.0.0.0 interface loopback0 ip address 10.0.254.1/32
Figure 9: OSPF neighbor configuration to N95k
The OSPF configuration passive-interface default command is used to suppress routing updates on all interfaces except of the interfaces that are manually configured. Finally, all networks have been configured to the default backbone area (Area 0). The Area ID is a 32-bit value that you can enter as a number or in dotted decimal notation, such as 10.10.10.1. Cisco NX-OS always displays the area in dotted decimal notation.
sh ip ospf neighbors OSPF Process ID 1 VRF default Total number of neighbors: 2 Neighbor ID Pri State Up Time Address Interface 10.0.254.1 1 FULL/DROTHER 1w0d 10.138.254.1 Vlan102 10.138.255.2 1 FULL/BDR 1w0d 10.138.254.4 Vlan102
Figure 10: OSPF neighbors from N93k-A
sh ip ospf neighbors OSPF Process ID 1 VRF default Total number of neighbors: 2 Neighbor ID Pri State Up Time Address Interface 10.0.254.1 1 FULL/DROTHER 1w0d 10.138.254.1 Vlan102 10.138.255.1 1 FULL/DR 1w0d 10.138.254.3 Vlan102
Figure 11: OSPF neighbors from N93k-B
sh ip ospf ne OSPF Process ID 1 VRF default Total number of neighbors: 3 Neighbor ID Pri State Up Time Address Interface 10.138.255.1 1 FULL/DR 1w0d 10.138.254.3 Vlan102 10.138.255.2 1 FULL/BDR 1w0d 10.138.254.4 Vlan102
Figure 12: OSPF neighbors from N95k
The OSPF area network type is set to broadcast. OSPFv2 As is illustrated above the Nexus switches establish a DR and BDR that controls LSA flooding on the network. OSPFv2 uses the well-known IPv4 multicast addresses 18.104.22.168 and a MAC address of 0100.5300.0005 to communicate with neighbors. All other nexus switches establish adjacency with the DR and the BDR and use the IPv4 multicast address 22.214.171.124 to send LSA updates to the DR and BDR.
Finally, the routing table below illustrates the received OSPF routes of the N95k Nexus switch. The ECMP (equal routes) works as expected and each route has two next hops toward to each of the Nexus N93k vPC peer switches.
sh ip route ospf-1 IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%<string>' in via output denotes VRF <string> 10.138.0.0/24, ubest/mbest: 2/0 *via 10.138.254.3, Vlan102, [110/80], 6d13h, ospf-1, intra *via 10.138.254.4, Vlan102, [110/80], 6d13h, ospf-1, intra 10.138.1.0/24, ubest/mbest: 2/0 *via 10.138.254.3, Vlan102, [110/80], 6d13h, ospf-1, intra *via 10.138.254.4, Vlan102, [110/80], 6d13h, ospf-1, intra
Figure 13: OSPF routes from N95k
The routing table from a Nexus vPC peer switch N93k switch is below is where all the OSPF routes learnt via the N95k have a single next hop which is the IP address of the N95k.
sh ip route ospf-1 IP Route Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%<string>' in via output denotes VRF <string> 10.0.254.1/32, ubest/mbest: 1/0 *via 10.138.254.1, Vlan102, [110/20], 1w0d, ospf-1, type-2 10.139.1.0/24, ubest/mbest: 1/0 *via 10.138.254.1, Vlan102, [110/83], 1w0d, ospf-1, inter 10.139.17.0/24, ubest/mbest: 1/0 *via 10.138.254.1, Vlan102, [110/83], 1w0d, ospf-1, inter
Figure 14: OSPF routes from N93k switches
Cisco Nexus and Indeni
Indeni currently analyzes several important metric related to the OSPF routing over vPC feature like these described below:
- vPC role configuration/operation status
- vPC type 2 consistency status
- vPC advanced features status such as peer-gateway, vPC graceful consistency check, auto-recovery
- vPC peer link status or any of the members link vPC status
- vPC peer keepalive status
- OSPF neighbor status
This article describes the Layer 3 Routing feature recently introduced to Cisco NX-OS. The main advantages of this feature are described before be tested at the Lab. The steps to configure OSPF over vPC are described and implemented to the Nexus Lab environment. Testing and verification commands proves the fully operation of this feature. Finally, it is mentioned how the Indeni analyzes and get insight of the Routing over vPC feature.
This article is part of a Series of articles which highlight cool NX-OS features and powerful analysis capabilities that Indeni offers for the Nexus switches. More articles for Cisco Nexus and Indeni are coming soon. Stay tuned ☺
Learn more by joining the Indeni Community. If you liked this article please share it by clicking on one of the social media sharing icons at the top of this page. Thank you to Vasileios Bouloukos for his contributions to this article.