Topology
![clip_image002[4]](https://i1.wp.com/www.blog.madnetworks.fr/wp-content/uploads/2013/03/clip_image0024.jpg "clip_image002[4]")
Hardware Review
Nexus 7K
First let’s review the hardware used for this lab :
N7K-1-1# show module
Mod Ports Module-Type Model Status
— —– ———————————– —————— ———-
1 32 1/10 Gbps Ethernet Module N7K-F132XP-15 ok
3 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 ok
5 0 Supervisor module-1X N7K-SUP1 active *
Xbar Ports Module-Type Model Status
— —– ———————————– —————— ———-
1 0 Fabric Module 1 N7K-C7010-FAB-1 ok
2 0 Fabric Module 1 N7K-C7010-FAB-1 ok
3 0 Fabric Module 1 N7K-C7010-FAB-1 ok
So we have F1 card and M1 card, this is a chassis in mixed mode. FabricPath is only supported on F1/F2 cards.
N5K
N5K-p1-1# sh mod
Mod Ports Module-Type Model Status
— —– ——————————– ———————- ————
1 32 O2 32X10GE/Modular Universal Pla N5K-C5548UP-SUP active *
3 0 O2 Non L3 Daughter Card N55-DL2 ok
The Nexus 5500 is a 32 ports 10GE.
LAB
VLAN Configuration
First let’s configure a very simple Layer 2 topology with the VLAN 100 : The VLAN will be allowed on trunks going between Nexus 5500 and Nexus 7000 :
N7K-1-1(config)# vlan 100
N7K-1-1(config-vlan)# exit
N7K-1-1(config)# int e1/1-8
N7K-1-1(config-if-range)# switchport mode trunk
N7K-1-1(config-if-range)# switchport trunk allowed vlan 100
N7K-1-1(config-if-range)# no sh
N5K-p1-1(config)# vlan 100
N5K-p1-1(config-vlan)# int e1/1-8
N5K-p1-1(config-if-range)# switchport mode trunk
N5K-p1-1(config-if-range)# switchport trunk allowed vlan 100
N5K-p1-1(config-if-range)# no sh
N7K-1-1# sh span vlan 100
VLAN0100
Spanning tree enabled protocol rstp
Root ID Priority 32868
Address 0024.98e8.01c2
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32868 (priority 32768 sys-id-ext 100)
Address 0024.98e8.01c2
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
—————- —- — ——— ——– ——————————–
Eth1/1 Desg FWD 2 128.129 P2p
Eth1/2 Desg FWD 2 128.130 P2p
Eth1/3 Desg FWD 2 128.131 P2p
Eth1/4 Desg FWD 2 128.132 P2p
Eth1/5 Desg FWD 2 128.133 P2p
Eth1/6 Desg FWD 2 128.134 P2p
Eth1/7 Desg FWD 2 128.135 P2p
Eth1/8 Desg FWD 2 128.136 P2p
N5K-p1-1# sh span vlan 100
VLAN0100
Spanning tree enabled protocol rstp
Root ID Priority 32868
Address 0024.98e8.01c2
Cost 2
Port 129 (Ethernet1/1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32868 (priority 32768 sys-id-ext 100)
Address 547f.ee22.81fc
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
—————- —- — ——— ——– ——————————–
Eth1/1 Root FWD 2 128.129 P2p
Eth1/2 Altn BLK 2 128.130 P2p
Eth1/3 Altn BLK 2 128.131 P2p
Eth1/4 Altn BLK 2 128.132 P2p
Eth1/7 Desg FWD 2 128.135 P2p
Nothing complicated here !
What we see is that STP kicks in to avoid any layer 2 loops. This result as usual as blocking some ports. Know from a bandwidth and availability point of view it means we lose 3 10GE uplinks and we are prone to RSTP timers.
RSTP is the default mode on Nexus switches.
FabricPatch configuration
Now let’s configure FabricPath. The basic configuration is very simple.
First we need to check the licence, FabricPath doesn’t come for free and the ENHANCED_LAYER2 feature is needed :
N5K-p1-2# sh license usage
Feature Ins Lic Status Expiry Date Comments
Count
——————————————————————————–
FCOE_NPV_PKG No – Unused –
FM_SERVER_PKG No – Unused –
ENTERPRISE_PKG Yes – Unused Never –
FC_FEATURES_PKG Yes – Unused Never –
VMFEX_FEATURE_PKG No – Unused –
ENHANCED_LAYER2_PKG No – Unused Grace 99D 10H
LAN_BASE_SERVICES_PKG No – Unused –
LAN_ENTERPRISE_SERVICES_PKG Yes – Unused Never –
N7K-1-1# sh license usage
Feature Ins Lic Status Expiry Date Comments
Count
——————————————————————————
MPLS_PKG No – Unused –
STORAGE-ENT No – Unused –
ENTERPRISE_PKG No – Unused –
FCOE-N7K-F132XP No 0 Unused –
ENHANCED_LAYER2_PKG Yes – Unused Never –
SCALABLE_SERVICES_PKG No – Unused –
TRANSPORT_SERVICES_PKG Yes – Unused Never –
LAN_ADVANCED_SERVICES_PKG Yes – Unused Never –
LAN_ENTERPRISE_SERVICES_PKG Yes – Unused Never –
——————————————————————————
By default the features comes with a grace period, so if you do not have purchased the right licence you still can run the feature for a limited 120 day period.
On Nexus 7000 we need to activate the feature and then turn the VLAN into FabricPath mode. What this change compared to CE mode (Classical Ethernet) is that we switch on a new MAC Learning paradigm where the Source MAC Address of an incoming frame is learned only if the Destination MAC is already known. This is Conversational MAC Learning.
This is the major feature of FabricPath that enable the scalability of the protocol. From a practical point of view, it means that you only learn MACs from systems that have a bidirectionnal communication and you will not learn the MAC addresses of remote systems that do not talk with system that you host (« you » being the device running FabricPath).
N7K-1-1(config)# feature-set fabricpath
N7K-1-1(config)# vlan 100
N7K-1-1(config-vlan)# mode fabricpath
FabricPath will randomly assign a Switch ID to the device. The Switch ID has the same purpose as the Router ID in OSPF, the goal is to allow ISIS to build and LSDB and uniquely identify the device.
The Switch ID can be statically assigned but in this case FabricPath will lose the capability to automagically resolve the potential conflicts.
N7K-1-1(config)# show fabricpath switch-id
FABRICPATH SWITCH-ID TABLE
Legend: ‘*’ – this system
=========================================================================
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED
———-+—————-+————+———–+——————–
*1398 0024.98e8.01c2 Primary Confirmed No No
Total Switch-ids: 1
Next step is to enable the interfaces for the FabricPath mode. This will identify which interfaces needs to encapsulates/decapsulates trafic.
FabricPath is NOT Ethernet, these interfaces will NOT be able to switch standard Ethernet Frame after the switch in FabricPath mode.
N7K-1-1(config)# int e1/1-8
N7K-1-1(config-if-range)# switchport mode fabricpath
2013 Mar 2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/8 up in fabricpath topology 0
2013 Mar 2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/8 is up in mode fabricpath
2013 Mar 2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/7 is up in mode fabricpath
2013 Mar 2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/7 up in fabricpath topology 0
2013 Mar 2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/6 is up in mode fabricpath
2013 Mar 2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/5 is up in mode fabricpath
2013 Mar 2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/4 is up in mode fabricpath
2013 Mar 2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/3 is up in mode fabricpath
2013 Mar 2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/6 up in fabricpath topology 0
2013 Mar 2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/2 is up in mode fabricpath
2013 Mar 2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/5 up in fabricpath topology 0
2013 Mar 2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/4 up in fabricpath topology 0
2013 Mar 2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/3 up in fabricpath topology 0
2013 Mar 2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/2 up in fabricpath topology 0
2013 Mar 2 17:31:08 N7K-1-1 %ETHPORT-5-IF_UP: Interface Ethernet1/1 is up in mode fabricpath
2013 Mar 2 17:31:08 N7K-1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet1/1 up in fabricpath topology 0
Now, STP has vanished for VLAN 100 because FabricPath is activated. This is another key point, unlike VPC STP does not run on the top of FabricPath.
The looped design is managed by the new mechanisms of FabricPath, the Conversational Learning, the fact that FabricPath knows the entire topology and the TTL value that is part of the FabricPath header.
N7K-1-1(config-if-range)# do sh span vlan 100
Spanning tree instance(s) for vlan does not exist.
Same operations on Nexus 5500, the only slight difference is that we need to install the feature before activating it :
N5K-p1-2# sh span vlan 100
VLAN0100
Spanning tree enabled protocol rstp
Root ID Priority 32868
Address 0005.73ba.637c
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32868 (priority 32768 sys-id-ext 100)
Address 0005.73ba.637c
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
—————- —- — ——— ——– ——————————–
Eth1/1 Desg FWD 2 128.129 P2p
Eth1/3 Desg FWD 2 128.131 P2p
Eth1/4 Desg FWD 2 128.132 P2p
Eth1/5 Desg FWD 2 128.133 P2p
Eth1/6 Desg FWD 2 128.134 P2p
Eth1/7 Desg FWD 2 128.135 P2p
Eth1/8 Desg FWD 2 128.136 P2p
N5K-p1-1(config)# install feature-set fabricpath
N5K-p1-1(config)# feature-set fabricpath
N5K-p1-1(config)# vlan 100
N5K-p1-1(config-vlan)# mode fabricpath
N5K-p1-1(config-vlan)# int e1/1-8
N5K-p1-1(config-if-range)# switchport mode fabricpath
2013 Mar 2 12:04:35 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE: isis_fabricpath-default [3736] P2P adj L1 0024.98e8.01c2 over Ethernet1/3 – DOWN (New) on MT-0
2013 Mar 2 12:04:35 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE: isis_fabricpath-default [3736] P2P adj L1 0024.98e8.01c2 over Ethernet1/3 – UP on MT-0
2013 Mar 2 12:04:36 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE: isis_fabricpath-default [3736] P2P adj L1 0024.98e8.01c2 over Ethernet1/2 – DOWN (New) on MT-0
2013 Mar 2 12:04:36 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE: isis_fabricpath-default [3736] P2P adj L1 0024.98e8.01c2 over Ethernet1/2 – UP on MT-0
2013 Mar 2 12:04:36 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE: isis_fabricpath-default [3736] P2P adj L1 0024.98e8.01c2 over Ethernet1/1 – DOWN (New) on MT-0
2013 Mar 2 12:04:36 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE: isis_fabricpath-default [3736] P2P adj L1 0024.98e8.01c2 over Ethernet1/1 – UP on MT-0
2013 Mar 2 12:04:37 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE: isis_fabricpath-default [3736] P2P adj L1 N7K-1-1 over Ethernet1/4 – DOWN (New) on MT-0
2013 Mar 2 12:04:37 N5K-p1-1 %ISIS_FABRICPATH-5-ADJCHANGE: isis_fabricpath-default [3736] P2P adj L1 N7K-1-1 over Ethernet1/4 – UP on MT-0
Now we can check the new type of the interfaces activated for FabricPath :
N5K-p1-1# sh int brief
——————————————————————————
Ethernet VLAN Type Mode Status Reason Speed Port
Interface
——————————————————————————
Eth1/1 1 eth f-path up none 10G(D) –
Eth1/2 1 eth f-path up none 10G(D) –
Eth1/3 1 eth f-path up none 10G(D) –
Eth1/4 1 eth f-path up none 10G(D) –
Eth1/5 1 eth f-path up none 10G(D) –
Eth1/6 1 eth f-path up none 10G(D) –
Eth1/7 1 eth f-path up none 10G(D) –
Eth1/8 1 eth f-path up none 10G(D) –
Like in OSPF we can check the status of the interface and the various values like the network type, the status, the circuit, the MTU and the Metric.
By default the reference metric is 400000M which means that 10GE interfaces have a metric of 40 (400000M / 10000M). Of course this value can be tuned if necessary.
N7K-1-1# sh fabricpath isis interface brief
Fabricpath IS-IS domain: default
Interface Type Idx State Circuit MTU Metric Priority Adjs/AdjsUp
——————————————————————————–
Ethernet1/1 P2P 4 Up/Ready 0x01/L1 1500 40 64 1/1
Ethernet1/2 P2P 1 Up/Ready 0x01/L1 1500 40 64 1/1
Ethernet1/3 P2P 2 Up/Ready 0x01/L1 1500 40 64 1/1
Ethernet1/4 P2P 3 Up/Ready 0x01/L1 1500 40 64 1/1
Ethernet1/5 P2P 5 Up/Ready 0x01/L1 1500 40 64 0/0
Ethernet1/6 P2P 6 Up/Ready 0x01/L1 1500 40 64 0/0
Ethernet1/7 P2P 7 Up/Ready 0x01/L1 1500 40 64 0/0
Ethernet1/8 P2P 8 Up/Ready 0x01/L1 1500 40 64 0/0
The detail of protocol can be obtained, this is very similar (again) to OSPF
N7K-1-1# show fabricpath isis
Fabricpath IS-IS domain : default
System ID : 0024.98e8.01c2 IS-Type : L1
SAP : 432 Queue Handle : 11
Maximum LSP MTU: 1492
Graceful Restart enabled. State: Inactive
Last graceful restart status : none
Metric-style : advertise(wide), accept(wide)
Start-Mode: Complete [Start-type configuration]
Area address(es) :
00
Process is up and running
CIB ID: 4
Interfaces supported by Fabricpath IS-IS :
Ethernet1/1
Ethernet1/2
Ethernet1/3
Ethernet1/4
Ethernet1/5
Ethernet1/6
Ethernet1/7
Ethernet1/8
Level 1
Authentication type and keychain not configured
Authentication check specified
MT-0 Ref-Bw: 400000
Address family Swid unicast :
Number of interface : 8
Distance : 115
L1 Next SPF: Inactive
The adjacencies can be check with the following command, note that FabricPath build L1 adjacencies only so from an ISIS perspective this is a flat L1 intra-area.
N7K-1-1# show fabricpath isis adjacency
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID SNPA Level State Hold Time Interface
N5K-p1-1 N/A 1 UP 00:00:30 Ethernet1/1
N5K-p1-1 N/A 1 UP 00:00:22 Ethernet1/2
N5K-p1-1 N/A 1 UP 00:00:28 Ethernet1/3
N5K-p1-1 N/A 1 UP 00:00:31 Ethernet1/4
The switch IDs known to participate in the FabricPath domain can be checked. This is because the ISIS LSDB is able to identify all the switch IDs, the entire topology is known:
N7K-1-1# show fabricpath switch-id
FABRICPATH SWITCH-ID TABLE
Legend: ‘*’ – this system
=========================================================================
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED
———-+—————-+————+———–+——————–
*1398 0024.98e8.01c2 Primary Confirmed No No
2857 547f.ee22.81fc Primary Confirmed No No
Total Switch-ids: 2
Tips and Tricks, on Nexus there is a global command that can be used to turn all switchports into FabricPath interfaces.
N7K-1-2(config)# system default switchport fabricpath
FabricPath switch IDs can be manually assigned. There is no disruption of service when changing the switch ID because FabricPath runs some kind of interim switch ID will it ensure that the old one is flushed out the database.
N5K-p1-2(config)# fabricpath switch-id 52
The entire topology can be seen from any devices in the FabricPath domain. What we see is that FabricPath builds two trees.
The first one is built for unknown unicast, broadcast and multicast.
The second one is built for multicast.
Standard known unicast doesn’t use a tree, the frame will be moved based on the FabricPath route.
The first tree is built by electing a root, very much like STP the root is elected based on :
– Highest Root Priority
– Highest System ID
– Highest Switch ID
One the first tree is build, an Ftag is assigned to it.
Then the first elected root elects the second one which will be assigned a second Ftag.
N7K-1-1# show fabricpath isis topology summary
Fabricpath IS-IS domain: default FabricPath IS-IS Topology Summary
MT-0
Configured interfaces: Ethernet1/1 Ethernet1/2 Ethernet1/3 Ethernet1/4 Ethernet1/5 Ethernet1/6 Ethernet1/7 Ethernet1/8
Number of trees: 2
Tree id: 1, ftag: 1, root system: 547f.ee22.81fc, 51
Tree id: 2, ftag: 2, root system: 0024.98e8.01c2, 71
The root election can be done on a predictive manner by changing the root-priority inside the fabricpath domain:
N7K-1-1(config)# fabricpath domain default
N7K-1-1(config-fabricpath-isis)# root-priority ?
<1-255> Root priority value per topology
*Default value is 64
N7K-1-1(config-fabricpath-isis)# root-priority 255
N7K-1-2(config)# fabricpath domain default
N7K-1-2(config-fabricpath-isis)# root-priority ?
<1-255> Root priority value per topology
*Default value is 64
N7K-1-2(config-fabricpath-isis)# root-priority 254
N7K-1-1# sh fabricpath isis topology summary
Fabricpath IS-IS domain: default FabricPath IS-IS Topology Summary
MT-0
Configured interfaces: Ethernet1/1 Ethernet1/2 Ethernet1/3 Ethernet1/4 Ethernet1/5 Ethernet1/6 Ethernet1/7 Ethernet1/8
Number of trees: 2
Tree id: 1, ftag: 1, root system: 0024.98e8.01c2, 71
Tree id: 2, ftag: 2, root system: 001b.54c2.67c2, 72
The entire tree can be seen at any point of the FabricPath domain:
N7K-1-1# show fabricpath isis topology 0 trees
Fabricpath IS-IS domain: default
Note: The metric mentioned for multidestination tree is from the root of that tree to that switch-id
MT-0
Topology 0, Tree 1, Swid routing table
51, L1
via Ethernet1/4, metric 40
52, L1
via Ethernet1/8, metric 40
72, L1
via Ethernet1/8, metric 80
Topology 0, Tree 2, Swid routing table
51, L1
via Ethernet1/4, metric 40
52, L1
via Ethernet1/4, metric 40
72, L1
via Ethernet1/4, metric 0
N5K-p1-1# show fabricpath isis topology 0 trees
Fabricpath IS-IS domain: default
Note: The metric mentioned for multidestination tree is from the root of that tree to that switch-id
MT-0
Topology 0, Tree 1, Swid routing table
52, L1
via Ethernet1/4, metric 40
71, L1
via Ethernet1/4, metric 0
72, L1
via Ethernet1/4, metric 80
Topology 0, Tree 2, Swid routing table
52, L1
via Ethernet1/8, metric 40
71, L1
via Ethernet1/4, metric 80
72, L1
via Ethernet1/8, metric 0
N5K-p1-2# show fabricpath isis topology 0 trees
Fabricpath IS-IS domain: default
Note: The metric mentioned for multidestination tree is from the root of that tree to that switch-id
MT-0
Topology 0, Tree 1, Swid routing table
51, L1
via Ethernet1/8, metric 40
71, L1
via Ethernet1/8, metric 0
72, L1
via Ethernet1/4, metric 80
Topology 0, Tree 2, Swid routing table
51, L1
via Ethernet1/4, metric 40
71, L1
via Ethernet1/4, metric 80
72, L1
via Ethernet1/4, metric 0
N7K-1-2# show fabricpath isis topology 0 trees
Fabricpath IS-IS domain: default
Note: The metric mentioned for multidestination tree is from the root of that tree to that switch-id
MT-0
Topology 0, Tree 1, Swid routing table
51, L1
via Ethernet1/4, metric 40
52, L1
via Ethernet1/4, metric 40
71, L1
via Ethernet1/4, metric 0
Topology 0, Tree 2, Swid routing table
51, L1
via Ethernet1/8, metric 40
52, L1
via Ethernet1/4, metric 40
71, L1
via Ethernet1/8, metric 80
FabricPath allows for ECMP load-balacing, up to 16 paths by default. The way ECMP is done can be modified, the default behavior is to mix L3/L4 information.
N7K-1-1(config)# fabricpath load-balance unicast ?
<CR>
destination Include destination parameters
include-vlan Use vlan
layer3 Only Layer-3 parameters considered
layer4 Only Layer-4 parameters considered
mixed Mix of Layer-3 and Layer-4 paramaters (default)
rotate-amount Rotate amount for hash string
source Include source parameters
source-destination Include source and destination parameters
symmetric Symmetric (default)
xor Include ex-or of source and destination parameters
N7K-1-1# show fabricpath load-balance
ECMP load-balancing configuration:
L3/L4 Preference: Mixed
Hash Control: Symmetric
Rotate amount: 0 bytes
Use VLAN: TRUE
Ftag load-balancing configuration:
Hash Control: Symmetric
Rotate amount: 0 bytes
Use VLAN: TRUE
The entire LSDB of the FabricPath domain can be looked at any point in the domain…
N7K-1-1# show fabricpath isis database
Fabricpath IS-IS domain: default LSP database
LSPID Seq Number Checksum Lifetime A/P/O/T
N5K-p1-2.00-00 0x0000000E 0xA7C6 723 0/0/0/1
N7K-1-2.00-00 0x00000021 0x07BC 1156 0/0/0/1
N7K-1-1.00-00 * 0x00000021 0xA324 1158 0/0/0/1
N5K-p1-1.00-00 0x00000012 0x8C91 1075 0/0/0/1
N7K-1-1# show fabricpath isis database detail
Fabricpath IS-IS domain: default LSP database
LSPID Seq Number Checksum Lifetime A/P/O/T
N5K-p1-2.00-00 0x0000000E 0xA7C6 702 0/0/0/1
Instance : 0x0000000C
Area Address : 00
NLPID : 0xC0
Hostname : N5K-p1-2 Length : 8
Extended IS : N7K-1-1.00 Metric : 40
Extended IS : N7K-1-2.00 Metric : 40
Extended IS : N7K-1-1.00 Metric : 40
Extended IS : N7K-1-2.00 Metric : 40
Extended IS : N7K-1-2.00 Metric : 40
Extended IS : N7K-1-2.00 Metric : 40
Extended IS : N7K-1-1.00 Metric : 40
Extended IS : N7K-1-1.00 Metric : 40
Capability : Device Id: 52 Base Topology
Base Topo Root Pri :
Trees desired: 2 Trees computed: 2 Trees usable 2
Nickname :
Priority: 0 Nickname: 52 BcastPriority: 64
Nickname Migration :
Swid: 52 Sec. Swid: 0
Digest Offset : 0
N7K-1-2.00-00 0x00000021 0x07BC 1135 0/0/0/1
Instance : 0x0000001C
Area Address : 00
NLPID : 0xC0
Hostname : N7K-1-2 Length : 7
Extended IS : N5K-p1-2.00 Metric : 40
Extended IS : N5K-p1-2.00 Metric : 40
Extended IS : N5K-p1-2.00 Metric : 40
Extended IS : N5K-p1-2.00 Metric : 40
Extended IS : N5K-p1-1.00 Metric : 40
Extended IS : N5K-p1-1.00 Metric : 40
Extended IS : N5K-p1-1.00 Metric : 40
Extended IS : N5K-p1-1.00 Metric : 40
Capability : Device Id: 72 Base Topology
Base Topo Root Pri :
Trees desired: 2 Trees computed: 2 Trees usable 2
Nickname :
Priority: 0 Nickname: 72 BcastPriority: 254
Nickname Migration :
Swid: 72 Sec. Swid: 0
Digest Offset : 0
N7K-1-1.00-00 * 0x00000021 0xA324 1137 0/0/0/1
Instance : 0x00000021
Area Address : 00
NLPID : 0xC0
Hostname : N7K-1-1 Length : 7
Extended IS : N5K-p1-2.00 Metric : 40
Extended IS : N5K-p1-2.00 Metric : 40
Extended IS : N5K-p1-2.00 Metric : 40
Extended IS : N5K-p1-2.00 Metric : 40
Extended IS : N5K-p1-1.00 Metric : 40
Extended IS : N5K-p1-1.00 Metric : 40
Extended IS : N5K-p1-1.00 Metric : 40
Extended IS : N5K-p1-1.00 Metric : 40
Capability : Device Id: 71 Base Topology
Base Topo Ftag :
Graph 1: Root: N7K-1-1 Primary: 1, Secondary: 0 Nickname 71
Graph 2: Root: N7K-1-2 Primary: 2, Secondary: 0 Nickname 72
Base Topo Roots :
Graph 1: Root Nickname: 71
Graph 2: Root Nickname: 72
Base Topo Root Pri :
Trees desired: 2 Trees computed: 2 Trees usable 2
Nickname :
Priority: 0 Nickname: 71 BcastPriority: 255
Nickname Migration :
Swid: 71 Sec. Swid: 0
Digest Offset : 0
N5K-p1-1.00-00 0x00000012 0x8C91 1054 0/0/0/1
Instance : 0x00000010
Area Address : 00
NLPID : 0xC0
Hostname : N5K-p1-1 Length : 8
Extended IS : N7K-1-2.00 Metric : 40
Extended IS : N7K-1-2.00 Metric : 40
Extended IS : N7K-1-2.00 Metric : 40
Extended IS : N7K-1-2.00 Metric : 40
Extended IS : N7K-1-1.00 Metric : 40
Extended IS : N7K-1-1.00 Metric : 40
Extended IS : N7K-1-1.00 Metric : 40
Extended IS : N7K-1-1.00 Metric : 40
Capability : Device Id: 51 Base Topology
Base Topo Root Pri :
Trees desired: 2 Trees computed: 2 Trees usable 2
Nickname :
Priority: 0 Nickname: 51 BcastPriority: 64
Nickname Migration :
Swid: 51 Sec. Swid: 0
Digest Offset : 0
vPC+
Let’s see now of vPC can be integrated with FabricPath.
The issue with the design is that the control plane of the vPC Peer is not merge. So from a FabricPath point of view the two peers should have two separate Switch IDs… Which is wrong because vPC is designed to appear like a single switch.
This is the issue that vPC+ resolves by allowing two vPC peers to share a unique and common FabricPath switch ID.
First the definition of the vPC domain and the configuation of the FabricPath Switch ID under the vPC domain :
N5K-p1-1# sh run vpc
!Command: show running-config vpc
!Time: Sat Mar 2 13:51:58 2013
version 5.1(3)N1(1)
feature vpc
vpc domain 1
role priority 1000
fabricpath switch-id 1
N5K-p1-2(config-vpc-domain)# sh run vpc
!Command: show running-config vpc
!Time: Sat Mar 2 13:24:27 2013
version 5.1(3)N1(1)
feature vpc
vpc domain 1
role priority 2000
fabricpath switch-id 1
The Peer Keepalive link can be configured as usual:
N5K-p1-1(config-vpc-domain)# peer-keepalive destination 10.0.8.202 source 10.0.8.201 vrf management
N5K-p1-2(config-vpc-domain)# peer-keepalive destination 10.0.8.201 vrf management source 10.0.8.202
N5K-p1-1(config-vpc-domain)# sh vpc
Legend:
(*) – local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
vPC+ switch id : 1
Peer status : peer link not configured
vPC keep-alive status : peer is alive
vPC fabricpath status : peer not found
Configuration consistency status: failed
Per-vlan consistency status : failed
Configuration consistency reason: vPC peer-link does not exist
Type-2 consistency status : failed
Type-2 consistency reason : vPC peer-link does not exist
vPC role : none established
Number of vPCs configured : 0
Peer Gateway : Disabled
Dual-active excluded VLANs : –
Graceful Consistency Check : Disabled (due to peer configuration)
The peer-link must be defined and must run as a FabricPath switchport:
N5K-p1-2(config-if)# sh run int po1
!Command: show running-config interface port-channel1
!Time: Sat Mar 2 13:32:33 2013
version 5.1(3)N1(1)
interface port-channel1
switchport mode fabricpath
speed 10000
vpc peer-link
N5K-p1-1(config-if)# sh run int po1
!Command: show running-config interface port-channel1
!Time: Sat Mar 2 14:00:28 2013
version 5.1(3)N1(1)
interface port-channel1
switchport mode fabricpath
speed 10000
vpc peer-link
Now vPC is aware that the peer is available through FabricPath:
N5K-p1-1# sh vpc
Legend:
(*) – local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
vPC+ switch id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
vPC fabricpath status : peer is reachable through fabricpath
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 0
Peer Gateway : Disabled
Dual-active excluded VLANs : –
Graceful Consistency Check : Enabled
vPC Peer-link status
———————————————————————
id Port Status Active vlans
— —- —— ————————————————–
1 Po1 up 100
N5K-p1-2(config-if)# sh vpc
Legend:
(*) – local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
vPC+ switch id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
vPC fabricpath status : peer is reachable through fabricpath
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary
Number of vPCs configured : 0
Peer Gateway : Disabled
Dual-active excluded VLANs : –
Graceful Consistency Check : Enabled
vPC Peer-link status
———————————————————————
id Port Status Active vlans
— —- —— ————————————————–
1 Po1 up 100
The ISIS FabricPath adjacency table has been modified to reflect the new design:
N5K-p1-2# show fabricpath isis adjacency
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID SNPA Level State Hold Time Interface
N5K-p1-1 N/A 1 UP 00:00:28 port-channel1
N7K-1-2 N/A 1 UP 00:00:29 Ethernet1/1
N7K-1-2 N/A 1 UP 00:00:27 Ethernet1/2
N7K-1-2 N/A 1 UP 00:00:24 Ethernet1/3
N7K-1-2 N/A 1 UP 00:00:27 Ethernet1/4
N7K-1-1 N/A 1 UP 00:00:33 Ethernet1/5
N7K-1-1 N/A 1 UP 00:00:24 Ethernet1/6
N7K-1-1 N/A 1 UP 00:00:27 Ethernet1/7
N7K-1-1 N/A 1 UP 00:00:22 Ethernet1/8
One very important design requirement of FabricPath is that devices doing the separation between CE and FP must be root of the STP domain and configured with the same priority. Cisco recommends 8192.
N5K-p1-1(config)# spanning-tree vlan 100 priority 8192
N5K-p1-2(config)# spanning-tree vlan 100 priority 8192
Now vPC member port can be configured towards end hosts (or FEXes).
N5K-p1-2# sh run int e1/11
!Command: show running-config interface Ethernet1/11
!Time: Sat Mar 2 13:40:30 2013
version 5.1(3)N1(1)
interface Ethernet1/11
switchport access vlan 100
speed 1000
channel-group 6 mode active
N5K-p1-2# sh run int po6
!Command: show running-config interface port-channel6
!Time: Sat Mar 2 13:40:41 2013
version 5.1(3)N1(1)
interface port-channel6
switchport access vlan 100
speed 1000
vpc 6
N5K-p1-1# sh run int e1/11
!Command: show running-config interface Ethernet1/11
!Time: Sat Mar 2 14:08:38 2013
version 5.1(3)N1(1)
interface Ethernet1/11
switchport access vlan 100
speed 1000
channel-group 6 mode active
N5K-p1-1# sh run int po6
!Command: show running-config interface port-channel6
!Time: Sat Mar 2 14:08:45 2013
version 5.1(3)N1(1)
interface port-channel6
switchport access vlan 100
speed 1000
vpc 6
N5K-p1-1# sh vpc
Legend:
(*) – local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
vPC+ switch id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
vPC fabricpath status : peer is reachable through fabricpath
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : –
Graceful Consistency Check : Enabled
vPC Peer-link status
———————————————————————
id Port Status Active vlans
— —- —— ————————————————–
1 Po1 up 100
vPC status
—————————————————————————
id Port Status Consistency Reason Active vlans vPC+ Attrib
— ———- —— ———– —— ———— ———–
6 Po6 up success success 100 DF: Partial
N5K-p1-2# sh vpc
Legend:
(*) – local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
vPC+ switch id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
vPC fabricpath status : peer is reachable through fabricpath
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : –
Graceful Consistency Check : Enabled
vPC Peer-link status
———————————————————————
id Port Status Active vlans
— —- —— ————————————————–
1 Po1 up 100
vPC status
—————————————————————————
id Port Status Consistency Reason Active vlans vPC+ Attrib
— ———- —— ———– —— ———— ———–
6 Po6 up success success 100 DF: Partial
Note that the two peers are STP root of their STP domain and they share a common Bride ID, c84c.75fa.6000:
N5K-p1-1# sh span vlan 100
VLAN0100
Spanning tree enabled protocol rstp
Root ID Priority 8292
Address c84c.75fa.6000
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 8292 (priority 8192 sys-id-ext 100)
Address c84c.75fa.6000
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
—————- —- — ——— ——– ——————————–
Po6 Desg FWD 1 128.4101 (vPC) P2p
VLAN0100
Spanning tree enabled protocol rstp
Root ID Priority 8292
Address c84c.75fa.6000
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 8292 (priority 8192 sys-id-ext 100)
Address c84c.75fa.6000
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
—————- —- — ——— ——– ——————————–
Po6 Desg FWD 1 128.4101 (vPC) P2p
N5K-p1-1# show fabricpath switch-id
FABRICPATH SWITCH-ID TABLE
Legend: ‘*’ – this system
=========================================================================
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED
———-+—————-+————+———–+——————–
1 547f.ee22.81fc Primary Confirmed No Yes
1 0005.73ba.637c Primary Confirmed No Yes
*51 547f.ee22.81fc Primary Confirmed Yes No
52 0005.73ba.637c Primary Confirmed Yes No
71 0024.98e8.01c2 Primary Confirmed Yes No
72 001b.54c2.67c2 Primary Confirmed Yes No
Total Switch-ids: 6
N7K-1-1# sh fabricpath route
FabricPath Unicast Route Table
‘a/b/c’ denotes ftag/switch-id/subswitch-id
‘[x/y]’ denotes [admin distance/metric]
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id
FabricPath Unicast Route Table for Topology-Default
0/71/0, number of next-hops: 0
via —- , [60/0], 0 day/s 01:47:48, local
1/1/0, number of next-hops: 8
via Eth1/1, [115/40], 0 day/s 00:12:52, isis_fabricpath-default
via Eth1/2, [115/40], 0 day/s 00:12:52, isis_fabricpath-default
via Eth1/3, [115/40], 0 day/s 00:12:52, isis_fabricpath-default
via Eth1/4, [115/40], 0 day/s 00:12:52, isis_fabricpath-default
via Eth1/5, [115/40], 0 day/s 00:12:52, isis_fabricpath-default
via Eth1/6, [115/40], 0 day/s 00:12:52, isis_fabricpath-default
via Eth1/7, [115/40], 0 day/s 00:12:52, isis_fabricpath-default
via Eth1/8, [115/40], 0 day/s 00:12:52, isis_fabricpath-default
1/51/0, number of next-hops: 4
via Eth1/1, [115/40], 0 day/s 01:47:35, isis_fabricpath-default
via Eth1/2, [115/40], 0 day/s 01:47:35, isis_fabricpath-default
via Eth1/3, [115/40], 0 day/s 01:47:35, isis_fabricpath-default
via Eth1/4, [115/40], 0 day/s 01:47:35, isis_fabricpath-default
1/52/0, number of next-hops: 4
via Eth1/5, [115/40], 0 day/s 01:01:29, isis_fabricpath-default
via Eth1/6, [115/40], 0 day/s 01:01:29, isis_fabricpath-default
via Eth1/7, [115/40], 0 day/s 01:01:29, isis_fabricpath-default
via Eth1/8, [115/40], 0 day/s 01:01:29, isis_fabricpath-default
1/72/0, number of next-hops: 8
via Eth1/1, [115/80], 0 day/s 01:47:23, isis_fabricpath-default
via Eth1/2, [115/80], 0 day/s 01:47:23, isis_fabricpath-default
via Eth1/3, [115/80], 0 day/s 01:47:23, isis_fabricpath-default
via Eth1/4, [115/80], 0 day/s 01:47:23, isis_fabricpath-default
via Eth1/5, [115/80], 0 day/s 01:01:29, isis_fabricpath-default
via Eth1/6, [115/80], 0 day/s 01:01:29, isis_fabricpath-default
via Eth1/7, [115/80], 0 day/s 01:01:29, isis_fabricpath-default
via Eth1/8, [115/80], 0 day/s 01:01:29, isis_fabricpath-default
Let’s see the result of the MAC address table now.
0050.568b.002d, 0014.1cad.fb0a and 0014.1cad.fb0a are end hosts. What we see is that the reachability of the remote hosts is not a port but is a FabricPath Switch ID.
For example from N5K-p1-1 perspective, 0014.1cad.fb0a is reacheable through a switch that has the FabricPath Switch ID 52.0.0.
The reachability of this FabricPath host can be resolved through the ISIS database and the unicast frame will be forwared directly to that device.
N5K-p1-1# sh mac address-table dynamic
Legend:
* – primary entry, G – Gateway MAC, (R) – Routed MAC, O – Overlay MAC
age – seconds since last seen,+ – primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
———+—————–+——–+———+——+—-+——————
+ 100 0014.1cad.fb0a dynamic 0 F F 52.0.0
* 100 0015.1758.17dc dynamic 10 F F Po6
* 100 0050.568b.002d dynamic 20 F F Eth1/12
+ 100 0050.568b.002e dynamic 0 F F 52.0.0
* 100 58bc.27b7.1d96 dynamic 10 F F Eth1/12
N5K-p1-2# sh mac address-table dynamic
Legend:
* – primary entry, G – Gateway MAC, (R) – Routed MAC, O – Overlay MAC
age – seconds since last seen,+ – primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
———+—————–+——–+———+——+—-+——————
* 100 0014.1cad.fb0a dynamic 0 F F Eth1/12
* 100 0015.1758.17dc dynamic 40 F F Po6
+ 100 0050.568b.002d dynamic 0 F F 51.0.0
* 100 0050.568b.002e dynamic 20 F F Eth1/12
+ 100 58bc.27b7.1d96 dynamic 0 F F 51.0.0
What we see here is that Nexus 7000 have learned the MAC addresses of end hosts too which is not what Conversational Learning should allow because the Nexus 7000 doesn’t host the end device and therefore do not know their MAC addresses.
The reason is that the 7000s are running in mixed chassis mode (with both F and M cards). M cards are capable of L3 routing and can do proxy routing for F cards. So when the broadcast is received in the F modules, M card will learn the MAC.
If the Nexus 7000 switches were only hosting F cards, this learning would not have happened.
N7K-1-1# sh mac address-table dynamic
Legend:
* – primary entry, G – Gateway MAC, (R) – Routed MAC, O – Overlay MAC
age – seconds since last seen,+ – primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
———+—————–+——–+———+——+—-+——————
100 0014.1cad.fb0a dynamic 210 F F 52.0.0
100 0015.1758.17dc dynamic 60 F F 1.0.0
100 0050.568b.002d dynamic 90 F F 51.0.0
100 0050.568b.002e dynamic 30 F F 52.0.0
100 58bc.27b7.1d96 dynamic 240 F F 51.0.0
N7K-1-2# sh mac address-table dynamic
Legend:
* – primary entry, G – Gateway MAC, (R) – Routed MAC, O – Overlay MAC
age – seconds since last seen,+ – primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
———+—————–+——–+———+——+—-+——————
100 0014.1cad.fb0a dynamic 600 F F 52.0.0
100 0015.1758.17dc dynamic 30 F F 1.0.0
100 0050.568b.002d dynamic 450 F F 51.0.0
100 0050.568b.002e dynamic 420 F F 52.0.0
100 58bc.27b7.1d96 dynamic 600 F F 51.0.0
Next time the configuration will be complete with the addition of vPC+ on the 7Ks and L3 function like HSRP !
Hi Romain! Thank you so much. This is very helpful in my future deployments. I was trying to find the vPC+ with HSRP in your blog but I can’t find it. Do you also have that information? Thanks again.
Hello Bill,
HSRP can run over vPC+ as HSPR anycast for example.
Thanks,
in Fabricpath, can we also do a dual sided vPC+?
Hello Bill,
Yes you can do dual sided vPC+.
However I’m not sure to see what advantages it would have over running fabricpath between the 4 units.
Thanks,