Introduction

What I often hear about MTU on Nexus 7000 and Nexus in general is that nothing needs to be done and Nexus devices are ready for jumbo MTU right out of the box.

The truth is not that simple and this is what I’ll try to explain here.

Layer 2 MTU

MTU can be modified on layer 2 interface by using the “MTU” command and the default MTU is 1500 as it can be seen on the following output :

N7K-1# sh int e3/9
Ethernet3/9 is up
admin state is up, Dedicated Interface
  Hardware: 10000 Ethernet, address: e4c7.2217.8240 (bia e4c7.2217.8240)
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is broadcast
  Port mode is trunk

If I try to ping an SVI from this Nexus 7000 to another Nexus 7000 SVI connected to it via a layer 2 trunk interface it can not success with the DF-BIT set :

N7K-1# ping 10.10.10.101 packet-size 1600 df-bit
PING 10.10.10.101 (10.10.10.101): 1600 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out

The packet size is too big to be able to transit through the trunk interface without being fragmented.

This demonstrates that the Nexus 7000 is not jumbomtu enable right out of the box. This administrator still needs to specify the MTU at the interface level.

What is true is that we can specify the MTU at 9216 without modifying anything at the system level and this is because the system jumbomtu is already set up at 9216 by default :

N7K-1# show running-config all | inc jumbomtu
system jumbomtu 9216

But it is not possible to specify another value, for example 8000 :

N7K-1(config)# int e3/9
N7K-1(config-if)# mtu 8000
ERROR: Ethernet3/9: MTU on L2 interfaces can only be set to default or system-jumboMTU

The reason why is because the system jumbomtu command dictates what can be entered as MTU :

N7K-1(config-if)# inter e3/9
N7K-1(config-if)# mtu ?
  <1500-9216>  Enter MTU

If you change the system jumbomtu value, it change the maximum value you can specify :

N7K-1(config-if)# exit
N7K-1(config)# system jumbomtu 8000
N7K-1(config)# interface e3/9
N7K-1(config-if)# mtu ?
  <1500-8000>  Enter MTU

Once the MTU is large enough to support the packet, the ping will success. Of course this needs to be done on both side and it also need to be done on the SVI :

N7K-1(config-if)# exit
N7K-1(config)# system jumbomtu 9216
N7K-1(config)# interface e3/9
N7K-1(config-if)# mtu 9216
N7K-1(config-if)# show interface e3/9
Ethernet3/9 is up
admin state is up, Dedicated Interface
  Hardware: 10000 Ethernet, address: e4c7.2217.8240 (bia e4c7.2217.8240)
  MTU 9216 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, medium is broadcast
  Port mode is trunk
  full-duplex, 10 Gb/s, media type is 10G
…

N7K-1# ping 10.10.10.101 packet-size 7000 df-bit
PING 10.10.10.101 (10.10.10.101): 7000 data bytes
7008 bytes from 10.10.10.101: icmp_seq=0 ttl=254 time=1.958 ms
7008 bytes from 10.10.10.101: icmp_seq=1 ttl=254 time=5.401 ms
7008 bytes from 10.10.10.101: icmp_seq=2 ttl=254 time=5.673 ms

One thing that need to be said is that Layer 3 interfaces are not subject to this rule. On a Layer 3 interface any MTU on a range from 64 to 9216 can be specify and it does not depend on the system jumbomtu value.