VMware: Uplink used by a VM in a LB NIC Teaming

In this post I will explain how to show the uplink used by a VM in a vSwitch with NIC Teaming Load Balancing.

Since we all put our uplinks in a Load Balancing NIC Teaming sometimes it could be useful for troubleshooting purpouses to see what's the specific physical uplink used by that particular VM to access the network.

To explain this I will use a real case scenario that happened to me today.

A customer reported that after vMotioning a VM from one physical host to another this VM was unable to reach a specific IP Address in his network (a router, fyi).

I checked the basic networking settings from vSphere Client like vSwitch configuration and if every vmnic reported network activity (i.e. plugged cable) and everything was fine.

vSwitch configuration was similar to this:

vm4 was the VM reporting issues.

Analyzing vSwitch0 settings this was the result:

As you can see all vmnics (vmnic0, vmnic1, vmnic2, vmnic3) are set as Active and Route Based on originating virtual port ID was selected as Load Balancing method.

Next step was to take a look at physical networking. Customer has a blade enclosure with 3 hosts, each blade (i.e. ESXi physical host) with  4 NIC: vmnic0, vmnic1, vmnic2 and vmnic3.

Each NIC was connected to a different physical switch to achieve path realiability and every switch is connected to the one 'core switch' (second switch from the top in picture below) EXCEPT switch 4...and here all yours bells and whistles will ring.

To make this story short, as most of you have already understood, this was the flaw in this configuration. By per se it's not an issue having a switch not connected to other switches but in this scenario it's a fault because it's used in conjunction with Route Based on originating virtual port ID Load Balancing.

Route Based on originating virtual port ID it's based on a Round Robin uplink assignation. This means that every VM powered on recieve in sequence the next available uplink (vmnic) in the vSwitch.

Take a look at this picture for better understandment:

VM power-on sequence in this case was:

POWER-ON SEQ.  UPLINK ASSIGNED
vm1            vmnic0
vm2            vmnic1
vm3            vmnic2
vm4            vmnic3
vm5            vmnic0

...and so on...let's pretend for example a new vm6 will be powered on this VM will use vmnic1. If vm3  and vm6 are powered off and vm6 is powered on back again it will use vmnic2 as uplink. Pretty simple, right?

In my case vm4 reported network issues reaching the router IP Address (the rightmost grey block on the picture above) because after vMotion from one host to another vm4 ended on a host's vSwitch which assigned it to vmnic3 which is connected to an isolated physical switch. This is easily solvable either by adding the missing intra-switch link or by removing vmnic3 from the Load Balancing active adapters.

After this theorethical prologue let's dive a bit into practice by explaining how it's possible to display which uplink (in a LB NIC Team) is used by a VM.

Let's login into our ESXi host.

To retrieve port informations for a particular VM we need to use the following command:

~ # esxcli network vm port list -w <World_ID>

Where <World_ID> is the World ID of our VM. This World ID uniquely identificates a VM in a host and changes every time a VM is vMotioned and a power status change occurs (i.e. vm is powered off and back on again).

To display World ID for our VMs we need to execute following command. Informations about every VM running in the host are retrieved. To keep things clean I only show you the output of vm4.

~ # esxcli vm process list

vm4
   World ID: 3307679
   Process ID: 0
   VMX Cartel ID: 3307678
   UUID: 42 19 4a 84 a3 52 c7 ca-d9 28 93 bf 20 6c 2d bc
   Display Name: vm4
   Config File: /vmfs/volumes/51dc35ee-c4b96dc0-1c1c-b4b53f5110c0/vm4/vm4.vmx


World ID: 3307679 if the value of interest.

If we run the command above this time specifying World ID we get:

~ # esxcli network vm port list -w 3307679
   Port ID: 33554444
   vSwitch: vSwitch0
   Portgroup: VM Network
   DVPort ID:
   MAC Address: 00:50:56:99:45:3b
   IP Address: 0.0.0.0
   Team Uplink: vmnic3
   Uplink Port ID: 33554438
   Active Filters:

As you can see Team Uplink: vmnic3 indicates that vmnic3 is used by this VM.

That's all!