VMware: IP-Hash LB in IP Storage

In this post I would like to explain IP-Hash Load Balancing principles used for connecting VMware hosts to IP storage, like NFS, for better understanding how traffic balancement across various NICs occurs.

As reported in previous VMware: Uplink used by a VM in a LB NIC Teaming article Load Balancing policies are effective between vmnics and physical storage and all pSwitches inbetween.

Load Balancing does not occur between VMs, VMkernels and Port Groups in vSwitches.

IP-Hash Load Balancing is the LB technique that could potentially achieve the highest degree of efficiency since it uses different vmnics based on an algorithm that considers the destination IP address the packet has to be sent to and, unlike Route Based on originating virtual port ID it doesen't use a Round Robin vmnic assignation methodology.

Although IP-Hash Load Balancing lead to a better traffic load balancing it also brings the most complex set-up because it also requires particular configuration changes in all pSwitches packets will traverse to reach IP storage.

In pSwitches inbetween your VMware hosts and IP storage you will need to enable IEEE 802.3ad LACP, or Etherchannel if you deal with Cisco equipment, in order to benefit IP-Hash Load Balancing. LACP basically aggregates from 2 up to 8 ethernet channels providing a link with more aggregate bandwidth and resiliency.

Here's a simple connection schema, please bear in mind that in a real case scenario using a single pSwitch introduces a single point of failure in your infrastructure.

You should also consider VLAN implementation in order to isolate storage traffic. This, as you certainly know, is a thumb rule not just for storage traffic such as NFS or iSCSI but even for management traffic, vMotion traffic, FT traffic, etc.

Since my post is VMware related I will assume you have already configured LACP properly in your physical network. For more info on LACP setup please have a look at 1004048 KB.

Let's start by setting up our virtual networking. For sake of simplicity we will use just two vmnics.
We create a new vSwitch and a VMkernel for IP Storage (NFS in my case) assigning these two vmnics to it.

Please be sure that all vmnics are set as Active and that there are no vmnics set as Standby or Unused. IP-Hash Load Balancing must be set at vSwitch level and not overridden at VMkernel/PortGroup level.

IP-Hash Load Balancing algorithm chooses which vmnic utilize for any IP connection based on destination IP address upon the following equation:

vmnic used = [HEX(IP VMkernel) xor HEX(IP Storage)] mod (Number of vmnics)

where:

HEX indicates that the IP address has been converted in hexadecimal format. This is done octet by octet which means that for example 10.11.12.13 IP address in HEX base is 0A.0B.0C.0D that will be represented as 0x0A0B0C0D.

IP VMkernel is the IP address assigned to the VMkernel used for IP Storage

IP Storage is the IP address assigned to the NFS storage

xor is the exclusive or operand

mod is modulo operation

Now that we know how vmnic are choosen for data transfer let's examine how we can wisely assign IP addresses to our IP storage systems.

In this example I will use a NFS server with two NICs (no vIP). I've already assigned IP 192.168.116.10 to VMkernel responsible for carrying IP storage data.

1) Bad IP addresses choice

VMkernel = 192.168.116.10
NFS 1 = 192.168.116.50
NFS 2 = 192.168.116.60

Let's do some math:

vmnic used for NFS1 = [HEX(192.168.116.10) xor HEX(192.168.116.50)] mod (2)

= [0xC0A8740A xor 0xC0A87432] mod (2)

= [38] mod (2)

= 0 -> vmnic0 will be used


vmnic used for NFS2 = [HEX(192.168.116.10) xor HEX(192.168.116.60)] mod (2)

= [0xC0A8740A xor 0xC0A8743C] mod (2)

= [36] mod (2)

= 0 -> vmnic0 will be used

As you can see assigning these two IP addresses to NFS storage was a bad choice becase communications toward both NFS1 and NFS2 will utilize vmnic0 leaving vmnic1 unused.

2) Good IP addresses choice

VMkernel = 192.168.116.10
NFS 1 = 192.168.116.50
NFS 2 = 192.168.116.51

Let's do some math again:

vmnic used for NFS1 = [HEX(192.168.116.10) xor HEX(192.168.116.50)] mod (2)

= [0xC0A8740A xor 0xC0A87432] mod (2)

= [38] mod (2)

= 0 -> vmnic0 will be used


vmnic used for NFS2 = [HEX(192.168.116.10) xor HEX(192.168.116.51)] mod (2)

= [0xC0A8740A xor 0xC0A87433] mod (2)

= [39] mod (2)

= 1 -> vmnic1 will be used

These was a wise choice since communications with NFS1 will use vmnic0 and communications with NFS2 will use vmnic1 achieving traffic balance and both uplinks utilization.

Some useful links:

Best Practices for running VMware vSphere on Network Attached Storage

Sample configuration of EtherChannel / Link Aggregation Control Protocol (LACP) with ESXi/ESX and Cisco/HP switches

That's all!!