VMware NSX Load Balancing 503 Error from VIP

Creation of the VMware NSX load balancer is pretty straight forward.  Ultimately you create an Application Profile, A Serverfarm Pool, a VIP, and you are off to the races.

Of course there is a bit more to it, but if you have load balancing experience already it’s pretty straight forward.  I did however find myself scratching my head for a bit after installing a few Ubuntu LAMP servers to participate in an HTTP load balancing 2 armed scenario. (NOTE: the DNAT just represents a Public IP). Let me set the stage:

With a relatively vanilla network, I went to my CST1 Edge and flipped to the Load Balancing tab.  I clicked Enable Load balancer in the load balancer Global Configuration, created a straight forward Application Profile for HTTP, made sure the service monitor was set to default_http_monitor.

I then created a pool with both LAMP load balancers by clicking on “Select” in the IP Address / VC Container field, the Object Type “Virtual Machine” and selecting my VMs (My VMs are named CST1-DMZ01 and 02 in vCenter, LAMP01-02 are just their tags)

And Finally I built the Virtual Server with a public IP address that was attached to the TenantTransit01 uplink on the ESG.

This is a pretty run of the mill setup, so I was a bit astounded when I opened my web browser and received a 503 Service Unavailable- No server is available to handle this request. 

I did some cursory troubleshooting, checked to make sure my Pool and Member Status looked good.  Sure enough it did.  

I then double-checked firewall rules and routes to make sure I could indeed get into the environment from the outside.  Everything looked good.  I was able to ping my VIP, I was able to browse to the individual servers themselves by giving them NATs on the FW, so I knew that routes / firewall rules / and the service on the VMs themselves were good.  So what went wrong?

After a bit more head scratching, I decided to install VMware Tools on the Ubuntu LAMP01 server.  All of a sudden my VIP started working, but only balancing to LAMP01.  Ah hah!  You must have VMware Tools installed in order to use the load balancer right?  Not entirely.  While this did fix my issue, I wanted to make sure I could still balance on other servers that for one reason or another would not be able to have the VMware toolset installed.  That’s when I remembered my pool allocation.

When I allocated the serverfarm pool, I created each pool member by selecting them using the “Select” option and then chose Object Type: Virtual Machine.  Since the VMs did not have VMware tools installed, NSX was unable to map the Object ID to the correct IP address of the LAMP servers.  I was able to get around having to install VMware tools by manually entering the IP address in the Edit Member -> IP Address / VC Container field.  

When this was completed, both of my VMs started participating in the VIP rotation.

The pool member status now reflects the IP Address instead of the VC Container ID.

In summary: Install VMware tools on your VM’s when possible.  When not possible, manually enter the IP address of the Pool Member when configuring your Pools.