Load Balancing and Protecting Cloud Director with Avi Networks

This article covers protecting and load balancing the Cloud Director application with Avi Networks. It covers SSL termination. health monitoring and layer 7 HTTP filtering. It can also be used as a reference for other load balancer products such as F5 LTM or NGINX.

Overview

The Avi Vantage platform is built on software-defined principles, enabling a next generation architecture to deliver the flexibility and simplicity expected by IT and lines of business. The Avi Vantage architecture separates the data and control planes to deliver application services beyond load balancing, such as application analytics, predictive autoscaling, micro-segmentation, and self-service for app owners in both on-premises or cloud environments. The platform provides a centrally managed, dynamic pool of load balancing resources on commodity x86 servers, VMs or containers, to deliver granular services close to individual applications. This allows network services to scale near infinitely without the added complexity of managing hundreds of disparate appliances.

https://avinetworks.com/docs/18.2/architectural-overview/

Avi components

Controllers – these are the management appliances that are responsible for state data, Service Engines are deployed by the controllers. The controllers run in a management network.

Service Engines – the load balancing services run in here. These generally run in a DMZ network. Service Engines can have one or more network adaptors connected to multiple networks. At least one network with routing to the controllers, and the remaining networks as data networks.

Deployment modes

Avi can be installed in a variety of deployment types. For VMware Cloud on AWS, it is not currently possible to deploy using ‘write access’ as vCenter is locked-down in VMC and it also has a different API from vSphere 6.7 vCenter Server. You’ll also find that other tools may not work with vCenter in a VMware Cloud on AWS SDDC, such as govc.

Instead Avi needs to be deployed using ‘No Access’ mode.

You can refer to this link for instructions to deploy Avi Controllers in ‘No Access’ mode.

Since it is only possible to use ‘No Access’ mode with VMC based SDDCs, its also a requirement to deploy the service engines manually. To do this follow the guide in this link, and start at the section titled Downloading Avi Service Engine on OVA.

If you’re using Avi with on-premises deployments of vCenter, then ‘Write Mode’ can be used to automate the provisioning of service engines. Refer to this link for more information on the different modes.

Deploying Avi Controller with govc

You can deploy the Avi Controller onto non VMware Cloud on AWS vCenter servers using the govc tool. Refer to this other post on how to do so. I’ve copied the JSON for the controller.ova for your convenience below.

{
    "DiskProvisioning": "flat",
    "IPAllocationPolicy": "dhcpPolicy",
    "IPProtocol": "IPv4",
    "PropertyMapping": [
        {
            "Key": "avi.mgmt-ip.CONTROLLER",
            "Value": ""
        },
        {
            "Key": "avi.mgmt-mask.CONTROLLER",
            "Value": ""
        },
        {
            "Key": "avi.default-gw.CONTROLLER",
            "Value": ""
        },
        {
            "Key": "avi.sysadmin-public-key.CONTROLLER",
            "Value": ""
        }
    ],
    "NetworkMapping": [
        {
            "Name": "Management",
            "Network": ""
        }
    ],
    "MarkAsTemplate": false,
    "PowerOn": false,
    "InjectOvfEnv": false,
    "WaitForIP": false,
    "Name": null
}

Architecture

For a high-level architecture overview, this link provides a great starting point.

Figure 1. Avi architecture

Service Engine Typical Deployment Architecture

Generally in legacy deployments, where BGP is not used. The service engines would tend to have three network interfaces. These are typically used for frontend, backend and management networks. This is typical of traditional deployments with F5 LTM for example.

For our example here, I will use three networks for the SEs as laid out below.

Network nameGateway CIDRPurpose
sddc-cgw-vcd-dmz1 10.104.125.1/24Management
sddc-cgw-vcd-dmz210.104.126.1/24Backend
sddc-cgw-vcd-dmz310.104.127.1/24Frontend

The service engines are configured with the following details. It is important to make a note of the MAC addresses in ‘No access’ mode as you will need this information later.

Service Engineavi-se1avi-se2
ManagementIP Address 10.104.125.11
Mac Address 00:50:56:8d:c0:2e
IP Address 10.104.125.12
Mac Address 00:50:56:8d:38:33
BackendIP Address 10.104.126.11
Mac Address 00:50:56:8d:8e:41
IP Address 10.104.126.12
Mac Address 00:50:56:8d:53:f6
FrontendIP Address 10.104.127.11
Mac Address 00:50:56:8d:89:b4
IP Address 10.104.127.12
Mac Address 00:50:56:8d:80:41

The Management network is used for communications between the SEs and the Avi controllers. For the port requirements, please refer to this link.

The Backend network is used for communications between the SEs and the application that is being load balanced and protected by Avi.

The Frontend network is used for upstream communications to the clients, in this case the northbound router or firewall towards the Internet.

Sample Application

Lets use VMware Cloud Director as the sample application for configuring Avi. vCD as it is more commonly named (to be renamed VMware Cloud Director), is a cloud platform which is deployed with an Internet facing portal. Due to this, it is always best to protect the portal from malicious attacks by employing a number of methods.

Some of these include, SSL termination and web application filtering. The following two documents explain this in more detail.

vCloud Director Security and VMware vCloud Director Security Hardening Guide.

The vCD application is configured as below:

vCD Appliance 1vCD Appliance 2
namevcd-fe1vcd-fe2
eth0 ip address10.104.123.2110.104.123.22
static route10.104.123.1 10.104.126.0/2410.104.123.1 10.104.126.0/24
eth1 ip address10.104.124.2110.104.124.22

You’ll notice that the eth0 and eth1 interfaces are connected to two different management networks 10.104.123.0/24 and 10.104.124.0/24 respectively. For vCD, it is generally good practice to separate the two interfaces into separate networks.

Network nameGateway CIDRPurpose
sddc-cgw-vcd-mgmt-110.104.123.1/24vCD Frontend
UI/API/VM Remote Console
sddc-cgw-vcd-mgmt-210.104.124.1/24vCD Backend
PostgreSQL, SSH etc.

For simplicity, I also deployed my Avi controllers onto the sddc-cgw-vcd-mgmt-2 network.

The diagram below summarises the above architecture for the HTTP interface for vCD. For this guide, I’ve used VMware Cloud on AWS together with Avi Networks to protect vCD running as an appliance inside the SDDC. This is not a typical deployment model as Cloud Director Service will be able to use VMware Cloud on AWS SDDC resource soon, but I wanted to showcase the possibilities and constraints when using Avi with VMC based SDDCs.

Figure 2 . vCD HTTP Diagram

Configuring Avi for Cloud Director

After you have deployed the Avi Controllers and the Service Engines, there are few more steps needed before vCD is fully up and operational. The proceeding steps can be summarised as follows:

  1. Setup networking for the service engines by assigning the right IP address to the correct MAC addresses for the data networks
  2. Configure the network subnets for the service engines
  3. Configure static routes for the service engines to reach vCD
  4. Setup Legacy HA mode for the service engine group
  5. Setup the SSL certificate for the HTTP service
  6. Setup the Virtual Services for HTTP and Remote Console (VMRC)
  7. Setup the server pools
  8. Setup health monitors
  9. Setup HTTP security policies

Map Service Engine interfaces

Using the Avi Vantage Controller, navigate to Infrastructure > Service Engine, select one of the Service Engines then click on the little pencil icon. Then map the MAC addresses to the correct IP addresses.

Configure the network subnets for the service engines

Navigate to Infrastructure > Networks and create the subnets.

Configure static routes

Navigate to Infrastructure > Routing and setup any static routes. You’ll notice from figure 2 that since the service engine has three network interfaces on different networks, we need to create a static route on the interface that does not have the default gateway. This is so the service engines knows which gateway to use to route traffic for particular traffic types. In this case, the gateway for the service engine to route the HTTP and Remote Console traffic southbound to the vCD cells.

Setup Legacy HA mode for the service engine group

Navigate to Infrastructure > Service Engine Group.

Setup the HA mode to Legacy HA. This is the simplest configuration, you can use Elastic HA if you wish.

Configure the HTTP and Remote Console Virtual Services

Navigate to Applications > Virtual Services.

Creating a Virtual Service, has a few sub tasks which include the creation of the downstream server pools and SSL certificates.

Create a new Virtual Service for the HTTP service, this is for the Cloud Director UI and API. Please use this example to create another Virtual Service for the Remote Console.

For the Remote Console service, you will need to accept TCP 443 on the load balancer but connect southbound to the Cloud Director appliances on port TCP 8443. TCP 8443 is the port that VMRC uses as it shares the same IP addresses as the HTTP service.

You may notice that the screenshot is for an already configured Virtual Service for the vCD HTTP service. The server pool and SSL certificate is already configured. Below are the screenshots for those.

Certificate Management

You may already have a signed HTTP certificate that you wish to use with the load balancer for SSL termination. To do so, you will need to use the JAVA keytool to manipulate the HTTP certificate, obtaining the private key and convert from JCEKS to PCKS12. JAVA keytool is available in the vCD appliance at /opt/vmware/vcloud-director/jre/bin/.

Figure 3. SSL termination on load balancer

For detailed instructions on creating a signed certificate for vCD, please follow this guide.

Convert the keystore file certificates.ks file from JCEKS to PKCS12

keytool -importkeystore -srcstoretype JCEKS -srckeystore certificates.ks -destkeystore certificates_pkcs12.ks -deststoretype PKCS12

Export private key for the HTTP certificate from the certificates_pkcs12.ks file

keytool -importkeystore -srckeystore certificates_pkcs12.ks -srcalias http -destalias http -destkeystore httpcert.p12 -deststoretype PKCS12

Now that you have the private key for the HTTP certificate, you can go ahead and configure the HTTP certificate on the load balancer.

For the certificate file, you can either paste the text or upload the certificate file (.cer, .crt) from the certificate authority for the HTTP certificate.

For the Key (PEM) or PKCS12 file, you can use the httpcert.p12 file that you extracted from the certificates_pkcs12.ks file above.

The Key Passphrase is the password that you used to secure the httpcert.p12 file earlier.

Note that the vCD Remote Console (VMRC) must use pass-through for SSL termination, e.g., termination of the VMRC session must happen on the Cloud Director cell. Therefore, the above certificate management activities on Avi are not required for the VMRC.

Health Monitors

Navigate to Applications > Pools.

Edit the HTTP pool using the pencil icon and click on the Add Active Monitor green button.

Health monitoring of the HTTP service uses

GET /cloud/server_status HTTP/1.0

With an expected server response of

Service is up.

And a response code of 200.

The vCD Remote Console Health monitor is a lot simpler as you can see below.

Layer 7 HTTP Security

Layer 7 HTTP Security is very important and is highly recommended for any application exposed to the Internet. Layer 3 fire-walling and SSL certificates is always never enough in protecting and securing applications.

Navigate to Applications > Virtual Services.

Click on the pencil icon for the HTTP virtual service and then click on the Policies tab. Then click on the HTTP Security policy. Add a new policy with the following settings. You can read more about Layer 7 HTTP policies here.

Allowed StringsRequired by
/tenantTenant use
/loginLogin
/networkAccess to networking
/tenant-networkingAccess to networking
/cloudFor SAML/SSO logins
/transferUploads/Downloads of ISO and templates
/apiGeneral API access
/cloudapiGeneral API access
/docsSwagger API browser
Blocked Strings
/cloudapi/1.0.0/sessions/providerSpecifically block admin APIs from the Internet

This will drop all provider side services when accessed from the Internet. To access provider side services, such as /provider or admin APIs, use an internal connection to the Cloud Director cells.

Change Cloud Director public addresses

If not already done so, you should also change the public address settings in Cloud Director.

Testing the Cloud Director portal

Try to access https://vcloud.vmwire.com/provider

You won’t be able to access it as /provider is not on the list of allowed URI strings that we configured in the L7 HTTPS Security settings.

However, if you try to access https://vcloud.vmwire.com/tenant/vmwire, you will be able to reach the tenant portal for the organisation named VMwire.

Many thanks to Mikael Steding, our Avi Network Systems Engineer for helping me with setting this up.

Please reach out to me if you have any questions.

Author: Hugo Phan

@hugophan

20 thoughts on “Load Balancing and Protecting Cloud Director with Avi Networks”

  1. Hi Hugo, this is a great article and I was looking for a similar example of protecting cloud director via AVI Loadbalancer. Everything is clear in the above example except one thing. You mentioned about creating another service for vmrc console and provided the health monitors as well however what is not clear is how you can create this virtual service using the same public IP as for http.

    In mycase we are using cloud appliance 10.2.1 and we will have only one IP which will be used for both http and vmrc however its not clear how can i create service for vmrc with same IP.

    Can you please provide some guidance here? what I am missing?

    1. Hi Ajit,

      When you use the vCD appliance, the UI and API services runs on port 443. The VMRC Service runs on 8443. This is because the appliance has one vmnic for this frontend. The other vmnic is for backend services such as PostgreSQL.

      You have two options:
      1. Use one fqdn…. vcloud.vmwire.com, this points to one public IP address, change the vmrc virtual service in Avi to port 8443. You save one public IP address with this option but this may cause issues if clients cannot egress from their corporate networks on port 8443.

      2. Use two fqdns…. vcloud.vmwire.com and vmrc.vmwire.com, point to two public IP addresses respectively, create http virtual service in Avi on port 443 and point this to the vcloud server pool. Create vmrc virtual service in avi on port 443 and point it to the vmrc server pool. This is the option documented in this article and is best if clients have firewall restrictions that only allow them to egress on port 443.

      Hope this helps.

  2. Hi Hugo, thanks for this great article. I also have some issues with VMRC load balancing I’m trying with “Option 1”. I created a virtual service with TCP/UDP profile “System-TCP-Profile” , App Profile “System-Secure-HTTP”, Service Port “8443” and a specific pool for the cells with active monitors “System-Ping”, “System-TCP” and Default Server Port set to 8443. Also the servers added to the pool have port 8443 specified. In VCD public addresses for the console I’ve specified port 8443 (vcd.domain.com:8443) but when I try to launch a console the connection is immediately closed and see “disconnected” in the console. Where I’m wrong? Please, can you post the avi config for the virtual service and pool for the vmdc part? Thanks!

    1. Hi Nagnelli,

      for VMRC console proxy settings should be as given below :

      Application Profile – System-L4-Application
      TCP/UDP Profile – System-TCP-Proxy

      Since this is a layer 4 connection application profile has to be as mentioned above and it should work after this.

      Regards,
      Ajit

      1. Hi Ajit,
        unfortunately it doesn’t not work. I’ve already tried with L4-Application (I’ve tried to “replicate” the configuration I’ve on NSX-T LB that works). Thanks, regards,
        N.

        1. Are you able to share config details or screenshots for the configurations. I had faced similar issue but got that working after rectifying the configs. I was making mistakes in selecting wrong application profile.
          which version of AVI you are using?
          Do you see any traffic hitting this application at all on the Service Engine?

          1. I think it’s not possible to add screenshots here. Anyway my config is:

            Virtual Services:
            IPv4 VIP: shared with https virtual service
            TCP/UDP Profile: System-TCP-Proxy
            Application Profile: System-L4-Application
            Service Port: 8443
            Pool: vcd-console-pool (same ips added as in https pool)

            Pool:
            Default Server Port: 8443
            Load Balance: Consisten Hash -> Source IP address
            Health Monitors: Passive Health Monitor checked – System-Ping, System-TCP
            Enable SSL: NOT checked

            The error I see in console-proxy.log:

            2021-07-06 13:09:04,486 | DEBUG | consoleproxy queue : 1 | SSLHandshakeTask | Exception during handshake: javax.net.ssl.SSLHandshakeException: no cipher suites in common | java.nio.channels.SocketChannel[connected local=/10.12.31.101:8443 remote=/100.64.224.1:6228]

            I’m using AVI Version: 20.1.6 Build: 9132

            Thanks for helping me!

            1. Did you get resolve the issue ? I have similar issues with the vrmc configuration with my VCD appliances.

            2. Hi Nagnelli,

              Based on the error you have pasted its pointing towards SSL issue and this is pointing to config issue on AVI as I had handshakes errors as well on vmrc and I realize that i had not applied the certificates in Cloud Director “public Address” configuration page.
              After applying the correct certificate there and updating AVI config, immediately VMRC was up for me. May be worthwhile to check that setting.

              Regards,
              Ajit

              1. Hi Ajit,

                I already have certs (same imported in avi) in public address Config page. I’ve also tried to re-add it but nothing changed.

                Thanks, regards,
                Nicola

                1. Same to me. I will paste tomorrow all my configurations, here in Spain it is night. The balance worked sometimes but only worked with one cell. If I started the services in more than one cell, the balancer does not work over the console. The trick if check from outside: if the htttps:/console.tudominio.com:8443 gives you the public certificate good thing, while the balancer does not give you this, it never will work.

                  I will speak you tomorrow.

            3. Here are my VMRC virtual service settings:

              Settings tab
              virtual service name: vmrc.vmwire.com
              Traffic Enabled: Enabled
              IPv4 VIP: 10.104.127.101
              Application Profile: System-L4-Application
              TCP/UDP Profile: System-TCP-Proxy
              Service Port: 443 (in your case use 8443 as you are using a single public IP address and FQDN)
              Pool: vmrc.vmwire.com-pool

              Advanced tab
              Weight: 1
              Fairness: Throughput And Delay Fairness
              SE Group: Default-Group
              Auto Gateway: Enabled

              Here are my VMRC pool settings:

              Settings tab
              Name: vmrc.vmwire.com-pool
              Enabled: True
              Default Server Port: 8443
              Load Balance: Consistent Hash -> Source IP Address
              Health Monitors: Passive Health Monitor with
              System-Ping: leave defaults
              System-TCP: leave defaults
              Analytics Profile: System-Analytics-Profile
              SSL to Backend Servers: Disable (don’t tick the “Enable SSL” box).

              If you are using these settings and the console is still not working, then it might be a certificates issue for the VMRC service on your VCD cells.

              Do you have the certificates.ks file configured correctly?

              The LB service for the VMRC service will pass-through traffic from the Avi load balancer straight down to the VCD cells. This may be a reason why the remote console is not connecting.

              Check this other article, it demonstrates how to configure certificates for VCD. https://vmwire.com/2020/09/29/using-lets-encrypt-certificates-with-cloud-director/

              1. Hello all,
                I’ve been able to find my root problem. Thank to your suggestion I’ve started to look deeper at the certificate itself and found that the ca siging the certs in my lab was using EC and not RSA but accordingly to https://kb.vmware.com/s/article/2057736 only RSA-based server public key is supported. After redeploying CA and cert it worked. Thank all!

              2. Perfect, I am going to check along these days. But my issue was with VCD certificates. I had to run /opt/vmware/vcloud-director/bin/configure in all my cells ( appliances ) because two cells gave me the local certificate instead of my wildcard. I recommend use

                openssl s_client -connect CELDA1:443 | grep -i CN = *.miwilcardomian.com
                openssl s_client -connect CELDA1:8443 | grep -i CN = *..miwilcardomian.com
                openssl s_client -connect CELDA2443 | grep -i CN = *.miwilcardomian.com
                openssl s_client -connect CELDA2:8443 | grep -i CN = *..miwilcardomian.com
                ..

                You ensure the cells are returning you the wildcard.

                The avi configuration seen works correctly.

                Great, thank a lot.

                1. ahh … I recommend using different range segments in cells to NIC0 and NIC1(bbdd).
                  If you use the same range in both NICs, the gateway is the same ( you have two routes) and the traffic goes out by NIC1.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s