Load Balancing and Protecting Cloud Director with Avi Networks

This article covers protecting and load balancing the Cloud Director application with Avi Networks. It covers SSL termination. health monitoring and layer 7 HTTP filtering. It can also be used as a reference for other load balancer products such as F5 LTM or NGINX.

Overview

The Avi Vantage platform is built on software-defined principles, enabling a next generation architecture to deliver the flexibility and simplicity expected by IT and lines of business. The Avi Vantage architecture separates the data and control planes to deliver application services beyond load balancing, such as application analytics, predictive autoscaling, micro-segmentation, and self-service for app owners in both on-premises or cloud environments. The platform provides a centrally managed, dynamic pool of load balancing resources on commodity x86 servers, VMs or containers, to deliver granular services close to individual applications. This allows network services to scale near infinitely without the added complexity of managing hundreds of disparate appliances.

https://avinetworks.com/docs/18.2/architectural-overview/

Avi components

Controllers – these are the management appliances that are responsible for state data, Service Engines are deployed by the controllers. The controllers run in a management network.

Service Engines – the load balancing services run in here. These generally run in a DMZ network. Service Engines can have one or more network adaptors connected to multiple networks. At least one network with routing to the controllers, and the remaining networks as data networks.

Deployment modes

Avi can be installed in a variety of deployment types. For VMware Cloud on AWS, it is not currently possible to deploy using ‘write access’ as vCenter is locked-down in VMC and it also has a different API from vSphere 6.7 vCenter Server. You’ll also find that other tools may not work with vCenter in a VMware Cloud on AWS SDDC, such as govc.

Instead Avi needs to be deployed using ‘No Access’ mode.

You can refer to this link for instructions to deploy Avi Controllers in ‘No Access’ mode.

Since it is only possible to use ‘No Access’ mode with VMC based SDDCs, its also a requirement to deploy the service engines manually. To do this follow the guide in this link, and start at the section titled Downloading Avi Service Engine on OVA.

If you’re using Avi with on-premises deployments of vCenter, then ‘Write Mode’ can be used to automate the provisioning of service engines. Refer to this link for more information on the different modes.

Deploying Avi Controller with govc

You can deploy the Avi Controller onto non VMware Cloud on AWS vCenter servers using the govc tool. Refer to this other post on how to do so. I’ve copied the JSON for the controller.ova for your convenience below.

{
    "DiskProvisioning": "flat",
    "IPAllocationPolicy": "dhcpPolicy",
    "IPProtocol": "IPv4",
    "PropertyMapping": [
        {
            "Key": "avi.mgmt-ip.CONTROLLER",
            "Value": ""
        },
        {
            "Key": "avi.mgmt-mask.CONTROLLER",
            "Value": ""
        },
        {
            "Key": "avi.default-gw.CONTROLLER",
            "Value": ""
        },
        {
            "Key": "avi.sysadmin-public-key.CONTROLLER",
            "Value": ""
        }
    ],
    "NetworkMapping": [
        {
            "Name": "Management",
            "Network": ""
        }
    ],
    "MarkAsTemplate": false,
    "PowerOn": false,
    "InjectOvfEnv": false,
    "WaitForIP": false,
    "Name": null
}

Architecture

For a high-level architecture overview, this link provides a great starting point.

Figure 1. Avi architecture

Service Engine Typical Deployment Architecture

Generally in legacy deployments, where BGP is not used. The service engines would tend to have three network interfaces. These are typically used for frontend, backend and management networks. This is typical of traditional deployments with F5 LTM for example.

For our example here, I will use three networks for the SEs as laid out below.

Network nameGateway CIDRPurpose
sddc-cgw-vcd-dmz1 10.104.125.1/24Management
sddc-cgw-vcd-dmz210.104.126.1/24Backend
sddc-cgw-vcd-dmz310.104.127.1/24Frontend

The service engines are configured with the following details. It is important to make a note of the MAC addresses in ‘No access’ mode as you will need this information later.

Service Engineavi-se1avi-se2
ManagementIP Address 10.104.125.11
Mac Address 00:50:56:8d:c0:2e
IP Address 10.104.125.12
Mac Address 00:50:56:8d:38:33
BackendIP Address 10.104.126.11
Mac Address 00:50:56:8d:8e:41
IP Address 10.104.126.12
Mac Address 00:50:56:8d:53:f6
FrontendIP Address 10.104.127.11
Mac Address 00:50:56:8d:89:b4
IP Address 10.104.127.12
Mac Address 00:50:56:8d:80:41

The Management network is used for communications between the SEs and the Avi controllers. For the port requirements, please refer to this link.

The Backend network is used for communications between the SEs and the application that is being load balanced and protected by Avi.

The Frontend network is used for upstream communications to the clients, in this case the northbound router or firewall towards the Internet.

Sample Application

Lets use VMware Cloud Director as the sample application for configuring Avi. vCD as it is more commonly named (to be renamed VMware Cloud Director), is a cloud platform which is deployed with an Internet facing portal. Due to this, it is always best to protect the portal from malicious attacks by employing a number of methods.

Some of these include, SSL termination and web application filtering. The following two documents explain this in more detail.

vCloud Director Security and VMware vCloud Director Security Hardening Guide.

The vCD application is configured as below:

vCD Appliance 1vCD Appliance 2
namevcd-fe1vcd-fe2
eth0 ip address10.104.123.2110.104.123.22
static route10.104.123.1 10.104.126.0/2410.104.123.1 10.104.126.0/24
eth1 ip address10.104.124.2110.104.124.22

You’ll notice that the eth0 and eth1 interfaces are connected to two different management networks 10.104.123.0/24 and 10.104.124.0/24 respectively. For vCD, it is generally good practice to separate the two interfaces into separate networks.

Network nameGateway CIDRPurpose
sddc-cgw-vcd-mgmt-110.104.123.1/24vCD Frontend
UI/API/VM Remote Console
sddc-cgw-vcd-mgmt-210.104.124.1/24vCD Backend
PostgreSQL, SSH etc.

For simplicity, I also deployed my Avi controllers onto the sddc-cgw-vcd-mgmt-2 network.

The diagram below summarises the above architecture for the HTTP interface for vCD. For this guide, I’ve used VMware Cloud on AWS together with Avi Networks to protect vCD running as an appliance inside the SDDC. This is not a typical deployment model as Cloud Director Service will be able to use VMware Cloud on AWS SDDC resource soon, but I wanted to showcase the possibilities and constraints when using Avi with VMC based SDDCs.

Figure 2 . vCD HTTP Diagram

Configuring Avi for Cloud Director

After you have deployed the Avi Controllers and the Service Engines, there are few more steps needed before vCD is fully up and operational. The proceeding steps can be summarised as follows:

  1. Setup networking for the service engines by assigning the right IP address to the correct MAC addresses for the data networks
  2. Configure the network subnets for the service engines
  3. Configure static routes for the service engines to reach vCD
  4. Setup Legacy HA mode for the service engine group
  5. Setup the SSL certificate for the HTTP service
  6. Setup the Virtual Services for HTTP and Remote Console (VMRC)
  7. Setup the server pools
  8. Setup health monitors
  9. Setup HTTP security policies

Map Service Engine interfaces

Using the Avi Vantage Controller, navigate to Infrastructure > Service Engine, select one of the Service Engines then click on the little pencil icon. Then map the MAC addresses to the correct IP addresses.

Configure the network subnets for the service engines

Navigate to Infrastructure > Networks and create the subnets.

Configure static routes

Navigate to Infrastructure > Routing and setup any static routes. You’ll notice from figure 2 that since the service engine has three network interfaces on different networks, we need to create a static route on the interface that does not have the default gateway. This is so the service engines knows which gateway to use to route traffic for particular traffic types. In this case, the gateway for the service engine to route the HTTP and Remote Console traffic southbound to the vCD cells.

Setup Legacy HA mode for the service engine group

Navigate to Infrastructure > Service Engine Group.

Setup the HA mode to Legacy HA. This is the simplest configuration, you can use Elastic HA if you wish.

Configure the HTTP and Remote Console Virtual Services

Navigate to Applications > Virtual Services.

Creating a Virtual Service, has a few sub tasks which include the creation of the downstream server pools and SSL certificates.

Create a new Virtual Service for the HTTP service, this is for the Cloud Director UI and API. Please use this example to create another Virtual Service for the Remote Console.

For the Remote Console service, you will need to accept TCP 443 on the load balancer but connect southbound to the Cloud Director appliances on port TCP 8443. TCP 8443 is the port that VMRC uses as it shares the same IP addresses as the HTTP service.

You may notice that the screenshot is for an already configured Virtual Service for the vCD HTTP service. The server pool and SSL certificate is already configured. Below are the screenshots for those.

Certificate Management

You may already have a signed HTTP certificate that you wish to use with the load balancer for SSL termination. To do so, you will need to use the JAVA keytool to manipulate the HTTP certificate, obtaining the private key and convert from JCEKS to PCKS12. JAVA keytool is available in the vCD appliance at /opt/vmware/vcloud-director/jre/bin/.

Figure 3. SSL termination on load balancer

For detailed instructions on creating a signed certificate for vCD, please follow this guide.

Convert the keystore file certificates.ks file from JCEKS to PKCS12

keytool -importkeystore -srcstoretype JCEKS -srckeystore certificates.ks -destkeystore certificates_pkcs12.ks -deststoretype PKCS12

Export private key for the HTTP certificate from the certificates_pkcs12.ks file

keytool -importkeystore -srckeystore certificates_pkcs12.ks -srcalias http -destalias http -destkeystore httpcert.p12 -deststoretype PKCS12

Now that you have the private key for the HTTP certificate, you can go ahead and configure the HTTP certificate on the load balancer.

For the certificate file, you can either paste the text or upload the certificate file (.cer, .crt) from the certificate authority for the HTTP certificate.

For the Key (PEM) or PKCS12 file, you can use the httpcert.p12 file that you extracted from the certificates_pkcs12.ks file above.

The Key Passphrase is the password that you used to secure the httpcert.p12 file earlier.

Note that the vCD Remote Console (VMRC) must use pass-through for SSL termination, e.g., termination of the VMRC session must happen on the Cloud Director cell. Therefore, the above certificate management activities on Avi are not required for the VMRC.

Health Monitors

Navigate to Applications > Pools.

Edit the HTTP pool using the pencil icon and click on the Add Active Monitor green button.

Health monitoring of the HTTP service uses

GET /cloud/server_status HTTP/1.0

With an expected server response of

Service is up.

And a response code of 200.

The vCD Remote Console Health monitor is a lot simpler as you can see below.

Layer 7 HTTP Security

Layer 7 HTTP Security is very important and is highly recommended for any application exposed to the Internet. Layer 3 fire-walling and SSL certificates is always never enough in protecting and securing applications.

Navigate to Applications > Virtual Services.

Click on the pencil icon for the HTTP virtual service and then click on the Policies tab. Then click on the HTTP Security policy. Add a new policy with the following settings. You can read more about Layer 7 HTTP policies here.

Allowed StringsRequired by
/tenantTenant use
/loginLogin
/networkAccess to networking
/tenant-networkingAccess to networking
/cloudFor SAML/SSO logins
/transferUploads/Downloads of ISO and templates
/apiGeneral API access
/cloudapiGeneral API access
/docsSwagger API browser
Blocked Strings
/cloudapi/1.0.0/sessions/providerSpecifically block admin APIs from the Internet

This will drop all provider side services when accessed from the Internet. To access provider side services, such as /provider or admin APIs, use an internal connection to the Cloud Director cells.

Change Cloud Director public addresses

If not already done so, you should also change the public address settings in Cloud Director.

Testing the Cloud Director portal

Try to access https://vcloud.vmwire.com/provider

You won’t be able to access it as /provider is not on the list of allowed URI strings that we configured in the L7 HTTPS Security settings.

However, if you try to access https://vcloud.vmwire.com/tenant/vmwire, you will be able to reach the tenant portal for the organisation named VMwire.

Many thanks to Mikael Steding, our Avi Network Systems Engineer for helping me with setting this up.

Please reach out to me if you have any questions.

Securing VMware Cloud on AWS remote access to your SDDC with an SSL VPN

Not all customers want to setup site-to-site VPNs using IPSEC or Route-based VPNs between their on-premises data centre to an SDDC on VMware Cloud on AWS. Using a client VPN such as an SSL VPN to enable a client-side device to setup an SSL VPN tunnel to the SDDC.

The Use Case

What is an SSL VPN?

An SSL VPN (Secure Sockets Layer virtual private network) is a form of VPN that can be used with a standard Web browser. In contrast to the traditional Internet Protocol Security (IPsec) VPN, an SSL VPN does not require the installation of specialised client software on the end user’s computer. -www.bitpipe.com

 

Why?

  • SSL VPN is not an available feature by the Management Gateway or Compute Gateway in VMware Cloud on AWS
  • Enable client VPN connections over SSL to an SDDC in VMware Cloud on AWS for secure access to the resources
  • Avoid site-to-site VPN configurations between on-premises and the Management Gateway
  • Avoid opening vCenter to the Internet

Not all customers want to setup site-to-site VPNs using IPSEC or Route-based VPNs between their on-premises data centre to an SDDC on VMware Cloud on AWS. Using a client VPN such as an SSL VPN to enable a client-side device to setup an SSL VPN tunnel to the SDDC.

Benefits

  • Improve remote administrative security
  • Enable users to access SDDC resource including vCenter over a secure SSL VPN from anywhere with an Internet connection

Summary

This article goes through the requirements and steps needed to get OpenVPN up and running. Of course, you can use any SSL VPN software, OpenVPN is a freely available open source alternative that is quick and easy to setup and is used in this article as a working example.

Review the following basic requirements before proceeding:

  • Access to your VMware Cloud on AWS SDDC
  • Basic knowledge of Linux
  • Basic knowledge of VMware vSphere
  • Basic knowledge of firewall administration

Steps

vCenter Server

In this section you’ll deploy the OpenVPN appliance. The steps can be summarised below:

  • Download the OpenVPN appliance to the SDDC. The latest VMware version is available with this link:

https://openvpn.net/downloads/openvpn-as-latest-vmware.ova

Make a note of the IP address of the appliance, you’ll need this to NAT a public IP to this internal IP using the HTTPS service later. My appliance is using an IP of 192.168.1.201.

  • Log in as root with password of openvpnas to change a password for the openvpn user. This user is used for administering the admin web interface for OpenVPN.

VMware Cloud on AWS

In this section you’ll need to create a number of firewall rules as summarised in the tables further below.

Here’s a quick diagram to show how the components relate.

What does the workflow look like?

  1. A user connects to the SSL VPN to OpenVPN using the public IP address 3.122.197.159.
  2. HTTPS (TCP 443) is NAT’d from 3.122.197.159 to the OpenVPNAppliance with an IP of 192.168.1.201 also to the HTTPS service.
  3. OpenVPN is configured with subnets that VPN users are allowed to access. 192.168.1.0/24 and 10.71.0.0/16 are the two allowed subnets. OpenVPN configures the SSL VPN tunnel to route to these two subnets.
  4. The user can open up a browser session on his laptop and connect to vCenter server using https://10.71.224.4.

Rules Configured on Management Gateway

Rule # Rule name Source Destination Services Action
1 Allow the OpenVPN appliance to access vCenter only on port 443 OpenVPN appliance vCenter HTTPS Allow

The rule should look similar to the following.

Rules Configured on Compute Gateway

Rule # Rule name Source Destination Services Action
2 Allow port 443 access to the OpenVPN appliance Any OpenVPN appliance HTTPS Allow
3 Allow the OpenVPN-network outbound access to any destination OpenVPN-network Any Any Allow

The two rules should look similar to the following.

I won’t go into detail on how to create these rules. However, you will need to create a few User Defined Groups for some of the Source and Destination objects.

NAT Rules

Rule name Public IP Service Public Ports Internal IP Internal Ports
NAT HTTPS Public IP to OpenVPN appliance 3.122.197.159 HTTPS 443 192.168.1.201 443

You’ll need to request a new Public IP before configuring the NAT rule.

The NAT rule should look similar to the following.

OpenVPN Configuration

We need to configure OpenVPN before it will accept SSL VPN connections. Ensure you’ve gone through the initial configuration detailed in this document

https://openvpn.net/vpn-server-resources/deploying-the-access-server-appliance-on-vmware-esxi/

  • Connect to the OpenVPNAppliance VM using a web browser. The URL is for my appliance is https://192.168.1.201:943
  • Login using openvpn and use the password you set earlier.

  • Click on the Admin button

Configure Network Settings

  • Click on Network Settings and enter the public IP that was issued by VMware Cloud on AWS earlier.
  • Also, only enable the TCP daemon.

  • Leave everything else on default settings.
  • Press Save Settings at the bottom.
  • Press the Update Running Server button.

Configure Routing

  • Click on VPN Settings and enter the subnet that vCenter runs on under the Routing section. I use the Infrastructure Subnet. 10.71.0.0/16.

  • Leave all other settings default, however this depends on what you configured when you deployed the OpenVPN appliance initially. My settings are below:

  • Press Save Settings at the bottom.
  • Press the Update Running Server button.

Configure Users and Users’ access to networks

  • Click on User Permissions and add a new user
  • Click on the More Settings pencil icon and configure a password and add in the subnets that you want this user to be able to access. I am using 192.168.1.0/24 – this is the OpenVPN-network subnet and also 10.71.0.0/16 – this is the Infrastructure Subnet for vCenter, ESXi in the SDDC. This will allow clients connected through the SSL VPN to connect directly to vCenter.

If you don’t know the Infrastructure Subnet you can obtain it by going to Network & Security > Overview

  • Press Save Settings at the bottom.
  • Press the Update Running Server button.

Installing the OpenVPN SSL VPN client onto a client device

The desktop client is only required if you do not want to use the web browser to initiate the SSL VPN. Unfortunately, we need signed certificates configured on OpenVPN to use the browser. I don’t have any for this example, so we will use the desktop client to connect instead.

For this section I will use my laptop to connect to the VPN.

  • Open up a HTTPS browser session to the public IP address that was provisioned by VMware Cloud on AWS earlier. For me this is https://3.122.197.159.
  • Accept any certificates to proceed. Of course, you can use real signed certificates with your OpenVPN configuration.
  • Enter the username of the user that was created earlier, the password and select the Connect button.

  • Click on the continue link to download the SSL VPN client

  • Once downloaded, launch the installation file.
  • Once complete you can close the browser as it won’t connect automatically as we are not using signed certificates.

Connecting to the OpenVPN SSL VPN client from a client device

Now that the SSL VPN client is installed we can open an SSL VPN tunnel.

  • Launch the OpenVPNConnect client, I’m on OSX, so SPACEBAR “OpenVPNConnect” will bring up the client.
  • Once launched, you can click on the small icon at the top of your screen.

  • Connect to the public IP relevant to your OpenVPN configuration.
  • Enter the credentials then click on Connect.
  • Accept all certificate prompts and the VPN should now be connected.

Connect to vCenter

Open up a HTTPS browser session and use the internal IP address of vCenter. You may need to add a hosts file entry for the public FQDN for vCenter to redirect to the internal IP instead. That’s it! You’re now accessing vCenter over an SSL VPN.

It’s also possible to use this method to connect to other network segments. Just follow the procedures above to add additional network segments and rules in the Compute Gateway and also add additional subnets to the Access Control section when adding/editing users to OpenVPN.

Call to Action

Learn more with these resources: