Updating Let’s Encrypt SSL Certificates for vCenter Server

Updating Let’s Encrypt SSL Certificates for vCenter Server

I prefer to use wildcard certificates for my environment to reduce the number of certificates that I need to manage. This is due to Let’s Encrypt limiting their certificates to 90 days. This means that you’ll need to renew each certificate every <90 days or so. Using a wildcard certificate reduces your operational overhead. However, vCenter does not support wildcard certificates.

After you’ve prepped the fullchain.pem file according to the previous article, you can now update the vCenter SSL certificate using vCenter’s Certificate Management tool.

Navigate to Menu then Administration and click on Certificate Management.

Under the Machine SSL Certificate, click on Actions and choose Import and Replace Certificate.

Select the Replace with external CA certificate (requires private key).

Copy the section for the Subscriber Certificate part into the Machine SSL Certificate box, and then the rest into the Chain of trusted root certificates box.

Copy the contents of the privkey.pem file into the Private Key box.

Once you click on Replace, vCenter will restart its services and you can open a new browser window to the FQDN of vCenter and enjoy a secured vCenter session.

Preparing Let’s Encrypt SSL Certificates for vCenter, NSX-T Manager and Avi Controller

The Let’s Encrypt DST Root CA X3 certificate is missing from the fullchain.pem and chain.pem files, therefore errors such as the following prevent certificates from being imported by VMware appliances such as NSX-T and vCenter.

This post summarizes how to fix this issue.

Let’s Encrypt is a great service that provides free SSL certificates. I recently rebuilt my lab and decided to use SSL certs for my management appliances. However, non of the management appliances would accept the certificates issued by Let’s Encrypt due to an incomplete chain. This post summarizes how to fix this issue.

TL;DR the Let’s Encrypt DST Root CA X3 certificate is missing from the fullchain.pem and chain.pem files, therefore errors such as the following prevent certificates from being imported by VMware appliances such as NSX-T and vCenter.

Certificate chain validation failed. Make sure a valid chain is provided in order leaf,intermediate,root certificate. (Error code: 2076)

Get your certbot tool up and running, you can read more with this link.

Grab your files from the /etc/letsencrypt/live folder for your vCenter certificate. My one is in /etc/letsencrypt/live/vcenter.vmwire.com

You should now have the following files.

cert.pem
chain.pem
fullchain.pem
privkey.pem

A note on Let’s Encrypt certificate chain. If you look at the Certification Path for Let’s Encrypt certificates, you’ll notice something like this.

figure 1.

vcenter.vmwire.com is issued by the R3 CA certificate. This is Let’s Encrypt’s Intermediate certificate.

R3 is issued by the DST Root CA X3 certificate. This is Let’s Encrypt root certificate.

Then the DST Root CA X3 certificate needs to be trusted by all of our management appliances, vCenter, NSX-T and Avi Controller.

What I found is that, this is not the case and trying to import a Let’s Encrypt certificate without the root certificate that issued the DST Root CA X3 certificate will fail. Here’s an example from NSX-T when importing the chain.pem certificate.

figure 2. Importing the chain.pem certificate to NSX

The chain.pem file contains the R3 certificate and the DST Root CA X3 certificate. When you open it in notepad++ it looks like this.

figure 3. chain.pem

So we have a problem. We need the certificate that issued the DST Root CA X3 certificate to complete the chain and pass the chain validation.

Lets take a look at Let’s Encrypt certificates on their website.

ISRG Certificate Hierarchy Diagram, as of December 2020

So looking up the chain, it appears that my certificate vcenter.vmwire.com corresponds to the Subscriber Cert, which is issued by R3. This confirms the assumptions above in figure 1. However, it looks like the R3 certificate is not issued by the DST Root CA X3 certificate but in fact another certificate named ISRG Root X1.

Lets test this theory and import each of the certificates in the chain.pem file individually using NSX-T.

After importing, you can see that this is in fact the ISRG Root X1 certificate that is issued by the DST Root CA X3 certificate. My assumption from figure 3. is then incorrect.

So what is the top certificate in the chain.pem file?

Lets’ import it and find out. Yup, its the R3 certificate.

So where is the DST Root CA X3 certificate that we need to complete the validation chain?

We can obtain this from the Let’s Encrypt website. Scroll all the way down to the bottom of that page and you’ll see the following:

Clicking on that link will get you the the following page with this link.

And we will get closer to our DST Root CA X3 certificate when we click on that link above.

Clicking on that link gets us to this page here.

Then clicking on that link will get us to this page here.

We can now grab our certificate with this link highlighted here.

When you click on this link, you’ll be able to download a file named 8395.crt, this is the DST Root CA X3 certificate that we need to complete the chain. However, it is in a .crt format but we need to work with .pem.

To convert a crt certificate to pem use the following command.

openssl x509 -in 8395.crt -out 8395.pem -outform PEM

Once we import it into NSX-T we can see that it is indeed our missing certificate.

Looking at all the certificates in NSX-T, it all looks good.

You can see from my terrible Apple Pencil skills that it all now looks good.

So how do we use all these certificates together?

Well, the easiest is to take the fullchain.pem file, this file contains the following certificates in the following order.

Subscriber Certificate
R3 Certificate
ISRG Root X1 Certificate

That means we just need to append our new DST Root CA X3 certificate to the bottom of the fullchain.pem file to get a valid chain. It will now look like this.

Subscriber Certificate
R3 Certificate
ISRG Root X1 Certificate
DST Root CA X3 Certificate

Once we import this new fullchain into NSX-T in goes through successfully.

Now that we have a working fullchain.pem, we can use that file or its contents to import these signed certificates into our management appliances.

Read on – the following articles summarize how to use these certificates with vCenter, NSX-T Manager and the Avi Controller.

Updating vCenter to use Let’s Encrypt certificates.

Updating NSX-T to use Let’s Encrypt certificates.

Updating Avi Controller to use Let’s Encrypt certificates.

Configure NFS Storage on the VMware vCenter Server Appliance

This post highlights some best practices on the management of the vCSA log and core files. VMware recommends that these files are stored on an NFS share external to the vCSA due to the possibility of the default log and core locations filling up.

When this happens, vCenter services will be impacted.

For more information about the vCSA, please see the resources listed here https://vmwire.com/vmware-vcenter-server-virtual-appliance-vcsa/.

There may be trouble ahead

This screenshot shows what happens when this is not done, the partitions for /storage/core will fill up over time and will impact the availability of vCenter Server.

Figure 1 – Local core storage full!

Configuring NFS storage on the vCSA

You can add the NFS shares for the log and core files by logging into the VMware Studio management interface of the vCSA, normally https://<vcsa>:5480.

The default username and password is root | vmware.

Click on the vCenter Server tab, and then click on Storage.

Figure 2 – Configuring NFS storage on the vCSA

Using the correct syntax for the NFS storage

The correct syntax for adding the storage is

<NFS_Server>:<NFS_Export>

So if my NFS_Server is 192.168.200.21 and my NFS_Export is /mnt/vg01/vcsa_core/vcsa_core/, I would enter the following in the box for NFS share for core files:

192.168.200.21:/mnt/vg01/vcsa_core/vcsa_core/

Make sure that the NFS export on the NFS Server is configured with a UID/GID mapping of no_root_squash. For example, use the command on the NFS server:

exportfs -vo rw,no_root_squash,sync :/mnt/vg01/vcsa_core/vcsa_core/

Once done, click on Test Settings to verify that the vCSA can successfully store files to the specified NFS shares, then click on Save Settings, then restart the vCSA.

Browsing to the NFS storage

You can also see what is created in the NFS share if you listed the contents of the core files share.

Figure 3 – Core logs

You can also see what is created in the NFS share if you listed the contents of the log files share. The screenshots below show the directory structure on the NFS server. On the vCSA the directories are mounted at /storage.

Figure 4 – All other Logs

Configuring vCenter Server Virtual Appliance to use an Oracle database

In previous posts I blogged about what the vCenter Server Virtual Appliance (vCSA) is, its features and benefits, feature parity with the Windows vCenter Server and also how to quickly deploy the vCSA. For more information about the vCSA, please see the resources listed here https://vmwire.com/vmware-vcenter-server-virtual-appliance-vcsa/.

This post extends the series with how to configure an external Oracle database for use by the vCSA.

Why use an Oracle database?

The vCSA comes preinstalled with an embedded DB2 database which has similar use cases as the Windows vCenter Server when configured with SQL Express – intended for small deployments of 5 ESX/ESXi servers or less. The ability for the vCSA to utilise an external Oracle database allows customers to scale and manage larger vSphere infrastructures equivalent to environments with Windows vCenter Servers backed by SQL or Oracle databases.

This post shows how quickly and easily it is to use an external Oracle database instead of the embedded DB2 database. Hopefully you’ll see the benefits of how much quicker it is to configure the Oracle connectivity between the vCSA and the Oracle server vs installing the Oracle 64-bit Client onto a Window Server and configuring tnsnames.ora, followed by configuration of ODBC settings.

Configure an Oracle Database and User

  1. Log into SQL*Plus session with the system account. I’m using Oracle 11g R2 x64 on Windows Server 2008.
    C:`>sqlplus sys/<password> as SYSDBA
  2. Run the following SQL commands to create a vCenter Server database. Note that your directory structure may be different.

    CREATE SMALLFILE TABLESPACE “VPX” DATAFILE ‘e:/app/oracle/oradata/orcl/vpx01.dbf’ SIZE 1G AUTOEXTEND ON NEXT 10M MAXSIZE UNLIMITED LOGGING EXTENT MANAGEMENT LOCAL SEGMENT SPACE MANAGEMENT AUTO;

  3. Run the following SQL command to create a vCenter Server database user with the correct permissions. I will create a new user named “VPXADMIN” with a password of “oracle”.
    CREATE USER "VPXADMIN" PROFILE "DEFAULT" IDENTIFIED BY "oracle" DEFAULT TABLESPACE "VPX" ACCOUNT UNLOCK;
    grant connect to VPXADMIN;
    grant resource to VPXADMIN;
    grant create view to VPXADMIN;
    grant create sequence to VPXADMIN; 
    grant create table to VPXADMIN; 
    grant create materialized view to VPXADMIN;
    grant execute on dbms_lock to VPXADMIN;
    grant execute on dbms_job to VPXADMIN;
    grant select on dba_tablespaces to VPXADMIN;
    grant select on dba_temp_files to VPXADMIN;
    grant select on dba_data_files to VPXADMIN;
    grant unlimited tablespace to VPXADMIN;

Configure the vCSA

  1. Log into the vCSA VMware Studio management interface at https://<vcsa>:5480/
  2. Navigate to the vCenter Server tab, then click on Database.
  3. Select oracle as the Database Type using the drop-down menu and enter your environment information into the fields and then click on Save Settings. Note how easy that was, no messing about with installing the Oracle Client, no need to configure tnsnames.ora and no need for any ODBC configuration either.

  4. Wait for around 5 minutes for the vCSA to create the database schema.
  5. Now it’s safe to start the vCenter services, navigate to the Status tab and click on Start vCenter.

  6. You can then start using vCenter when the Service Status reports as Running.

Cleaning up the Oracle configuration

After you’ve tested that everything is working, you can revoke the following privileges using SQL*Plus again.

revoke select on dba_tablespaces from VPXADMIN;
revoke select on dba_temp_files from VPXADMIN;
revoke select on dba_data_files from VPXADMIN;

Total configuration time ~approx 10 minutes.

References

vSphere Installation and Setup Guide

VMware vCenter Server Virtual Appliance (vCSA) Feature Parity

In a previous article I wrote about the vCSA’s features and benefits.  This post lists the interoperability or feature parity of the vCSA and the Windows vCenter Server.  For more information about the vCSA, please see the resources listed here https://vmwire.com/vmware-vcenter-server-virtual-appliance-vcsa/.

A few readers have asked what works with the vCSA and what does not.

The vCSA supports all vCenter features – DRS, SDRS, HA, Host Profiles, dvSwitches, etc.

Secondary architecture features like supported DB, View Composer are not yet at feature parity with the Windows vCenter Server.

Not supported yet:

  • Microsoft SQL as the database for vCenter – requires stable ODBC driver for Linux that can scale.
  • vCenter Server Linked Mode – requires ADAM.
  • vCenter Server Heartbeat – requires Windows.
  • IPv6.
  • Single sign-on using Windows session credentials.
  • VMware View Composer (Linked Clones) – installed on Windows vCenter Server only.
  • vSphere Storage Appliance – VSA Manager & VSA Cluster Server installed on Windows vCenter Server.
  • VIX Plugin for vCenter Orchestrator – VMware Tools API only works with Windows vCenter Server.

Other VMware products that work with the vCSA:

  • vCenter Operations.
  • vCenter Orchestrator.
  • vCenter CapacityIQ.
  • SRM5.
  • VMware View 5 (no Linked Clones).
  • Auto Deploy.
  • vCenter Update Manager.
  • vMA.
  • vSphere Client.
  • vSphere Web Client.
  • VMware vCloud Director.
  • PowerCLI.
  • vSphere Client for iPad & vCMA.

If I find anything else, I’ll update the article.

VMware vCenter Server Virtual Appliance (vCSA) features and benefits

The VMware vCenter Server Virtual Appliance (vCSA) provides an alternative option for organizations that chose not to run the Windows vCenter Server but still require centralised management of VMware vSphere deployments in the enterprise.

It provides exactly the same functionality as the traditional Windows vCenter Server but packaged in a Linux distribution. I know that some of my pure UNIX and LINUX customers have been asking for this for a while.

It’s been available as a technology preview since 2009 as “vCenter 2.5 on Linux” but has finally arrived with vSphere 5 to give customers’ an alternative to the Windows vCenter Server. Expect to see it available for download when vSphere 5 goes GA.

*UPDATE* vSphere5 is now GA, and the vCSA is available to download here.

For more information about the vCSA, please see the resources listed here https://vmwire.com/vmware-vcenter-server-virtual-appliance-vcsa/.

I’ve been using it for a while now in the lab and have found it very easy to deploy and use. vCenter services start a lot quicker and the user experience with the VMware vSphere Client is exactly the same.

vCenter Server Virtual Appliance features and benefits

  • Installed on SUSE Linux Enterprise Server 11 x64.
  • OVF when deployed is configured with 2vCPUs and 8Gb memory, LSI Logic Parallel, VMXNET 3, 15Gb and 60Gb VMDKs and VMware Tools.
  • Includes embedded DB2 database that is suitable for evaluation or for environments with less than 5 ESXi hosts or 50 virtual machines (equivalent to Windows vCenter Server + MSSQL Express).
  • Supports external Oracle database for large environments.
  • Includes Active Directory (AD) and Network Information Services (NIS) authentication.
  • vSphere Web Client support is built into the vCenter Server Virtual Appliance. vSphere Web Client is OS agnostic and the interface is highly customisable.
  • Windows vSphere Client is still supported.
  • Includes a pre-configured Auto Deploy server therefore reducing operational costs with the installation of Auto Deploy.
  • Can use NFS mounts to store vCenter Server Virtual Appliance core and log files.
  • vCSA can act as a syslog server for ESXi system logs.
  • Can be used as a network collector for ESXi kernel core dumps.
  • Simplified and rapid deployment, approximately 15 minutes deployment time.
  • Lower TCO by eliminating Windows OS dependency and licenses.
  • Reduces operational costs – vCSA is easier to upgrade – just deploy a new appliance and connect to the external Oracle database or
  • Import configuration data from previous installation.
  • Patches can be installed using the vCSA web interface.

Not yet feature parity with Windows vCenter Server

vCenter Server Virtual Appliance provides all features as the Windows vCenter Server but does not support the following features:

  • Microsoft SQL as the database for vCenter.
  • vCenter Server Linked Mode.
  • vCenter Server Heartbeat.
  • IPv6.

For details on what products are supported with the vCSA please see this post.

I’ve provided a quick start guide including a 10-minute how-to video demonstrating the deployment and administration in this post.