Replacing NSX-T Controller SSL Certificates with Lets Encrypt

Another certificate management post as Lets Encrypt have removed their old root certificates. This post shows how to update signed certificates for NSX-T.

NSX-T checks certificate chains when importing certificates, and unless the full chain of trust is available, you will not be able to use the certificate with NSX-T.

Another certificate management post as Lets Encrypt have removed their old root certificates. This post shows how to update signed certificates for NSX-T.

NSX-T checks certificate chains when importing certificates, and unless the full chain of trust is available, you will not be able to use the certificate with NSX-T.

This link here shows the chain of trust for Lets Encrypt certificates.

https://letsencrypt.org/certificates/

The certificate chain for Lets Encrypt is as follows:

your-certificate -> R3 -> ISRG Root X1

Your certificate is delivered to you after you request a certificate using Lets Encrypt services, the file that contains your certificate is named cert.pem.

The R3 certificate can be downloaded with this link

https://letsencrypt.org/certs/lets-encrypt-r3.pem

The ISRG Root X1 certificate can be downloaded with this link

https://letsencrypt.org/certs/isrgrootx1.pem

To create a complete chain that NSX-T can accept, combine the contents of the files above into a single file in this order: cert, R3, X1. Like this

-----BEGIN CERTIFICATE-----
cert.pem content
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
R3.pem content
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
X1.pem content
-----END CERTIFICATE-----

Now you can use this new file to import and use this new certificate in NSX-T. It will look similar to this if successful.

Updating Let’s Encrypt SSL Certificates for NSX-T Manager

Updating Let’s Encrypt SSL Certificates for NSX-T Manager

Updating NSX-T Manager to use a CA signed SSL certificate is a little bit different from how we updated the vCenter certificate. It requires interacting with the NSX-T API.

First lets import the certificate into NSX-T. Again, you’ll need the fullchain.pem file but with the appended DST Root CA X3 certificate that was prepared in this article.

Navigate to System and then under Settings, click on the Certificates link.

First we need to import each of the CA certificates in the chain before we import the certificate for NSX-T Manager.

Again the certificates in the fullchain.pem file in order are

Subscriber Certificate
R3 Certificate
ISRG Root X1 Certificate
DST Root CA X3 Certificate

Click on IMPORT, Import CA Certificate. Then import each certificate individually starting from the bottom, make sure to deselect the Service Certificate slider, as we are not using these certificates for virtual services.

Its important to import bottom up as this enables NSX-T to check the issuer for subsequent certificates that you import. So import in reverse order of the fullchain.pem file. Start importing with this order

DST Root CA X3 Certificate
ISRG Root X1 Certificate
R3 Certificate
Subscriber Certificate

Once you’ve imported all three of the CA root and intermediate certificates – DST Root CA X3 certificate, ISRG Root X1 CA and the R3 CA certificate, you can then import the Subscriber Certificate *.vmwire.com last, once all done you’ll see the following.

Summarized in the following table.

Order in fullchain.pemName in NSX-TIssued By
Subscriber Certificatestar-vmwire-comR3
R3 CertificateR3ISRG Root X1
ISRG Root X1 CertificateISRG Root X1DST Root CA X3
DST Root CA X3 CertificateDST Root CA X3DST Root CA X3

You’ll need the certificate ID for the certificate star-vmwire-com to use to update the NSX-T Manager certificate.

Click on the ID column of that certificate and copy the ID to your clipboard.

Now you’ll need to open a tool such as Postman to make the change.

First lets validate that our certificate is OK by using this GET against the NSX-T API, paste in the certificate ID into the URL.

GET https://nsx.vmwire.com/api/v1/trust-management/certificates/21fd7e8a-3a2e-4938-9dc7-5f3eccd791e7/?action=validate

If the status is “OK”, we’re good to continue.

Next use will POST the certificate ID against the following URL.

POST https://nsx.vmwire.com/api/v1/node/services/http?action=apply_certificate&certificate_id=21fd7e8a-3a2e-4938-9dc7-5f3eccd791e7

Once done, close your NSX-T Manager browser session, and enjoy using a CA signed certificate with NSX-T.

Preparing Let’s Encrypt SSL Certificates for vCenter, NSX-T Manager and Avi Controller

The Let’s Encrypt DST Root CA X3 certificate is missing from the fullchain.pem and chain.pem files, therefore errors such as the following prevent certificates from being imported by VMware appliances such as NSX-T and vCenter.

This post summarizes how to fix this issue.

Let’s Encrypt is a great service that provides free SSL certificates. I recently rebuilt my lab and decided to use SSL certs for my management appliances. However, non of the management appliances would accept the certificates issued by Let’s Encrypt due to an incomplete chain. This post summarizes how to fix this issue.

[Updated here]

TL;DR the Let’s Encrypt DST Root CA X3 certificate is missing from the fullchain.pem and chain.pem files, therefore errors such as the following prevent certificates from being imported by VMware appliances such as NSX-T and vCenter.

Certificate chain validation failed. Make sure a valid chain is provided in order leaf,intermediate,root certificate. (Error code: 2076)

Get your certbot tool up and running, you can read more with this link.

Grab your files from the /etc/letsencrypt/live folder for your vCenter certificate. My one is in /etc/letsencrypt/live/vcenter.vmwire.com

You should now have the following files.

cert.pem
chain.pem
fullchain.pem
privkey.pem

A note on Let’s Encrypt certificate chain. If you look at the Certification Path for Let’s Encrypt certificates, you’ll notice something like this.

figure 1.

vcenter.vmwire.com is issued by the R3 CA certificate. This is Let’s Encrypt’s Intermediate certificate.

R3 is issued by the DST Root CA X3 certificate. This is Let’s Encrypt root certificate.

Then the DST Root CA X3 certificate needs to be trusted by all of our management appliances, vCenter, NSX-T and Avi Controller.

What I found is that, this is not the case and trying to import a Let’s Encrypt certificate without the root certificate that issued the DST Root CA X3 certificate will fail. Here’s an example from NSX-T when importing the chain.pem certificate.

figure 2. Importing the chain.pem certificate to NSX

The chain.pem file contains the R3 certificate and the DST Root CA X3 certificate. When you open it in notepad++ it looks like this.

figure 3. chain.pem

So we have a problem. We need the certificate that issued the DST Root CA X3 certificate to complete the chain and pass the chain validation.

Lets take a look at Let’s Encrypt certificates on their website.

ISRG Certificate Hierarchy Diagram, as of December 2020

So looking up the chain, it appears that my certificate vcenter.vmwire.com corresponds to the Subscriber Cert, which is issued by R3. This confirms the assumptions above in figure 1. However, it looks like the R3 certificate is not issued by the DST Root CA X3 certificate but in fact another certificate named ISRG Root X1.

Lets test this theory and import each of the certificates in the chain.pem file individually using NSX-T.

After importing, you can see that this is in fact the ISRG Root X1 certificate that is issued by the DST Root CA X3 certificate. My assumption from figure 3. is then incorrect.

So what is the top certificate in the chain.pem file?

Lets’ import it and find out. Yup, its the R3 certificate.

So where is the DST Root CA X3 certificate that we need to complete the validation chain?

We can obtain this from the Let’s Encrypt website. Scroll all the way down to the bottom of that page and you’ll see the following:

Clicking on that link will get you the the following page with this link.

And we will get closer to our DST Root CA X3 certificate when we click on that link above.

Clicking on that link gets us to this page here.

Then clicking on that link will get us to this page here.

We can now grab our certificate with this link highlighted here.

When you click on this link, you’ll be able to download a file named 8395.crt, this is the DST Root CA X3 certificate that we need to complete the chain. However, it is in a .crt format but we need to work with .pem.

To convert a crt certificate to pem use the following command.

openssl x509 -in 8395.crt -out 8395.pem -outform PEM

Once we import it into NSX-T we can see that it is indeed our missing certificate.

Looking at all the certificates in NSX-T, it all looks good.

You can see from my terrible Apple Pencil skills that it all now looks good.

So how do we use all these certificates together?

Well, the easiest is to take the fullchain.pem file, this file contains the following certificates in the following order.

Subscriber Certificate
R3 Certificate
ISRG Root X1 Certificate

That means we just need to append our new DST Root CA X3 certificate to the bottom of the fullchain.pem file to get a valid chain. It will now look like this.

Subscriber Certificate
R3 Certificate
ISRG Root X1 Certificate
DST Root CA X3 Certificate

Once we import this new fullchain into NSX-T in goes through successfully.

Now that we have a working fullchain.pem, we can use that file or its contents to import these signed certificates into our management appliances.

Read on – the following articles summarize how to use these certificates with vCenter, NSX-T Manager and the Avi Controller.

Updating vCenter to use Let’s Encrypt certificates.

Updating NSX-T to use Let’s Encrypt certificates.

Updating Avi Controller to use Let’s Encrypt certificates.