Let’s Encrypt is a great service that provides free SSL certificates. I recently rebuilt my lab and decided to use SSL certs for my management appliances. However, non of the management appliances would accept the certificates issued by Let’s Encrypt due to an incomplete chain. This post summarizes how to fix this issue.
[Updated here]
TL;DR the Let’s Encrypt DST Root CA X3 certificate is missing from the fullchain.pem and chain.pem files, therefore errors such as the following prevent certificates from being imported by VMware appliances such as NSX-T and vCenter.
Certificate chain validation failed. Make sure a valid chain is provided in order leaf,intermediate,root certificate. (Error code: 2076)
Get your certbot tool up and running, you can read more with this link.
Grab your files from the /etc/letsencrypt/live folder for your vCenter certificate. My one is in /etc/letsencrypt/live/vcenter.vmwire.com
You should now have the following files.
cert.pem
chain.pem
fullchain.pem
privkey.pem
A note on Let’s Encrypt certificate chain. If you look at the Certification Path for Let’s Encrypt certificates, you’ll notice something like this.
vcenter.vmwire.com is issued by the R3 CA certificate. This is Let’s Encrypt’s Intermediate certificate.
R3 is issued by the DST Root CA X3 certificate. This is Let’s Encrypt root certificate.
Then the DST Root CA X3 certificate needs to be trusted by all of our management appliances, vCenter, NSX-T and Avi Controller.
What I found is that, this is not the case and trying to import a Let’s Encrypt certificate without the root certificate that issued the DST Root CA X3 certificate will fail. Here’s an example from NSX-T when importing the chain.pem certificate.
The chain.pem file contains the R3 certificate and the DST Root CA X3 certificate. When you open it in notepad++ it looks like this.
So we have a problem. We need the certificate that issued the DST Root CA X3 certificate to complete the chain and pass the chain validation.
Lets take a look at Let’s Encrypt certificates on their website.
So looking up the chain, it appears that my certificate vcenter.vmwire.com corresponds to the Subscriber Cert, which is issued by R3. This confirms the assumptions above in figure 1. However, it looks like the R3 certificate is not issued by the DST Root CA X3 certificate but in fact another certificate named ISRG Root X1.
Lets test this theory and import each of the certificates in the chain.pem file individually using NSX-T.
After importing, you can see that this is in fact the ISRG Root X1 certificate that is issued by the DST Root CA X3 certificate. My assumption from figure 3. is then incorrect.
So what is the top certificate in the chain.pem file?
Lets’ import it and find out. Yup, its the R3 certificate.
So where is the DST Root CA X3 certificate that we need to complete the validation chain?
We can obtain this from the Let’s Encrypt website. Scroll all the way down to the bottom of that page and you’ll see the following:
Clicking on that link will get you the the following page with this link.
And we will get closer to our DST Root CA X3 certificate when we click on that link above.
Clicking on that link gets us to this page here.
Then clicking on that link will get us to this page here.
We can now grab our certificate with this link highlighted here.
When you click on this link, you’ll be able to download a file named 8395.crt, this is the DST Root CA X3 certificate that we need to complete the chain. However, it is in a .crt format but we need to work with .pem.
To convert a crt certificate to pem use the following command.
openssl x509 -in 8395.crt -out 8395.pem -outform PEM
Once we import it into NSX-T we can see that it is indeed our missing certificate.
Looking at all the certificates in NSX-T, it all looks good.
You can see from my terrible Apple Pencil skills that it all now looks good.
So how do we use all these certificates together?
Well, the easiest is to take the fullchain.pem file, this file contains the following certificates in the following order.
Subscriber Certificate
R3 Certificate
ISRG Root X1 Certificate
That means we just need to append our new DST Root CA X3 certificate to the bottom of the fullchain.pem file to get a valid chain. It will now look like this.
Subscriber Certificate
R3 Certificate
ISRG Root X1 Certificate
DST Root CA X3 Certificate
Once we import this new fullchain into NSX-T in goes through successfully.
Now that we have a working fullchain.pem, we can use that file or its contents to import these signed certificates into our management appliances.
Read on – the following articles summarize how to use these certificates with vCenter, NSX-T Manager and the Avi Controller.
Updating vCenter to use Let’s Encrypt certificates.
Updating NSX-T to use Let’s Encrypt certificates.
Updating Avi Controller to use Let’s Encrypt certificates.
VMwire 