Another certificate management post as Lets Encrypt have removed their old root certificates. This post shows how to update signed certificates for NSX-T.
NSX-T checks certificate chains when importing certificates, and unless the full chain of trust is available, you will not be able to use the certificate with NSX-T.
Another certificate management post as Lets Encrypt have removed their old root certificates. This post shows how to update signed certificates for NSX-T.
NSX-T checks certificate chains when importing certificates, and unless the full chain of trust is available, you will not be able to use the certificate with NSX-T.
This link here shows the chain of trust for Lets Encrypt certificates.
The certificate chain for Lets Encrypt is as follows:
your-certificate -> R3 -> ISRG Root X1
Your certificate is delivered to you after you request a certificate using Lets Encrypt services, the file that contains your certificate is named cert.pem.
The R3 certificate can be downloaded with this link
Updating Let’s Encrypt SSL Certificates for NSX-T Manager
Updating NSX-T Manager to use a CA signed SSL certificate is a little bit different from how we updated the vCenter certificate. It requires interacting with the NSX-T API.
First lets import the certificate into NSX-T. Again, you’ll need the fullchain.pem file but with the appended DST Root CA X3 certificate that was prepared in this article.
Navigate to System and then under Settings, click on the Certificates link.
First we need to import each of the CA certificates in the chain before we import the certificate for NSX-T Manager.
Again the certificates in the fullchain.pem file in order are
Click on IMPORT, Import CA Certificate. Then import each certificate individually starting from the bottom, make sure to deselect the Service Certificate slider, as we are not using these certificates for virtual services.
Its important to import bottom up as this enables NSX-T to check the issuer for subsequent certificates that you import. So import in reverse order of the fullchain.pem file. Start importing with this order
Once you’ve imported all three of the CA root and intermediate certificates – DST Root CA X3 certificate, ISRG Root X1 CA and the R3 CA certificate, you can then import the Subscriber Certificate *.vmwire.com last, once all done you’ll see the following.
Summarized in the following table.
Order in fullchain.pem
Name in NSX-T
Issued By
Subscriber Certificate
star-vmwire-com
R3
R3 Certificate
R3
ISRG Root X1
ISRG Root X1 Certificate
ISRG Root X1
DST Root CA X3
DST Root CA X3 Certificate
DST Root CA X3
DST Root CA X3
You’ll need the certificate ID for the certificate star-vmwire-com to use to update the NSX-T Manager certificate.
Click on the ID column of that certificate and copy the ID to your clipboard.
Now you’ll need to open a tool such as Postman to make the change.
First lets validate that our certificate is OK by using this GET against the NSX-T API, paste in the certificate ID into the URL.
GET https://nsx.vmwire.com/api/v1/trust-management/certificates/21fd7e8a-3a2e-4938-9dc7-5f3eccd791e7/?action=validate
If the status is “OK”, we’re good to continue.
Next use will POST the certificate ID against the following URL.
POST https://nsx.vmwire.com/api/v1/node/services/http?action=apply_certificate&certificate_id=21fd7e8a-3a2e-4938-9dc7-5f3eccd791e7
Once done, close your NSX-T Manager browser session, and enjoy using a CA signed certificate with NSX-T.
The Let’s Encrypt DST Root CA X3 certificate is missing from the fullchain.pem and chain.pem files, therefore errors such as the following prevent certificates from being imported by VMware appliances such as NSX-T and vCenter.
This post summarizes how to fix this issue.
Let’s Encrypt is a great service that provides free SSL certificates. I recently rebuilt my lab and decided to use SSL certs for my management appliances. However, non of the management appliances would accept the certificates issued by Let’s Encrypt due to an incomplete chain. This post summarizes how to fix this issue.
TL;DR the Let’s Encrypt DST Root CA X3 certificate is missing from the fullchain.pem and chain.pem files, therefore errors such as the following prevent certificates from being imported by VMware appliances such as NSX-T and vCenter.
Certificate chain validation failed. Make sure a valid chain is provided in order leaf,intermediate,root certificate. (Error code: 2076)
Get your certbot tool up and running, you can read more with this link.
Grab your files from the /etc/letsencrypt/live folder for your vCenter certificate. My one is in /etc/letsencrypt/live/vcenter.vmwire.com
You should now have the following files.
cert.pem
chain.pem
fullchain.pem
privkey.pem
A note on Let’s Encrypt certificate chain. If you look at the Certification Path for Let’s Encrypt certificates, you’ll notice something like this.
figure 1.
vcenter.vmwire.com is issued by the R3 CA certificate. This is Let’s Encrypt’s Intermediate certificate.
R3 is issued by the DST Root CA X3 certificate. This is Let’s Encrypt root certificate.
Then the DST Root CA X3 certificate needs to be trusted by all of our management appliances, vCenter, NSX-T and Avi Controller.
What I found is that, this is not the case and trying to import a Let’s Encrypt certificate without the root certificate that issued the DST Root CA X3 certificate will fail. Here’s an example from NSX-T when importing the chain.pem certificate.
figure 2. Importing the chain.pem certificate to NSX
The chain.pem file contains the R3 certificate and the DST Root CA X3 certificate. When you open it in notepad++ it looks like this.
figure 3. chain.pem
So we have a problem. We need the certificate that issued the DST Root CA X3 certificate to complete the chain and pass the chain validation.
Lets take a look at Let’s Encrypt certificates on their website.
So looking up the chain, it appears that my certificate vcenter.vmwire.com corresponds to the Subscriber Cert, which is issued by R3. This confirms the assumptions above in figure 1. However, it looks like the R3 certificate is not issued by the DST Root CA X3 certificate but in fact another certificate named ISRG Root X1.
Lets test this theory and import each of the certificates in the chain.pem file individually using NSX-T.
After importing, you can see that this is in fact the ISRG Root X1 certificate that is issued by the DST Root CA X3 certificate. My assumption from figure 3. is then incorrect.
So what is the top certificate in the chain.pem file?
Lets’ import it and find out. Yup, its the R3 certificate.
So where is the DST Root CA X3 certificate that we need to complete the validation chain?
We can obtain this from the Let’s Encrypt website. Scroll all the way down to the bottom of that page and you’ll see the following:
Clicking on that link will get you the the following page with this link.
And we will get closer to our DST Root CA X3 certificate when we click on that link above.
Clicking on that link gets us to this page here.
Then clicking on that link will get us to this page here.
We can now grab our certificate with this link highlighted here.
When you click on this link, you’ll be able to download a file named 8395.crt, this is the DST Root CA X3 certificate that we need to complete the chain. However, it is in a .crt format but we need to work with .pem.
To convert a crt certificate to pem use the following command.
That means we just need to append our new DST Root CA X3 certificate to the bottom of the fullchain.pem file to get a valid chain. It will now look like this.
VMwire 