Preparing Let’s Encrypt SSL Certificates for vCenter, NSX-T Manager and Avi Controller

The Let’s Encrypt DST Root CA X3 certificate is missing from the fullchain.pem and chain.pem files, therefore errors such as the following prevent certificates from being imported by VMware appliances such as NSX-T and vCenter.

This post summarizes how to fix this issue.

Let’s Encrypt is a great service that provides free SSL certificates. I recently rebuilt my lab and decided to use SSL certs for my management appliances. However, non of the management appliances would accept the certificates issued by Let’s Encrypt due to an incomplete chain. This post summarizes how to fix this issue.

TL;DR the Let’s Encrypt DST Root CA X3 certificate is missing from the fullchain.pem and chain.pem files, therefore errors such as the following prevent certificates from being imported by VMware appliances such as NSX-T and vCenter.

Certificate chain validation failed. Make sure a valid chain is provided in order leaf,intermediate,root certificate. (Error code: 2076)

Get your certbot tool up and running, you can read more with this link.

Grab your files from the /etc/letsencrypt/live folder for your vCenter certificate. My one is in /etc/letsencrypt/live/vcenter.vmwire.com

You should now have the following files.

cert.pem
chain.pem
fullchain.pem
privkey.pem

A note on Let’s Encrypt certificate chain. If you look at the Certification Path for Let’s Encrypt certificates, you’ll notice something like this.

figure 1.

vcenter.vmwire.com is issued by the R3 CA certificate. This is Let’s Encrypt’s Intermediate certificate.

R3 is issued by the DST Root CA X3 certificate. This is Let’s Encrypt root certificate.

Then the DST Root CA X3 certificate needs to be trusted by all of our management appliances, vCenter, NSX-T and Avi Controller.

What I found is that, this is not the case and trying to import a Let’s Encrypt certificate without the root certificate that issued the DST Root CA X3 certificate will fail. Here’s an example from NSX-T when importing the chain.pem certificate.

figure 2. Importing the chain.pem certificate to NSX

The chain.pem file contains the R3 certificate and the DST Root CA X3 certificate. When you open it in notepad++ it looks like this.

figure 3. chain.pem

So we have a problem. We need the certificate that issued the DST Root CA X3 certificate to complete the chain and pass the chain validation.

Lets take a look at Let’s Encrypt certificates on their website.

ISRG Certificate Hierarchy Diagram, as of December 2020

So looking up the chain, it appears that my certificate vcenter.vmwire.com corresponds to the Subscriber Cert, which is issued by R3. This confirms the assumptions above in figure 1. However, it looks like the R3 certificate is not issued by the DST Root CA X3 certificate but in fact another certificate named ISRG Root X1.

Lets test this theory and import each of the certificates in the chain.pem file individually using NSX-T.

After importing, you can see that this is in fact the ISRG Root X1 certificate that is issued by the DST Root CA X3 certificate. My assumption from figure 3. is then incorrect.

So what is the top certificate in the chain.pem file?

Lets’ import it and find out. Yup, its the R3 certificate.

So where is the DST Root CA X3 certificate that we need to complete the validation chain?

We can obtain this from the Let’s Encrypt website. Scroll all the way down to the bottom of that page and you’ll see the following:

Clicking on that link will get you the the following page with this link.

And we will get closer to our DST Root CA X3 certificate when we click on that link above.

Clicking on that link gets us to this page here.

Then clicking on that link will get us to this page here.

We can now grab our certificate with this link highlighted here.

When you click on this link, you’ll be able to download a file named 8395.crt, this is the DST Root CA X3 certificate that we need to complete the chain. However, it is in a .crt format but we need to work with .pem.

To convert a crt certificate to pem use the following command.

openssl x509 -in 8395.crt -out 8395.pem -outform PEM

Once we import it into NSX-T we can see that it is indeed our missing certificate.

Looking at all the certificates in NSX-T, it all looks good.

You can see from my terrible Apple Pencil skills that it all now looks good.

So how do we use all these certificates together?

Well, the easiest is to take the fullchain.pem file, this file contains the following certificates in the following order.

Subscriber Certificate
R3 Certificate
ISRG Root X1 Certificate

That means we just need to append our new DST Root CA X3 certificate to the bottom of the fullchain.pem file to get a valid chain. It will now look like this.

Subscriber Certificate
R3 Certificate
ISRG Root X1 Certificate
DST Root CA X3 Certificate

Once we import this new fullchain into NSX-T in goes through successfully.

Now that we have a working fullchain.pem, we can use that file or its contents to import these signed certificates into our management appliances.

Read on – the following articles summarize how to use these certificates with vCenter, NSX-T Manager and the Avi Controller.

Updating vCenter to use Let’s Encrypt certificates.

Updating NSX-T to use Let’s Encrypt certificates.

Updating Avi Controller to use Let’s Encrypt certificates.

My VCDX Journey – 5 simple steps to VCDX

I’ve just recently been awarded the VCDX4 certification after completing my defence in Frankfurt. It is part of the final stage in the VCDX certification culminating in a journey over the past year. Defence experiences have been shared by others such as Duncan Epping, Jason Boche, Scott Lowe and Kenneth van Ditmarsch and I found that mine was very similar so this is a post on how I prepared for my VCDX and by careful planning how it can be achieved within 12 months.

For information regarding the VCDX certification path, please see the VCDX page on VMware.com.

First a quick thanks to all those that helped in true Oscar style, namely Steve Byrne my manager at VMware for supporting my journey, my colleagues at VMware for your help with the mock panels, you were awesome – @simonlong_, @repping, @ady189, @baecke & John Pollard. A shout out to @frankdenneman for the motivational support and advice.

Fail to plan? Then plan to fail, preparation is key, so this was how I planned my journey in 5 easy steps.

Step 1 – Gain support from your employer and family

This is critical as the certification path is not an easy one, there is a minimum of one course to attend (vSphere ICM), three exams (VCP, VCAP-DCA, VCAP-DCD) and fees for the VCDX submission and defence. Not to mention the expenses of travelling to the defences themselves. It’s also good to agree time to study, work on your defence materials as well as any time you need to actually attend the defence. Remember that taking time out to study and prepare would mean your company would take the hit on your productivity. So having a mutual agreement benefits all.

Support from your family is also a must as it will be a huge investment in your time.

Step 2 – Set clear objectives

Sit down with your manager and discuss clear objectives that are SMART. Agree on what your objectives are, and plan to achieve them. An example:

Objective Estimated Completion Date Resources
VCP Q1 ICM course, lab practice
VCAP-DCA Q2 Courses (optional), lab practice
VCAP-DCD Q3 Design Workshop (optional), read PDFs, lab practice
Create a vSphere Design Q2-Q3 Work on real design for a customer with real world requirements and use this as your VCDX submission
Complete VCDX Submission Q4 Choose a VCDX defence date and aim to submit your VCDX materials in time

Step 3 – Keep a track of your progress

Remember to keep a track of your progress, if you pass the exams, share the news with your team, it keeps you motivated. If you fail, then your timeline objectives may need tweaking. Keep your manager in the loop with progress, as ultimately, funding needs to come from somewhere for your fees and expenses right?

Step 4 – Work on your VCDX materials and then submit

Read the VCDX requirements and register your intention to pursue the VCDX on myLearn and make sure that you meet all the requirements before sending in your submission. Make sure to get some colleagues to review your documents first.

If everything goes well, your submission may well be accepted by VMware and you’re invited to defend.

Step 5 – Prepare for your defence

At this stage you should have been invited to defend. This is the most critical stage of the process, all the work that you’ve done so far has ultimately come down to this. So no pressure.

There are many ways to prepare, but here’s how I made myself ready for the defence.

1. Request peer reviews from your colleagues and virtualisation friends. Ask them to review all of your documents and materials again, especially the design.

2. Run Webex sessions with your peers to go over your 15 minute VCDX presentation. Record this, it will help you review your performance, note the duration and your tone of voice, did you project well?

3. Conduct a mock defence session with your peers. Invite them to ask as many questions that they could think of, even the obvious ones. Record this as well, note your performance, how you responded to the questions, tone of voice, setup a BS counter. Too much BS means that you don’t know your design well enough and you’ll be at risk when it comes to your real defence. Just remember to be – clear – concise – calculated.

4. Practice white boarding, you will have at least one whiteboard at your defence and it’s your most powerful tool, so learn to use it like it’s second nature.

5. Know your design inside out, not just the technical aspects. If you can justify the technical design decisions back to the business and technical requirements and constraints then you’re on the right track.

6. If you feel that you’re not ready or you can’t make it to your defence, you can postpone it to the next defence dates without submitting your application again. I was initially scheduled to defend in Singapore but could not travel so defended in Frankfurt instead.

Well that’s my advice, I hope this information is useful and that it helps more people being able to attain the VCDX certification. Who knows I might see you on the other side of the table in 12 month’s time. 😀