Preparing Let’s Encrypt SSL Certificates for vCenter, NSX-T Manager and Avi Controller

The Let’s Encrypt DST Root CA X3 certificate is missing from the fullchain.pem and chain.pem files, therefore errors such as the following prevent certificates from being imported by VMware appliances such as NSX-T and vCenter.

This post summarizes how to fix this issue.

Let’s Encrypt is a great service that provides free SSL certificates. I recently rebuilt my lab and decided to use SSL certs for my management appliances. However, non of the management appliances would accept the certificates issued by Let’s Encrypt due to an incomplete chain. This post summarizes how to fix this issue.

TL;DR the Let’s Encrypt DST Root CA X3 certificate is missing from the fullchain.pem and chain.pem files, therefore errors such as the following prevent certificates from being imported by VMware appliances such as NSX-T and vCenter.

Certificate chain validation failed. Make sure a valid chain is provided in order leaf,intermediate,root certificate. (Error code: 2076)

Get your certbot tool up and running, you can read more with this link.

Grab your files from the /etc/letsencrypt/live folder for your vCenter certificate. My one is in /etc/letsencrypt/live/vcenter.vmwire.com

You should now have the following files.

cert.pem
chain.pem
fullchain.pem
privkey.pem

A note on Let’s Encrypt certificate chain. If you look at the Certification Path for Let’s Encrypt certificates, you’ll notice something like this.

figure 1.

vcenter.vmwire.com is issued by the R3 CA certificate. This is Let’s Encrypt’s Intermediate certificate.

R3 is issued by the DST Root CA X3 certificate. This is Let’s Encrypt root certificate.

Then the DST Root CA X3 certificate needs to be trusted by all of our management appliances, vCenter, NSX-T and Avi Controller.

What I found is that, this is not the case and trying to import a Let’s Encrypt certificate without the root certificate that issued the DST Root CA X3 certificate will fail. Here’s an example from NSX-T when importing the chain.pem certificate.

figure 2. Importing the chain.pem certificate to NSX

The chain.pem file contains the R3 certificate and the DST Root CA X3 certificate. When you open it in notepad++ it looks like this.

figure 3. chain.pem

So we have a problem. We need the certificate that issued the DST Root CA X3 certificate to complete the chain and pass the chain validation.

Lets take a look at Let’s Encrypt certificates on their website.

ISRG Certificate Hierarchy Diagram, as of December 2020

So looking up the chain, it appears that my certificate vcenter.vmwire.com corresponds to the Subscriber Cert, which is issued by R3. This confirms the assumptions above in figure 1. However, it looks like the R3 certificate is not issued by the DST Root CA X3 certificate but in fact another certificate named ISRG Root X1.

Lets test this theory and import each of the certificates in the chain.pem file individually using NSX-T.

After importing, you can see that this is in fact the ISRG Root X1 certificate that is issued by the DST Root CA X3 certificate. My assumption from figure 3. is then incorrect.

So what is the top certificate in the chain.pem file?

Lets’ import it and find out. Yup, its the R3 certificate.

So where is the DST Root CA X3 certificate that we need to complete the validation chain?

We can obtain this from the Let’s Encrypt website. Scroll all the way down to the bottom of that page and you’ll see the following:

Clicking on that link will get you the the following page with this link.

And we will get closer to our DST Root CA X3 certificate when we click on that link above.

Clicking on that link gets us to this page here.

Then clicking on that link will get us to this page here.

We can now grab our certificate with this link highlighted here.

When you click on this link, you’ll be able to download a file named 8395.crt, this is the DST Root CA X3 certificate that we need to complete the chain. However, it is in a .crt format but we need to work with .pem.

To convert a crt certificate to pem use the following command.

openssl x509 -in 8395.crt -out 8395.pem -outform PEM

Once we import it into NSX-T we can see that it is indeed our missing certificate.

Looking at all the certificates in NSX-T, it all looks good.

You can see from my terrible Apple Pencil skills that it all now looks good.

So how do we use all these certificates together?

Well, the easiest is to take the fullchain.pem file, this file contains the following certificates in the following order.

Subscriber Certificate
R3 Certificate
ISRG Root X1 Certificate

That means we just need to append our new DST Root CA X3 certificate to the bottom of the fullchain.pem file to get a valid chain. It will now look like this.

Subscriber Certificate
R3 Certificate
ISRG Root X1 Certificate
DST Root CA X3 Certificate

Once we import this new fullchain into NSX-T in goes through successfully.

Now that we have a working fullchain.pem, we can use that file or its contents to import these signed certificates into our management appliances.

Read on – the following articles summarize how to use these certificates with vCenter, NSX-T Manager and the Avi Controller.

Updating vCenter to use Let’s Encrypt certificates.

Updating NSX-T to use Let’s Encrypt certificates.

Updating Avi Controller to use Let’s Encrypt certificates.

A List of VMware Employee Tweeps (people on Twitter)

Following on from the PSO NEMEA twitter list, I decided to go further and produce this list of VMware employees that are on Twitter, sorted alphabetically by Twitter ID as of 29/06/2011.

Let me know if I have missed you out or you follow someone that works for VMware.

Twitter ID Name Blog
Adrian Roberts  
Alan Renouf www.virtu-al.net
Andrew Mitchell  
Andy Banta  
Arnim van Lieshout www.van-lieshout.com
Kamau Wanguhu www.borgcube.com
Brian Thomas Rice  
Chris Colotti www.chriscolotti.us
Christophe Decanini www.vcoteam.info
Christoph Harding www.thatsmyview.net
Brittany Coulson  
Carter Shanklin  
Dale Carter www.delboycarter.com
Dave Hill www.virtual-blog.com
Douglas Phillips  
Richard Damoser  
Duncan Epping www.yellow-bricks.com
Eric Gray www.vcritical.com
Frank Denneman www.frankdenneman.nl
Frank Wegner  
Hany Michael www.hypervizor.com
Andy Troup  
Stephen Herrod www.vmware.com/company/leadership.html
Pablo Roesch  
Hugo Strydom www.vroem.co.za
Hugo Phan www.vmwire.com
Jean-Francois Richard  
Jerry Chen  
Johnny Krogsboll  
Joe Sarabia  
John Troyer  
Julie Escott  
Greg A Lato www.latogalabs.com
Lode Vermeiren lodev.name
Max Daneri  
Manish Patel  
Mark Verhagen  
Martyn Storey  
Matthew Meyer  
Dave McCrory blog.mccrory.me
Matt Coppinger
Michael Haines  
Mike DiPetrillo www.mikedipetrillo.com
Massimo Re Ferre’ it20.info
Nadyne Richmond www.nadynerichmond.com
Peter Giordano petergiordano.com
Paul Nothard  
Rawlinson Rivera www.punchingclouds.com
Rasmus Jensen www.vpeeling.com
Ray Heffer www.rayheffer.com
Raymon Epping  
Richard McDougall blog.richardmcdougall.com
Rick Blythe www.vmwarewolf.com
Robin Prudholm  
Rob Upham  
Safouh Kharrat  
Scott Davis blogs.vmware.com/view-point
Simon Long www.simonlong.co.uk
Steve Jin www.doublecloud.org
Scott Sauer unhub.com/ssauer
Stan Hutten Czapski  
Susan Gudenkauf  
Burke Azbill www.vcoteam.info
Tedd Fox about.me/teddfox
Richard Garsthagen www.run-virtual.com
John Dodge www.dodgeretort.com
Tom Ralph about.me/TomRalph
Tony Dunn  
Tristan Todd  
Timo Sugliani  
Jason Miles  
John Arrasjid  
Alexander Thoma  
Vegard Sagbakken   
Vic Camacho wefollow.com/Virtual_Vic
Andrew Johnson  
Irfan virtualscoop.org
Todd Muirhead  
Mark C  
Josh Liebster vmsupergenius.com
Vittorio Viarengo journeytocloud.com
Wade Holmes  
Willem van Engeland  
Jian Zhen zhen.org

VMware PSO NEMEA Twitter List

A list of VMware PSO consultants covering NEMEA that are on Twitter, sorted alphabetically by Twitter ID.

Follow us for tweets from the real world.

Map created using templates from http://www.presentationmagazine.com.

vShield Manager Notes

Most administrative changes to vShield Manager can be done using the command line interface (CLI) by initiating a console session to the vShield Manager virtual machine.  You can log in to the CLI by using the default user name admin and password default.

You can also access the CLI by enabling SSH.

To enable SSH:

  • Log in to the CLI by using the default user name and password
  • Enter configuration mode by typing

manager# en

manager# configure terminal

manager(config)# ssh start

manager(config)# cli ssh allow

 

To change the hostname of vShield Manager

vShield Manager uses manager as the default hostname but there is no easy way to change the hostname using the web interface or the vSphere plugin.  You can only change vShield Manager’s hostname using the CLI.

  • Log in to the CLI by using the default user name and password
  • Enter configuration mode by typing

manager# en

manager# configure terminal

manager# hostname newhostname

  • vShield will then restart its web services and accept the changes

 

More to follow….