The growth of Let’s Encrypt is phenomenal – 7 million certificates in last four months. The remaining hurdle for automation is verification of domain ownership. Well, actually it is NOT true. We were doing syntax testing – hoping to get the right kind of verification error … only to discover we have been successfully verified without providing any information.

 

Whatever you say about let’s encrypt project, you can’t deny its success. Issuance of certificates has been commoditized for a while and definitely ripe for disruption. Let’s Encrypt is the disruption and its adoption, despite shorter validity of certificates issued by Let’s Encrypt CA, is exponential and really fast.

We played with Let’s Encrypt for our Cloud PKI service (https://enigmabridge.com/amazonpki), which we hope to launch on Amazon AWS Marketplace as soon as we get through administrative hurdles.

Our goal was to use DNS-based challenge verification as that would allow us to create a new PKI instance with browser-trusted HTTPS as part of an instance initialization. Dusan was testing the command line syntax of certbot so that we could start integration with DNS records’ updates.

To his and subsequently everone’s confusion, he realized that he got issued a certificate without any challenge verification at all. All he had to do was to run the following command:

certbot certonly --agree-tos --text \
--email [email protected] \
--authenticator manual --preferred-challenges dns \
-d durham1.pki.enigmabridge.com

He quickly did a verification – copied his test account key to another server (a random Amazon EC2 instance) and got another valid certificate for the “durham1.pki.enigmabridge.com” domain.

It frightened me, it didn’t look right, it didn’t smell right. I decided to investigate. With help of Kenn White I reached out to security guys at LE as well as to Brad Warren working on the Certbot project. Brad provided us with some new information and links to documentation that showed this behaviour was

a feature of Let’s Encrypt.

Where is the problem? Expectations!

We tried to figure out why we had freaked out and thought it was a bug. The answer was relatively simple. We always expected that every single certification request would include fresh verification of the domain ownership. This is not true any more with re-usable AUTHZ tokens (see Upcoming change: valid authz reuse).

As we understand the process, we need two things to get a new Let’s Encrypt certificate:

1. account key [stored on our server]- it’s and RSA key that we generate and which is stored on the server requesting a certificate (e.g., in /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/…)
2. authorization token (Authz) [kept by Let’s Encrypt] – it’s created by Boulder (i.e., the registration authority of Let’s Encrypt) upon successful execution of a domain ownership verification.

The surprising aspect is that Authz has a validity of 300 days (which is likely to be increased). It means that within 299 days of issuance of an Authz token, all we need to get a new certificate is the account key.

Update: I was wrong here, it seems that the validity is now 90 days.

The matter of expectations is that we always assumed that the account key is important – it creates the JWS signature on certification requests – but it’s always coupled with on-the-fly verification that we control the domain name for which a new certificate was requested.

Let’s have a look at the diagram below. Disclaimer: this is as I understand the mechanism, I’d love to hear from anyone who can confirm this or show my fallacy.

You can see that attackers only need to compromise the “Super important server” once. As long as this “Super important server” uses Let’s Encrypt, it will keep updating the authz token. The “Bad guys to launch MITM” or some kind of masquerade attacks only have to be good enough not-to-lose the stolen account key. The Authz token updates will be done by the “Super important server”!

What Are Were My Expectations

I certainly didn’t expect the importance of the account key to be paramount. It now appears it is the single piece of secret needed to get a new Let’s Encrypt certificate.

I didn’t expect there would be a “master key” with infinite validity and that I should look after it as such.

I did expect that each request includes a fresh verification of the domain control. And I also expect that substantial majority of people out there believe in this process / data flow as that’s how certificates have always been verified – for each new certificate, you either have to send an email from a given domain or put a file to a web server’s folder.

I do expect this is an accident waiting to happen situation that needs to be rectified – the sooner the better. Whether it’s a technology change or re-adjustment of user expectations.

I also expect that our Enigma Bridge Cloud HSM service (check our PKI product with hardware key security) can securely control storage and use of the account key to prevent thefts of the keys and their subsequent misuse.