Getting TLS certificates from Let’s Encrypt is easy, but that’s just part of the story. It may sound paranoid, but being able to revoke certs is almost equally important. The premise is that there’s always a chance that your private keys will leak. Maybe it will never happen to me, but it will eventually happen to someone. The risk is certainly higher for companies and organizations: team members come and go all the time, and some of them might gain knowledge of the keys. That’s why revoking certificates should be part of off-boarding procedures. Well, I’m running this blog alone, and there’s not much to protect here. Nevertheless I wanted to learn how to get this right and played through that whole revocation exercise.
Like everything else around Let’s Encrypt, certificate revocation is automated via the ACME protocol. Of course, you wouldn’t want anyone else to revoke your certs — that would effectively block all HTTPS traffic to your side. So before revoking a cert, Let’s Encrypt needs some proof that you are the legitimate owner. Generally there are 3 alternatives:
- A: Prove control of the private key that belongs to the cert.
- B: Prove control of the Let’s Encrypt account that has issued the cert, via the account private key.
- C: Prove ownership of all the domains listed in the certificate.
Alternative A requires that I never lose any private key, at least not before the corresponding cert expires. (Btw, I’d recommend against reusing the same private key for consecutive certs. Luckily certbot generates a new private key when renewing a cert.) But my servers and discs are cattle, not pets. I intend to throw them away at anytime and recreate them from scratch or replace them with something better. So my private keys can go away any time, too. If I wanted to use them for certificate revocation, I’d have to back them up in some way. Ideally to some some key management solution like Hashicorp Vault or Ansible Vault (which I’m already using for other parts of my infrastructure). But each copy of the private key increases the risk that it will get leaked.
Alternative B is similar, although the account private key is somewhat more stable. It makes sense to use the same account for consecutive certs. But relying on the account for revocation would require rock solid backups of the key. (Writing these lines I realize that leaking the account private key itself would also be problematic. Let’s Encrypt associates the account with a list of authorized domains and might skip challenges for a certain period of time. I should research ACME specs and Let’s Encrypt policies on this…)
So I wanted to experiment with alternative C, which works even after I have lost all of my private keys. But when I lose the private keys, I’ll probably lose the certs themselves at the same time. So how do I even know which certs to revoke?
CT to the rescue
Wouldn’t it be nice, if Let’s Encrypt published a list of all certs that they ever issued? Actually, wouldn’t it be nice, if all CAs did that? Turns out they are already doing that. It’s what the Certificate Transparency (CT) standard (RFC 6962) is for. The main purpose of CT is, ensuring that CAs (and domain owners) can verify that other CAs are not issuing certificates fraudulently. CT defines an append-only data-structure (based on Merkle trees) which CAs can use to log all the certs that they issue.
Ideally browsers (and other TLS clients) should only trust certificates that have been published in a CT log. Chrome is already enforcing that, but Firefox seems to be lagging behind.
CT logs can be operated by CAs themselves (like Let’s Encrypt does) or by third parties (like Cloudflare or Google). Different CAs can submit their issued certs to different logs. TLS clients consume the CT log entries (a.k.a. Signed Certificate Timestamps a.k.a. SCT) via various channels (e.g. an X.509 extension or a TLS extension or OCSP) but none of those contain the URL of the CT log itself. Which is weird, because CT logs have a clearly defined REST API which is rooted at an HTTPS URL. Lack of URLs makes it harder to query the CT logs of Let’s Encrypt directly (let alone those of all CAs).
Moreover, searching the CT logs for certain domain names (like meeque.de) is not trivial. You’d basically have to download the whole log and search it entry by entry (or create your own search index over all entries). There might be open-source software that that could help (e.g. Let’s Encrypt’s own CT Woodpecker) but I haven’t checked that out yet.
Luckily, others have solved all these problems and are providing CT log monitoring as a service. The nice people at Sectigo provide such service for free at crt.sh. I’ve been using their web UI for manual certificate searches for a while now. E.g. here’s a list of all certs for my meeque.de domain, including its sub-domains. crt.sh provides data in machine friendly formats, too: JSON for search results and PEM for certs. So I wrote a small bash script that queries crt.sh for my domains and downloads the relevant cert chains into a local tracking directory. I’ve also setup a cron-job to do that on a regular basis.
So, let’s revoke already
With that we can circle back to the revocation topic. My script also helps me using certbot for sending ACME revocation requests for any tracked cert. All of that works without knowing the certificate’s private key or the private key of a Let’s Encrypt account. Certbot will simply create a new Let’s Encrypt account under the hood whenever it cannot find an existing one in local configuration. And Let’s Encrypt allows you to revoke a certificate that has been issued to another account, as long as you can prove ownership of all the domains included in the certificate. You just have to fulfill the domain ownership challenges, similar to when requesting a new cert.
There’s a small catch though: ACME supports pre-authorization, which allows clients to prove domain ownership (by fulfilling challenges) as a stand-alone operation. But that operation seems to be optional and I’m not sure, if Let’s Encrypt supports it. In any case, certbot does not support stand-alone pre-authorization as of today (or does not document it).
My workaround is simple: I’ll just ask certbot to obtain a new cert for my domains and fulfill all necessary challenges. After that, my brand new Let’s Encrypt account will have authorization for my domains. With that, Let’s Encrypt will allow me to revoke any of the certificates that I’ve found in CT logs. Well, at least for those domains that I still own.
So far, this only works for revoking certificates that have been issued by Let’s Encrypt. I guess it would be easy to add support for other CAs that implement the ACME protocol. I’d just need some kind of mapping between CA root certs and ACME URLs… For now, I’d have to take manual action, if I ever find a cert from another CA in the CT logs for my domain. To find out, if that ever happens, my next step will be setting up alert notifications that will warn me whenever an unknown cert shows up…