Let’s Encrypt! [en]

… is where I’m getting the TLS certificates for this blog nowadays (after moving away from CAcert). I’ve been using Let’s Encrypt at work now and then. Many colleagues in my department are heavy users and my employer is a sponsor. So I knew what to expect and how to get started. Nevertheless, here’s a couple of experiences that I’d like to share…

Let’s Encrypt is all about automation, which eventually lead to the ACME standard. There seems to be a myriad of ACME clients out there, but Let’s Encrypt recommends certbot for getting started. So I defaulted to that one.

Certbot is fairly opinionated and makes it easy to get going without thinking too much about configuration details. No need to tinker with keys and CSRs, certbot does it under the hood. There’s some decisions to be made though…

Domain Ownership Challenges

First decision was, which challenge to use to prove my ownership of meeque.de? Using a DNS challenge would be nice, because it’s the only way to get wildcard certs from Let’s Encrypt. However, it would require giving certbot access to my DNS configuration, which I’ve recently moved to Route 53. But I don’t want to get too tight with the evil empire and I don’t need wildcard certs for now. So I opted for an HTTP challenge.

Even then, certbot offers a plethora of plugins that fulfill the HTTP challenge and install the resulting certificates in a web server. E.g. the standalone plugin spins up it’s own little web server, just for fulfilling challenges. There’s also a certbot plugin for nginx, the web server that I’m using for TLS-termination. But letting certbot tinker with my nginx configuration seems kinda invasive. Besides, it would require manual wiring, because I’m running nginx isolated in a Docker container.

So I picked the certbot webroot plugin instead and I like its minimalism. You just point it to a local directory where it can create challenge files. You can then use any web server you like to expose these files via HTTP, so that Let’s Encrypt can verify them. I’ve extended my existing nginx configuration for that. And since I’m running nginx in a container, I’m using Docker mounts to ensure that both certbot and nginx can access the challenge files. Works like a charm!

Deploying Certificates

Some certbot plugins go further than just sending ACME requests and fulfilling challenges. They also install the certs in a web-server. The webroot plugin does not do that though.

But that’s not a problem: certbot puts new certificates (and the corresponding private keys) in a predictable directory. All I need to do is point my nginx configuration to the right files. Once again, I’m using Docker mounts to ensure that both certbot and nginx can access the cert files.

In a typical nginx setup, you’d use separate virtual servers for HTTP and HTTPS traffic. And you’d repeat that for all DNS domains that you want to host. That causes a small bootstrapping problem:

• In order to serve HTTPS traffic you need a certificate.
• But in order to get a certificate from Let’s Encrypt you need a running HTTP server to fulfill ACME challenges.

There are several ways around this problem, e.g. bootstrapping with self-signed certs or spinning up a temporary web server just for the challenges. I went a step further, by using two full-fledged nginx servers, both isolated in their own Docker container:

• The first nginx server (“challenger”) only serves HTTP traffic. And only ACME HTTP challenge files. It redirects all other requests to HTTPS.
  This server starts successfully, even when I recreate my hosting environment from scratch.
• The second nginx server (“webfront”) only serves HTTPS traffic. It performs TLS termination for this blog (and could be extended to serve other virtual hosts).
  This server fails to start after I recreate the hosting environment from scratch, because nginx will not find the cert and key files that I’ve configured. However, a quick run of certbot will fix that condition and restart this server.

In practice, I’ve automated all of the above, with Terraform and Ansible. I can easily delete the whole host from my environment and get everything running from scratch within a couple of minutes. Including Let’s Encrypt certificates! (I’ve tested that just last week.)

What else?

Yupp, using another nginx instance just for ACME challenges seems a little wasteful. But I think there’s some elegance to it. And Docker makes it easy to implement this setup.

That covers my basic certificate needs. I can obtain certs from scratch and renew them regularly. I have a simple cron job for that part.

But there’s more to working with certificates. Just think monitoring and revocation. More on that soon…