Let’s Encrypt! [en]

… is where I’m getting the TLS certificates for this blog nowadays (after moving away from CAcert). I’ve been using Let’s Encrypt at work now and then. Many colleagues in my department are heavy users and my employer is a sponsor. So I knew what to expect and how to get started. Nevertheless, here’s a couple of experiences that I’d like to share…

Let’s Encrypt is all about automation, which eventually lead to the ACME standard. There seems to be a myriad of ACME clients out there, but Let’s Encrypt recommends certbot for getting started. So I defaulted to that one.

Certbot is fairly opinionated and makes it easy to get going without thinking too much about configuration details. No need to tinker with keys and CSRs, certbot does it under the hood. There’s some decisions to be made though…

Domain Ownership Challenges

First decision was, which challenge to use to prove my ownership of meeque.de? Using a DNS challenge would be nice, because it’s the only way to get wildcard certs from Let’s Encrypt. However, it would require giving certbot access to my DNS configuration, which I’ve recently moved to Route 53. But I don’t want to get too tight with the evil empire and I don’t need wildcard certs for now. So I opted for an HTTP challenge.

Even then, certbot offers a plethora of plugins that fulfill the HTTP challenge and install the resulting certificates in a web server. E.g. the standalone plugin spins up it’s own little web server, just for fulfilling challenges. There’s also a certbot plugin for nginx, the web server that I’m using for TLS-termination. But letting certbot tinker with my nginx configuration seems kinda invasive. Besides, it would require manual wiring, because I’m running nginx isolated in a Docker container.

So I picked the certbot webroot plugin instead and I like its minimalism. You just point it to a local directory where it can create challenge files. You can then use any web server you like to expose these files via HTTP, so that Let’s Encrypt can verify them. I’ve extended my existing nginx configuration for that. And since I’m running nginx in a container, I’m using Docker mounts to ensure that both certbot and nginx can access the challenge files. Works like a charm!

Deploying Certificates

Some certbot plugins go further than just sending ACME requests and fulfilling challenges. They also install the certs in a web-server. The webroot plugin does not do that though.

But that’s not a problem: certbot puts new certificates (and the corresponding private keys) in a predictable directory. All I need to do is point my nginx configuration to the right files. Once again, I’m using Docker mounts to ensure that both certbot and nginx can access the cert files.

In a typical nginx setup, you’d use separate virtual servers for HTTP and HTTPS traffic. And you’d repeat that for all DNS domains that you want to host. That causes a small bootstrapping problem:

  • In order to serve HTTPS traffic you need a certificate.
  • But in order to get a certificate from Let’s Encrypt you need a running HTTP server to fulfill ACME challenges.

There are several ways around this problem, e.g. bootstrapping with self-signed certs or spinning up a temporary web server just for the challenges. I went a step further, by using two full-fledged nginx servers, both isolated in their own Docker container:

  • The first nginx server (“challenger”) only serves HTTP traffic. And only ACME HTTP challenge files. It redirects all other requests to HTTPS.
    This server starts successfully, even when I recreate my hosting environment from scratch.
  • The second nginx server (“webfront”) only serves HTTPS traffic. It performs TLS termination for this blog (and could be extended to serve other virtual hosts).
    This server fails to start after I recreate the hosting environment from scratch, because nginx will not find the cert and key files that I’ve configured. However, a quick run of certbot will fix that condition and restart this server.

In practice, I’ve automated all of the above, with Terraform and Ansible. I can easily delete the whole host from my environment and get everything running from scratch within a couple of minutes. Including Let’s Encrypt certificates! (I’ve tested that just last week.)

What else?

Yupp, using another nginx instance just for ACME challenges seems a little wasteful. But I think there’s some elegance to it. And Docker makes it easy to implement this setup.

That covers my basic certificate needs. I can obtain certs from scratch and renew them regularly. I have a simple cron job for that part.

But there’s more to working with certificates. Just think monitoring and revocation. More on that soon…

Goodbye CAcert [en]

When I started this blog back in 2010, I wanted HTTPS, but I didn’t want to pay extra for it. Back then that wasn’t as easy as it is today. So I compromised and got my TLS certificates from CAcert. Problem was that almost no OS or browser vendor trusted their certs. Debian and Ubuntu did make it part of their ca-certificates package, which worked fine for myself, but very few other people.

Anyway, according to the changelog, Debian has removed CAcert back in 2014 after lots of discussion. Looks like Ubuntu followed immediately, because they have a very similar changelog entry. And I only realized today, over 7 years later. The CAcert website and wiki also look pretty stale by now.

The other problem was that requesting certs from CAcert was a manual task. They had a nice web interface, and they would send email reminders before your cert expired. According to my mail archive, my last CAcert certificate had expired back in 2018. And I just never bothered to request a new one.

Of course, a lot has changed since those ancient times. Let’s Encrypt has done a very good job filling CAcert’s niche. Ever since I revived this blog a couple of months ago, I’ve been using their certs. But that’s another story…

Die Stones leben noch [de]

… aber MCA ist tot. Keith Flint ist tot. Sebastian Hackert ist tot. Und jetzt ist auch noch einer meiner Lieblingsmoderatoren bei FM4 gestorben: Martin Blumenau.

Er bleibt mir vor allem als Moderator der Wunschsendung FM4 Zimmerservice in Erinnerung. Er hatte da diesen gewissen Wiener Schmäh. (Zumindest erschien einen außenstehenden Piefke wie mir das so.) Er konnte mit den Hörerwünschen zum Teil sehr hart ins Gericht gehen. Auch mal mitten im Song abbrechen: “Nein, das geht jetzt überhaupt nicht.” Immer hatte er eine gute Begründung für jeden Verriss parat. Aber auch die richtigen lobenden Worte für die Sachen, die er dann doch ausgespielt hat.

Habe erst gestern beim Nachhören von Davidecks zufälling erfahren, dass Martin Blumenau letzte Woche verstorben ist. Und, dass er damals in den Neunzigern einer der Mitgründer von FM4 gewesen ist.

DNA replication lifehack [en]

I’ve recently posted about these talks on “DNA: The Code of Life” that I found. I really enjoyed them, even though most of the contents were not fundamentally new to me. However, I want to highlight one specific topic that I did learn, and that kinda blew my mind…

I had not even been aware of this particular DNA replication problem, let alone the solution. One part of that replication process is adding nucleosides to match the single-stranded-DNA template and forming phosphodiester bonds in the backbone of the daughter strand. That’s the job of DNA polymerase. It turns out that these highly conserved enzymes only work in one direction: they always traverse the DNA template from ‘3 end to ‘5 end. (In other words, they move along the growing daughter strand in 5’ to ‘3 direction and add nucleosides at the 3’ end.)

That’s pretty straight-forward for one of the daughter strands, called the leading strand. There, the DNA polymerase sits right behind the replication fork and traverses the DNA template in its ‘3 to ‘5 direction. The leading strand “grows” in the same direction, so no problem here.

However, the other daughter strand, called the lagging strand, is anti-parallel to that. It “grows” in ‘5 to 3’ direction of the template strand. But DNA polymerase does not work in that direction! Instead, it has to work backwards on the lagging strand, assembling the daughter strand segment by segment. There’s an intricate machinery of other enzymes that coordinate that, which includes forming temporary loops of DNA. Here’s an illustration of what it looks like:

My description and illustration are not very intuitive. It becomes more clear when watching the video from this original source:

What’s truly weird is that there are no fundamental chemical or physical reasons that would prevent the existence of a polymerase enzyme that works in the other direction. It just seems that a polymerase that traverses the DNA template from ‘3 end to ‘5 end evolved first, alongside the “hack” described above. Since then, different variations of DNA polymerases have evolved in various branches of the tree of life. But they are all variations on the same theme, whereas it would require a major re-haul for a polymerase to work the other way round. I think this is similar to the homochirality observed elsewhere in biology.

There are many more molecular biology videos at that WEHI place. I’ve seem similar ones elsewhere, but I’m surprised how old these are. Looks pretty neat for 2003. It seems that some of that has been used in a Björk video a while back.

Biting my style [en]

So I’m working at a huge software company and in my department we have this nice tradition of lunch-talks. The company buys pizza and we all eat it, while one hungry person gives ~1h talk. (Well, that was before the pandemic, now everything is remote and we have to fend for ourselves.)

Most of the topics are work related, e.g. about new technology, cool open-source projects, and latest developments in our own software. But there is also a lot of other stuff: hobby IT projects, soft skills, charity, you name it…

I’ve given lunch-talks myself in the past, mostly on security topics (e.g. XSS and other injection vulnerabilities). I’ve also had this idea floating around for years, that I’d like to give a lunch-talk about analogies between biology and computer science. But I’m not a great presenter and it takes me a lot of preparation work to compensate for that. So I’ve just been too lazy to implement this idea. Well, it turns out that someone has beaten me to it. And he did a brilliant job at it. And not just one talk, but two, but more about that later…

Or biting his style?

Let me first explain, how I found out about it in the first place. I was searching for information on RNA vaccines, and somehow I stumbled upon this article about Reverse Engineering the source code of the BioNTech/Pfizer SARS-CoV-2 Vaccine.

That’s a very cool article by itself, but it also pointed me to this other one, by the same author: DNA seen through the eyes of a coder. Turns out that page has been around since the early 2000s and it’s been constantly updated and improved.

Not only that, but the author (Bert Hubert) has also put most of the contents into two talks:

There’s also an dedicated article summarizing both talks, with videos embedded, slides for download, and heaps of links.

These talks touch on almost every topic that I had envisioned for my own lunch-talk. They cover DNA and RNA basics, chromosomes, proteins, ribosomes, the genetic code, the central dogma, genes and other types of DNA, promotion, repression, regulation, splicing, HOX genes, CRISPR-Cas9, etc. And they point out really interesting analogies to computing phenomena.

The talks also cover other biology topics, like evolution, the tree of life, RNA-world, LUCA, the three domains, viruses, horizontal gene transfer, epigenetics, endosymbiosis, and more. All of that is enriched with great examples and numbers, both from bio-technology (sequencing, editing) and from life itself. This includes comparisons of genome sizes, reproduction rates, the bacterial flagellum and how it is regulated in search for food, and the good old comparison between human and octopus eyes.

I’ll try to share some thoughts about specific contents of these talks. In the meantime, let me end with a random quote of the presenter:

Are we anthropomorphising nature, which it really hates?

— Bert Hubert, 2017

It’s been a while [en]

My last post here has been over 4 years ago. I’ve really neglected this blog. Let’s see, if I can change that…

Part of the problem was the rotten technology underneath. Even apart from WordPress. It was running on some weird virtual server at a local hosting provider. Weird, because it was neither full virtualization nor light-weight containerization. It felt like it combined the worst of both worlds.

It certainly wasn’t fun to work with it, so I didn’t. The server was down for several months in a row at multiple times. I never installed a TLS certificate that would be accepted by mainstream browsers. I had no backups whatsoever. And everything was set up manually.

During the past few weeks, I have changed most of that. By now, the technology is in good enough shape that I feel comfortable blogging again. I hope to share more about that shortly. And I guess there will be more cooking recipes, too.