Small Money in Tanzania [en]

One thing that stands out here in Tanzania (yupp, I’m still here, greetings from Zanzibar!) is the huge wealth gap. Tourists can easily pay European prices at restaurants and hotels. Wealthy locals drive big SUVs and own generous houses (and other real estate). But many Tanzanians live in tremendously poor conditions, especially in rural areas. You’ll see improvised fishermen’s huts along the coast and farmers’ tiny mud dwellings in the interior. Many of them mostly employ subsistence agriculture. They may have less than 1 EUR per day at their disposal.

As one tourist guide book put it: you’ll probably have more cash in your pocket right now than your Tanzanian street vendor makes in a year. When dealing with chapati vendors, boda boda drivers, or even small shops and restaurants, you’ll have trouble getting correct change, because they simply do not have enough money around. And that is despite the fact that the biggest bill (10000 TZS) is currently worth less than 4 EUR. At the other end of the spectrum this means that successful local businesswomen will wander around with big piles of money, otherwise only seen in rap videos. Here’s one of my own bundles:

(Un)availability of modern technology reflects the poverty. Many people have no access to the power grid, let alone landline phones. Mobile phones on the other hand seem attractive and cheap enough that they proliferated widely. Even in remote villages, at least a few persons may have a mobile phone. In more urban areas you’ll see Android phones, presumably many of them are second-hand. You’ll still see many 90ies Nokia-style phones with little to no mobile Internet capabilities.

However, even those simple phones give access to some form of mobile banking (while most poor people cannot afford real bank accounts, credit cards, etc.). When I bought my Vodacom pre-paid SIM card, I automatically gained access to their M-Pesa service. That is a mobile phone based money-transfer and micro-payment service. While M-Pesa was one of the front-runners (originally in Kenya) all other Tanzanian mobile network operators offer similar services today.

The M-Pesa service is primarily bound to your mobile phone number. While mobile apps are available, you do not need one to use M-Pesa. Instead M-Pesa can be used via USSD, an obscure (to me) part of the GSM standard. USSD is an interactive protocol: mobile phones can initiate USSD via dialing special numbers (containing lots of # and *, similar to GSM codes). Then you’ll be presented a list of options and you can select them by pressing numbers on the phone. You can also specify other numeric inputs, e.g. the amount of money or the recipient’s phone number in an M-Pesa transaction. This is what it looks like in the phone app of a modern Android phone:

Keep in mind that Android (or iOS) is not required for any of that. Most simple GSM phones will support USSD, and later 3G/4G/5G phones are backwards compatible with GSM. (Though very old GSM phones from the 90ies may not support USSD yet.) You do not need proper mobile Internet either, and if your mobile data package is billed by the byte, USSD will not count against that. However, some M-Pesa operations are subject to fees, including moderate transaction fees for sending money.

You can also transfer money between classic bank accounts or credit cards and M-Pesa. Allegedly that does not work well with foreign accounts or cards though. The main interface to the world of cash money are the abundant “Wakala” kiosks. The term Wakala seems to come from Islamic finance, but here in Tanzania, Wakala typically means this:

Several road-side wakala kiosks

So far, I haven’t used M-Pesa much. Just for topping up my prepaid phone services. And for paying the entrance fee for the National Museum of Tanzania, who did not accept cash — presumably because of the Covid19 pandemic?

For the future, here is my Tanzania USSD cheat sheet:

  • *150*00# — M-Pesa
  • *149*01# — Vodacom prepaid services
  • *#100# — Show own phone number (standard GSM code)

As an information security expert, I have to wonder how secure a payment system ultimately based on GSM can be? I’ve heard about so many problems with the underlying networking standards. But I guess that real-life attacks would require an IMSEI-catcher. Widespread fraud might just not pay off here. In particular, because M-Pesa seems to impose some limits on account balance and transaction amounts.

Usambara Mountains [en]

After two weeks, I’ve left Kigamboni behind (for now) and started to travel north-west.

First stop: Usambara mountains. A friend recommended a good guide, Subira from Kigamboni-Eco-Cultural Tours. She is native to the Usambara region, and you can reach her via phone/WhatsApp at +255627944103 or via email at kigamboniecoculturaltours@gmail.com. Here’s some of our impressions…

Bus ride from Dar-es-Salaam to Usambara Mountains:

Bus ride up to the Usambara Mountians:

Our accommodation nearby Lushoto — St. Eugene’s Hostel:

Making our own coffee at Magamba Eco-Cultural Tourism:

Hike from Magamba to Kigulu Hakwewa Peak, Irente View, and Maguzoni (GPX):

Sugar cane wine (“boha”) making and tasting in Maguzoni:

Kisasa Waterfall (GPX):

Mambo view:

On to Arusha:

Next stop: Kilimanjaro.

Update 2021-12-13: Added more photos, links, and geo tracks.

Kite Kigamboni [en]

I’ve brought my kite equipment to Tanzania, so I can do some kiting over on Zanzibar in a couple of weeks. Originally I didn’t plan to kite on the mainland. But when visiting the beach in Kigamboni, I was happy to realize that it was somewhat windy.

I didn’t see any other kiters or wind surfers though, and I wasn’t sure, if the wind would be enough. I’ve only brought a 12m² kite, and I’m still struggling in low-wind conditions. So I watched the forecasts for a couple of days, and waited for the best opportunity.

That finally came last week Monday, with side-on wind at about 12 knots. So I grabbed my kite stuff, and went to Mikadi Beach Club (whose manager had told me that he hadn’t seen any kiters during the past 2 years). I met Mo — one of the guests whom I had got to know the night before — and asked him to help me launch. Since the beach is rather narrow and somewhat busy at Mikadi, we walked south-east (close to Tanzanite Beach Resort). I began to setup my stuff and explained to Mo how to help launch a kite…

Here’s some footage:

Last pre-launch alignment
The launch
Engines running
On the water…
… riding, but had to move the kite a lot.
Safe landing

Overall a great success, though I didn’t stay on the water for long, since it was getting dark soon. I also pumped just too little air into the kite, which made it somewhat unstable during quick turns. My brand new board worked fine though.

The next day, I tried it again, yet a few meters down the beach (close to Malaika Beach Club). This time without a camera woman. And without anyone helping me launch. My first self-launch worked, though I felt very unconfident. And unfortunately the wind was just a little to weak this time. I drifted downwind, and had to give up and walk back on the beach.

Let’s hope for more wind once I get back to Kigamboni in about 2 weeks. And later on Zanzibar…

Africa! [en]

I’m gonna be in Tanzania for the next few weeks. This is my first time in Sub-Saharan Afrika. Let’s hope that I don’t turn into this mamafat guy.

I’ve arrived almost 2 weeks ago and I’ve been staying with a friend in Kigamboni. That’s a suburb of Dar es Salaam, but separated from the city by a narrow estuary. Let me share my impressions so far…

The vibe reminds me of prior trips to places like Indonesia or Sri Lanka. There’s overlap in climate, landscape, infrastructure, traffic, etc. Tuk-tuks are called bajaji here and they are abundant. So are boda boda.

I have not found my favorite local food yet. Most traditional food is fairly simple, e.g. based on rice, beans, and vegetables. Local fast-food has international influences: chips and grilled meat sticks can be found on every street corner. The latter are called sate, but sadly they come without the sate source seen in Southeast Asia. Baked goods are also common, most notably chapati. There’s a lot’s of fish, too, but that category is still not my favorite. Fresh tropical fruits, that’s more like it.

The people have been very friendly so far. On average, they speak better English than I would have expected. Almost everyone understands a few chunks — enough to negotiate prices and agree on goods or destinations. Many shop keepers, waiters, drivers, etc. know way more than that, and will happily discuss a wide range of topics in English. Everyone who is “well-educated” and/or “upper-class” speaks English fluently.

The locals also seem fairly health-conscious. I hardly ever see people smoke. On the other hand, I do see many people jogging at the beach. Speaking of beaches, this is what the nearby ones look like:

There’s also a couple of beach clubs, who watch their particular section of beach and provide amenities to tourists. However, there’s a surprisingly small number of tourists around. At least when it comes to international tourists. Yes, we’ve met a couple of Europeans at one of the beach clubs, but elsewhere foreigners are pretty rare.

Last week I’ve spent all day walking Kigamboni’s center (nearby the ferry terminal to Dar es Salaam), main road, and beaches without meeting another melanin-deficient person — except for the mannequins of tiny apparel shops that is. (Does nearby Zanzibar really have much better beaches? I guess I’ll find out in a couple of weeks…)

Another thing that I notice is, how much infrastructure work is going on. The Kigamboni Bridge, which connects us to Dar es Salaam, has only been finished 5 years ago. Many highways that connect to it are still under construction. Another big bridge is being build across a bay in Dar es Salaam itself. New long-distance railroad tracks are also in process.

The Dar es Salaam port also looks very busy. Many big container ships are idling off the coast though. More than usually I’m being told. Presumably that has to do with the ongoing worldwide supply-chain issues. There’s also a small shipyard in Kigamboni. (I think it’s run by Sorongoro Marine Ltd.) They are building, repairing, and tearing down ships in a rather improvised manner. No proper docks, just hauling ships on and off the beach. Let’s hope that their front does not fall off.

Fall of Civilizations [en]

A while ago, the YouTube algorithm suggested an episode of the Fall of Civilizations podcast. Not exactly sure why, because I rarely watch history topics online. (I do sometimes read up on history topics on Wikipedia, so maybe that’s why? Although I cannot see any obvious 3rd-party tracking code on Wikipedia.) Even more rarely do I follow overly random YouTube suggestions, but this time I clicked the video and started watching. Here’s the trailer:

And somehow I got hooked. By now, I’ve watched all 12 episodes that are available today. Some of them are several hours long and more episodes are in the making. Here’s my top 5 so far:

  1. The Bronze Age Collapse
    Hell yeah, “sea peoples”. Also see this talk by some other guy.
  2. The Sumerians
    I like my history ancient and this is about as ancient as it gets.
  3. Byzantium
    A.k.a. the Eastern Roman Empire, definitely a big one – I wonder if there will be an episode about the collapse of the western part?
  4. The Aztecs
    And also The Inca and The Mayans. Sorry, for lumping them together. The non-European perspective is often marginalized where I live.
  5. Easter Island
    I’m fascinated by the history of Polynesia, maybe because I lived there for a while. Well, that was New Zealand, one of the other corners of that huge triangle.
    (This episode got featured on Fefe’s Blog recently, which is also very random.)

Not sure why I like that shit so much. Back in school I was never particularly keen on history. And frankly, a fair portion of my history knowledge comes from the first two Age of Empires games — Wololo! — not just the re-enacted battles, but also the background info for the campaigns. Well that, and Wikipedia, and having visited a fair amount of historic places.

XSS Demo [en]

Finally some good use for my new infrastructure. I’ve had this small Angular app lying around, which I wrote for a presentation/demo on XSS a couple of years ago. So far, I’ve run it locally to demonstrate XSS vulnerabilities and how to exploit them. Now I have a place to put it and share it with others. You can find it here:

The Docker image doesn’t do TLS termination itself. Instead I’m using an nginx reverse-proxy for that, same as with this blog. Just had to add another nginx config file. And add the dedicated subdomain to my Let’s Encrypt configuration. There, done.

The XSS demo app may not be very intuitive, because the original target audience was just myself. But it comes with a brief guide. Just play around from there. I may add more inline documentation later…

Right now the focus is on XSS vulnerabilities in plain HTML, via the DOM, and via Angular. Not saying the latter are vulnerable themselves. But they are all prone to XSS, if used carelessly. I’m thinking about extending the list to other web front-end libs/components/frameworks/technologies, e.g. a WYSIWYG editor or a Markdown processor.

The B in BYOK stands for Bullshit [en]

I’ve recently encountered someone, who insisted on a Bring-Your-Own-Key (BYOK) setup for compliance reason. I’ve always been skeptical about that and I didn’t have to search long for confirmation. This is what the Wikipedia article on BYOK has to say:

[…] a cloud computing security marketing model […]

[…] gives the enterprise the perceived control of its own keys […]

Data owners may believe their data is secured […]

I’ve rarely seen anything that snarky on Wikipedia. Yet so accurate.

Let’s Revoke! [en]

Getting TLS certificates from Let’s Encrypt is easy, but that’s just part of the story. It may sound paranoid, but being able to revoke certs is almost equally important. The premise is that there’s always a chance that your private keys will leak. Maybe it will never happen to me, but it will eventually happen to someone. The risk is certainly higher for companies and organizations: team members come and go all the time, and some of them might gain knowledge of the keys. That’s why revoking certificates should be part of off-boarding procedures. Well, I’m running this blog alone, and there’s not much to protect here. Nevertheless I wanted to learn how to get this right and played through that whole revocation exercise.

Like everything else around Let’s Encrypt, certificate revocation is automated via the ACME protocol. Of course, you wouldn’t want anyone else to revoke your certs — that would effectively block all HTTPS traffic to your side. So before revoking a cert, Let’s Encrypt needs some proof that you are the legitimate owner. Generally there are 3 alternatives:

  • A: Prove control of the private key that belongs to the cert.
  • B: Prove control of the Let’s Encrypt account that has issued the cert, via the account private key.
  • C: Prove ownership of all the domains listed in the certificate.

Alternative A requires that I never lose any private key, at least not before the corresponding cert expires. (Btw, I’d recommend against reusing the same private key for consecutive certs. Luckily certbot generates a new private key when renewing a cert.) But my servers and discs are cattle, not pets. I intend to throw them away at anytime and recreate them from scratch or replace them with something better. So my private keys can go away any time, too. If I wanted to use them for certificate revocation, I’d have to back them up in some way. Ideally to some some key management solution like Hashicorp Vault or Ansible Vault (which I’m already using for other parts of my infrastructure). But each copy of the private key increases the risk that it will get leaked.

Alternative B is similar, although the account private key is somewhat more stable. It makes sense to use the same account for consecutive certs. But relying on the account for revocation would require rock solid backups of the key. (Writing these lines I realize that leaking the account private key itself would also be problematic. Let’s Encrypt associates the account with a list of authorized domains and might skip challenges for a certain period of time. I should research ACME specs and Let’s Encrypt policies on this…)

So I wanted to experiment with alternative C, which works even after I have lost all of my private keys. But when I lose the private keys, I’ll probably lose the certs themselves at the same time. So how do I even know which certs to revoke?

CT to the rescue

Wouldn’t it be nice, if Let’s Encrypt published a list of all certs that they ever issued? Actually, wouldn’t it be nice, if all CAs did that? Turns out they are already doing that. It’s what the Certificate Transparency (CT) standard (RFC 6962) is for. The main purpose of CT is, ensuring that CAs (and domain owners) can verify that other CAs are not issuing certificates fraudulently. CT defines an append-only data-structure (based on Merkle trees) which CAs can use to log all the certs that they issue.

Ideally browsers (and other TLS clients) should only trust certificates that have been published in a CT log. Chrome is already enforcing that, but Firefox seems to be lagging behind.

CT logs can be operated by CAs themselves (like Let’s Encrypt does) or by third parties (like Cloudflare or Google). Different CAs can submit their issued certs to different logs. TLS clients consume the CT log entries (a.k.a. Signed Certificate Timestamps a.k.a. SCT) via various channels (e.g. an X.509 extension or a TLS extension or OCSP) but none of those contain the URL of the CT log itself. Which is weird, because CT logs have a clearly defined REST API which is rooted at an HTTPS URL. Lack of URLs makes it harder to query the CT logs of Let’s Encrypt directly (let alone those of all CAs).

Moreover, searching the CT logs for certain domain names (like meeque.de) is not trivial. You’d basically have to download the whole log and search it entry by entry (or create your own search index over all entries). There might be open-source software that that could help (e.g. Let’s Encrypt’s own CT Woodpecker) but I haven’t checked that out yet.

Luckily, others have solved all these problems and are providing CT log monitoring as a service. The nice people at Sectigo provide such service for free at crt.sh. I’ve been using their web UI for manual certificate searches for a while now. E.g. here’s a list of all certs for my meeque.de domain, including its sub-domains. crt.sh provides data in machine friendly formats, too: JSON for search results and PEM for certs. So I wrote a small bash script that queries crt.sh for my domains and downloads the relevant cert chains into a local tracking directory. I’ve also setup a cron-job to do that on a regular basis.

So, let’s revoke already

With that we can circle back to the revocation topic. My script also helps me using certbot for sending ACME revocation requests for any tracked cert. All of that works without knowing the certificate’s private key or the private key of a Let’s Encrypt account. Certbot will simply create a new Let’s Encrypt account under the hood whenever it cannot find an existing one in local configuration. And Let’s Encrypt allows you to revoke a certificate that has been issued to another account, as long as you can prove ownership of all the domains included in the certificate. You just have to fulfill the domain ownership challenges, similar to when requesting a new cert.

There’s a small catch though: ACME supports pre-authorization, which allows clients to prove domain ownership (by fulfilling challenges) as a stand-alone operation. But that operation seems to be optional and I’m not sure, if Let’s Encrypt supports it. In any case, certbot does not support stand-alone pre-authorization as of today (or does not document it).

My workaround is simple: I’ll just ask certbot to obtain a new cert for my domains and fulfill all necessary challenges. After that, my brand new Let’s Encrypt account will have authorization for my domains. With that, Let’s Encrypt will allow me to revoke any of the certificates that I’ve found in CT logs. Well, at least for those domains that I still own.

So far, this only works for revoking certificates that have been issued by Let’s Encrypt. I guess it would be easy to add support for other CAs that implement the ACME protocol. I’d just need some kind of mapping between CA root certs and ACME URLs… For now, I’d have to take manual action, if I ever find a cert from another CA in the CT logs for my domain. To find out, if that ever happens, my next step will be setting up alert notifications that will warn me whenever an unknown cert shows up…

Let’s Encrypt! [en]

… is where I’m getting the TLS certificates for this blog nowadays (after moving away from CAcert). I’ve been using Let’s Encrypt at work now and then. Many colleagues in my department are heavy users and my employer is a sponsor. So I knew what to expect and how to get started. Nevertheless, here’s a couple of experiences that I’d like to share…

Let’s Encrypt is all about automation, which eventually lead to the ACME standard. There seems to be a myriad of ACME clients out there, but Let’s Encrypt recommends certbot for getting started. So I defaulted to that one.

Certbot is fairly opinionated and makes it easy to get going without thinking too much about configuration details. No need to tinker with keys and CSRs, certbot does it under the hood. There’s some decisions to be made though…

Domain Ownership Challenges

First decision was, which challenge to use to prove my ownership of meeque.de? Using a DNS challenge would be nice, because it’s the only way to get wildcard certs from Let’s Encrypt. However, it would require giving certbot access to my DNS configuration, which I’ve recently moved to Route 53. But I don’t want to get too tight with the evil empire and I don’t need wildcard certs for now. So I opted for an HTTP challenge.

Even then, certbot offers a plethora of plugins that fulfill the HTTP challenge and install the resulting certificates in a web server. E.g. the standalone plugin spins up it’s own little web server, just for fulfilling challenges. There’s also a certbot plugin for nginx, the web server that I’m using for TLS-termination. But letting certbot tinker with my nginx configuration seems kinda invasive. Besides, it would require manual wiring, because I’m running nginx isolated in a Docker container.

So I picked the certbot webroot plugin instead and I like its minimalism. You just point it to a local directory where it can create challenge files. You can then use any web server you like to expose these files via HTTP, so that Let’s Encrypt can verify them. I’ve extended my existing nginx configuration for that. And since I’m running nginx in a container, I’m using Docker mounts to ensure that both certbot and nginx can access the challenge files. Works like a charm!

Deploying Certificates

Some certbot plugins go further than just sending ACME requests and fulfilling challenges. They also install the certs in a web-server. The webroot plugin does not do that though.

But that’s not a problem: certbot puts new certificates (and the corresponding private keys) in a predictable directory. All I need to do is point my nginx configuration to the right files. Once again, I’m using Docker mounts to ensure that both certbot and nginx can access the cert files.

In a typical nginx setup, you’d use separate virtual servers for HTTP and HTTPS traffic. And you’d repeat that for all DNS domains that you want to host. That causes a small bootstrapping problem:

  • In order to serve HTTPS traffic you need a certificate.
  • But in order to get a certificate from Let’s Encrypt you need a running HTTP server to fulfill ACME challenges.

There are several ways around this problem, e.g. bootstrapping with self-signed certs or spinning up a temporary web server just for the challenges. I went a step further, by using two full-fledged nginx servers, both isolated in their own Docker container:

  • The first nginx server (“challenger”) only serves HTTP traffic. And only ACME HTTP challenge files. It redirects all other requests to HTTPS.
    This server starts successfully, even when I recreate my hosting environment from scratch.
  • The second nginx server (“webfront”) only serves HTTPS traffic. It performs TLS termination for this blog (and could be extended to serve other virtual hosts).
    This server fails to start after I recreate the hosting environment from scratch, because nginx will not find the cert and key files that I’ve configured. However, a quick run of certbot will fix that condition and restart this server.

In practice, I’ve automated all of the above, with Terraform and Ansible. I can easily delete the whole host from my environment and get everything running from scratch within a couple of minutes. Including Let’s Encrypt certificates! (I’ve tested that just last week.)

What else?

Yupp, using another nginx instance just for ACME challenges seems a little wasteful. But I think there’s some elegance to it. And Docker makes it easy to implement this setup.

That covers my basic certificate needs. I can obtain certs from scratch and renew them regularly. I have a simple cron job for that part.

But there’s more to working with certificates. Just think monitoring and revocation. More on that soon…