Shoot yourself in the foot: crypttab edition [en]

I’m running Ubuntu on my laptop, using the standard disk-encryption that the Ubuntu installer provides. (Well, the one it provided a couple of years back, when I last installed from scratch.) This setup uses cryptsetup with LUKS on the main partition. This in turn contains an LVM physical volume, which contains a volume group with two logical volumes: one for root, one for swap. So far so good.

Problem

Two days ago, I let my laptop run out of power. But when I restarted it, I wouldn’t get the usual prompt asking me for the disk-encryption passphrase. Instead the Ubuntu startup screen would just show a wait animation for several minutes and then drop me to the initramfs shell (BusyBox). One of the error messages complained that it couldn’t find the expected logical volume with the root filesystem. I had no idea why, so I suspected (wrongly) that some recent Ubuntu update broke something.

Workaround

I fired up my phone and my work laptop and started searching for solutions. At first, I couldn’t find any good advice. So I resorted to my work laptop for a couple of days and waited until the weekend would give me more time to deal with this.

And indeed, today I found a great solution to my problem. Unlike other suggestions elsewhere, it didn’t even need to run a live Linux distro. All I had to do was wait until the Ubuntu boot dropped me to the shell and then decrypt my main partition. In my case, this command did the trick:

$ cryptsetup open /dev/nvme0n1p3 crypt-disk

I entered my passphrase when prompted, then ran exit and the boot process continued normally.

Solution

But I still didn’t know what was going on. And I was pretty sure that the same thing would happen on the next boot. Suspecting broken packages, I first ran an apt upgrade, but I didn’t see anything suspicious. Then I went back to this other proposed solution, which recommended running update-initramfs. When I did, I got this:

$ sudo update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.15.0-78-generic
cryptsetup: WARNING: target 'crypt-disk' not found in /etc/crypttab

And indeed, the respective entry (which had always been there) was missing from my crypttab file. So I added the following:

# <target name>	<source device>		<key file>	<options>
crypt-disk UUID=e6d0b181-b36f-4833-b87d-c5c4946f88e7 none luks

When I ran update-initramfs again, I got no more warnings. So I took my chances an re-booted, and … it worked!

Root Cause

But how did the crypt-disk entry disappear from my crypttab in the first place? It dawned on me immediately:

I’m using Ansible to manage my boxes, including my laptop. I also use it to manage crypttab entries for some encrypted external USB disks. But I do not manage the crypt-disk entry in the crypttab file with Ansible — I just stick with whatever the Ubuntu installer has given me.

This isn’t a problem per-se. The Ansible crypttab module only manages individual entries, not the whole crypttab file. It leaves manual entries in peace. But a couple of weeks ago, I did some manual testing and deleted the crypttab file. Just wanted to verify that the next ansible-playbook run would fix everything. And yupp, it worked.

But I totally forgot that I had manual entries in the crypttab file. When checking the results of the playbook run, I didn’t realize that something important was missing. And the problem only manifested weeks later, when I booted my laptop.

So in the end it was all my fault. Nothing to blame on Ubuntu, as was my first instinct. Also: I should reboot more often.

Iberia Impressions: Cantabria [en]

From the Picos de Europa I headed back to the Bay of Biscay. Cantabria was the last section of Spains northern coast that I hadn’t visited yet.

Santander

I only spent a few hours in Cantabria’s capital Santander, because I had to return my rental car there. As ever so often, the most apparent landmarks included old churches and modern architecture by the shore. Here, the latter was represented by Centro Botín, a stilted museum looking like some kind of space-ship. Oh, and they have the straightest clouds that I’ve ever seen.

San Vicente

The last surf destination on my trip was at San Vicente de la Barquera, about half way between the town and neighboring Comillas. While all of northern Spain was way greener than I had expected, the area around San Vicente was particularly colorful. White-blue skies, turquoise waters, yellow sands, green meadows, black forests on the slopes of the distant Picos, and purple sunsets — almost too kitsch.

It was still windy, but way less windy then in the Picos. The waves had grown stronger again and the offshore breezes didn’t seem to disturb them too much. A great setting for my last week in Spain.

Well, San Vicente was not quite my last surfing destination on this trip. On my way back east I had another brief stop in Donostia, but I’ve already shared my impressions from there. After that, I was ready for my long train-ride home to Munich.

Iberia Impressions: Picos de Europa [en]

I hadn’t known about the Picos de Europa mountains before planing my trip to Iberia. But a Spanish colleague from work recommended them to me, when I asked for hiking recommendations in the Pyrenees. Pictures on the Internet looked promising and it was not far off my route along the coast. Since the swell forecast looked flat for a couple of days, I decided to pay the “Picos” a visit.

I struggled to find good train or bus connections (maybe because it was off-season?) so I rented a tiny car for a couple of days. I picked it up in Gijón and dropped it off in Santander afterwards, because that was the closest city to my next surfing destination.

Lagos de Covadonga, Vega de Ario, Belbín

My first stop was the Lagos de Covadonga (Lago Enol and Lago de la Ercina) which are situated in the western part of the mountain range. This place must get crowded in summer. I hear the road gets closed and you have to take shuttle buses from the valley. I visited in the end of October, so I could drive all the way up to the lakes myself. It was pretty deserted up there, just a few dozen tourists by the lakes.

But I wanted to venture further up, to the Refugio Vega de Ario. I had downloaded some GPS tracks for this trip. On the way up, I followed the main hiking trail. The Picos are mostly limestone and this part forms something like a plateau. It is kinda hilly and rocky, but there’s no big climbs. It reminded me of Steinernes Meer back home at the border of Germany and Austria. But here in the Picos, you get views of the actual sea, not just a sea of rocks.

It was super windy. Though the whether forecast said it would remain dry, I saw dark clouds moving in from the south. Luckily they dissipated once they reached the main mountain range to the south, and I didn’t get caught in any rain.

It took me about 3 hours to reach the Refugio. And after leaving the Lagos, I didn’t meet another person. The Refugio itself was closed down for winter. A small room in the entrance area was left open and served as an emergency shelter. And there was some outdoor seating, which I used to have a very windy lunch.

On my way back I tried to follow a different GPS track, which was kinda tricky. While there was traces of human and animal traffic in some sections, there was no visible clues of a trail in other parts. I got off track for a while and I found it cumbersome to get back again. After I joined the GPS track I checked it on my phone every few meters, in order to not get lost again. (Luckily I had brought a battery pack, but had my phone broken down, I could have been in real trouble.) On the plus side, I had great views of the nearby rock formations, the sea in the background, and even some rainbows.

I got back onto a proper trail about half-way back to the Lagos. There, I also encountered the only people that I’d meet on my hike: two cowboys who were driving a herd of cattle down the hills. The trail led me through the picturesque Majada de Belbín. Old stone huts surrounded by free roaming animals, like a donkey, a couple of dogs, several cows, and a big scary looking bull. From here a comfortable gravel road led back to the Lagos, with horses grazing nearby. No more need for GPS.

Covadonga

The village of Seattle Covadonga is located in a deep valley, further down from the Lagos the Covadonga. There’s a holy cave (not pictured) and an impressive christian church located on a big hill in the middle of the valley.

From Bulnes towards Picu Urriellu

The next day I drove across the Rio Cares into the central massif of the Picos de Europa. I was heading to the remote village of Bulnes, which can be reached easily by a cable-car leading through a long tunnel. Bulnes is a small village (well, actually two villages) with tiny old houses built from stone. Its lower part is situated inside a deep valley, with towering cliffs all around. Its upper part is located further up the slopes, right beneath one of these cliffs.

But Bulnes wasn’t my only destination for the day. Instead, I hiked further uphill through another deep valley towards a mountain called Naranjo de Bulnes (Spanish) or Picu Urriellu (Asturian). It is not the highest mountain in the Picos, but it has a very distinctive peak. I’ve seen many mountains all around the world, but Picu Urriellu is certainly one of most sightly ones.

I didn’t have much time left, so I didn’t expect to make it all the way up to the Refugio de Urriellu. Let alone climb the peak itself. But I got fairly close to it, about half way between Bulnes and the Refugio. This gave me a good view of the mountain and its surroundings. Compared to Lagos the Covadonga area, the terrain here is much more mountainous. Still limestone, but steep rocks rather than a plateau.

I hiked up for about 2h and after leaving the village of Bulnes I didn’t meet a single person. The weather looked threatening, but in the end it only rained a couple of drops. And the valley that I passed was somewhat sheltered from the wind. I descended by the same route, so here’s some pictures from top to bottom:

Iberia Impressions: Galicia & Asturias [en]

After two weeks of surfing in Portugal, I headed back north, to the Spanish region of Galicia. They must have great beaches and surf-spots, but I hear that many of them are pretty remote. Traveling by bus/train and not having infinite time left, I decided to leave Galician waves for some other time and to head further east. But since I was coming through anyway, I decided to have quick peek at Santiago…

Santiago de Compostela

Galicia’s capital is world-famous for being the destination of the Camino de Santiago. So whenever you’re at a train-station, bus-stop, hostel, or country road in northern Spain or northern Portugal, you’ll have a good chance to meet pilgrims on their hike to Santiago. I’m not overly keen on the religious background of “The Camino”, but some secular folks who do it, too. They just enjoy the hiking, the scenery along the way, or their timeout from day-to-day life.

I arrived at Santiago in the late afternoon and journeyed on early next morning. So I only had one night to see the old town. By the time I got there night had fallen. The dark blue sky and the yellow lights reflecting off stone walls gave the place Mêlée-Island vibe. Though most buildings are certainly more recent, all looks kinda medieval. And yes, Santiago is soaked in christian architecture and symbolism. It’s not just the great cathedral, there’s churches everywhere. And a penis shaped pillar for some reason, must be a religious thing.

Gijón (Xixón)

After having left the Basque Country more than 2 weeks earlier, I was back to the Bay of Biscay. My first stop was the Asturian port of Gijón. Though bigger, the geography kinda reminded me of Donostia: there’s a peninsula with big hill in the middle. The old town is located right underneath, on the land-facing side. From here, big bays stretch along the coastline in both directions. The city itself rises right behind their beaches.

So there’s clearly an urban vibe in the surf line-up. But the scenery looks quite pleasant, especially when big swells are pushing in. Like elsewhere in Spain, there’s a lot of (monumental) artworks sprinkled in and around the city. A good mix of nature, history, and modernism.

Ribadesella (Ribeseya)

After Gijón, I headed further east along the cost. Took a lunch-break in the small coastal town of Ribadesella. They’ve got a nice beach there and I hear they get good surf. But for the next few days the Cantabrian Sea would remain flat. So I turned inland from here and headed into the mountains.

Iberia Impressions: Portugal’s North [en]

After Madrid, I headed for the coast of northern Portugal. Less sight-seeing and photographing, more surfing and partying. Didn’t even make it nearby Porto. Well, I’ll get to see it some other time…

Esposende/Ofir

First stop was the Ofir peninsula near the town of Esposende (the proper Portuguese pronunciation of which I will never learn). My accommodation was basically in the dunes right behind the beach. There’s also an estuary/lagoon right by the house. A good kite spot in theory, but as ever so often there was barely any wind during my visit. So I focused on wave surfing.

Viana do Castello

Just am small coastal town, half-way between Esposende and Moledo. I came through it twice, because I had to change between train and bus here. Looked lovely, but I couldn’t spend much time.

Moledo & Âncora

Next destination was Moledo do Minho, a coastal village, whose scenic beach stretches all the way to the river that forms the border to Spain. The beach of nearby Vila Praia de Âncora wasn’t bad either. And we had decent surf conditions at both.

From here, I headed right back to northern Spain.

Iberia Impressions: Madrid Stopover [en]

I hadn’t been to Madrid before and the city has always been a mystery to me. How could one of the most notorious maritime empires have been governed from a landlocked place like this? The geographic location seemed unappealing — no harbor, no impressive mount-range serving as backdrop, not even a noteworthy river flowing through.

But it looks like in Spain all roads lead to Madrid, rather than Rome. So for bridging the distance between Basque Country and Portugal, a stopover in Madrid just seemed convenient. Other than this, I didn’t expect much, but this is what I saw…

Let’s do some Tourism

I booked an overpriced hostel, near Plaza de España. It turned out that it wasn’t far from the Royal Palace, the Almudena Cathedral, or Plaza Mayor either. So, expect for a short train ride to the area of El Retiro Park, I explored all of that by foot. I have to say that the city has an impressive vibe. Everything looks big: the streets, the buildings, the monuments, the parks. Not big in the sense of skyscrapers, but in a sense of grandeur that cities at home lack.

The parks in particular. Parque Casa de Campo starts right by the palace and the cathedral and it seems to stretch to the horizon. It’s probably the biggest park in Madrid, but not the only one by far.

Madrid Street-Art Special

I had dinner someplace around the Malasaña quarter, which quite lively at night. I also noticed loads of art on the walls and shutters. So I just had to come back the next day during daylight and have a closer look…

Iberia Impressions: Euskal Herria (Basque Country) [en]

After visiting Biarritz on the French side of Basque Country, I finally arrived in Spain and on the Iberian peninsula.

Donostia (San Sebastian)

First stop was in Donostia, a city famous for it’s surf beaches, like Zurriola. It’s located by a river mouth and two bays. In the middle, there’s a small hill raising from the sea and the old town is right beneath it. Great views from the top.

Sopela

Moved west to Sopela, a small coastal suburb of Bilbao. Great beaches surrounded by small cliffs are in walking distance. Unfortunately, it was all rainy and stormy during my stay. Except for the day I arrived and the day that I left:

Bilbao

The bad weather on the beaches gave me plenty of opportunity to visit nearby Bilbao, the largest city of Basque Country. The most famous tourist attraction still seems to be the Guggenheim museum. I must have seen hundreds of pictures of it, but I had never noticed some of the details. E.g. how it integrates with the bridge over the Nervión river, or the weird arachnid-inspired sculpture by its side. Here’s my impression inside and around the museum on a rainy day:

The rest of Bilbao is also quite charming. Am mix of old and new. Even the housing blocks that rise on the the slopes of the river valley have something to them:

Next stop: Madrid.

Iberia Impressions: In Transit through France [en]

I’ve already written way too much about my recent Interrail surf-trip to Spain and Portugal. So this time let my just dump some photos from the places that I came through. Let’s start with my journey through France…

Paris

Two days stop-over. Always liked La Defense and the modern stuff (we don’t have much of this back home in Munich):

I’ve been to the Eiffel Tower before. But this was the first time I was touristy enough go up and enjoy the views:

Other than that, I was just strolling around enjoying the pleasant fall weather:

Biarritz

My first stop on the Basque coast. Picturesque town in between the sea and the cliffs. Forgot to take pictures at Grande Plage, so these don’t give a full impression:

Next up: the Spanish side of the Basque Country.

The Gotchas of Railroad-Safety [en]

They say trains are one of the safest modes of transportation. And while I can think of several horrific train crashes in the past, I generally believe it’s true. But now I’m wondering about all the small things that could go wrong? How common are they?

On my last surf-trip, I witnessed this small incident after getting off a Renfe train in A Coruña. Everyone was heading for the exit of the terminal, when suddenly there was some screaming. As I turned around, I saw that someone had got their foot jammed in a retracting footboard. Other passengers tried to alert the railroad staff, but it took a while till they arrived at the scene.

Since I still speak zero Spanish, so I figured that there was not much I could do. So I decided to document the situation instead:

When you look beneath the conductor in the suit, you’ll see these two feet in white slippers. And one of them is stuck between the wagon and the footboard, which is trying to tilt 90° upwards. You cannot see much of the trapped woman herself. Somehow she managed to fall backward into the into the train, leaving her foot at an awkward angle. It did look quite painful.

By now all other passengers had moved on and it was clear that the railroad crew didn’t know how to deal with the situation either. I was thinking that it might be a good idea to lift the accident victim over to the outside before trying to pull her foot out. But she looked fairly heavy and I feared that it could make her situation even worse. Besides I couldn’t communicate with anyone in Spanish (and I figured now was not the time to distract them with English).

The conductor and me tried to put our own weight on the footboard, but it didn’t move a bit. Next the conductor tried to grab the foot by the heel and yank it out. I joined in and started pulling by the toes. I was really worried we could twist her ankle, but I didn’t hear any screams. And slowly the stuck foot started moving and we got it loose after a few seconds.

The footboard immediately continued tilting upwards. Still putting weight on it, me and the conductor almost got our own feet stuck. But we managed to get out in time. The women looked shocked, but after a short while she could get up again. Looked like she could also walk normally. She didn’t look too pissed either.

In hindsight I wonder, if the emergency door release would also release the footboard? The release lever was right next to the door, but I hadn’t noticed it in time:

The railroad staff hadn’t made any attempt to use it either. So maybe it’s really just for the door?

I’m still baffled that the footboard didn’t have any safety-mechanism built in. Shouldn’t the retraction mechanism back off when it detects resistance? I’d sure expect automatic doors to back off, so why not the footstep?

I also wonder how other trains deal with this? I think the footboards of most ICE trains at home do not tilt, but rather slide. I guess they could still squeeze your foot against the platform, but you’d really have to be in the wrong place to allow this to happen. Gotta be more careful around trains, I guess!