Meeque.de

Apa Kabar, Lombok? [en]

When I visited Indonesia back in 2003, I spent most of my time on the island of Lombok. On my latest trip, I saved Lombok for the last three weeks. I sorta knew what to expect there, so travel planning was less of a hassle, as compared to Sumbawa or Flores/Komodo.

The phrase “Apa kabar?” means “How are you?” in Bahasa Indonesia. The answer is usually “baik” or “bagus”. So, I was wondering: How is Lombok doing nowadays?

Lombok Then And Now

I surely knew that a lot would have changed over the past 20+ years. So let’s start by drawing some visual comparisons:

Back then, the town of Kuta Lombok was barely visible from the hills above, …

Mandalika

Even when I first visited Lombok ages ago, I had heard about government plans to boost tourism on Lombok. Seems to make sense, considering the over-tourism on neighboring Bali.

Only now did I learn about the scale of tourism development. One of the drivers behind this seems to be the newly established Mandalika Special Economic Zone, which now covers a huge chunk of Lombok’s south coast. (According to OSM, it stretches between Kuta and Gerupuk.)

I guess that all of this is aiming at creating a (high-end) resort area, similar to Nusa Dua on Bali. However, development is not quite there yet – and there are several distinguishing factors. Notably, one of the first projects that they implemented is a motorsports race track, the Mandalika International Street Circuit (a.k.a. Pertamina Mandalika International Circuit a.k.a. Sirkuit Internasional Mandalika). Completed in the early 2020’s, it regularly hosts MotoGP races since 2023, and other international competitions.

This phase of development has also created a new road network in the area. Mostly multi-laned roads connecting at massive roundabouts. These are generally arranged in a rather confusing manner. Other developments – big hotels, golf course, amusement park – are lagging behind, so all the new roads feel ridiculously oversized and sometimes end in the middle of nowhere:

Everything has “Mandalika” in it’s name now.

I’ve heard rumors that locals have been driven off their land for some of the ongoing construction projects. And there’s certainly some impact on nature (which includes mangroves). However, my overall impression is that this whole Mandalika thingy could be reasonably sustainable in the end. It’s certainly not a super-sized bridge-to-nowhere situation. And some of it seems to benefit the locals directly, not just those working in tourism.

But where does the name Mandalika come from? When I visited Lombok in 2003, I’d heard of place names like Kuta, Gerupuk, Mataram, Lembar, Mawun, Senaru, Gunung Rinjani, Segara Anak, Gili Trawangan, etc. In fact, I had visited most of these. But I cannot recall hearing of any place called Mandalika. So why is it being used so prominently today? I forgot to ask the locals before I left, so I had to look it up just now. Turns out that it’s named after Princess Mandalika, a character from Sasak folklore. Apparently, her story is still being commemorated by the annual Bau Nyale festival.

Gerupuk

When arriving on Lombok a couple of weeks ago, I decided to skip busy Kuta. Instead, I headed straight to Gerupuk, a small fishing village located in a deep bay a few kilometers further east.

I had visited Gerupuk before, because of its famous surf spots. But back then in 2003, we did not stay over night, because there just wasn’t any suitable accommodation there. Not even a cafe or a warung. There only was a small shack by the shore, where you could rent surfboards (from a very limited collection, not at all beginner friendly). I think that most houses had been built of wood and reed, though I missed the opportunity of taking photos of the village itself.

Much has changed since. By rule of thumb I’d say that Gerupuk today is similar to Kuta Lombok twenty years ago. There’s several guest houses, restaurants, warungs, mini-markets, surf-shops, etc. But most of it is at a fairly small scale and locally owned. There is also a couple of hotels nearby, but no oversized resorts yet:

Gerupuks shore, near Aussie Skin Cafe (I think).

I decided to stay at Edo’s Homestay, which is a great starting point for boat trips to Gerupuk’s surf spots. They have a nice restaurant right at the beach, plus surf and scooter rental. I liked it so much, that I stayed with them for my whole Lombok sojourn (about 3 weeks).

Wet Season Road Conditions

Like for most of the Lesser Sunda Islands, wet season hits Lombok between October and April. My whole Indonesia trip fell right into this period, and I was constantly on edge about it. Whenever heavy rainfall occurred (mostly in the evening or at night) I was thinking to myself: well this is it, it’s gonna be like this for the rest of the trip. But then there would be a couple of dry days and I’d forget about raining season altogether.

I do actually enjoy a moderate rain shower in the surf lineup, it can be rather peaceful. But I also wanted to move around by scooter, and wet season road conditions turned out to be challenging at times. Generally, Lombok has a pretty decent road network. Not just the shiny new roads of Mandalika, even small country roads between rice paddies and up the slopes of Gunung Rinjani are in good shape. But occasionally, the rains do have impact:

A little water on a sealed road is no big deal.

Unfortunately, local authorities didn’t seem overly eager to fix these problems right away. I guess that they’ll do a big round of cleanup at the end of wet season.

Tetebatu

One of my day-trips took me to the small village of Tetebatu. It’s situated on the southern slopes of Gunung Rinjani, right at the border of Rinjani National Park:

Making my way through the plains of central Lombok, …

I didn’t take many pictures in the village of Tetebatu itself. It’s certainly a nice village, but it’s kinda spread out and most of the buildings simply blend into the surrounding trees. That said, I’ve only seen a small part of it and I may have missed something interesting.

Awang & Sade

Another day trip took me to Awang, the next big bay east of Gerupuk:

The road to Awang winds through many hills and is more modest then Mandelika’s roads.

These tsunami escape route signs (as seen in that last pic) have been pervasive throughout my Indonesia trip. I’ve seen them in all coastal regions, be it on Bali, Lombok, Sumbawa, or Flores. And frankly, I’m not surprised that Indonesians are nervous about tsunamis after the disaster of 2004.

I didn’t stay long in the Awang area, just enjoyed the landscape and had a little snack at a roadside warung. Then I headed back west, bypassing Gerupuk and Kuta, and made my way to Sade. This is a small traditional village, about 6km inland from Kuta. Frankly, it’s a bit of a tourist trap, but I enjoyed it nevertheless:

Sade is located on one of the many hills in this area, …

I really love these straw roofs. On Bali they are often incorporated into modern buildings, in particular in touristy areas. But on Lombok, Sumbawa, and Flores they are pretty rare. The guide in Sade told me, that the straw cover needs to be replaced after around ten years. I guess tin roofs are less of a hassle in most contexts.

Lombok’s South Coast

Apart from the trips to Tetebatu and Sade, I mostly stuck to Lombok’s south coast. This is not one long beach like the one that stretches from Kuta Bali on to Seminyak and Canggu. Instead, it’s all hills, cliffs, capes, and bays. Many of these bays harbor amazing beaches (“pantai”). Here’s some of them, roughly from east to west:

Pantai Muluq Indah Permai, right next to Gerupuk,, where seaweed is farmed and dried.

Many of the above beaches have surf spots, but I haven’t tried all of them. Let’s have a look at those that I’ve surfed…

The Surf

Pantai Mawan is not famous for surfing, but there are small peaks at the flanks of the bay. I tried the right-hander on the western side. Short rides, but no crowds at all. Despite the many warungs at the beach, surf-rental opportunities are rare – had to ask around for a board.

One spot that I visited repeatedly was Areguling, the next bay west from Kuta. There’s a channel in the middle of the bay, with surf breaks on each side. Not too far to paddle out yourself, but you can get a boat-ride for a reasonable price, and there’s plenty of surfboards to rent. It’s a reef break, and the right-hander can break pretty fast even on smaller swells. The surrounding hills shield some of the westerly winds that seem prevalent during wet season. And it’s usually less crowded than the spots at Gerupuk.

There’s long mellow a-frame in the middle of the bay at Pantai Aan, not far from Gerupuk. This spot is popular with beginners and long-boarders. Here was the only time that I picked a proper long-board, a heavy 9 footer. Despite some wind, the spot worked surprisingly well. Took me a couple of waves to get the hang of turning such a big board though.

Having established my base right in Gerupuk, I stuck to the surf spots in Gerupuk bay on most days. All of these are best reached by boat from Gerupuk. I’ve visited Inside, Don Don, and Outside Rights, but skipped Outside Lefts (too far) and Kiddies (only works on bigger swells nowadays).

Most often, I’d end up at Gerupuk Inside, which worked better at low to mid tides. It’s fairly exposed to westerly side-shore winds, which can also create a bit of a current. I was too lazy to get up early and beat the wind, but usually it calmed down a little in the late afternoon, too.

Gerupuk Inside is an a-frame, though usually the right offers longer rides and an easier paddle out. That said, I had some excellent waves to the left. The main peak can be powerful, suitable for shorter boards. I was on a 7’6″ fun-board most of the time though.

Further down the line is better suited for long-boarders and beginners, so it gets pretty crowded. Many people book surf guides who push them into the waves, and have little regard for priority rules. At the main peak, surfers were more considerate on average. But the peak is a bit shifty, and I found it hard to tell whether people would go left or right. Even when both sides were clear, I sometimes struggled to decide which way to go myself.

There is a second peak about 100m to the west of Gerupuk inside. Some locals call it Coco, because of the coconut trees in front of it. I had one great session there, with just 5 people in the lineup. But this wave tends to get too soft when conditions aren’t just right.

Another spot, called Don Don, is about half way between Gerupuk village and Inside. When conditions check out, this a-frame works equally well in either direction. The crowd factor is similar to Gerupuk Inside. One day, there was a photographer around and I got some rare footage of surfing on Lombok:

Gerupuk Outside (more precisely: Outside Rights) is a little different. First off, the scenery is amazing, with a big cliff towering over the lineup. The cliff shields some of the westerly wind, and wind from north-west can create excellent off-shore conditions. Gerupuk Outside is more exposed to the ocean can pick up more swell than Inside or Don Don.

During my stay, there were usually three distinct sections at Gerupuk Outside. The first section produced huge waves (2 meters or more) but looked rather soft, with wide shoulders. Nevertheless, I skipped this section, fearing to get washed away when a bigger set rolls through.

The second section was smaller but tended to break faster and close out after a short ride. After catching a wave, I had a hard time paddling out again. Especially with a bigger board that cannot be duck-dived. On the other hand, I had trouble catching enough waves on shorter boards (less than 7 foot). I got a little scared of the stronger waves at times and got tired quickly by all the paddling.

The third section was yet smaller and a little mellower. And less prone to close out, so I could ride it out all the way to the right for an easy exit and paddle back out. I often opted for these more mellow waves, which worked well with the 7’6″ fun-board that I was using most of the time.

While Gerupuk Outside did draw massive crowds, these tended to spread out across the different sections. Less experienced surfers (and their guides) mostly picked the last section and surfers at the other sections were more respectful of priority rules. By the way, Gerupuk Outside worked best at high to mid tide. It got quite shallow at lower tide, but more advanced surfers could get faster breaking waves and even barrels.

Gerupuk Outside Rights as seen from nearby hills, …

Despite the huge crowds, Gerupuk is still a great surf destination. With so many spots, there’s always one that’s gonna work.

It’s a little annoying to depend on a boat all the time. But with some patience, there are enough people around with whom to share a boat. Prices are OK, too, for both board rental and boat charter – even compared to Bali or Sumbawa. That said, there’s too many middle-men involved and prices can vary widely. Once I had been around for some while and got to know the locals, we settled on a standard price that I found reasonable. I guess more aggressive haggling could reduce prices further, but there was no need to overdo it.

I’ve had a look at some online reviews recently, and some speak of a “Gerupuk Surf Mafia” that drives up prices and makes the overall surf experience more painful. Some people claim that they have been pressured to book a surf guide or lessons, and experienced hostilities in the lineup when going without a guide. This has never happened to me, even though I never booked a guide.

It helps knowing the different spots and having a general idea which one works best for your skill level and current wave conditions. Otherwise you might end up on a boat towards a session that you will not enjoy. When in doubt, you can have a look at multiple spots first, or change to a different spot after a while. (Most boat captains ask for a small extra fee in such situations.)

As mentioned above, some of the surf guides have little concern for priority and push their clients into waves that are already taken. But this is no different from the surf guides at an average spot on Bali. Overall, the vibe stayed friendly and welcoming in every lineup that I’ve visited on Lombok.

Odds And Ends

My first trip to Lombok back in 2003 has been very special to me. But I’m happy to say that I still enjoyed Lombok now as much as back then.

So I’m ending this post with some of the creative signage that I’ve observed on Lombok:

Unfortunately, the soup-dispenser did not work. Bernie would have been disappointed.

Shoot yourself in the foot: WordPress UAM vs. HTTP Cache Headers [en]

Ever since I had reactivated this blog a couple of years ago, I noticed that something was wrong with its content caching. On each page reload, the browser would download all images from scratch. Even though it should have been easy to cache these images.

There are various standards govern how web browsers should cache resources. And, which HTTP response headers web servers can set to advise on desirable caching behavior.

I had kinda expected that the WordPress media system or the underlying Apache httpd server would set reasonable defaults for the most important cache headers. In particular, for static resources (like images) that are big and do not change often. But my WordPress did not send response headers like ETag or Last-Modified when serving images that I had uploaded.

As a work-around, I added a hard-coded Cache-Control header with a max-age=3600 directive to all HTTP responses that serve media. This is not a very efficient mechanism, because it limits caching to 1 hour and does not take changes to individual resources into account. But it seemed a good-enough compromise, and I ignored the remaining problem for several years.

Let’s revisit this…

I’m currently traveling Indonesia and I’ve been posting about it. Including heaps of photos, so this image caching problem started to bother me more and more. I decided to take some time and have another look at it.

I started with a quick web search for how to enable ETag headers in WordPress, but strangely I could not find any good results. Searches for wordpress caching yielded loads of hits about various server-side caches and on advanced cache headers tweaking. But nothing about simply getting the basics running.

Next, I searched for solutions at the Apache httpd server level and found its FileETag directive. Looks like this is a core part of Apache httpd and is active by default. This default generates ETags based on resource modification time and size in the underlying filesystem. But somehow I could not see any of these ETag headers in the HTTP responses.

So I checked my Apache httpd config (most of which is coming from the wordpress:apache Docker image) for FileETag overrides, but I couldn’t find any. I did find a .htaccess file that is part of my WordPress installation and directs most HTTP requests to WordPress’ own index.php script. However, this explicitly excludes resources that can be served directly from the underlying filesystem. These should still be handled by the Apache httpd.

But why didn’t it work for me? I finally decided to do some testing with a brand-new WordPress installation. This came with the same configuration in that .htaccess file, but to my surprise it served reasonable cache headers out-of-the-box. So I had to dig further, and figure out what’s different in the WordPress configuration of my blog…

What does the UAM plugin have to do with this?

Eventually, I found another .htaccess file in the wp-content/uploads/ directory, where the WordPress media system keeps all images and such. A quick look inside revealed that this config file directs HTTP requests to a script that is owned by the WordPress User Access Manager (UAM) plugin.

Ok, this finally made sense! I had installed the UAM plugin ages ago, because I wanted to share certain contents with friends only. That is, people with a user account for my blog. Obviously, the UAM plugin needs to intercept HTTP requests for media resources in order to run authorization checks.

A quick look into the UAM settings revealed that the UAM plugin had indeed installed this other .htaccess file. I even found a way to regenerate the file, but unfortunately this did not solve the cache headers problem.

Not sure why UAM does not set reasonable cache headers? Its authorization concept does not seem to prevent caching. Maybe UAM does support cache headers after all, but I clicked through its numerous settings and could not figure out how. Superficial search in the docs did not yield a solution either.

Bye-bye, UAM!

In the end, I decided to de-activate the UAM plugin. I don’t claim that this was the only solution, but it did work for me.

I only had a couple of posts (and about a hundred photos) that were protected by UAM authorization rules. These were all 15 years old, and I didn’t need them anymore. So I deleted them, cleaned up any dangling resources and configuration, then parted with the UAM plugin.

Now that the UAM plugin is de-activated, my WordPress blog is sending reasonable cache headers. In particular ETag or Last-Modified. Jumping back and forth between blog posts feels much smoother, now that the browser caches images. In the current setup, the browser still asks my WordPress server for changes before using a cached image. But these HTTP requests are lightweight and fast.

When it comes to sharing photos and other resources with friends, I don’t need to rely on WordPress and the UAM plugin anymore. In recent years I’ve been using my Nextcloud server instead, which has great file-sharing and access management capabilities.

Komodo & Flores – Last Chance To See? [en]

It’s been a while since I’ve read Douglas Adams’ (et. al.) Last Chance to See, but I remember that it had Komodo Dragons in it. The German edition prominently features a Komodo Dragon on its cover, while the English editions feature the diverse animals that are discussed in the book in a more balanced manner. If I remember correctly, Adams was surprised how accessible Komodo Dragons were, as compared to the other animals in the book. And maybe not threatened quite as severely as some of the others.

When Adams visited Komodo island (presumably in the 1980s), Komodo National Park had already been established. The park comprises the two islands of Komodo and Rinca and many smaller islands in the area. It also protects the marine environments that surround these islands, and I had heard good things about scuba diving there.

I bet that even back in the day many tour operators offered Komodo trips to tourists.

Labuan Bajo Base Camp

The Komodo National Park lies between the islands of Sumbawa and Flores. It’s part of the Indonesian province of Nusa Tenggara Timur (NTT), the same province as Flores. After spending a couple of weeks on Sumbawa, I thought Komodo would be worth a detour.

Initially I wanted to book a liveaboard cruise to visit Komodo National Park. There are numerous operators who offer such cruises, with or without a diving package. However, the price range is rather high, and I heard that most of these cruise boats are crammed. Next I thought, I could settle in on the eastern end of Sumbawa and do day trips to Komodo. However, there are hardly any operators who offer such trips from Sumbawa.

Turns out, that Komodo and Rinca are much are further away from Sumbawa than they are from Flores. They are separated from Sumbawa by the Sape Strait. This is why almost all Komodo tourism goes through the town of Labuan Bajo, located on the north of the western tip of Flores.

I arrived at Labuan Bajo by overnight ferry from Sape on Sumbawa. The ferry was surprisingly comfortable and arrived surprisingly early in the morning. After breakfast, I started looking for accommodation and moved into Manta Manta Guesthouse, situated on the hillside right above the town center. Had great views on the town, the marina, and surrounding islands right from my room.

Overall, Labuan Bajo is a tremendously picturesque. At nighttime, it reminded me of some scenes from Monkey Island:

Compared to Sumbawa, Labuan Bajo is very much a touristy place. It wasn’t overly crowded, because I visited during wet season. But it’s got everything that tourists desire, without obvious signs of over-tourism. I’ve heard that the Indonesian government has invested into local infrastructure in recent years, and I think it shows. Labuan Bajo is also one of the cleanest towns that I’ve seen in Indonesia so far.

Komodo Day Trip

The Komodo National Park is Labuan Bajo’s main tourist attraction, and numerous tour operators have offices in the town center. Most of them seem to offer similar Komodo day-trips, that include land visits and snorkeling. Itineraries and prices only seem to vary marginally.

So I just booked a random tour and hoped for the best. The tour started at a modern passenger terminal at Labuan Bajo’s marina. I was surprised how many boats were leaving here, despite low season. Ours was a modern fast-boat, designed for about 30 tourists, and I think it was fully booked.

Our first two stops were on Pulau Padar, where we first hiked up to a small hilltop to enjoy the view. Then we headed over to “pink beach” for some sun-bathing and swimming:

Next, the boat took us to Pulau Komodo, arguably the main attraction of our trip. There are two major tourist outposts in its southern bay, each equipped with a massive jetty. We headed to the less crowded one, deeper in the bay. I think there’s no real village there, just a couple of warungs by the beach.

Here we met with our obligatory national park guides, and went for a short walk further inland. Right in the beginning, I saw a couple of deer and boars, who came surprisingly close to the human settlement, but quickly fled as they noticed our group. This gave rise to optimism, since Komodo Dragons like them as pray.

However, it took a little longer till we found any dragon. It was around noon by now, and most of them would be resting in the shady forest to avoid the heat. They were just lying there, showing no signs of activity whatsoever. The first one had its eyes closed initially, and frankly I wasn’t sure, if it’s even alive.

I found this slightly disappointing, but I’m not sure what else I had expected? I guess it’s unlikely to find Komodo Dragons roaming the forest, hunting, or feeding during such a short visit. On the bright side, their lack of activity allowed us to approach the Komodo Dragons very closely.

Many tourists took photos with the dragons, always protected by our guides’ defensive sticks. This does not mean that the guides held down the dragons. They didn’t touch them at all. But they were prepared to deflect the dragons in case of an attack, which is really rare though.

After watching two dragons resting in the forest, we finished our short round-trip and headed back to the settlement by the beach. To my surprise, we met three more dragons here, right at the beach. In fact, I almost stumbled over one of these dragons, since I was looking in a different direction. Locals had to warn me before I realized that the dragon was lying right behind to me.

This dragon was slightly more active than the ones we had seen further inland. It was propped up on its front legs, sticking its tongue out to smell. Another one came walking down the beach just moments later. Not too bad after all:

After leaving Komodo island, we made three more stops in Komodo National Park for snorkeling. Our hope was to see reef manta rays and we were not disappointed. At our last stop (close to Pulau Mauwang, I think) we met a large manta circling above the reef at a cleaning station for at least 20 minutes.

Soon, a big crowd of snorkelers had aggregated above the manta, who did not seem bothered. The manta was at a depth of about two to four meters. I got carried away and dove down to the manta’s level a couple of times. I tried to keep to its side and give it enough space though:

Diving in Komodo National Park

The next day, I headed to Scuba Junkie Komodo and spent a couple of days exploring the marine part of Komodo National Park more closely. This dive resort is located to the south-west of Labuan Bajo, near the village of Warloka. It’s on the mainland of Flores, not within the national park itself. But its only 1 km away from the island of Rinca, thus very close to the park. It’s best reached by boat from Labuan Bajo, but due to rough weather the journey took longer than expected.

I have very few photos of my stay at the dive center, none of them under water. The reason is that I was busy taking a course to get the Advanced Open Water certification.

Frankly, this was harder than expected. While I’ve always felt comfortable diving, I struggled focusing on other activities in parallel. E.g. using the compass and counting fin strokes for under-water navigation, while maintaining buoyancy at the same time. I’m not great at multitasking in general. But in the end, it was all good enough to pass.

The whole course and some additional dives took place at many amazing dive sites in Komodo National Park. We got to see more mantas, but also stingrays, reef sharks, moray eels, seahorses, frogfishes, turtles, tunicates, echinoderms, octopuses, squid, nudibranchs, mantis shrimps, lobsters, sea anemones, corals, … and many Teleost fishes (which I’m still struggling to tell apart).

And All the Rest?

Unfortunately, Flores is not famous for surfing, so I decided to move over to Lombok after all the diving. That said, Flores left a good impression on me. And it certainly has great beaches to enjoy, exotic wildlife to watch, interesting villages to visit, and mighty volcanoes to climb.

One thing that I was itching to see was Liang Bua cave, the site where Homo floresienses was first discovered. But in the end I decided against this detour (almost 4 hours by car from Labuan Bajo). There probably isn’t much to see in the cave itself, though the small museum might be nice.

Either way, this will have to wait for another time. Just hoping that I haven’t missed my last chance to see all of it.

Back to Wallacea: Sumbawa [en]

After my Singapore stopover, I headed straight to the Indonesian island of Sumbawa. Sumbawa lies between Lombok in the west and Flores in the east. Its location can be classified in many different ways. Amongst others, Sumbawa is part of the Pacific Ring of Fire, the Lesser Sunda Islands, the province of Nusa Tenggara Barat (NTB), and Wallacea.

As a biology nerd, I find this latter classification the most interesting one. Wallecea (which I had visited before) is separated from Asia by the Wallace Line and from Sahul by the Lydekker Line. This means that the flora and fauna of Sumbawa is distinct from both the Asian and the Australian/Papuan flora and fauna. Instead, it’s something in-between.

I’m certainly not skilled to confirm this distinction by my own observations. But when it comes to climate, Sumbawa seemed just a little dryer than Bali. I heard that Sumbawa hosts fewer rain forests, but more Savannah, which makes it popular for horse husbandry.

Sumbawa is also way more mountainous than I had expected. And it is populated very sparsely. Therefore, there are very few roads that cut through the mountains and these roads tend to be narrow and winding.

Sumbawa vs. Bima

Sumbawa is split in half ethnically and linguistically. The west is populated by the Sumbawa people, their biggest city being Sumbawa Besar (SWQ). The east is populated by the Bimanese people, whose biggest city is Bima (BMU).

To make matters more complicated, there are also many Sasak people living on Sumbawa. I’ve heard that whole coastal villages in West Sumbawa have been populated by people from Lombok (which is the Sasak homeland). Allegedly this migration may have taken place as late as the 1980’s.

The Surf

Sumbawa’s surf spots follow a similar east/west split. Some are located on Sumbawa’s west coast, mostly near Sasak villages. Others are located in Bimanese sourth-central Sumbawa, around Lakey. The rest of the island seems to be uncharted when it comes to surfing. The northern coast just doesn’t receive any Indian Ocean swells and many parts of the southern coast may be too inaccessible.

West Sumbawa

I first headed to West Sumbawa and picked Scar Reef Homestay in Jelenga as my base. Unsurprisingly, Scar Reef is the most prominent surf spot at Pantai Jelenga. Less known spots include Little Bingin (named after Bali’s Bingin) and Phantoms (at the very north of the long beach).

All these spots are best reached by boat, though one can paddle out to Little Bingin in reasonable time. That’s why I mostly surfed there, visiting Phantoms only once and skipping Scar Reef altogether. Maybe I was deterred by its name, maybe the swell was too small during most of my stay.

Some of the spots further south are more exposed to the swell, in particular Yo-Yos and Tropical. Both are about an hour from Jelenga by scooter.

My first day-trip was to the Yo-Yos, located near Pantai Gili Dua. However, I had difficulties renting a suitable board there for a reasonable price. In the end, I picked a 6’8″ with 42.4 liters and 4 fins. This is way smaller than any board that I had used before – used successfully, that is. I didn’t expect to catch any waves, but I didn’t want the long journey to go to waste either.

I did catch a couple of waves after all. It was tough to paddle to the shifty peaks and build up enough speed. But then the popup worked just fine. Only a couple short rides that day, but those felt really great. Plus I could duck-dive this board. It also helped that there were no crowds that day – at one point I was alone in the line-up, otherwise maybe 2 to 5 other surfers.

Two days later I made my way to Tropical, which is named after the small beach resort nearby (not to be confused with Hazel Brugger’s Tropical). There’s a left and a right, separated by a strong channel. The beach-side Rock’n’Roll Cafe (I think this was the name, cannot find them online) is a good starting point for either of these waves.

I felt emboldened by recent experiences and picked a 6’6″ board with 44.6 liters. I was struggling with the current though, either getting sucked back into the channel or caught in the soup. The slightly bigger volume made it more challenging to duck-dive. In the end, I got only a couple of waves on the right-hander. Did not try the left-hander, because locals said the currents there are even stronger.

I left Jelenga with mixed feelings. The beach and the village are really nice. There’s no over-tourism yet, but there’s sufficient infrastructure, in particular for surfers. Overall the vibe reminded me of southern Lombok twenty years earlier. But the local surf spots do not seem very consistent, at least not during wet season. And all the other west coast spots are stretched out way too far.

Lakey

After ten days on the west coast, I made my way to Lakey, mostly by bus. It’s a long trip, since you have to go back north all the way to Sumbawa Besar first. There are no suitable roads along the south coast or through the inland mountains.

I booked a room at Lucky Laykey Surf House, right in front of Lakey Peak. Slightly more pricey than my accommodation in Jelenga, but the beachfront location is worth it. Most rooms have ocean view, and the terrace on the 2nd floor is great for checking wave conditions.

The name of the local village is actually Hu’u, though that’s a couple of kilometers away from Pantai Lakey. Seems like the name Lakey emerged from a misunderstanding, when surfers told the locals about this “lucky” peak.

Lakey Peak is a very consistent a-frame reef break. You can recognize it on photos by the two watchtowers out on the reef. Apparently those have been build ages ago for some sort of surfing competitions. There are many other surf-spots nearby, e.g. Lakey Pipe, Nungas, or Periscopes. These and others can be reached on scooter in a couple of minutes. Boat rides are not necessary to reach any of them, but there is some paddling involved.

Overall, Lakey seems slightly more convenient for surfers as compared to West Sumbawa. However, convenience comes at the cost of bigger crowds and more tourism in general. Then again, the sunsets at local beaches are amazing – and so are the sunsets at nearby rice paddies.

What else?

There is certainly more to Sumbawa than just surfing. But that’s what I focused on this time. And it’s what most of the local tourism industry caters to.

When going off the beaten track, Sumbawa is more challenging than say Bali. English is well understood in tourist hot-spots, less so anywhere else. It’s hard to find reliable information on stuff like bus or ferry schedules. The usual ride-hailing services (such as Grab) do not work here (except maybe for airport transfer). On the other hand, living costs are still comparatively low here. Some money and some patience can get you a long way.

I think I might come back to Sumbawa eventually. There’s lots of stuff left to do:

Climbing Gunung Tambora
Watching a water buffalo race (pic search)
Visiting Moyo Island
Swimming with whale sharks (offered by several tour operators near Sumbawa Besar, not sure if this is sustainable)

Singapore Stopover – No Smoking, No Durians! [en]

I hate air travel. I just cannot afford it. Nobody can. But many of us are doing it anyway. At least those who can afford it financially. Sadly, this includes myself occasionally.

For damage control, I’ve self-imposed a flight reduction policy, which I’ve been trying to follow in recent years:

No more than 1 return flight every 2 years.
No short-distance flights, if there are feasible alternatives.
Make it worthwhile:
Stay at least 4 weeks when flying mid-distance,
and at least 6 weeks for flying long-distance.

In theory, my ongoing Indonesia trip adheres to this policy. However, bureaucratic constraints made it necessary to leave Indonesia for a couple of days. Just to apply for a new visa afterwards. And the most pragmatic solution seemed to be a short flight from Bali to Singapore and back.

Since I had to make this flight trip anyway, I decided to have a quick look at Singapore along the way…

Beach Road

Accommodation in Singapore is much more expensive than in Indonesia. That’s why I booked a rather humble room at The Pod on Beach Road. It was a private room, but technically I still shared a bunk bed with someone else. Their lower bunk was simply opening into a separate room. This efficiency didn’t surprise me in a crowded city-state like Singapore.

What did surprise me was their strict policy against durian fruits. Not just at the hotel, but also on public transport, and many other places in Singapore. Fine with me, since I’m not a durian aficionado.

I had more trouble with the non-smoking policy in Singapore. Smoking is forbidden almost everywhere, even outdoors. Smokers are restricted to dedicated smoking areas, usually just around a public trash can with an integrated ashtray. Whole parts of Singapore lack these smoking areas altogether (see 🚭 below).

Anyway, my accommodation near the corner of Beach Road and Arab Street turned out to be a good pick. As the latter street name suggests, this quarter has a Middle Eastern flair, with some Mediterranean and Indian mixed in. Loads of food options and night life.

On my first day, I made my way towards Marina Bay on foot. South of Beach Road the small narrow streets give way to high-rise office buildings, malls, and hotels. Not quite as shiny as the skyscrapers in the downtown banking district, but generously spaced, with some greenery mixed in.

Plus I found this weird magic-carpet skiing/snowboarding venue in one of the malls. I was itching to give it a go, but I resisted the urge. Gonna have plenty of the real deal once I get back home right in the middle of winter.

Marina Bay

Next I crossed the Helix Bridge towards the Marina Bay Sands resort, which seems to have become a major landmark of Singapore. While the hotel towers are the eye-catcher here, the ArtScience museum looks nice, too. And the surrounding promenades offer great views of the banking district across the bay:

The Marina Bay Sands also hosts a huge mall, mostly housing fashion and jewelry stores. All a little bit too kitsch. I guess the hotel itself wouldn’t be to my taste either, the lobby underneath the hotel towers does look impressive though. I might have enjoyed the view from the observation deck, but my timing was off. They also have a casino, but this sort of thing is way beyond my comprehension.

Gardens By The Bay

Just south of the Marina Bay Sands are the Gardens by the Bay 🚭, a huge park area built on reclaimed land. This park is really huge, so I only focused on some of its main attractions. In particular, the “Super-Tree Grove”, the “Cloud Forrest”, and the “Flower Dome”:

I guess that I liked the Cloud Forest best. A closer look revealed that the lush green just formed a thin facade and that the cloud climate was artificially generated. I did not care for the Jurassic Park exhibits that they’ve sprinkled in, but luckily these were unobtrusive. Overall, it was a pleasant experience.

The gardens and the surrounding areas are nerved by cycle paths, and I noticed many shared bicycles on these. So I signed up for one of these sharing services, and explored Singapore by bike for the rest of the evening. I went up the Singapore River for a bit, where there’s lots of bars and restaurants. But in the end I decided to make my way back to my hotel and grab dinner nearby.

Orchard Road

The next day I cycled to the shopping district on Orchard Road 🚭. I had come with a small shopping list, and spent most of the day ticking of the few items on it. I did not enjoy this shopping experience there at all. Then again, I never enjoy shopping. In the end, I got all that I needed.

Changi Airport

The next day, I headed back to Bali. The flight was in the afternoon, so I had some time to explore Changi Airport, which is a tourist attraction in its own right. In particular The Jewel 🚭, which combines some of the things that I had seen elsewhere in Singapore: a shopping mall and an impressive indoor garden with a waterfall.

After going through airport security, I paid a couple of visits to the outdoor smoking areas there. These are placed in beautiful gardens, too, though they are much smaller in scale:

While I shamefully traveled by air, I also learned how important shipping is for Singapore. There are huge docks and container terminals just south of Marina Bay. And I think I’ve never seen more maritime traffic than in the waters around Singapore. Like here, right in front of the Gardens By The Bay:

Bali Picdump [en]

I’m currently on a trip to Indonesia. I’ve spent the first month on Bali, mostly surfing. But I’ve also had a chance to do some other tourist activities. So, let’s dump some pictures here, with just a little textual context…

Uluwatu

Uluwatu (on the Bukit peninsula in Bali’s south) has some of the most popular surf spots on Bali. While I didn’t surf there myself, I dropped by to enjoy the sunset from the cliffs right above. If you look closely, you can see crowds of surfers out in the line up:

Earlier that day, I visited the Uluwatu temple (Pura Luhur Uluwatu), which is famous for its monkey inhabitants:

Garuda Wisnu Kencana

The Garuda Wisnu Kencana Cultural Park (GWK) is another tourist attraction on the Bukit peninsula. While it lacks the long history of Bali’s temples, it is also considered a site of religious significance. Like any place on Bali, it holds a couple of Hindu shrines.

The main attraction is clearly the ginormous statue of the deities Vishnu and Garuda though. It can be seen in the distance from Kuta’s beaches and anywhere else in southern Bali. I also liked the artificial cliffs that they’ve carved into the limestone rocks of the park.

Gunung Batur & Gunung Agung

Gunung Agung is Bali’s highest mountain, towering over the eastern part of the island. I had seen it from across the Lombok Strait when climbing Gunung Rinjani back in 2003. Ever since then I wanted to climb this “Great Mountain”, if I ever spent more time on Bali.

However, Gunung Agung isn’t the only volcano on Bali. There’s also the Bedugul volcanic area with Gunung Batukaru, and Gunung Batur, and some others. Gunung Batur has an impressive double caldera around it, with Lake Batur inside and Gunung Abang as the highest point on the caldera rim. Most of the caldera rim is inhabited and easily accessible, so I stayed at Batur View Homestay (close to Kintamani) in preparation of my mountain hike. I could see the whole Batur caldera right from my room, and even Gunung Agung looming in the distance:

For Gunung Agung itself, I booked a packaged tour with a local guide. Traveling alone, a “1-day-hike” without on-mountain accommodation seemed the only reasonable option. “1-night-hike” may be more accurate, since most guides offer this as a sunrise hike: you climb all night to reach the summit around sunrise.

The tour started near Besakih, approaching the Great Mountain from the west. There are multiple western routes up the mountain, we took the “Edelweiss Route”. It’s almost 2000m up from the trail-head to the summit, most of it in fairly steep terrain.

We made good progress initially, but we wanted to avoid arriving at the summit too early. No point freezing our asses off while waiting for sunrise. So we decided to take a short break (and a nap) at a campsite about 600m below the summit, in the wind-sheltered forest zone. We still had a third of the ascent left afterwards, measured in altitude. See the GPS metrics:

GPX:	Ascent Pt.1	Ascent Pt.2	Descent
Distance:	5km	1.4km	6.8km
Start – End Altitudes:	1230m – 2486m (+1256m)	2499m – 3062m (+563m)	3063m – 1231m (-1832m)
Climb / Descent*:	1475m / 220m	616m / 53m	77m / 1908m
Time:	22:52 – 2:04 (3h 13m)	4:03 – 5:30 (1h 27m)	6:04 – 8:56 (2h 53m)

* Overall climb and descent as calculated from GPS data. Numbers seem exaggerated, because GPS altitude measurements have a lot of jitter. Apparently more than the latitude/longitude measurements used to calculate distance.

After a steep climb, we arrived at the more mellow ridge to the western summit around dawn. Afaict, this is the highest peak of Gunung Agung. The view to the east was partially clouded, so we didn’t bother continuing to the other peak that lies about 5 minutes to the east.

It had been a fairly clear night, almost at full moon. And we could see a lot of lights below us, not just in the urban centers around Denpasar, but all over the slopes and foothills of the mountains. Just shows how densely populated most of Bali is. Doesn’t show well on photos though, so I’ll start with snapshots from the summit… Did I mention that the guide’s dog had followed us all the way up? Also, that (for reasons) we were the only ones on the summit this day?

If you look closely, you can spot parts of Lombok to the east, even get a glimpse of Gunung Rinjani and the Gilies. And Nusa Penida to the south, located closer to Bali.

Looking to the west, I just love the shadow that stratovolcanoes cast on their surroundings early in the day. Plus we had great views of Gunung Batur and many of the other volcanoes on Bali. And even the volcanic peaks on the eastern tip of Java.

We descended on the same route that we had come up, that is to the west. Due to the mountains own shadow, it didn’t get too hot initially. The descent was way less strenuous, but still challenging due to all the loose gravel and mud in the forest zone.

Further down the mountain, we encountered first signs of agriculture. Including plantations of Javanese Edelweiss, which explained the name of our route:

Around Ubud

After hiking the Great Mountain, I returned to my accommodation on the Batur caldera rim, and I had to rest for the remainder of the day. The next day, I headed back to the coast, to Seminyak. But I wanted to do more sight-seeing, too, so I stopped at Ubud along the way.

And I found a great way to get from Gunung Batur to Ubud: a relaxing mountain-bike tour, all down-hill, but at a mellow pace. The good folks from Dewa Bike Tours brought their bikes and a guide right to my homestay and took my bulky luggage on their van. They later dropped me off in the center of Ubud, and even kept my luggage safe at their base, where I could pick it up in the evening.

My bike was in good shape and seemed just right for this tour. It had a solid suspension fork and good disc breaks. I only struggled with the break levers being mounted the “wrong” way around. At home, we usually have the rear break on the right hand, but here it was on the left hand. (This seems to be the case more often in countries that drive on the left side of the road, though it does not seem to be a strict rule.) Anyway, I managed to avoid pulling the wrong break for a slide and going over the handlebars instead.

It was mostly sunny, but not too hot. We were still rather high up on the gentle mountain slopes, and the wind chill did the rest. We did a couple of sight-seeing stops, but frankly I enjoyed the landscape most. Palm trees and rice-paddies – my favorite tone of green:

I’ve got mixed feelings about Ubud itself. In a guide book I’ve read that visitors should take at least a week to explore this town and its surroundings. But frankly, I think my afternoon stopover was just about right.

Sure, Ubud may have the nicest temples and palaces on the island. And I really do like their style – as elaborate as European Baroque, but not quite as tacky. Then again, I can see this anywhere else on Bali. Plus I still feel reluctant to visit active religious sites.

So I picked just one such site at random, guided by geographic proximity. Turned out to be the Water Palace, which I really enjoyed. Traditional attire was mandatory, but rental garments were included in the entrance fee:

Otherwise I was just strolling through town. Loads of tourist-focused bars and restaurants. But not a single humble warung or bakso stand along my random walk. In the early afternoon it started raining, so I took shelter at some indoor souvenir stands. A superficial inspection did not reveal a single item that I’d wish to posses or gift to someone else.

Ubud is also famous for yoga and meditation. Nothing wrong with that, and it’s on me that I didn’t bring enough time for it. Then again, why not do this anywhere else on the planet?

In the end I just admired all the temple gates in passing – and the elaborate electric wiring:

Despite fearing this would be a treacherous tourist trap, I eventually made my way to the Sacred Monkey Forest in the south of Ubud. This turned out to be a rather pleasant experience. I do like monkeys, and though they are all over the place on Bali, these ones were particularly playful and approachable.

But it’s the forest itself that I’ve enjoyed most – or rather the well-maintained park. It had nice little gorge with a creek running through. And all sort of amazing trees, including many Banyan trees (or similar kinds of strangler figs). Human infrastructure and decoration complemented this nicely:

The Surf

Apart from all of the above, I did do loads of surfing. I was staying at two different surf camps for most of the time, each for about 2 weeks. One was Kima Surf in Seminyak, the other was Rapture Surf Greenbowl, in the south of the Bukit peninsula.

Here’s some impressions from the camps, mostly Rapture:

Both Kima and Rapture had great surf guiding (and coaching), which gave me a chance to get to know many different surf spots in southern Bali. This included the western beach breaks around Seminyak, Legian, and Kuta. Also many reef breaks on the Bukit peninsula, such as Balangan, Dreamland, Greenball, and several spots around Nusa Dua (Mushroom, Black Stone, Geger, Nico). We also had great boat-trips to Torotoro (close to Airport Rights).

I didn’t take detailed notes on the surf sessions, but one thing that I can say is that we had surfable waves almost every day. Except for three days towards the end of my stay on Bali, when it was just too flat everywhere.

While I did heaps of shopping on Bali, I still haven’t bought my own surfboard. Instead, I rented different boards, starting with an 8 foot foamy for the first couple of days, then various hard tops between 8’2″ and 7’6″.

Both Kima and Rapture had camera crews around on several sessions. This means that there now is tenfold the surfing footage of me than before this trip. Including video footage that helped me seeing my own mistakes, but also some nice pics that I can share. Still loads of room for improvement, but looking pretty good occasionally. At least on smaller waves:

Epilogue

As said initially, I’m still in Indonesia as I’m posting this. However, I have moved on from Bali by now. As I had learned on previous trips, other parts of Indonesia have quite a different vibe. Then again, even the more remote regions of Bali are very different from the tourist centers in southern Bali. I guess it’s fair to speak of over-tourism there. Of course, I was part of this problem myself.

I want to ends this post with some creative signage that I spotted on Bali. And with one of the mini gas-stations that are all over the place. Not sure why I hadn’t noticed these on previous trips to Indonesia:

Once upon a time in Indonesia [en]

Today I’m on my way to my yearly surf-trip. This year it’s gonna be Indonesia. My third trip to the country.

I might share some impressions later. But right now, I want to revisit my first trip to Indonesia. That was more than 20 years ago, back in 2003. It was my first big journey without my parents. We were a group of 6 friends, all students with loads of spare time and energy, but little money.

We spent most of our time on the island of Lombok. At the beaches in the south, on the Gilis, and on Mount Rinjani.

South Lombok

Here’s some pics from the area around Kuta Lombok:

Notably, the last few pics are from Gerupuk (back then a quiet fishing village) and the nearby surf spots. I think it might be Don Don and Kiddies (a.k.a. Children’s Point?).

Gunung Rinjani

“Gunung” stands for “mountain” in Bahasa Indonesia. And Rinjani is quite an impressive one. We spent 3 days on our Rinjani trek:

Up to the northern crater rim.
Down into the crater, to lake Segare Anak and the hot springs, then up to the eastern crater rim.
Night ascent reaching Rinjani’s summit around sunrise, then all the way down towards civilization.

Those 3 days offered impressive views, so I took more pics than on the rest of the trip combined. Here are some of them, in roughly chronological order:

And the rest?

After Rinjani, we headed to “The Gilies”. Unfortunately, that’s when my camera stopped working. So no material from that part of our trip.

Looking back at all of my photos (not just the selection above) I realize how few pictures I was taking. Where today I’d try 5 or 10 variations of a shot, I only had taken one or two back then. And I remember the reason, too: I had to contend with a 512 MB(!) compact flash memory card. For the whole trip. No other devices to shuffle around data, either. Needless to say, that memory card didn’t go into a smart-phone, but into a good old-fashioned digital compact camera. Those were the days;)

Shoot yourself in the foot: Docker, Nextcloud & git [en]

I did something stupid again. I noticed it, when I received notifications from my Prometheus monitoring. The probe that sends HTTP requests to my Nextcloud server was failing. Strangely, no other probes were failing. No high load, no memory exhaustion, no filesystem running full.

Manual testing confirmed that all my other web-apps worked as expected, only Nextcloud was responding with 502 errors. It looked like my nginx reverse-proxy could not reach the Nextcloud backend.

I ssh connected into the underlying host, and re-checked the vitals. They looked fine. Next I checked all the Docker containers that run my web-apps. All of them looked fine, except for my Nextcloud container, which was caught in a restart loop. And in the container’s logs, dozens of messages like this one had piled up:

Can't start Nextcloud because the version of the data (31.0.9.1) is higher than the docker image version (31.0.8.1) and downgrading is not supported. Are you sure you have pulled the newest image version?

Kudos to the maintainers of the Nextcloud Docker image! I wish all software had such precise and helpful log messages. At least in my case, this told me exactly what had gone wrong and how to fix it.

Indeed, I had accidentally downgraded to an older image version. I quickly upgraded to the latest one, which fixed the whole problem. Case closed.

Background – doing Docker the weird way

But how had I managed to downgrade my Nextcloud Docker image in the first place? Before answering this, let me share some background…

I’m using Ansible to manage most of my personal IT infrastructure. It’s a rather hybrid setup. It manages everything that runs here on meque.de, but also my laptop computer, some desktop VMs, and a couple of Rapberry Pis. In particular, I’m using Ansible’s community.docker.docker_container module to manage my Docker containers.

When configuring my Docker containers, I’m referencing Docker images by digest rather than by tag. For example, instead of this Docker image reference:

nextcloud:apache

I would use this one:

nextcloud@sha256:11f158050216614d585886600445e0a1b75ee224d6a6dddc5eba996cc9499fa6

This approach gives me more control about what code is actually running on my infra at any given time. I’ve worked with organizations that had policies that mandated this approach. And I was one of the guys who was supposed to encourage the implementation of these policies. Since I like to live what I preach, I started to adopt this policy for my own infra. Also, it kinda aligns with my control-freak tendencies.

There are some benefits to referencing software artifacts by immutable cryptographic identifiers (like Docker image digests) rather than by ephemeral aliases (like Docker tags). If something goes horribly wrong, it can make forensic analysis much easier. E.g. after a vulnerability becomes known for a specific image version – be it caused by neglect or by a deliberate supply-chain attack.

However, I believe that it is far more important to update to new software versions as early as possible. In general, this far more beneficial for information security than compliance to software traceability policies.

Here is the low-tech solution that I use to address trade-off:

In my Ansible IaC I’m maintaining a Docker image catalog (in an Ansible variables file in YAML format). Each image catalogue entry contains:
1. an image repository name,
2. a specific image digest,
3. an image tag that I want to follow.
When configuring a Docker container, my Ansible code references an image catalog entry to specify the image. It only uses repo name and digest (i.e. a and b), but not the tag (c).
I’m also using Ansible to setup Prometheus monitoring for all images in the catalog. Whenever the upstream Docker registry (i.e. Docker Hub) moves a Docker image tag to a new Docker image digest, this will show in my Prometheus alerts dashboard.
(I’m using Skopeo and Prometheus Script Exporter to implement this.)
Whenever I see alerts about outdated Docker image digests, I run a small update script on my Ansible code base. This just walks through my image catalog and updates all image digests (b) so that they follow the desired image tag (c) again.
(This script also leverages Skopeo.)
Then I run my main Ansible playbook, which recreates all Docker containers, based on the new image digest from the catalog. It also adjusts the Prometheus config to match the updated Docker image catalog.
Finally, I check for new Prometheus alerts and do some manual smoke-testing. If I’m happy with the result, I commit the updated Docker image catalog to the git repo that holds my Ansible code.
(If I’m not happy with the result, I can roll back by adjusting the image catalog manually.)

This approach balances my personal needs for control vs timely updates quite nicely. Most of it is automated, but it still requires human interaction.

Root cause & fix

Generally, my Docker image update approach works well. In itself, it was not the root cause of my Nextcloud outage. Instead, I had to make some mistakes on top of it…

In my Ansible git repo, I’m currently working with multiple branches. Recently, I was mostly working on a feature branch to implement new functionality that is not related to Nextcloud at all. However, I did update the image catalog in this branch and I did run it against my production infrastructure. This ensured that all my Docker containers were running up-to-date image versions. So far so good.

Yesterday, I switched back to my default branch, because I wanted to roll out some small improvement that were not related to my new feature. I was planning to merge these small changes into my feature branch later. But first, I ran my main Ansible playbook on the default branch. I had totally forgotten that my default branch contained an outdated version of my Docker image catalog. Once Ansible was done, my Nextcloud container was suddenly running an outdated Docker image version. But as the log messages later showed, Nexcloud does not like to do this (and I don’t blame it). So, we’re finally at the root cause of my Nextcloud outage.

To fix the problem, I simply ran my update script again on the default branch. Since the script is idempotent, it updated my Docker image catalog to the same image digests that I already had on my feature branch. After re-running the Ansible playbook, Nextcloud came up again with no complaints.

Lessons learned

According to my Prometheus metrics, the Nextcloud outage lasted about half an hour. This is not great, and I should be more careful in the future. And I should respond to Prometheus alerts more quickly. But overall, it wasn’t a big deal for a Nextcloud instance that’s just for my personal use.

Maybe I’ll need more automation, testing, rollbacks, etc. in the future. And a dedicated staging environment for all my infra. But for the time being, I’ll just be more careful when switching between branches. Always update the image catalog after switching to a new branch! Or, always update it on the default branch and merge it into all active feature branches afterwards.

I’ve had Nextcloud up and running for almost 2 years now. Afaict, this was the worst outage so far. This is fine.

Kodi on Raspberry Pi: First Impressions [en]

Background

I’ve been living without a proper TV for the past 20 years. Nevertheless, I’m consuming a shitload of video content. Mostly streaming services; the usual suspects. But also select contents from Germany’s public TV stations.

So far, I’ve been watching such videos on my laptop computer – I get a lot of screen-time there, far more than on my phone. I’ve always spent the extra buck on laptops with top notch displays and speakers. Living alone, this has been fine and I never felt the need for buying a TV or a stereo. But now I’ve moved into a bigger place and I’ve been busy buying all sorts of furniture and decor. I think that a big TV screen will ultimately become part of my setup.

Trouble is, I do not like those modern “smart” TVs. I’m almost exclusively using open-source software on all of my devices. And I’m trying to sandbox proprietary software (and software from other dubious sources) in VMs and containers as much as I can. I really hate the idea of having hardware with commercial software and an Internet connection (and probably cameras and microphones) running in my living room.

Kodi to the rescue?

So, the plan is buying a huge (but “dumb”) computer display and linking it up to a Raspberry Pi with some open-source media center software. There seem to be plenty of open-source projects that could do the software part. Yesterday, I started tinkering with one of them: Kodi, formerly known as XBMC.

I did not want to run a dedicated Kodi Linux distribution. Instead, I’m sticking to good old Raspberry Pi OS (previously “Raspbian”). I’m already using this on a few other Pi devices, and I’ve got good automation around it. I’m using Ansible scripts to manage these Pis and a few of other boxes, including my laptop and the servers that host this blog and everything else on meeque.de. These Ansible scripts take care of basic user configuration, ssh access, WiFi connection, software packages, etc.

Luckily, Raspberry Pi OS is based on Debian, and both come with a Kodi package. Or, more precisely, dozens of packages that contain various components of Kodi. So I knew that installing Kodi (and keeping it up to date) wouldn’t be a huge problem.

That said, I’m aiming for a lightweight Kodi deployment, without a full desktop environment. The Pi should simply boot right into Kodi. I was worried that this could be some hassle, but I found this guide that showed how easy it is. Kodi has a dedicated “standalone” mode that can be run from a terminal:

kodi-standalone

So I ssh-ed into the Pi, ran the above command, and the Kodi UI came up on the attached display. (Had to run it as root though, will try to set it up for a less privileged user later.)

Then I followed the same guide and set up a systemd service for Kodi. Just hacked it into my Ansible scripts right away. After reboot, the Kodi UI came up automatically, as desired. This was way more straight forward than I had feared.

Unfortunately, I do not have a USB keyboard at hand right now, but I found an old USB mouse and hooked it up to the Pi. This allowed me having a closer look at the Kodi UI. And getting familiar with some basic Kodi settings. Looks like it’s got all that I would expect from a media server, and I haven’t even tried any fancy add-ons yet. The default theme looks a bit old fashioned (early 2000s style, I reckon) but I bet there’s heaps of alternative out there.

Next I installed the Kore Official Remote for Kodi app on my Android phone. Following instructions in the app, I enabled the remote control HTTP API in the Kodi settings, configured a passphrase, and connected the app to it. Unfortunately, the app does not seem to allow managing the settings of the Kodi server itself. Guess I’ll have to resort to walking over to my Pi and configure Kodi from there. Or, find out how to work with Kodi config files over ssh, and mange them with Ansible…

The hardware – for now

So far, I haven’t settled on my TV hardware. I’m exploring Kodi on old hardware that I still had lying around. Namely an ancient 1600×1200 20″ LCD and a 1st generation Raspberry Pi with a USB WiFi dongle. The Pi only has 0.5 GB of RAM – I was very surprised that the Kodi UI can work with so little memory at all. The CPU is not the fastest either, so the UI feels very slow. Even the mouse pointer has significant lag. And I haven’t even tried playing any fancy media yet.

I’ve already ordered new hardware, which should make further exploration much more pleasant. I’ve opted for a Raspberry Pi 4 Model B with 4 GB of RAM. It’s not the latest model, but it looks like it can be operated with passive cooling. In contrast, active cooling (i.e. a fan) seems to be recommended for the latest Raspberry Pi 5.

What’s next?

I’ll still have a lot of exploration left to do. As mentioned before, I want to manage everything with Ansible asap. Including Kodi settings, Kodi add-ons, and updates. I couldn’t find a dedicated Ansible module for Kodi, so I’ll have to do lots of scripting myself. Let’s see how that will play out…

Ultimately I’d like to see the following capabilities in my new media center TV setup:

Playing media right from Kodi.
Both streamed media from the Internet and locally stored media.
Playing media from my laptop computer over Kodi, to leverage the attached screen and audio system.
There seems to be a myriad of options here, many of them proprietary. I guess a free AirPlay implementation like Shairplay will do the trick.
Using the screen attached to Kodi as a second display for my Laptop computer. This should work over the air using WiFi.
Again, there’s a myriad of options, many of them proprietary or even bound to special hardware. I might be able to use the AirPlay protocol here, too, or have a look at Miracast, Google Cast, etc.
Remote access to the Kodi UI from my laptop. I hope that this could be more convenient and more powerful than a remote control app on my phone.
A free remote desktop protocol should do the trick, maybe good old VNC or even Spice. (The latter is tailored to VM scenarios, not sure it makes sense for remote UI access over the network.)

There might be synergy between some of the above. I’m still hoping to solve most of it using Kodi add-ons. If that does not work out, I’ll have to run additional software next to Kodi. This might mean resorting to a full-fledged desktop environment after all.

If all goes well, I’ll be in the market for a big computer display and matching speakers soon. And for some kind flexible wall-mount, so I can rotate the display to various directions. I want to watch “TV” from the couch, but also work from my table. Who knows, maybe I’ll buy some DVB receiver hardware eventually, and rejoin the insanity that is broadcast television.

I might post further updates here…

Free at Last — XSS Edition [en]

Yesterday my employment contract ended, after more than ten years at a big software company. I feel free now. Weirdly. Cause it’s not like I wasn’t free in my work there. I had a lot of freedom in what I was working on and how I organized my work. And, I was working alongside some great colleagues from all around the world.

I haven’t left on bad terms either. In fact, I might have stayed longer, if it weren’t for some very favorable circumstances. As I’ve written in my good email to co-workers, the company made me an offer that I could not reject. (Not just me, I should mention. Many others got a similar offer, and some accepted it.)

Anyway, this week I finished my last work goal. Well, more of a side-project: I gave a talk about “The Browser’s Same Origin Policy“. For this talk, I’ve shown demos in the XSS Demo App that’s running alongside this blog (as I’ve announced a couple of years ago). For this new talk (but not just for this) I’ve made several improvements to this web-app:

Hosting on two distinct subdomains, in order to demo cross-origin stuff. Here they are:
- https://xss.meeque.de (same as before)
- https://yss.meeque.de (new)
Several stand-alone HTML probes and mocks that the XSS Demo App can interact with. Look for new payload presets that make use of these!
Payload outputs that are based on jQuery code, complementing the existing outputs (pure HTML, DOM APIs, Angular templates). Not sure why I had not added these earlier. Well, under the hood it’s all DOM anyway.
Numerous library upgrades. Most notably, upgraded Angular across several major release versions.
Lot’s of refactoring, which has made it much easier to implement the above improvements. Some of the code is still a little awkward. But I guess that’s what you get when you use a sophisticated framework like Angular for a web application that aims to show-case the intricacies of low-level browser behavior.
And an infinite increase in automated test coverage. Easy to achieve when you start at zero.

There’s also some new ideas that I haven’t implemented yet:

The style library that I’ve used is causing some pain. I’ll either need to fix the integration, or switch to a different one.
Some inline documentation would not hurt. For now, I’m the only one who knows how to use all of the app’s capabilities. Explanations for individual payload presets and payload outputs would not hurt either.
CSP support could be added to show how different policies can help prevent certain XSS attacks.
Saving custom payload presets could be helpful. Browser storage should be sufficient for this. And maybe some import/export capabilities.
Add some serious XSS challenges sounds like fun. Should be easy to implement as payload outputs.

Not sure, if any of this will ever materialize. I’m free for this kinda stuff now, at least for the next couple of months. But there’s also some other ideas floating around.