Thib's Blog

I realized that A cheap VPS is a good front

Fri, 22 May 2026 16:00:00 GMT

I have a server at home. It runs a Kubernetes cluster and a few services. I want to expose them to the Internet, so I can e.g. share public links from my Nextcloud, or synchronize my Kobo reader with Grimmory. But I don't want to expose my home IP to the world, and I want to have some reasonable protection against unsophisticated DoS attacks.

I realized that I can achieve that with a cheap VPS that acts as a front, HAProxy, and Wireguard.

I rented a tiny VPS for €4/month at Infrawire (1 vCPU, 2 GB RAM, 25 GB NVMe). I installed a Debian 13 on it, because I want that front server to be as stable and low maintenance as possible, and installed the Debian-packaged HAProxy onto it. I also installed Wireguard. The VPS has a publicly accessible IP, so it will be my Wireguard server: my server at home can reach the VPS to establish a tunnel, the opposite is not true.

On my k3s node, I've installed Wireguard as well. I configured Wireguard on the VPS and my k3s node to establish a tunnel between the two. I've also bound the sshd on my VPS to the wireguard address. Infrawire offers a console so I can unstick myself if I locked me out of my own server (e.g. by misconfiguring Wireguard on any side, or if my server at home had any failure).

I pointed all my DNS records to the VPS. The HAProxy is a "dumb" tcp forwarder, so I can keep operating like before on my cluster. In particular, HAProxy doesn't do TLS termination. My certificates are fetched on my cluster by cert-manager like before, using the http-01 challenge and Let's Encrypt. I could also move to dns-01 challenges, but http-01 just works and lets me switch to a registrar without an API if need be.

That way, I don't need a fixed IP at home, and I don't have to do any port-forwarding from my home router to my k3s cluster. Even better: the VPS has an anti-DDoS protection included, and I can also configure HAProxy to refuse too many connections from a same IP, I can make it close TCP connections that take too long to establish, and more. If my VPS gets hammered, I can still access my services from within my home network.

TIL that Yubikeys are convenient for Linux login

Tue, 28 Apr 2026 10:00:00 GMT

I got myself a Yubikey recently, and I wanted to use it as a nice convenience to:

Grant me sudo privileges
Unlock my session
Decrypt my LUKS-encrypted disk

I've only managed to do the first two, since they both rely on Linux Pluggable Authentication Modules (PAM). Luckily for me, one of PAM's modules supports U2F, the standard Yubikeys rely on.

First I need to install pam-u2f to add U2F support to PAM, and pamu2fcfg to configure my key.

$ sudo rpm-ostree install pam-u2f pamu2fcfg

Since I'm running an immutable OS I need to reboot, and then I can create the correct directory and file to dump an U2F key into it.

$ mkdir -p ~/.config/Yubico
$ pamu2fcfg > ~/.config/Yubico/u2f_keys

Then I make sure to have a root session open in case I lock myself out of sudoers.

$ sudo su
#

In a different terminal, I can edit the sudoers file to add this line

#%PAM-1.0
auth       sufficient   pam_u2f.so cue openasuser
auth       include      system-auth
account    include      system-auth
password   include      system-auth
session    optional     pam_keyinit.so revoke
session    required     pam_limits.so
session    include      system-auth

I save this file and open a new terminal. I type in sudo vi and it asks me to touch my FIDO authenticator before opening vi! If I touch the Yubikey, it indeed opens vi with root privileges.

Let's break down the line:

auth for authentication
sufficient passing this authentication challenge is enough (it's not an additional factor of authentication)
pam_u2f.so the module we load is for U2F, the standard Yubikeys use
cue print "Please touch the FIDO authenticator." when the user needs to authenticate
openasuser to fetch the authentication file without root privileges

It's also possible to use it to unlock my session, but it would be a bit reckless to allow anyone with my Yubikey to log into my laptop. If my backpack gets stolen and it has both my Yubikey and my laptop, anyone can log in.

It's possible to make the login screen require either my user password, or all of

The Yubikey itself
The PIN of the Yubikey
Me to touch the Yubikey

If someone fails more than three times to enter the correct PIN, the Yubikey will lock itself and require a PUK to be unlocked. This gives me an additional layer of security, and it's more convenient than having to type a full length passphrase.

I've added the following line to /etc/pam.d/greetd (the greeter I use):

#%PAM-1.0
auth       sufficient  pam_u2f.so cue openasuser pinverification=1 userpresence=1
auth       substack    system-auth
[...]

[!warning] I can lose my Yubikey

I use my Yubikey as a nice convenience to set up a weaker PIN while not compromising too much on security. I use it instead of a password, no in addition to it.

Since I can lose or break my Yubikey and I don't want to buy two of them, I make the U2F login sufficient but not required. This means I can still fallback to password authentication if I lose my Yubikey.

Finally, DankMaterialShell uses its own lockscreen manager too. I still want to be able to fallback to password authentication if need be, so I'll configure it to accept U2F OR the password, not both.

This means that the lockscreen will call /etc/pam.d/dankshell-u2f to know what to do when the screen is locked. Since this file doesn't exist, I can create it with the following content.

#%PAM-1.0
auth sufficient pam_u2f.so cue openasuser pinverification=1 userpresence=1

I need a fallback for when I don't have my Yubikey, so I also create the one for this occasion

#%PAM-1.0
auth include system-auth

Finally, I have a consistent setup where both my login and lock screen require me to plug my key, enter its PIN and touch it, or enter my full password. When it comes to sudo, I can only touch my key without requiring an PIN.

My next quest will be to use my Yubikey to unlock my LUKS-encrypted disk.

TIL that Minikube mounts volumes as root

Tue, 21 Apr 2026 07:00:00 GMT

When I have to play with a container image I have never met before, I like to deploy it on a test cluster to poke and prod it. I usually did that on a k3s cluster, but recently I've moved to Minikube to bring my test cluster with me when I'm on the go.

Minikube is a tiny one-node Kubernetes cluster meant to run on development machines. It's useful to test Deployments or StatefulSets with images you are not familiar with and build proper helm charts from them.

It provides volumes of the hostPath type by default. The major caveat of hostPath volumes is that they're mounted as root by default.

I usually handle mismatched ownership with a securityContext like the following to instruct the container to run with a specific UID and GID, and to make the volume owned by a specific group.

Typically in a StatefulSet it looks like this:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: myapp
# [...]
spec:
# [...]
  template:
# [...]
    spec:
      securityContext:
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
      containers:
        - name: myapp
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
# [...]

In this configuration:

Processes in the Pod myapp will run with UID 10001 and GID 10001.
The /data directory mounted from the data volume will belong to group 10001 as well.

The securityContext usually solves the problem, but that's not how hostPath works. For hostPath volumes, the securityContext.fsGroup property is silently ignored.

[!success] Init Container to the Rescue!

The solution in this specific case is to use an initContainer as root to chown the volume mounts to the unprivileged user.

In practice it will look like this.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: myapp
# [...]
spec:
# [...]
  template:
# [...]
    spec:
      securityContext:
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
      initContainers:
        - name: fix-perms
          image: busybox
          command:
            ["sh", "-c", "chown -R 10001:10001 /data"]
          securityContext:
            runAsUser: 0
          volumeMounts:
            - name: data
              mountPath: /data
      containers:
        - name: myapp
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
# [...]

It took me a little while to figure it out, because I was used to testing my StatefulSets on k3s. K3s uses a local path provisioner, which gives me local volumes, not hostPath ones like Minikube.

In production I don't need the initContainer to fix permissions since I'm deploying this on an EKS cluster.

TIL that Pagefind does great client-side search

Thu, 16 Apr 2026 10:00:00 GMT

I post more and more content on my website. What was visible at glance then is now more difficult to look for. I wanted to implement search, but it is a static website. It means that everything is built once, and then published somewhere as final, immutable pages. I can't send a request for search and get results in return.

Or that's what I thought! Pagefind is a neat javascript library that does two things:

It produces an index of the content right after building the static site.
It provides 2 web components to insert in my pages: <pagefind-modal> that is the search modal itself, hidden by default, and <pagefind-modal-trigger> that looks like a search field and opens the modal.

The pagefind-modal component looks up the index when the user types a request. The index is a static file, so there is not need for a backend that processes queries. Of course this only works for basic queries, but it's a great rool already!

Pagefind is also easy to customize via a list of CSS variables. Adding it to this website was very straightforward.

I realized that Niri can have gorgeous animation

Wed, 15 Apr 2026 06:30:00 GMT

I was a huge fan of Niri already. It's a scrolling tiling window manager. Roughly:

When I open an app it takes the full height and a pre-configured width on the screen.
When I open more apps and the screen was already full, it pushes the existing apps off screen.
It can stack windows in columns to have a more compact view

It means that windows always take the optimal amount of space, and they're very neatly organized. It's extremely pleasant to use and keyboard friendly.

Don't mind the apparent slowness: this was recorded on a 10 year old laptop, opening OBS is enough to make its CPU go brr. When OBS is not running, Niri is buttery smooth.

But now I've learned that Niri supports user-provided GLSL shaders for several animations. Roughly: you can animate how windows appear and disappear (and other events, but let's keep things simple).

Some people out there have created collections of shaders that work wonderfully for Niri:

My personal favorite is the glitchy one.

In a world of uniform UIs, these frivolous, unnecessary and creative ways to interact with users are a breath of fresh air! Those animations are healing my inner 14 year old.

TIL that Animated AVIFs make lightweight videos

Mon, 13 Apr 2026 07:00:00 GMT

Sometimes in my posts I need to show a screen recording. Videos can get heavy rapidly and take a lot of time to load.

I also write my posts in markdown which has syntax to include images:

![Alt text describing the image](path to the image)

Using that syntax for videos doesn't work though. Since html is valid markdown, it's possible to manually add <video> tags, but it's a bit more tedious.

It's also possible to use ffmpeg to convert a mp4 video into a looping animated AVIF. The command to do it is

$ ffmpeg -i demo.mp4 -loop 0 demo.avif

AVIF also compresses very well, without losing too much detail.

$ ls -lh
total 1.8M
-rw-r--r--. 1 thib thib 566K Apr 11 09:26 typst-live-preview.avif
-rw-r--r--. 1 thib thib 1.2M Apr  8 22:02 typst-live-preview.mp4

The support for AVIF in browsers is excellent, sitting at more than 94% as of writing.

My only remaining gripe is that Astro chokes on AVIF images when trying to optimize images in Markdown posts. A workaround for it is to store the AVIFs as static assets so Astro doesn't try to optimize them.

Update: The excellent Astro maintainers have fixed the AVIF issue and the workaround above is no longer needed.

TIL that Kubernetes can give you a shell into a crashing container

Fri, 10 Apr 2026 08:00:00 GMT

When a container crashes, it can be for several reasons. Sometimes the log won't tell you much about why the container crashed, and you can't get a shell into that container because... it has already crashed. It turns out that kubectl debug can let you do exactly that.

I was trying to ship Helfertool on our Kubernetes cluster. The firs step was to get it to work locally in my Minikube. The container I was deploying kept crashing, with an error message that put me on the right track: Cannot write to log directory. Exiting.

The container expected me to mount a volume on /log so it could write logs, which I did. I wanted to run a quick test from within the container to see if I could create a file in that directory. But when your container has already crashed you can't get a shell into it.

My better informed colleague Quentin told me about kubectl debug, a command that lets me create a copy of the crashing container but with a different COMMAND.

So instead of running its normal program, I can ask the container to run sh with the following command

$ kubectl debug mypod -it \
    --copy-to=mypod-debug \
    --container=my-pods-image \
    -- sh

And just like that I have shell inside a similar container. Using this trick I could confirm that I can't touch a file in that /log directory because it belongs to root while my container is running unprivileged.

That's a great trick to troubleshoot from within a crashing container!

TIL that Helix and Typst are a match made in heaven

Thu, 09 Apr 2026 08:00:00 GMT

I love Markdown with all my heart. It's a markup language so simple to understand that even people who are not software engineers can use it in a few minutes.

The flip side of that coin if that Markdown is limited. It can let you create various title levels, bold, italics, strikethrough, tables, links, and a bit more, but not so much.

When it comes to more complex documents, most people resort to full fledged office suite like Microsoft Office or LibreOffice. Both have their merits, but office file formats are brittle and heavy.

The alternative is to use another more complex markup language. Academics used to be into LaTeX but it's often tedious to use. Typst emerged more recently as a simpler yet useful markup language to create well formatted documents.

Tinymist is a language server for Typst. It provides the usual services a Language server provides, like semantic highlighting, code actions, formatting, etc.

But it really stands out by providing a live preview feature that keeps your cursor in sync in Helix when you are clicking around in the live preview!

I only had to install it with

$ brew install tinymist

I then configured Helix to use tinymist for Typst documents, enabling live preview along the way. This happens of course in ~/.config/helix/languages.toml

[language-server.tinymist]
command = "tinymist"
config = { preview.background.enabled = true, preview.background.args = ["--data-plane-host=127.0.0.1:23635", "--invert-colors=never", "--open"] }

[[language]]
name = "typst"
language-servers = ["tinymist"]

A warm thank you to my lovely friend Felix for showing me the live preview mode of tinymist!

TIL that Git can locally ignore files

Tue, 07 Apr 2026 17:35:00 GMT

When editing markdown, I love using Helix (best editor in the world). I rely on three language servers to help me do it:

rumdl to check markdown syntax and enforce the rules decided by a project
marksman to get assistance when creating links
harper-ls to check for spelling or grammar mistakes

All of these are configured in my ~/.config/helix/languages.toml configuration file, so it applies globally to all the markdown I edit. But when I edit This Week In Matrix at work, things are different.

To edit those posts, we let our community report their progress in a Matrix room, we collect them into a markdown file that we then editorialized. This is a perfect fit for Helix (best editor in the world) and its language servers.

Helix has two features that make it a particularly good fit for the job

The diagnostics view
Jumping to next error with ]d

It is possible to filter out pickers, but it becomes tedious to do so. For this project specifically, I want to disable harper-ls entirely. Helix supports per-project configuration by creating a .helix/languages.toml file at the project's root.

It's a good solution to override my default config, but now I have an extra .helix directory that git wants to track. I could add it to the .gitignore, but that would also add it to everyone else's .gitignore, even if they don't use Helix (best editor in the world) yet.

It turns out that there is a local-only equivalent to .gitignore, and it's .git/info/exclude. The syntax is the same as .gitignore but it's not committed.

Update: several people reached out to point out that there are global options to locally ignore files, if you don't need to do it per-project. Those options are:

The global ~/.config/git/ignore file, with the same syntax as .gitignore
The configuration variable core.excludesFile to specify a file that contains which patterns to ignore, like a .gitignore

I can't believe I didn't need this earlier in my life.

TIL that You can filter Helix pickers

Tue, 07 Apr 2026 17:20:00 GMT

Helix has a system of pickers. It's a pop up window to open files, or open diagnostics coming from a Language Server.

The diagnostics picker displays data in columns:

severity
source
code
message

Sometimes it can get very crowded, especially when you have plenty of hints but few actual errors. I didn't know it, but Helix supports filtering in pickers!

By typing %severity WARN I only get warnings. I can even shorten it to %se (and not %s, since source also starts with an s). The full syntax is well documented in the pickers documentation.

TIL that Sveltia is a good CMS for Astro

Tue, 31 Mar 2026 09:00:00 GMT

This website is built with the static site generator Astro. All my content is written in markdown and uploaded to a git repository. Once the content is merged into the main branch, Cloudflare deploys it publicly. The process to publish involves:

Creating a new markdown file.
Filling it with thoughts.
Pushing it to a new branch.
Waiting for CI to check my content respects some rules.
Pressing the merge button.

This is pretty involved and of course requires access to a computer. This goes directly against the goal I’ve set for myself to reduce friction to publish.

I wanted a simple solution to write and publish short posts directly from mobile, without hosting an additional service.

Such an app is called a git-based headless CMS. Decap CMS is the most frequently cited solution for git-based content management, but it has two show-stoppers for me:

It’s not mobile friendly (yet, since 2017) although there are community workarounds.
It’s not entirely client-side. You need to host a serverless script e.g. on a Cloudflare Worker to complete authentication.

Because my website is completely static, it’s easy to take it off GitHub and Cloudflare and move it elsewhere. I want the CMS solution I choose to be purely client-side, so it doesn’t get in the way of moving elsewhere.

It turns out that Sveltia, an API-compatible and self-proclaimed successor to Decap, is a good fit for this job, with a few caveats.

Sveltia is a mobile-friendly Progressive Web App (PWA) that doesn’t require a backend. It's a static app that can be added to my static website. It has a simple configuration file to describe what fields each post expects (title, publication date, body, etc).

Once the configuration and authentication are done, I have access to a lightweight PWA that lets me create new posts.

The authentication is straightforward for technical people. I need to paste a GitHub Personal Access Token (PAT) in the login page, and that's it. Sveltia will fetch the existing content and display it.

The PWA itself is also easy to deploy: I need to add a page served under the /admin route, that imports the app. I could just import it from a third party CDN, but there’s also a npm package for it. It allows me to serve the javascript as a first party instead, all while easily staying up to date.

I installed it with

$ pnpm add @sveltia/cms

I then created an Astro page under src/pages/admin/index.astro with the following content

title="src/pages/admin/index.astro"
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Content Manager – ergaster.org</title>
    <script>
    import { init } from "@sveltia/cms";
    init();
    </script>
  </head>
  <body></body>
</html>

I also created the config file under public/admin/config.yml with Sveltia Entry Collections matching my Astro content collections. The setup is straightforward and well documented.

Sveltia has a few caveats though:

It can only work on a single branch, and not create a new branch per post. According to the maintainer, it should be possible to create new branches with “Editorial Workflow” by Q2 or Q3 this year.
It pushes content directly to its target branch, including drafts. I still want to run CI checks before merging my content, so I’ve created a drafts branch and configured Sveltia to push content there. Once the CI checks have passed I merge the branch manually from the GitHub mobile app.
Having a single target branch also means I can only have one draft coming from Sveltia at a time. If I edited two drafts concurrently on the drafts branch, they would both be published the next time I merged drafts into main.
It’s clunky to rename a picture uploaded via Sveltia.

Those are not deal breakers to me. The maintainer seems reactive, and the Editorial Workflow feature coming in Q2 or Q3 will fix the remaining clunkiness.

Gear review: Garmin Forerunner 165

Mon, 30 Mar 2026 00:00:00 GMT

Last year I bought one of those rock solid, simple, sturdy Casio Watches that are supposed to last you a long time. I still love it with all my heart and sometimes wear it, but my primary watch is now the Garmin Forerunner 165.

A dozen years ago I got into running and installed an app on my phone to track my progress. I kept pushing harder to beat my previous records, and eventually injured myself badly. I used to think that the quantified self was the root of all evil and that it was the reason why I overtrained. It actually was out of ego, and it turns out that quantified self and coaching to make sense of the data can yield amazing results.

[!success] I got a Garmin Forerunner 165 and I am very happy with it.

I recommend caution nonetheless: the watch is a great tool to gather metrics about how you run, but it is a terrible replacement for a coach. Working with a professional will give you much better results.

Why I got it

In my mid 30s I realized that my metabolism wasn't what it used to be. I started gaining weight and felt uneasy in my body. After two kids and today's geopolitics, my mental health started to degrade too.

I've decided to get back into running to help with both aspects. Exercising makes the body burn calories and produce endorphins. Both are useful to feel better. Yes, it's only in my mid 30s that I realized that exercising was a physiological need. A need I had neglected for too long.

But the last time I got into running I injured myself badly. I'm the least competitive person against others, but I'm very competitive against myself. This means I'm subject to overtraining. I needed something to keep me on track.

Several friends had Garmin watches and told me that their watches actually prevented them from overtraining. That was my cue: I would buy one, and use it to get back in shape. Even better: the model I wanted, the Forerunner 165, could last 11 days on a single charge. It means I could wear it at night to follow my sleep patterns, and it would vibrate gently on my wrist to wake me up silently. This promised to be a low maintenance and useful watch.

[!info] In summary

I bought the Forerunner 165 to:

Get back into running.

Prevent overtraining.

Follow my sleep patterns.

Not babysit the watch or be constantly nagged by it.

First impressions

I bought the watch at the end of August 2025, and I started running immediately with it. In addition to the watch, I also bought a pair of Merrell Trail Glove 7: "barefoot" shoes that have such a thin sole that you have to land on the front of the foot and not on the heel.

I installed the Garmin Connect app on my phone, and I was delighted to see that I didn't any subscription to start a coaching plan. I enrolled in a Garmin Coach program for beginners, and I started following it. At the end of each run, the watch would ask me how I felt, and each run was more painful than the last. I thought it was just muscles building up so I kept following the program. And this is how I injured myself.

I went to a physiotherapist, and we started a specific training plan for barefoot shoes. Of course I quit the Garmin Coach one. He taught me that barefoot shoes require a higher cadence (number of steps per minute) than regular running shoes. I could configure the watch to keep track of my cadence during my runs. A gauge would tell me if I ran too few or too many steps per minute.

After the physiotherapy sessions ended, I could keep running normally. As of writing, I go running three times a week. Each session is between 6 and 12km long.

What I like

I didn't want a smart watch because I don't want it to pester me, and I don't want to charge it every day. On those two fronts the Forerunner delivered. I don't receive any of my phone notifications on my watch, and it doesn't pester me with anything during the day. I use my watch when I need it, not when it needs me.

As for the battery, it is fantastic for this type of watch. With 3 runs a week, my watch lasts 9 to 10 days on a single charge. I keep my watch at night so it monitors my sleep too. And charging is fast: I can charge it from 10 to 100% in about 1.5 hour.

I can confidently go to sleep with it and know it will still have plenty of battery to wake me up the next day with a gentle vibration on my wrist. I hate alarms that scream at you in the morning. This gentle nudge is infinitely better.

I have pathologically bad sleep and the watch does a good job at tracking it. It even helped me detect sleep apnea, that doctors later confirmed. The watch gives you several metrics for the night: how long you've slept, your average and resting heart rate, your average and lowest respiration and more. It can also measure your pulse oximetry, but that depletes the battery twice as fast as if it doesn't. That watch also supports tracking naps.

When it comes to exercising, I only use it for running. I can't say anything about its accuracy, but my physiotherapist seemed to believe that all the measures were plausible.

The wristband has many holes, making it easy to adjust. It is comfortable to wear, even during exercise when the wrist can swell and sweat a bit. It is also slightly elastic, so it can stretch a bit for extra comfort.

It is possible to use the watch only to track how you run, or to configure workouts in the app depending on your objectives. During my physiotherapy training I would make it track my cadence, but you can track a lot more metrics. You can also configure several steps, e.g. warm-up, light run, fast run, series, etc.

There are also built-in, free Garmin Coach programs depending on your needs, but I can't say I have a positive experience with them. If you're new to running I really recommend going to a professional coach or physiotherapist to get you started.

At €230, the watch is not cheap, but seems fairly priced for the amount of value I get from it. I also don't expect to replace it anytime soon.

What I don't like

The watch has an odd recovery time metric can be difficult to understand: you can workout lightly to recover. To this day I'm not entirely sure what it does.

Beyond that it's a good watch I can't complain about!

Conclusion

I’m very happy with my Forerunner 165. It’s important to bear in mind it’s just a tool, not something that can replace a human coach. If your knees or tendons hurt and the watch tells you to go running, don’t. Go see a professional.

I realized that You don't care

Sun, 29 Mar 2026 16:00:00 GMT

Quite a few of us maintain our own websites and publish our thoughts. We play in hard mode:

We need to build our website before even publishing our first post.
We don’t benefit from the network effect of bigger platforms to get eyeballs on our writing.
LLMs aggressively scrape the web and can serve our thoughts or expertise to their users without them visiting our websites.

And on top of that, you don’t care.

And I don’t expect you to care. Like the rest of us, you are flooded with information constantly. You’re fed so many words that you read the equivalent of whole books every day. How entitled would I be to expect you to care about my words when you have to filter through every story you’re bombarded with.

So why do we keep the small web alive?

I can’t speak for others, but I know why I maintain my website and why I publish my thoughts there. By increasing order of importance:

I keep my web development skills reasonably up to date.
I can shape my website to adapt to my content, and not the other way around.
I have freedom of tone and vocabulary. I don’t have to censor words like "suicide" or "sex".
I write long form posts that help me shape my thoughts, develop ideas, and receive feedback from my peers and readers.

If you can afford to, I can only encourage you to write and publish your thoughts on your own platform, as long as you don’t expect others to care in return.

I realized that I created too much friction to publish

Sat, 28 Mar 2026 10:00:00 GMT

I love writing on my blog. I love taking a complex topic, breaking it down, understanding how things work, and writing about how things clicked for me. It serves a double purpose:

I can organize my thoughts, ensure I understood the topic fully, and explain it to others.
It helps my future self: if I forgot about the topic, I can read about what made it click for me.

But as of writing, the last time I published something on my blog was 5 months ago.

The blogging process

My blog posts tend to be lengthy. My writing and publishing process is the following.

Take a nontrivial topic, something I didn't know about or didn't know how to do.
Understand it, break it down, and get a clear picture of how things work.
Write an outline for the post with the key points.
Ask my smarter friends if the outline makes sense.
Flesh out the outline into a proper blog posts, with all the details, code snippets, screenshots.
Ask my smarter friends to review the post again.
Get an illustrator to create a banner for the post, that also serves as an opengraph preview image.
Publish the post.

That is a lot of work. I have many posts stuck between step 3 and 5, because they take quite a bit of time. Asking an illustrator to create a banner for the post also creates more friction: obviously I need to pay the illustrator, but I also need to wait for him to be done with the illustration.

Not everything has to be a blog post

Sometimes I have quick thoughts that I want to jot down and share with the rest of the world, and I want to be able to find it back. There are two people I follow that write a lot, often in short format.

John Gruber on his blog Daring Fireball.
Simon Willison, on his Weblog.

Both of them have very short format notes. Willison even blogged about what he thinks people should write about.

Reducing friction and just posting

I don't think friction should be avoided at all costs. Take emails for example: there's a delay between when you send a message and your peer receives it, or the other way around. That friction encourages longer form messages, which gives more time to organize thoughts.

I also welcome the friction I have created for my own posts: I get through a proper review process and publish higher quality posts.

But there's also room for spontaneity. So I've updated my website to let me publish two smaller formats:

TILs. Those are short posts about something I've learned and found interesting.
Thoughts. Those are shorter posts I jot down in less than 20 minutes to develop simple thoughts.

TIL that Proxmox can provision Kubernetes Persistent Volumes

Wed, 25 Mar 2026 10:00:00 GMT

I wanted to dip my toes into Kubernetes for my homelab, but I knew I would need some flexibility to experiment. So instead of deploying k3s directly on my server, I

Installed a base Debian on my server, encrypting the disk with LUKS and using LVM to partition it.
Installed the Proxmox hypervisor on that base Debian
Spun up a Debian VM, and installed k3s on it.

Proxmox supports several storage plugins. It allows me to create LVM Local Volumes for the VM disks for example.

This setup allows me to spin up fresh VMs for my experiments, all while leaving my production k3s intact. This is great, but it came up with two problems:

When I provision the VM for k3s I need to allocate it a massive amount of disk space. This is because k3s uses a local path provisioner to provision new Persistent Volumes directly on the VM.
I can't take snapshots of the Persistent Volumes when doing backups. There's a risk that the data will change while I perform the backup.

The situation looks like the following.

On the LVM disk of the host, I create a VM for k3s. This VM has a virtual disk that doesn't rely on LVM, so it can't create LVM Logical Volumes. The local provisioner can only create volumes on the virtual disk, because it can't escape the VM to create volumes on the Proxmox host.

Because the volumes are created on the virtual disk that doesn't rely on LVM, I can't use LVM snapshots to take snapshots of my volumes.

[!question] Why not LVM Thin?

One solution to address the massive disk requirement could be to use LVM Thin: it would allow me to allocate a lot of space in theory, but in practice in only fills up as the VM storage gets used.

I don't want to use LVM Thin because it puts me at risk of overprovisioning. I could allocate more storage than I actually have, and it would be difficult to realize that my disks are filling up before it's too late.

My colleague Quentin mentioned the Proxmox CSI Plugin. It is a plugin that replaces k3s' local path provisioner. Instead of creating the kubernetes Persistent Volumes inside the VM, it calls the Proxmox host, asks it to create a LVM Logical Volume and binds it to a Persistent Volume in kubernetes.

Using the Proxmox CSI volume, the situation would look like this.

It solves the two problems for me:

I can now only provision a small disk for the k3s VM, since the Persistent Volumes will be created outside of the VM.
Since Proxmox will create LVM Logical Volumes to provision the Persistent Volumes, I can either do a LVM Snapshot from Proxmox or use Kubernete's Volume Snapshot feature, with some caveats.

Setting up the Proxmox-CSI-Plugin for k3s can be a bit involved, but I'm writing a longer blog post about it.

TIL that GNOME has launched a fellowship program

Tue, 24 Mar 2026 19:00:00 GMT

When open source nonprofits ask for donations, one common answer is "I only want to fund code, I don't want to fund anything else." GNOME has created a Fellowship Program to fund direct work on GNOME, a program entirely funded by donations. This is a testament to the Foundation's maturity, as it becomes a direct contributor to the project it stewards.

Let's take a step back to address the code-only argument. It is a misguided reaction, but I can see where its proponents are coming from. In the world of proprietary software, you pay to get your software. You don't realize that this bundles the marketing, accounting, legal, and even HR costs.

In the open source world, everyone can see who contributes code and how that code is built and packaged to create a software solution. A lot of things are not shown in git commits though. A few of them are:

What did it take to create the Human Interface Guidelines to have a coherent suite of applications? How many designers had to meet, what research did they have to do, did they have to meet in person?
What did it take to create the Developer Documentation to onboard new developers, help them make their first steps, and turn them into bigger contributors over the years?
What did it take to build a website to advertize all the cool apps that follow the GNOME HIG?
What did it take to set up the infrastructure the code lives on, and that builds the software we all love?

GNOME, like many other open source projects, is first and foremost a community. This is a group of people with diverse backgrounds, diverse opinions, who try to find common ground to solve problems. They don't always agree on how to solve problems, nor necessarily on what even is a problem in the first place.

The role of The GNOME Foundation is to provide a place to support its community. Its role is to help its contributors find common ground. Its role is to give them the tools and opportunities to do so.

Some people still don't value this, and want The GNOME Foundation to be a vendor for GNOME. They want to fund developers to produce code, because that's a very visible metric.

For them, and for everyone who's ever wanted to give back to GNOME without knowing how, The GNOME Foundation has created a Fellowship Program. It will directly fund a person to work on what few people want to do in their spare time: maintenance.

Round one focuses on sustainability: improving tooling, build systems, test infrastructure, automation, documentation, developer productivity, and ongoing maintainability. We are not funding feature development: the goal is for each fellowship to leave the project in a more efficient and sustainable state.

This is only fueled by our donations. If you want a direct pipeline between your money and GNOME development, this is it. Donate to GNOME, we can't afford not to have them when Big Tech has so much influence on our lives.

From VS Code to Helix

Wed, 29 Oct 2025 12:00:00 GMT

I created the website you're reading with VS Code. Behind the scenes I use Astro, a static site generator that gets out of the way while providing nice conveniences.

Using VS Code was a no-brainer: everyone in the industry seems to at least be familiar with it, every project can be opened with it, and most projects can get enhancements and syntactic helpers in a few clicks. In short: VS Code is free, easy to use, and widely adopted.

A Rustacean colleague kept singing Helix's praises. I discarded it because he's much smarter than I am, and I only ever use vim when I need to fiddle with files on a server. I like when things "Just Work" and didn't want to bother learning how to use Helix nor how to configure it.

Today it has become my daily driver. Why did I change my mind? What was preventing me from using it before? And how difficult was it to get there?

Automation is a double-edged sword

Automation and technology make work easier, this is why we produce technology in the first place. But it also means you grow more dependent on the tech you use. If the tech is produced transparently by an international team or a team you trust, it's fine. But if it's produced by a single large entity that can screw you over, it's dangerous.

VS Code might be open source, but in practice it's produced by Microsoft. Microsoft has a problematic relationship to consent and is shoving AI products down everyone's throat. I'd rather use tools that respect me and my decisions, and I'd rather not get my tools produced by already monopolistic organizations.

Microsoft is also based in the USA, and the political climate over there makes me want to depend as little as possible on American tools. I know that's a long, uphill battle, but we have to start somewhere.

I'm not advocating for a ban against American tech in general, but for more balance in our supply chain. I'm also not advocating for European tech either: I'd rather get open source tools from international teams competing in a race to the top, rather than from teams in a single jurisdiction. What is happening in the USA could happen in Europe too.

Why I feared using Helix

I've never found vim particularly pleasant to use but it's everywhere, so I figured I might just get used to it. But one of the things I never liked about vim is the number of moving pieces. By default, vim and neovim are very bare bones. They can be extended and completely modified with plugins, but I really don't like the idea of having extremely customize tools.

I'd rather have the same editor as everyone else, with a few knobs for minor preferences. I am subject to choice paralysis, so making me configure an editor before I've even started editing is the best way to tank my productivity.

When my colleague told me about Helix, two things struck me as improvements over vim.

Helix's philosophy is that everything should work out of the box. There are a few configs and themes, but everything should work similarly from one Helix to another. All the language-specific logic is handled in Language Servers that implement the Language Server Protocol standard.
In Helix, first you select text, and then you perform operations onto it. So you can visually tell what is going to be changed before you apply the change. It fits my mental model much better.

But there are major drawbacks to Helix too:

After decades of vim, I was scared to re-learn everything. In practice this wasn't a problem at all because of the very visual way Helix works.
VS Code "Just Works", and Helix sounded like more work than the few clicks from VS Code's extension store. This is true, but not as bad as I had anticipated.

After a single week of usage, Helix was already very comfortable to navigate. After a few weeks, most of the wrinkles have been ironed out and I use it as my primary editor. So how did I overcome those fears?

What Helped

Just Do It

I tried Helix. It can sound silly, but the very first step to get into Helix was not to overthink it. I just installed it on my mac with brew install helix and gave it a go. I was not too familiar with it, so I looked up the official documentation and noticed there was a tutorial.

This tutorial alone is what convinced me to try harder. It's an interactive and well written way to learn how to move and perform basic operations in Helix. I quickly learned how to move around, select things, surround them with braces or parenthesis. I could see what I was about to do before doing it. This has been epiphany. Helix just worked the way I wanted.

Better: I could get things done faster than in VS Code after a few minutes of learning. Being a lazy person, I never bothered looking up VS Code shortcuts. Because the learning curve for Helix is slightly steeper, you have to learn those shortcuts that make moving around feel so easy.

Not only did I quickly get used to Helix key bindings: my vim muscle-memory didn't get in the way at all!

Better docs

The built-in tutorial is a very pragmatic way to get started. You get results fast, you learn hands on, and it's not that long. But if you want to go further, you have to look for docs. Helix has officials docs. They seem to be fairly complete, but they're also impenetrable as a new user. They focus on what the editor supports and not on what I will want to do with it.

After a bit of browsing online, I've stumbled upon this third-party documentation website. The domain didn't inspire me a lot of confidence, but the docs are really good. They are clearly laid out, use-case oriented, and they make the most of Astro Starlight to provide a great reading experience. The author tried to upstream these docs, but that won't happen. It looks like they are upstreaming their docs to the current website. I hope this will improve the quality of upstream docs eventually.

After learning the basics and finding my way through the docs, it was time to ensure Helix was set up to help me where I needed it most.

Getting the most of Markdown and Astro in Helix

In my free time, I mostly use my editor for three things:

Write notes in markdown
Tweak my website with Astro
Edit yaml to faff around my Kubernetes cluster

Helix is a "stupid" text editor. It doesn't know much about what you're typing. But it supports Language Servers that implement the Language Server Protocol. Language Servers understand the document you're editing. They explain to Helix what you're editing, whether you're in a TypeScript function, typing a markdown link, etc. With that information, Helix and the Language Server can provide code completion hints, errors & warnings, and easier navigation in your code.

In addition to Language Servers, Helix also supports plugging code formatters. Those are pieces of software that will read the document and ensure that it is consistently formatted. It will check that all indentations use spaces and not tabs, that there is a consistent number of space when indenting, that brackets are on the same line as the function, etc. In short: it will make the code pretty.

Markdown

Markdown is not really a programming language, so it might seem surprising to configure a Language Server for it. But if you remember what we said earlier, Language Servers can provide code completion, which is useful when creating links for example. Marksman does exactly that!

Since Helix is pre-configured to use marksman for markdown files we only need to install marksman and make sure it's in our PATH. Installing it with homebrew is enough.

$ brew install marksman

We can check that Helix is happy with it with the following command

$ hx --health markdown
Configured language servers:
  ✓ marksman: /opt/homebrew/bin/marksman
Configured debug adapter: None
Configured formatter: None
Tree-sitter parser: ✓
Highlight queries: ✓
Textobject queries: ✘
Indent queries: ✘

But Language Servers can also help Helix display errors and warnings, and "code suggestions" to help fix the issues. It means Language Servers are a perfect fit for... grammar checkers! Several grammar checkers exist. The most notable are:

LTEX+, the Language Server used by Language Tool. It supports several languages but is quite resource hungry.
Harper, a grammar checker Language Server developed by Automattic, the people behind WordPress, Tumblr, WooCommerce, Beeper and more. Harper only support English and its variants, but they intend to support more languages in the future.

I mostly write in English and want to keep a minimalistic setup. Automattic is well funded, and I'm confident they will keep working on Harper to improve it. Since grammar checker LSPs can easily be changed, I've decided to go with Harper for now.

To install it, homebrew does the job as always:

$ brew install harper

Then I edited my ~/.config/helix/languages.toml to add Harper as a secondary Language Server in addition to marksman

[language-server.harper-ls]
command = "harper-ls"
args = ["--stdio"]

[[language]]
name = "markdown"
language-servers = ["marksman", "harper-ls"]

Finally I can add a markdown linter to ensure my markdown is formatted properly. Several options exist, and markdownlint is one of the most popular. My colleagues recommended the new kid on the block, a Blazing Fast equivalent: rumdl.

Installing rumdl was pretty simple on my mac. I only had to add the repository of the maintainer, and install rumdl from it.

$ brew tap rvben/rumdl
$ brew install rumdl

After that I added a new language-server to my ~/.config/helix/languages.toml and added it to the language servers to use for the markdown language.

[language-server.rumdl]
command = "rumdl"
args = ["server"]

[...]


[[language]]
name = "markdown"
language-servers = ["marksman", "harper-ls", "rumdl"]
soft-wrap.enable = true
text-width = 80
soft-wrap.wrap-at-text-width = true

Since my website already contained a .markdownlint.yaml I could import it to the rumdl format with

$ rumdl import .markdownlint.yaml
Converted markdownlint config from '.markdownlint.yaml' to '.rumdl.toml'
You can now use: rumdl check --config .rumdl.toml .

You might have noticed that I've added a little quality of life improvement: soft-wrap at 80 characters.

Now if you add this to your own config.toml you will notice that the text is completely left aligned. This is not a problem on small screens, but it rapidly gets annoying on wider screens.

Helix doesn't support centering the editor. There is a PR tackling the problem but it has been stale for most of the year. The maintainers are overwhelmed by the number of PRs making it their way, and it's not clear if or when this PR will be merged.

In the meantime, a workaround exists, with a few caveats. It is possible to add spaces to the left gutter (the column with the line numbers) so it pushes the content towards the center of the screen.

To figure out how many spaces are needed, you need to get your terminal width with stty

$ stty size
82 243

In my case, when in full screen, my terminal is 243 characters wide. I need to remove the content column with from it, and divide everything by 2 to get the space needed on each side. In my case for a 243 character wide terminal with a text width of 80 characters:

(243 - 80) / 2 = 81

As is, I would add 203 spaces to my left gutter to push the rest of the gutter and the content to the right. But the gutter itself has a width of 4 characters, that I need to remove from the total. So I need to subtract them from the total, which leaves me with 76 characters to add.

I can open my ~/.config/helix/config.toml to add a new key binding that will automatically add or remove those spaces from the left gutter when needed, to shift the content towards the center.

[keys.normal.space.t]
z = ":toggle gutters.line-numbers.min-width 76 3"

Now when in normal mode, pressing <kbd>Space</kbd> then <kbd>t</kbd> then <kbd>z</kbd> will add/remove the spaces. Of course this workaround only works when the terminal runs in full screen mode.

Astro

Astro works like a charm in VS Code. The team behind it provides a Language Server and a TypeScript plugin to enable code completion and syntax highlighting.

I only had to install those globally with

$ pnpm install -g @astrojs/language-server typescript @astrojs/ts-plugin

Now we need to add a few lines to our ~/.config/helix/languages.toml to tell it how to use the language server

[language-server.astro-ls]
command = "astro-ls"
args = ["--stdio"]
config = { typescript = { tsdk = "/Users/thibaultmartin/Library/pnpm/global/5/node_modules/typescript/lib" } }

[[language]]
name = "astro"
scope = "source.astro"
injection-regex = "astro"
file-types = ["astro"]
language-servers = ["astro-ls"]

We can check that the Astro Language Server can be used by helix with

$ hx --health astro
Configured language servers:
  ✓ astro-ls: /Users/thibaultmartin/Library/pnpm/astro-ls
Configured debug adapter: None
Configured formatter: None
Tree-sitter parser: ✓
Highlight queries: ✓
Textobject queries: ✘
Indent queries: ✘

I also like to get a formatter to automatically make my code consistent and pretty for me when I save a file. One of the most popular code formaters out there is Prettier. I've decided to go with the fast and easy formatter dprint instead.

I installed it with

$ brew install dprint

Then in the projects I want to use dprint in, I do

$ dprint init

I might edit the dprint.json file to my liking. Finally, I configure Helix to use dprint globally for all Astro projects by appending a few lines in my ~/.config/helix/languages.toml.

[[language]]
name = "astro"
scope = "source.astro"
injection-regex = "astro"
file-types = ["astro"]
language-servers = ["astro-ls"]
formatter = { command = "dprint", args = ["fmt", "--stdin", "astro"] }
auto-format = true

One final check, and I can see that Helix is ready to use the formatter as well

$ hx --health astro
Configured language servers:
  ✓ astro-ls: /Users/thibaultmartin/Library/pnpm/astro-ls
Configured debug adapter: None
Configured formatter:
  ✓ /opt/homebrew/bin/dprint
Tree-sitter parser: ✓
Highlight queries: ✓
Textobject queries: ✘
Indent queries: ✘

YAML

For yaml, it's simple and straightforward: Helix is preconfigured to use yaml-language-server as soon as it's in the PATH. I just need to install it with

$ brew install yaml-language-server

Is it worth it?

Helix really grew on me. I find it particularly easy and fast to edit code with it. It takes a tiny bit more work to get the language support than it does in VS Code, but it's nothing insurmountable. There is a slightly steeper learning curve than for VS Code, but I consider it to be a good thing. It forced me to learn how to move around and edit efficiently, because there is no way to do it inefficiently. Helix remains intuitive once you've learned the basics.

I am a GNOME enthusiast, and I adhere to the same principles: I like when my apps work out of the box, and when I have little to do to configure them. This is a strong stance that often attracts a vocal opposition. I like products that follow those principles better than those who don't.

With that said, Helix sometimes feels like it is maintained by one or two people who have a strong vision, but who struggle to onboard more maintainers. As of writing, Helix has more than 350 PRs open. Quite a few bring interesting features, but the maintainers don't have enough time to review them.

Those 350 PRs mean there is a lot of energy and goodwill around the project. People are willing to contribute. Right now, all that energy is gated, resulting in frustration both from the contributors who feel like they're working in the void, and the maintainers who feel like there at the receiving end of a fire hose.

A solution to make everyone happier without sacrificing the quality of the project would be to work on a Contributor Ladder. CHAOSS' Dr Dawn Foster published a blog post about it, listing interesting resources at the end.

Cloud tech makes sense on-prem too

Wed, 20 Aug 2025 10:00:00 GMT

In the previous post, we talked about the importance to have a flexible homelab with Proxmox, and set it up. Long story short, I only have a single physical server but I like to experiment with new setups regularly. Proxmox is a baremetal hypervisor: a piece of software that lets me spin up Virtual Machines on top of my server, to act as mini servers.

Thanks to this set-up I can have a long-lived VM for my (single node) production k3s cluster, and I can spin up disposable VMs to experiment with, without impacting my production.

But it's more complex to install Proxmox, spin up a VM, and install k3s on it, as compared to just installing Debian and k3s on my baremetal server. We have already automated the Proxmox install process. Let's now automate the VM provisioning and deploy k3s on it, to make it simple and easy to re-provision a fully functional Virtual Machine on top of Proxmox!

In this post we will configure opentofu so it can ask Proxmox to spin up a new VM, use cloud-init to do the basic pre-configuration of the VM, and use ansible to deploy k3s on it.

Provisioning and pre-configuring a VM

OpenTofu is software I execute on my laptop. It reads file describing what I want to provision, and performs the actual provisioning. I can use it to say "I want a VM with 4 vCPUs and 8GB of RAM on this Proxmox cluster," or "I want to add this A record to my DNS managed by Cloudflare." I need to write this down in .tf files, and invoke the tofu CLI to read those files and apply the changes.

Opentofu is quite flexible. It can connect to many different providers (e.g. Proxmox, AWS, Scaleway, Hetzner...) to spin up a variety of resources (e.g. a VM on Proxmox, an EC2 or EKS instance or AWS, etc). Proxmox, Amazon and other providers publish Provider plugins for opentofu, available in the OpenTofu Registry (and in the Terraform Registry since opentofu is backward compatible for now).

Configuring Opentofu for Proxmox

To use Opentofu with Proxmox, you need to pick and configure an Opentofu Provider for Proxmox. There seem to be two active implementations:

bpg/proxmox is maintained by an individual
telmate/proxmox is maintained by the Telmate organization

The former seems to have better test coverage, and friends have used it for months without a problem. I am taking a leap of faith and picking it.

The plugin needs to be configured so opentofu on my laptop can talk to Proxmox and spin up new VMs. To do so, I need to create a Proxmox service account that opentofu will use, so opentofu has sufficient privileges to create the VMs I ask it to create.

I will rely on the pveum (Proxmox Virtual Environment User Management) utility to create a role with the right privileges, create a new user/service account, and assign the role to the service account.

Once ssh'd into the Proxmox host, I can create the terraform user that opentofu will use

# pveum user add terraform@pve

[!info] I don't have to add a password

I will issue an API Key for opentofu to authenticate as this user. Not having a password reduces the attack surface by ensuring nobody can use this service account to log into the web UI.

Then let's create the role

# pveum role add Terraform -privs "Datastore.Allocate \
    Datastore.AllocateSpace \
    Datastore.AllocateTemplate \
    Datastore.Audit \
    Pool.Allocate \
    Sys.Audit \
    Sys.Console \
    Sys.Modify \
    VM.Allocate \
    VM.Audit \
    VM.Clone \
    VM.Config.CDROM \
    VM.Config.Cloudinit \
    VM.Config.CPU \
    VM.Config.Disk \
    VM.Config.HWType \
    VM.Config.Memory \
    VM.Config.Network \
    VM.Config.Options \
    VM.Console \
    VM.Migrate \
    VM.PowerMgmt \
    SDN.Use"

And now let's assign the role to the user

# pveum aclmod / -user terraform@pve -role Terraform

Finally I can create an API token

# pveum user token add terraform@pve provider --privsep=0
┌──────────────┬──────────────────────────────────────┐
│ key          │ value                                │
╞══════════════╪══════════════════════════════════════╡
│ full-tokenid │ terraform@pve!provider               │
├──────────────┼──────────────────────────────────────┤
│ info         │ {"privsep":"0"}                      │
├──────────────┼──────────────────────────────────────┤
│ value        │ REDACTED                             │
└──────────────┴──────────────────────────────────────┘

I now have a service account up and ready. Let's create a ~/Projects/infra/tofu folder that will contain my whole infrastructure's opentofu file. In that folder, I will create a providers.tf file to declare and configure the various providers I need. For now, this will only be Proxmox. I can configure my Proxmox provider so it knows where the API endpoint is, and what API key to use.

terraform {
  required_providers {
    proxmox = {
      source = "bpg/proxmox"
      version = "0.80.0"
    }
  }
}

provider "proxmox" {
  endpoint = "https://192.168.1.220:8006/"
  api_token = "terraform@pve!provider=REDACTED"
}

For some operations, including VM provisioning, the API is not enough and the Proxmox provider needs to ssh into the Proxmox host to issue commands. I can configure the Proxmox provider to use my ssh agent.

This way, when I call the tofu command on my laptop to provision VMs on the Proxmox host, the provider will use the ssh-agent of my laptop to authenticate against the Proxmox host. This will make opentofu use my ssh keypair to authenticate. Since my ssh key is already trusted by the Proxmox host, opentofu will be able to log in seamlessly.

provider "proxmox" {
  endpoint  = "https://192.168.1.220:8006/"
  api_token = "terraform@pve!provider=REDACTED"
  insecure  = true

  ssh {
    agent    = true
    username = "root"
  }
}

[!warning] Insecure but still somewhat secure

We add an insecure line to our configuration. It instructs opentofu to skip the TLS verification of the certificate presented by the Proxmox host. We do this because Proxmox generates a self-signed certificate our computer doesn't trust. We will understand what this means and fix that in a further blog post.

The main risk we're facing by doing so is to let another machine impersonate our Proxmox host. Since we're working on a homelab, in a home network, the chances of it happening are extraordinarily low, and this it can be considered a temporarily acceptable.

After moving to the tofu directory, running tofu init will install the provider

$ tofu init

Initializing the backend...

Initializing provider plugins...
- Finding bpg/proxmox versions matching "0.80.0"...
- Installing bpg/proxmox v0.80.0...
- Installed bpg/proxmox v0.80.0 (signed, key ID F0582AD6AE97C188)

And a tofu plan shouldn't return an error

$ tofu plan

No changes. Your infrastructure matches the configuration.

OpenTofu has compared your real infrastructure against your configuration and found no
differences, so no changes are needed.

Removing sensitive information

If you made it so far, you probably think I am completely reckless for storing credentials in plain text files, and you would be correct to think so. Credentials should never be stored in plain text. Fortunately opentofu can grab sensitive credentials from environment variables.

I use Bitwarden to store my production credentials and pass them to opentofu when I step into my work directory. You can find all the details on how to do it on this previous blog post. Bear in mind that this works well for a homelab but I wouldn't recommend it for a production setup.

We need to create a new credential in the Infra folder of our vault, and call it PROXMOX_VE_API_TOKEN. Its content is the following

terraform@pve!provider=yourApiKeyGoesHere

Then we need to sync the vault managed by the bitwarden CLI, to ensure it has the credential we just added.

$ bw sync
Syncing complete.

Let's update our ~/Projects/infra/.direnv to make it retrieve the PROXMOX_VE_API_TOKEN environment variable when we step into our work directory.

bitwarden_password_to_env Infra PROXMOX_VE_API_TOKEN

And let's make direnv allow it

$ direnv allow ~/Projects/infra/

We can now remove the credential from tofu/providers.tf

provider "proxmox" {
  endpoint  = "https://192.168.1.220:8006/"
  api_token = "terraform@pve!provider=REDACTED"
  insecure  = true

  ssh {
    agent    = true
    username = "root"
  }
}

Spinning up a new VM

Now I have a working proxmox provider for opentofu, it's time to spin up a first VM! I already use Debian for my Proxmox host, I'm familiar with Debian, it's very stable, and it has a reactive security team. I want to keep track of as few operating systems (OS) as possible, so whenever possible I will use it as the base OS for my VMs.

When I spin up a new VM, I can also pre-configure a few settings with cloud-init. Cloud-init defines standard files that my VM will read on first boot. Those files contain various instructions: I can use them to give a static IP to my VM, create a user, add it to the sudoers without a password, and add a ssh key to let me perform key-based authentication with ssh.

I need to use a "cloud image" of Debian for it to support cloud-init file. I can grab the link on Debian's official Download page. I could upload it manually to Proxmox, but we're here to make things tidy and reproducible! So let's create a tofu/cloud-images.tf file where we will tell opentofu to ask the Proxmox node to download the file.

resource "proxmox_virtual_environment_download_file" "debian_13_cloud_image" {
  content_type = "iso"
  datastore_id = "local"
  node_name    = "proximighty"
  url          = "https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2"
  file_name    = "debian-13-generic-amd64.img"
}

[!info] No include needed!

Opentofu merges all the files in the root of a directory into a single file before processing it. There is no need to include/import our tofu/providers.tf file into tofu/cloud-images.tf!

Let's run a tofu plan to see what opentofu would do.

$ tofu plan
OpenTofu used the selected providers to generate the following execution plan. Resource actions
are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

  # proxmox_virtual_environment_download_file.debian_13_cloud_image will be created
  + resource "proxmox_virtual_environment_download_file" "debian_13_cloud_image" {
      + content_type        = "iso"
      + datastore_id        = "local"
      + file_name           = "debian-13-generic-amd64.img"
      + id                  = (known after apply)
      + node_name           = "proximighty"
      + overwrite           = true
      + overwrite_unmanaged = false
      + size                = (known after apply)
      + upload_timeout      = 600
      + url                 = "https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2"
      + verify              = true
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Everything looks alright, let's apply it to actually make the Proxmox host download the image!

$ tofu apply

OpenTofu used the selected providers to generate the following execution plan. Resource actions
are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

  # proxmox_virtual_environment_download_file.debian_13_cloud_image will be created
  + resource "proxmox_virtual_environment_download_file" "debian_13_cloud_image" {
      + content_type        = "iso"
      + datastore_id        = "local"
      + file_name           = "debian-13-generic-amd64.img"
      + id                  = (known after apply)
      + node_name           = "proximighty"
      + overwrite           = true
      + overwrite_unmanaged = false
      + size                = (known after apply)
      + upload_timeout      = 600
      + url                 = "https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2"
      + verify              = true
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  OpenTofu will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

proxmox_virtual_environment_download_file.debian_13_cloud_image: Creating...
proxmox_virtual_environment_download_file.debian_13_cloud_image: Creation complete after 8s [id=local:iso/debian-13-generic-amd64.img]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Looking at the Proxmox UI I can see that the image has indeed been downloaded

Excellent! Now we can describe the parameters of the virtual machine we wan to create by creating a tofu/k3s-main.tf file that contains a virtual_environment_vm resource like so

resource "proxmox_virtual_environment_vm" "k3s-main" {
  name        = "k3s-main"
  description = "Production k3s' main VM"
  tags        = ["production", "k3s", "debian"]
  node_name   = "proximighty"
}

This is the meta-data of our VM, giving it a name and a Proxmox node to run onto. But we need to be more specific. Let's give it 4 CPUs, and 16 GB of RAM.

resource "proxmox_virtual_environment_vm" "k3s-main" {
  name        = "k3s-main"
  description = "Production k3s' main VM"
  tags        = ["production", "k3s", "debian"]
  node_name   = "proximighty"

  cpu {
    cores = 4
    type  = "x86-64-v4"
  }

  memory {
    dedicated = 16384
    floating  = 16384 # set equal to dedicated to enable ballooning
  }
}

To figure out the type of cpu to use for your VM, issue the following command on the Proxmox host

$ /lib64/ld-linux-x86-64.so.2 --help
[...]
Subdirectories of glibc-hwcaps directories, in priority order:
  x86-64-v4 (supported, searched)
  x86-64-v3 (supported, searched)
  x86-64-v2 (supported, searched)
[...]

We can then give it a 50 GB disk with the disk block.

resource "proxmox_virtual_environment_vm" "k3s-main" {
  name        = "k3s-main"

[...]

  disk {
    datastore_id = "local"
    interface    = "virtio0"
    iothread     = true
    size         = 50
    file_id      = proxmox_virtual_environment_download_file.debian_13_cloud_image.id
  }
}

I use the local datastore and not lvm-thin despite running QEMU because I don't want to allow over-provisioning. lvm-thin would allow me to allocate a disk of 500 GB to my VM, even if I only have 100 GB available, because the VM will only fill the Proxmox drive with the actual content it uses. You can read more about storage on Proxmox's wiki.

I use a virtio device, since a colleague told me "virtio uses a special communication channel that requires guest drivers, that are well supported out of the box on Linux. You can be way faster when the guest knows it's a VM and don't have to emulate something that was intended for actual real hardware. It's the same for your network interface and a bunch of other things. Usually if there is a virtio option you want to use that"

I set the file_id to the Debian cloud image we downloaded earlier.

I can then add a network interface that will use the vmbr0 bridge I created when setting up my Proxmox host. I also need an empty serial_device, or Debian crashes.

resource "proxmox_virtual_environment_vm" "k3s-main" {
  name        = "k3s-main"

[...]

  network_device {
    bridge = "vmbr0"
  }

  serial_device {}
}

Now, instead of spinning up an un-configured VM and manually retrieving the parameters set during boot, we will use cloud-init to pre-configure it. We will do the following:

Configure the network to get a static IP
Configure the hostname
Configure the timezone to UTC
Add a user thib that doesn't have a password
Add thib to sudoers, without a password
Add my the public ssh key from my laptop to the trust keys of thib on the VM, so I can login with a ssh key and never have to use a password.

The documentation of the Proxmox provider teach us that that Proxmox has native support for cloud-init. This cloud-init configuration is done in the initialization block of the virtual_environment_vm resource.

We will first give it an IP

resource "proxmox_virtual_environment_vm" "k3s-main" {
  name        = "k3s-main"

[...]

  initialization {
    datastore_id = "local"

    ip_config {
      ipv4 {
        address = "192.168.1.221/24"
        gateway = "192.168.1.254"
      }
    }
  }
}

I'm only specifying the datastore_id because by default it uses local-lvm, which I have not configured on my Proxmox host.

To create a user and give it ssh keys I could use the user_account block inside initialization. Unfortunately it doesn't support adding the user to sudoers, nor installing extra packages. To circumvent that limitation I will have to create a user config data file and pass it to cloud-init.

Let's start by creating the user config data file resource within tofu/k3s-main.tf

resource "proxmox_virtual_environment_file" "user_data_cloud_config" {
  content_type = "snippets"
  datastore_id = "local"
  node_name    = "proximighty"

  source_raw {
    data = <<-EOF
    #cloud-config
    hostname: mightykube
    timezone: UTC
    users:
      - default
      - name: thib
        lock_passwd: true
        groups:
          - sudo
        shell: /bin/bash
        ssh_authorized_keys:
          - ${trimspace("ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIGC++vbMTrSbQFKFgthj9oLaW1z5fCkQtlPCnG6eObB thib@ergaster.org")}
        sudo: ALL=(ALL) NOPASSWD:ALL
      - name: root
        lock_passwd: true
    package_update: true
    package_upgrade: true
    packages:
      - htop
      - qemu-guest-agent
      - vim
    runcmd:
      - systemctl enable qemu-guest-agent
      - systemctl start qemu-guest-agent
      - echo "done" > /tmp/cloud-config.done
    EOF

    file_name = "user-data-cloud-config.yaml"
  }
}

It's a bit inelegant to keep a copy of my ssh key inside this file. Let's ask opentofu to read it from the actual file on my laptop instead by creating a local_file resource for it, in tofu/k3s.tf

data "local_file" "ssh_public_key" {
  filename = "/Users/thibaultmartin/.ssh/id_ed25519.pub"
}

If I try to plan the change, I get the following error

$ tofu plan
╷
│ Error: Inconsistent dependency lock file
│ 
│ The following dependency selections recorded in the lock file are inconsistent with the
│ current configuration:
│   - provider registry.opentofu.org/hashicorp/local: required by this configuration but no version is selected
│ 
│ To update the locked dependency selections to match a changed configuration, run:
│   tofu init -upgrade

Like the error message says, I can fix it with tofu init

$ tofu init -upgrade
Initializing the backend...

Initializing provider plugins...
- Finding opentofu/cloudflare versions matching "5.7.1"...
- Finding bpg/proxmox versions matching "0.80.0"...
- Finding latest version of hashicorp/local...
- Installing hashicorp/local v2.5.3...
- Installed hashicorp/local v2.5.3 (signed, key ID 0C0AF313E5FD9F80)
- Using previously-installed opentofu/cloudflare v5.7.1
- Using previously-installed bpg/proxmox v0.80.0

Providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://opentofu.org/docs/cli/plugins/signing/

OpenTofu has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

OpenTofu has been successfully initialized!
[...]

I can now change the user_data_cloud_config resource to reference ssh_public_key

resource "proxmox_virtual_environment_file" "user_data_cloud_config" {
[...]
        ssh_authorized_keys:
          - ${trimspace("ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIGC++vbMTrSbQFKFgthj9oLaW1z5fCkQtlPCnG6eObB thib@ergaster.org")}
          - ${trimspace(data.local_file.ssh_public_key.content)}
[...]
}

Now let's update the initialization block of k3s-main to use that cloud-init file

resource "proxmox_virtual_environment_vm" "k3s-main" {
  name        = "k3s-main"

[...]

  initialization {
    datastore_id = "local"

    ip_config {
      ipv4 {
        address = "192.168.1.221/24"
        gateway = "192.168.1.254"
      }
    }

    user_data_file_id = proxmox_virtual_environment_file.user_data_cloud_config.id
  }
}

We can check with tofu plan that everything is alright, and then actually apply the plan with tofu apply

$ tofu apply
data.local_file.ssh_public_key: Reading...
data.local_file.ssh_public_key: Read complete after 0s [id=930cea05ae5e662573618e0d9f3e03920196cc5f]
proxmox_virtual_environment_file.user_data_cloud_config: Refreshing state... [id=local:snippets/user-data-cloud-config.yaml]
proxmox_virtual_environment_download_file.debian_13_cloud_image: Refreshing state... [id=local:iso/debian-13-generic-amd64.img]

OpenTofu used the selected providers to generate the following execution plan.
Resource actions are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

  # proxmox_virtual_environment_vm.k3s-main will be created
  + resource "proxmox_virtual_environment_vm" "k3s-main" {
    [...]
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  OpenTofu will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

proxmox_virtual_environment_vm.k3s-main: Creating...
proxmox_virtual_environment_vm.k3s-main: Creation complete after 5s [id=100]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Looking at Proxmox's console, I can see that the VM was created, it healthy, and I can even see that it has the mightykube hostname I had created for it.

Now I can try to ssh into the newly created VM from my laptop

$ ssh thib@192.168.1.221
The authenticity of host '192.168.1.221 (192.168.1.221)' can't be established.
ED25519 key fingerprint is SHA256:39Qocnshj+JMyt4ABpD9ZIjDpOHhXqdet94QeSh+uDo.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.1.221' (ED25519) to the list of known hosts.
Linux mightykube 6.12.41+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.41-1 (2025-08-12) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
thib@mightykube:~$

Let's check that I can perform actions as root without being prompted for a password

thib@mightykube:~$ sudo apt update
Get:1 file:/etc/apt/mirrors/debian.list Mirrorlist [30 B]
Get:2 file:/etc/apt/mirrors/debian-security.list Mirrorlist [39 B]
Hit:3 https://deb.debian.org/debian trixie InRelease
Hit:4 https://deb.debian.org/debian trixie-updates InRelease
Hit:5 https://deb.debian.org/debian trixie-backports InRelease
Hit:6 https://deb.debian.org/debian-security trixie-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.

Brilliant! Just like that, I have a VM on Proxmox, with a static IP address, a well-known user, ssh key authentication and no password to manage at all!

[!warning] No password means no password!

When creating the VM, cloud-init creates a user but doesn't give it a password. It means we can only rely on SSH to control the VM. If we lose our SSH key or mess up with the sshd config and can't ssh into the VM, we're (kind of) locked out!

We have access to the VM console via Proxmox, but without a password we can't log into it. It is possible to rescue it by booting a live system and chrooting into our actual system, but it can be tedious. We'll cover that in a future blog post.

We still have credentials in our .tf files so we can't commit them yet. We will extract the credentials a bit later, but first let's refactor our files for clarity.

A single place to attribute IPs

Since all my VMs will get a static IP, I want to make sure I keep a tidy list of all the IPs already used. This will help avoid IP clashes. Let's create a new ips.tf file to keep track of everything

locals {
  reserved_ips = {
    proxmox_host = "192.168.1.220/24"
    k3s_main     = "192.168.1.221/24"
    gateway      = "192.168.1.254"
  }
}

When spinning up the VM for the main k3s node, I will be able to refer to the local.reserved_ips.k3s_main local variable. So let's update the tofu/k3s-main.tf file accordingly!

resource "proxmox_virtual_environment_vm" "k3s-main" {
  name        = "k3s-main"
  description = "Production k3s' main VM"
  tags        = ["production", "k3s", "debian"]
  node_name   = "proximighty"

  [...]

  initialization {
    datastore_id = "local"

    ip_config {
      ipv4 {
        address = "192.168.1.221/24"
        gateway = "192.168.1.254"
        address = local.reserved_ips.k3s_main
        gateway = local.reserved_ips.gateway
      }
    }

    user_data_file_id = proxmox_virtual_environment_file.user_data_cloud_config.id
  }
}

We now have a single file to allocate IPs to virtual machines. We can see at a glance whether an IP is already used or not. That should save us some trouble! Let's now have a look at the precautions we need to take to save our files with git.

Keeping a safe copy of our state

What is tofu state

We used opentofu to describe what resources we wanted to create. Let's remove the resource "proxmox_virtual_environment_vm" "k3s-main" we have created, and run tofu plan to see how opentofu would react to that.

$ tofu plan
data.local_file.ssh_public_key: Reading...
data.local_file.ssh_public_key: Read complete after 0s [id=930cea05ae5e662573618e0d9f3e03920196cc5f]
proxmox_virtual_environment_download_file.debian_13_cloud_image: Refreshing state... [id=local:iso/debian-13-generic-amd64.img]
proxmox_virtual_environment_file.user_data_cloud_config: Refreshing state... [id=local:snippets/user-data-cloud-config.yaml]
proxmox_virtual_environment_vm.k3s-main: Refreshing state... [id=100]

OpenTofu used the selected providers to generate the following execution plan.
Resource actions are indicated with the following symbols:
  - destroy

OpenTofu will perform the following actions:

  # proxmox_virtual_environment_vm.k3s-main will be destroyed
  # (because proxmox_virtual_environment_vm.k3s-main is not in configuration)
  - resource "proxmox_virtual_environment_vm" "k3s-main" {
    [...]
    }

Plan: 0 to add, 0 to change, 1 to destroy.

If I remove a resource block, opentofu will try to delete it. But it might not be aware of other VMs I could have deployed. Hang on but that might be dangerous! If I already had 3 VMs running on Proxmox and started using opentofu after that, would it destroy them all, since I didn't describe them in my files?!

Fortunately for us, no. Opentofu needs to know what it is in charge of, and leave the rest alone. When I provision something via opentofu, it adds it to a local inventory of all the things it manages. That inventory is called a state file and looks like the following (prettified via jq)

{
  "version": 4,
  "terraform_version": "1.10.5",
  "serial": 13,
  "lineage": "24d431ee-3da9-4407-b649-b0d2c0ca2d67",
  "outputs": {},
  "resources": [
    {
      "mode": "data",
      "type": "local_file",
      "name": "ssh_public_key",
      "provider": "provider[\"registry.opentofu.org/hashicorp/local\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "content": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIGC++vbMTrSbQFKFgthj9oLaW1z5fCkQtlPCnG6eObB thib@ergaster.org\n",
            "content_base64": "c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUlHQysrdmJNVHJTYlFGS0ZndGhqOW9MYVcxejVmQ2tRdGxQQ25HNmVPYkIgdGhpYkBlcmdhc3Rlci5vcmcK",
            "content_base64sha256": "YjQvgHA99AXWCaKLep6phGgdlmkZHvXU3OOhRSsQvms=",
            "content_base64sha512": "tRp4/iG90wX0R1SghdvXwND8Hg6ADNuMMdPXANUYDa2uIjkRkLRgK5YPK6ACz5cbW+SbqvGPGzYpWNNFLGIFpQ==",
            "content_md5": "ed5ee6428ea7c048fe8019bb1a2206b3",
            "content_sha1": "930cea05ae5e662573618e0d9f3e03920196cc5f",
            "content_sha256": "62342f80703df405d609a28b7a9ea984681d9669191ef5d4dce3a1452b10be6b",
            "content_sha512": "b51a78fe21bdd305f44754a085dbd7c0d0fc1e0e800cdb8c31d3d700d5180dadae22391190b4602b960f2ba002cf971b5be49baaf18f1b362958d3452c6205a5",
            "filename": "/Users/thibaultmartin/.ssh/id_ed25519.pub",
            "id": "930cea05ae5e662573618e0d9f3e03920196cc5f"
          },
          "sensitive_attributes": []
        }
      ]
    },
    [...]
  ],
  "check_results": null
}

The tofu state is a local representation of what opentofu manages. It's absolutely mandatory for opentofu to work: this is how opentofu knows if it needs to deploy, update, or tear down resources. So we need to keep in a safe place, and my laptop is not a safe place at all. It can fail or get stolen. Since the state is a text based file, I could use git to keep remote copies of it.

But as you can see, the tofu state file contains my public key, that it read from a local file. The state file contains a structured view of what is in the .tf files it manages. So far we have not added any sensitive credentials, but we might do it and not realize they will end up in state, and thus on a git repo.

Fortunately, opentofu comes with tools that let us encrypt the state, so we can commit it to a remote git repository with more peace of mind.

Encrypting the tofu state

Before encrypting our state, the opentofu documentation has an important section to read so you understand what it entails.

We need to migrate our unencrypted plan to an encrypted one. Let's bear in mind that there's no way back if we screw up, so let's make a backup first (and delete it when we're done). Note that a properly encrypted state can be migrated to a decrypted one. A botched encrypted state will likely be irrecoverable. Let's just copy it in a different directory

$ cd ~/Project/infra/tofu
$ mkdir ~/tfbackups
$ cp terraform.tfstate{,.backup} ~/tfbackups/

To encrypt our state, we need to choose an encryption method: as a single admin homelabber I'm going for the simpler and sturdier method. I don't want to depend on extra infrastructure for secrets management, so I'm using PBKDF2, which roughly means "generating an encryption key from a long passphrase."

With that in mind, let's follow the documentation to migrate a pre-existing project. Let's open our providers.tf file and add an encryption block within the terraform one.

terraform {

  encryption {
    method "unencrypted" "migrate" {}

    key_provider "pbkdf2" "password_key" {
      passphrase = "REDACTED"
    }

    method "aes_gcm" "password_based" {
      keys = key_provider.pbkdf2.password_key
    }

    state {
      method = method.aes_gcm.password_based

      fallback {
        method = method.unencrypted.migrate
      }
    }
  }

  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "0.80.0"
    }
  }
}

provider "proxmox" {
  endpoint = "https://192.168.1.220:8006/"

  ssh {
    agent    = true
    username = "root"
  }
}

This block instructs terraform to encrypt the state with a key generated from our password. It also tells it to expect a pre-existing unencrypted state to exist, that it's okay to read and encrypt it.

Note that I've used the encryption passphrase directly in that block. We will move it to a safer place later, but for now let's keep things simple.

Let's now apply this plan to see if our state gets encrypted correctly, but make sure you do have a cleartext backup first.

$ cd ~/Projects/infra/tofu
$ tofu apply

After the apply, we can have a look at the terraform.tfstate file to check that it has indeed been encrypted.

{
  "serial": 13,
  "lineage": "24d431ee-3da9-4407-b649-b0d2c0ca2d67",
  "meta": {
    "key_provider.pbkdf2.password_key": "eyJzYWx0[...]"
  },
  "encrypted_data": "ONXZsJhz[...]",
  "encryption_version": "v0"
}

I know that opentofu people probably know what they're doing, but I don't like that password_key field. It starts with eyJ, so that must be a base64 encoded json object. Let's decode that

$ echo "eyJzYWx0[...]" | base64 -d
{"salt":"jzGRZLVANeFJpJRxj8RXg48FfOoB++GF/Honm6sIF9Y=","iterations":600000,"hash_function":"sha512","key_length":32}

All good, it's just the salt, iterations, hash function and key length parameters. Those are pretty much public, we can commit the file to our repo! But... what about the terraform.tfstate.backup file? Let's examine this one

{
  "version": 4,
  "terraform_version": "1.10.5",
  "serial": 12,
  "lineage": "24d431ee-3da9-4407-b649-b0d2c0ca2d67",
  "outputs": {},
  "resources": [
    {
      "mode": "data",
      "type": "local_file",
      "name": "ssh_public_key",
      "provider": "provider[\"registry.opentofu.org/hashicorp/local\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "content": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIGC++vbMTrSbQFKFgthj9oLaW1z5fCkQtlPCnG6eObB thib@ergaster.org\n",
            [...]
        }
      ]
    },
	[...]
  ],
  "check_results": null
}

Oh dear! That one is not encrypted! I didn't find any utility for it since terraform can't do "rollbacks", and I couldn't find docs for it. I've deleted the file and I could still perform tofu apply without a problem. The next iterations should be encrypted, but I will add it to my .gitignore just in case!

We're not quite ready to commit our files though. We still have a secret in plain text! We give away the encryption key we use. Let's extract it into an environment variable so we don't leak it.

Removing sensitive information

We need to create a new credential in the Infra folder of our vault, and call it TF_ENCRYPTION. Its content is the following

key_provider "pbkdf2" "password_key" { passphrase = "yourPassphraseGoesHere" }

Then we need to sync the vault managed by the bitwarden CLI, to ensure it has the credential we just added.

$ bw sync
Syncing complete.

Let's update our ~/Projects/infra/.direnv to make it retrieve the TF_ENCRYPTION environment variable

bitwarden_password_to_env Infra PROXMOX_VE_API_TOKEN TF_ENCRYPTION

And let's make direnv allow it

$ direnv allow ~/Projects/infra/

Let's remove the block that provided our password from the encryption block in providers.tf

terraform {

  encryption {
    method "unencrypted" "migrate" {}

    key_provider "pbkdf2" "password_key" {
      passphrase = "REDACTED"
    }

    method "aes_gcm" "password_based" {
      keys = key_provider.pbkdf2.password_key
    }

    state {
      method = method.aes_gcm.password_based

      fallback {
        method = method.unencrypted.migrate
      }
    }
  }

[...]
}

And let's try a tofu plan to confirm that opentofu could read the passphrase from the environment variable

$ tofu plan
╷
│ Warning: Unencrypted method configured
│ 
│   on  line 0:
│   (source code not available)
│ 
│ Method unencrypted is present in configuration. This is a security risk and
│ should only be enabled during migrations.
╵
data.local_file.ssh_public_key: Reading...
data.local_file.ssh_public_key: Read complete after 0s [id=930cea05ae5e662573618e0d9f3e03920196cc5f]
proxmox_virtual_environment_download_file.debian_13_cloud_image: Refreshing state... [id=local:iso/debian-13-generic-amd64.img]
proxmox_virtual_environment_file.user_data_cloud_config: Refreshing state... [id=local:snippets/user-data-cloud-config.yaml]
proxmox_virtual_environment_vm.k3s-main: Refreshing state... [id=100]

No changes. Your infrastructure matches the configuration.

OpenTofu has compared your real infrastructure against your configuration and found
no differences, so no changes are needed.

Brilliant! We can now also remove the migrate and fallback blocks so opentofu doesn't trust unencrypted content at all, which will prevent malicious actors from tampering with our file.

terraform {

  encryption {
    method "unencrypted" "migrate" {}

    method "aes_gcm" "password_based" {
      keys = key_provider.pbkdf2.password_key
    }

    state {
      method = method.aes_gcm.password_based

      fallback {
        method = method.unencrypted.migrate
      }
    }
  }

[...]
}

Finally we can delete our cleartext backup

$ rm -Rf ~/tfbackups

Voilà, we have an encrypted state that we can push to a remote Github repository, and our state will be reasonably safe for today's standards!

Fully configuring and managing the VM

As we've seen when setting up the Proxmox host, ansible can be used to put a machine in a desired state. It can write a playbook to install k3s and copy the kubeconfig file to my admin laptop.

Then there's the question of: how do we make opentofu (who provisions the VMs) and ansible (who deploys services on the VMs) talk to each other? In an ideal world, I would tell opentofu to provision the VM, and then to run an ansible playbook on the hosts it has created.

There's an ansible opentofu provider that's supposed to play this role. I didn't find intuitive to use, and most people around me told me they found it so cumbersome they didn't use it. There is a more flexible and sturdy solution: ansible dynamic inventories!

Creating a dynamic inventory for k3s VMs

Ansible supports creating inventories by calling plugins that will retrieve information from sources. The Proxmox inventory source plugin lets ansible query Proxmox and retrieve information about VMs, and automatically group them together.

Hang on. Are we really going to create a dynamic inventory for a single VM? I know we're over engineering things for the sake of learning, but isn't it a bit too much? As always, it's important to consider what problem we're trying to solve. To me, we're solving two different problems:

We make sure that there is a single canonical source of truth, and it is opentofu. The IP defined in opentofu, it's the one provisioned on Proxmox, and it's the one the dynamic inventory will use to perform operations on the VM. If the VM needs to change its IP, we only have to update it in opentofu, and ansible will follow along.
We build a sane foundation for more complex setups. It will be easy to extend when deploying more VMs to run complex clusters, while not adding unnecessary complexity.

So let's start by making sure we have the Proxmox plugin installed. It is part of the community.general collection on ansible-galaxy, so let's install it

$ ansible-galaxy collection install community.general

Then in the ~/Projects/infra/ansible/inventory directory, we can create a proximighty.proxmox.yaml. The file has to end with .proxmox.yaml for the Proxmox plugin to work.

plugin: community.general.proxmox
url: https://192.168.1.200:8006
user: "terraform@pve"
token_id: "provider"
token_secret: "REDACTED"

Let's break it down:

plugin tells ansible to use the Proxmox inventory source plugin.
url is the URL of the Proxmox cluster.
user is the Proxmox user we authenticate as. Here I'm reusing the same value as the service account we have created for opentofu.
token_id is the ID of the token we have issued for the user. I'm also reusing the same value as the API Key we have created for opentofu.
token_secret is the password for the API Key. Here again I'm reusing the same value as the API Key we have created for opentofu. I'm writting it in the plain text file for now, we will clean it up later.

Now we can try to pass that dynamic inventory configuration to ansible for it to build an inventory from Proxmox.

$ ansible-inventory -i proximighty.proxmox.yaml --list
[WARNING]:  * Failed to parse
/Users/thibaultmartin/Projects/infra/ansible/inventory/proximighty.proxmox.yaml
with auto plugin: HTTPSConnectionPool(host='192.168.1.200', port=8006): Max retries
exceeded with url: /api2/json/nodes (Caused by SSLError(SSLCertVerificationError(1,
'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local
issuer certificate (_ssl.c:1028)')))
[WARNING]:  * Failed to parse
/Users/thibaultmartin/Projects/infra/ansible/inventory/proximighty.proxmox.yaml
with yaml plugin: Plugin configuration YAML file, not YAML inventory
[WARNING]:  * Failed to parse
/Users/thibaultmartin/Projects/infra/ansible/inventory/proximighty.proxmox.yaml
with ini plugin: Invalid host pattern 'plugin:' supplied, ending in ':' is not
allowed, this character is reserved to provide a port.
[WARNING]: Unable to parse
/Users/thibaultmartin/Projects/infra/ansible/inventory/proximighty.proxmox.yaml as
an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
{
    "_meta": {
        "hostvars": {}
    },
    "all": {
        "children": [
            "ungrouped"
        ]
    }
}

And it fails! This is unfortunately not a surprise. We asked the plugin to look up into Proxmox and gave it a https URL. But when Proxmox runs for the first time, it generates a self-signed certificate. It is a perfectly fine certificate we can use to handle https requests. The only problem is that our laptop doesn't trust the Proxmox host, who signed the certificate for itself.

The good news is that Proxmox can retrieve certificates signed by authorities our laptop trusts! The bad news is that we need to understand what we're doing to do it properly. Like earlier, when we configured the Proxmox provider for opentofu, let's ask the Proxmox plugin to use the certificate even if it doesn't trust the authority who signed it. Since we're in a homelab on a home network, the risk of accidentally reaching a host that impersonates our Proxmox host is still fairly low, so it's acceptable to temporarily take this risk here again.

Let's add the following line to our dynamic inventory configuration to ignore the certificate signature

plugin: community.general.proxmox
url: https://192.168.1.200:8006
user: "terraform@pve"
token_id: "provider"
token_secret: "REDACTED"
validate_certs: false

And now, running the inventory command again

$ cd ~/Projects/infra/ansible/inventory
$ ansible-inventory -i proximighty.proxmox.yaml --list
{
    "_meta": {
        "hostvars": {}
    },
    "all": {
        "children": [
            "ungrouped",
            "proxmox_all_lxc",
            "proxmox_all_qemu",
            "proxmox_all_running",
            "proxmox_all_stopped",
            "proxmox_nodes",
            "proxmox_proximighty_lxc",
            "proxmox_proximighty_qemu"
        ]
    },
    "proxmox_all_qemu": {
        "hosts": [
            "k3s-main"
        ]
    },
    "proxmox_all_running": {
        "hosts": [
            "k3s-main"
        ]
    },
    "proxmox_nodes": {
        "hosts": [
            "proximighty"
        ]
    },
    "proxmox_proximighty_qemu": {
        "hosts": [
            "k3s-main"
        ]
    }
}

Great! We can see that our k3s-main VM appears! We didn't learn a lot about it though. Let's ask the Proxmox plugin to give us more information about the VMs with the want_facts parameter

plugin: community.general.proxmox
url: https://192.168.1.200:8006
user: "terraform@pve"
token_id: "provider"
token_secret: "REDACTED"
want_facts: true
validate_certs: false

Let's run it again and see if we get more interesting results

$ cd ~/Projects/infra/ansible/inventory
$ ansible-inventory -i proximighty.proxmox.yaml --list
{
    "_meta": {
        "hostvars": {
            "k3s-main": {
                "proxmox_acpi": 1,
                "proxmox_agent": {
                    "enabled": "0",
                    "fstrim_cloned_disks": "0",
                    "type": "virtio"
                },
                "proxmox_balloon": 16384,
                "proxmox_bios": "seabios",
                "proxmox_boot": {
                    "order": "virtio0;net0"
                },
                "proxmox_cicustom": {
                    "user": "local:snippets/user-data-cloud-config.yaml"
                },
                "proxmox_cores": 4,
                "proxmox_cpu": {
                    "cputype": "x86-64-v4"
                },
                "proxmox_cpuunits": 1024,
                "proxmox_description": "Production k3s' main VM",
                "proxmox_digest": "b68508152b464627d06cba6505ed195aa3d34f59",
                "proxmox_ide2": {
                    "disk_image": "local:100/vm-100-cloudinit.qcow2",
                    "media": "cdrom"
                },
                "proxmox_ipconfig0": {
                    "gw": "192.168.1.254",
                    "ip": "192.168.1.221/24"
                },
                "proxmox_keyboard": "en-us",
                "proxmox_memory": "16384",
                "proxmox_meta": {
                    "creation-qemu": "9.2.0",
                    "ctime": "1753547614"
                },
                "proxmox_name": "k3s-main",
                "proxmox_net0": {
                    "bridge": "vmbr0",
                    "firewall": "0",
                    "virtio": "BC:24:11:A6:96:8B"
                },
                "proxmox_node": "proximighty",
                "proxmox_numa": 0,
                "proxmox_onboot": 1,
                "proxmox_ostype": "other",
                "proxmox_protection": 0,
                "proxmox_qmpstatus": "running",
                "proxmox_scsihw": {
                    "disk_image": "virtio-scsi-pci"
                },
                "proxmox_serial0": "socket",
                "proxmox_smbios1": {
                    "uuid": "0d47f7c8-e0b4-4302-be03-64aa931a4c4e"
                },
                "proxmox_snapshots": [],
                "proxmox_sockets": 1,
                "proxmox_status": "running",
                "proxmox_tablet": 1,
                "proxmox_tags": "debian;k3s;production",
                "proxmox_tags_parsed": [
                    "debian",
                    "k3s",
                    "production"
                ],
                "proxmox_template": 0,
                "proxmox_virtio0": {
                    "aio": "io_uring",
                    "backup": "1",
                    "cache": "none",
                    "discard": "ignore",
                    "disk_image": "local:100/vm-100-disk-0.qcow2",
                    "iothread": "1",
                    "replicate": "1",
                    "size": "500G"
                },
                "proxmox_vmgenid": "e00a2059-1310-4b0b-87f7-7818e7cdb9ae",
                "proxmox_vmid": 100,
                "proxmox_vmtype": "qemu"
            }
        }
    },
    "all": {
        "children": [
            "ungrouped",
            "proxmox_all_lxc",
            "proxmox_all_qemu",
            "proxmox_all_running",
            "proxmox_all_stopped",
            "proxmox_nodes",
            "proxmox_proximighty_lxc",
            "proxmox_proximighty_qemu"
        ]
    },
    "proxmox_all_qemu": {
        "hosts": [
            "k3s-main"
        ]
    },
    "proxmox_all_running": {
        "hosts": [
            "k3s-main"
        ]
    },
    "proxmox_nodes": {
        "hosts": [
            "proximighty"
        ]
    },
    "proxmox_proximighty_qemu": {
        "hosts": [
            "k3s-main"
        ]
    }
}

That's a tonne of information! Probably more than we need, and we still don't know how to connect to a specific host. Let's add some order into that. First, let's group all the VMs that have k3s in their tags under a an ansible group called k3s

plugin: community.general.proxmox
url: https://192.168.1.200:8006
user: "terraform@pve"
token_id: "provider"
token_secret: "REDACTED"
want_facts: true
groups:
  k3s: "'k3s' in (proxmox_tags_parsed|list)"
validate_certs: false

And now let's tell ansible how to figure out what IP to use for a host. Since we are the ones provisioning the VMs, we know for sure that we have configured them to use a static IP, on the single virtual network interface we gave them.

Let's use the compose parameter to populate an ansible_host variable that contains the IP of the VM.

plugin: community.general.proxmox
url: https://192.168.1.200:8006
user: "terraform@pve"
token_id: "provider"
token_secret: "REDACT"
want_facts: true
groups:
  k3s: "'k3s' in (proxmox_tags_parsed|list)"
compose:
  ansible_host: proxmox_ipconfig0.ip | default(proxmox_net0.ip) | ansible.utils.ipaddr('address')
validate_certs: false

And finally let's test this again

$ cd ~/Projects/infra/ansible/inventory
ansible-inventory -i proximighty.proxmox.yaml --list
{
    "_meta": {
        "hostvars": {
            "k3s-main": {
                "ansible_host": "192.168.1.221",
                "proxmox_acpi": 1,
                "proxmox_agent": {
                    "enabled": "0",
                    "fstrim_cloned_disks": "0",
                    "type": "virtio"
                },
                [...]
            }
        }
    },
    "all": {
        "children": [
            "ungrouped",
            "proxmox_all_lxc",
			[...]
            "k3s"
        ]
    },
    "k3s": {
        "hosts": [
            "k3s-main"
        ]
    },
	[...]
}

Brilliant! We now have a k3s group, that contains our single k3s-main VM, and it's been able to retrieve its IP successfully! Let's create a simple playbook to try to execute on the VM one command that works and one that doesn't.

Let's create a ~/Projects/infra/ansible/k3s/test.yaml

---
- name: Execute commands on the k3s host
  hosts: k3s
  remote_user: thib
  tasks:
    - name: Echo on the remote server
      ansible.builtin.command: echo "It worked"
      changed_when: false
    - name: Get k3s installed version
      ansible.builtin.command: k3s --version
      register: k3s_version_output
      changed_when: false
      ignore_errors: true

The only two notable things here are

hosts is the name of the group we created in the dynamic inventory
remote_user is the user I have pre-configured via cloud-init when spinning up the VM

$ cd
$ ansible-playbook -i inventory/proximighty.proxmox.yaml k3s/test.yaml 

PLAY [Execute commands on the k3s host] ********************************************

TASK [Gathering Facts] *************************************************************
ok: [k3s-main]

TASK [Echo on the remote server] ***************************************************
ok: [k3s-main]

TASK [Get k3s installed version] ***************************************************
fatal: [k3s-main]: FAILED! => {"changed": false, "cmd": "k3s --version", "msg": "[Errno 2] No such file or directory: b'k3s'", "rc": 2, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
...ignoring

PLAY RECAP *************************************************************************
k3s-main                   : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=1

It works! Now that we know how to build an inventory based on Proxmox tags and could make a simply ansible playbook use it, let's move forward and actually deploy k3s on our VM!

Deploying k3s on the k3s-main VM

The k3s maintainers have created a k3s-ansible playbook that can preconfigure a machine to ensure it will be ready to make k3s run, and deploy a single or multi-node cluster. It's a great playbook and it's important not to reinvent the wheel. But I also like to understand what I execute, and keep things minimal to limit the risk of breakage.

Let's take inspiration from this excellent playbook to build one tailored for our (very simple) needs: deploying k3s-server on a single node. When trying to install k3s server via the playbook on Debian, it executes 2 roles:

prereq that performs a series of checks to ensure k3s can be installed and run well
k3s_server that downloads, preconfigures and install k3s

We know that the OS powering our virtual machine is always going to be a Debian cloud image. None of the checks in prereq are useful for a fresh vanilla Debian stable. So let's skip it entirely.

Let's have a closer look at what k3s_server does, and carry the important bits over to our playbook. We want to

Check whether k3s is already installed so we don't override it if it was already installed
Download the install script
Execute the install script to download k3s
Create a systemd service for k3s to start automatically
Enable the service
Copy the kubeconfig file generated by k3s to our laptop, and merge it with our kubeconfig under the cluster name and context mightykube

To do things cleanly, we will create a k3s_server role in a k3s directory.

---
- name: Get k3s installed version
  ansible.builtin.command: k3s --version
  register: k3s_server_version_output
  changed_when: false
  ignore_errors: true

- name: Set k3s installed version
  when: not ansible_check_mode and k3s_server_version_output.rc == 0
  ansible.builtin.set_fact:
    k3s_server_installed_version: "{{ k3s_server_version_output.stdout_lines[0].split(' ')[2] }}"

- name: Download and execute k3s installer if k3s is not already installed
  when: not ansible_check_mode and (k3s_server_version_output.rc != 0 or k3s_server_installed_version is version(k3s_server_version, '<'))
  block:
    - name: Download K3s install script
      ansible.builtin.get_url:
        url: https://get.k3s.io/
        timeout: 120
        dest: /usr/local/bin/k3s-install.sh
        owner: root
        group: root
        mode: "0755"

    - name: Install K3s binary
      ansible.builtin.command:
        cmd: /usr/local/bin/k3s-install.sh
      environment:
        INSTALL_K3S_SKIP_START: "true"
        INSTALL_K3S_VERSION: "{{ k3s_server_version }}"
      changed_when: true

    - name: Copy K3s service file [Single]
      ansible.builtin.template:
        src: "k3s-single.service.j2"
        dest: "/etc/systemd/system/k3s.service"
        owner: root
        group: root
        mode: "0644"
      register: k3s_server_service_file_single

    - name: Enable and check K3s service
      ansible.builtin.systemd:
        name: k3s
        daemon_reload: true
        state: started
        enabled: true

- name: Check whether kubectl is installed on control node
  ansible.builtin.command: "kubectl"
  register: k3s_server_kubectl_installed
  ignore_errors: true
  delegate_to: 127.0.0.1
  become: false
  changed_when: false

# Copy the k3s config to a second file to detect changes.
# If no changes are found, we can skip copying the kubeconfig to the control node.
- name: Copy k3s.yaml to second file
  ansible.builtin.copy:
    src: /etc/rancher/k3s/k3s.yaml
    dest: /etc/rancher/k3s/k3s-copy.yaml
    mode: "0600"
    remote_src: true
  register: k3s_server_k3s_yaml_file_copy

- name: Apply k3s kubeconfig to control node if file has change and control node has kubectl installed
  when:
    - k3s_server_kubectl_installed.rc == 0
    - k3s_server_k3s_yaml_file_copy.changed
  block:
    - name: Copy kubeconfig to control node
      ansible.builtin.fetch:
        src: /etc/rancher/k3s/k3s.yaml
        dest: "~/.kube/config.new"
        flat: true

    - name: Change server address in kubeconfig on control node
      ansible.builtin.shell: |
        KUBECONFIG=~/.kube/config.new kubectl config set-cluster default --server=https://{{ hostvars[groups['k3s'][0]]['ansible_host'] }}:6443
      delegate_to: 127.0.0.1
      become: false
      register: k3s_server_csa_result
      changed_when:
        - k3s_server_csa_result.rc == 0

    - name: Setup kubeconfig context on control node - mightykube
      ansible.builtin.replace:
        path: "~/.kube/config.new"
        regexp: "default"
        replace: "mightykube"
      delegate_to: 127.0.0.1
      become: false

    - name: Merge with any existing kubeconfig on control node
      ansible.builtin.shell: |
        TFILE=$(mktemp)
        KUBECONFIG=~/.kube/config.new:~/.kube/config kubectl config set-context mightykube --user=mightykube --cluster=mightykube
        KUBECONFIG=~/.kube/config.new:~/.kube/config kubectl config view --flatten > ${TFILE}
        mv ${TFILE} ~/.kube/config
      delegate_to: 127.0.0.1
      become: false
      register: k3s_server_mv_result
      changed_when:
        - k3s_server_mv_result.rc == 0

Let's also create a template file for the systemd service under ~/Projects/infra/ansible/k3s/roles/k3s_server/templates

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/etc/systemd/system/k3s.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s server

Let's create a ~/Projects/infra/ansible/k3s/k3s_server/default/main.yaml to set the version of k3s we want to install, and that we might change in the future when doing upgrades.

k3s_server_version: "v1.33.3+k3s1"

Finally, let's create a ~/Projects/infra/ansible/k3s/deploy.yaml that calls the role we just created on the k3s servers group.

---
- name: Install k3s
  hosts: k3s
  remote_user: thib
  tasks:
    - name: Install k3s server
      ansible.builtin.import_role:
        name: k3s_server

We can now use everything together by calling the playbook we created (and the role it calls) with the dynamic inventory generated by the Proxmox plugin. Let's try!

$ cd ~/Projects/infra/ansible
$ ansible-playbook -i inventory/proximighty.proxmox.yaml k3s/deploy.yaml

Using kubectl on my laptop, I can confirm that my single node cluster is ready

$ kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
mightykube   Ready    control-plane,master   4m   v1.33.3+k3s1

Great! We have used ansible to automate the installation of a single node k3s cluster, and how to control it from our laptop. Thanks to our dynamic inventory, ansible also figured out what VM to install it onto automatically. Look, Mom! We have a Kubernetes at home!

It's time to clean things up and remove sensitive credentials from our ansible scripts.

Removing sensitive information

When writing our ansible playbook, we didn't add new credentials. When setting up opentofu we created an API Key, and stored it in our Bitwarden vault under the name PROXMOX_VE_API_TOKEN. When configuring the dynamic inventory, we reused that same API key but wrote it in the plain text file.

There is a minor difference though. Opentofu uses the API Key formatted as

terraform@pve!provider=REDACTED

Ansible on the other hand uses the API Key formatted as

user: "terraform@pve"
token_id: "provider"
token_secret: "REDACTED"

The information is the same, but formatted differently. Fortunately for us, ansible supports searching strings with regular expressions. The regex to break it down into the three parts we need is rather simple:

([^!]+)!([^=]+)=(.+)

Ansible also has a lookup method to read environment variables. Let's put all the pieces together in our dynamic inventory file

plugin: community.general.proxmox
url: https://192.168.1.200:8006
user: "terraform@pve"
token_id: "provider"
token_secret: "REDACTED"
user: "{{ (lookup('ansible.builtin.env', 'PROXMOX_VE_API_TOKEN') | regex_search('([^!]+)!([^=]+)=(.+)', '\\1'))[0] }}"
token_id: "{{ (lookup('ansible.builtin.env', 'PROXMOX_VE_API_TOKEN') | regex_search('([^!]+)!([^=]+)=(.+)', '\\2'))[0] }}"
token_secret: "{{ (lookup('ansible.builtin.env', 'PROXMOX_VE_API_TOKEN') | regex_search('([^!]+)!([^=]+)=(.+)', '\\3'))[0] }}"
want_facts: true
groups:
  k3s: "'k3s' in (proxmox_tags_parsed|list)"
compose:
  ansible_host: proxmox_ipconfig0.ip | default(proxmox_net0.ip) | ansible.utils.ipaddr('address')
validate_certs: false

Voilà! Just like that we have removed the secrets from our files, and we're ready to commit them!

[!info] Why not use the Bitwarden plugin for ansible?

It's a good alternative, but I already rely on direnv to extract the relevant secrets from my vault and store them temporarily in environment variables.

Using the Bitwarden plugin in my playbook would tightly couple the playbook to ansible. By relying on the environment variables, only direnv is coupled to Bitwarden!

Now we can spin up a VM and install k3s on it in a handful of seconds! Our homelab is making steady progress. Next up we will see how to get services running on our cluster with GitOps!

A huge thanks to my friends and colleagues Davide, Quentin, Ark, Half-Shot, and Ben!

TIL that You can spot base64 encoded JSON, certificates, and private keys

Tue, 05 Aug 2025 13:00:00 GMT

I was working on my homelab and examined a file that was supposed to contain encrypted content that I could safely commit on a Github repository. The file looked like this

{
  "serial": 13,
  "lineage": "24d431ee-3da9-4407-b649-b0d2c0ca2d67",
  "meta": {
    "key_provider.pbkdf2.password_key": "eyJzYWx0IjoianpHUlpMVkFOZUZKcEpSeGo4UlhnNDhGZk9vQisrR0YvSG9ubTZzSUY5WT0iLCJpdGVyYXRpb25zIjo2MDAwMDAsImhhc2hfZnVuY3Rpb24iOiJzaGE1MTIiLCJrZXlfbGVuZ3RoIjozMn0="
  },
  "encrypted_data": "ONXZsJhz37eJA[...]",
  "encryption_version": "v0"
}

Hm, key provider? Password key? In an encrypted file? That doesn't sound right. The problem is that this file is generated by taking a password, deriving a key from it, and encrypting the content with that key. I don't know what the derived key could look like, but it could be that long indecipherable string.

I asked a colleague to have a look and he said "Oh that? It looks like a base64 encoded JSON. Give it a go to see what's inside."

I was incredulous but gave it a go, and it worked!!

$ echo "eyJzYW[...]" | base64 -d
{"salt":"jzGRZLVANeFJpJRxj8RXg48FfOoB++GF/Honm6sIF9Y=","iterations":600000,"hash_function":"sha512","key_length":32}

I couldn't believe my colleague had decoded the base64 string on the fly, so I asked. "What gave it away? Was it the trailing equal signs at the end for padding? But how did you know it was base64 encoded JSON and not just a base64 string?"

He replied,

Whenever you see ey, that's {" and then if it's followed by a letter, you'll get J followed by a letter.

I did a few tests in my terminal, and he was right! You can spot base64 json with your naked eye, and you don't need to decode it on the fly!

$ echo "{" | base64
ewo=
$ echo "{\"" | base64
eyIK
$ echo "{\"s" | base64
eyJzCg==
$ echo "{\"a" | base64
eyJhCg==
$ echo "{\"word\"" | base64
eyJ3b3JkIgo=

But there's even better! As tyzbit reported on the fediverse, you can even spot base64 encoded certificates and private keys! They all start with LS, which reminds of the LS in "TLS certificate."

$ echo -en "-----BEGIN CERTIFICATE-----" | base64
LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t

[!warning] Errata

As pointed out by gnabgib and athorax on Hacker News, this actually detects the leading dashes of the PEM format, commonly used for certificates, and a YAML file that starts with --- will yield the same result
$ echo "---\n" | base64
LS0tXG4K
This is not a silver bullet!

Thanks Davide and Denis for showing me this simple but pretty useful trick, and thanks tyzbit for completing it with certs and private keys!

Over engineering my homelab so I don't pay cloud providers

Mon, 04 Aug 2025 13:00:00 GMT

After years of self-hosting on a VPS in a datacenter, I've decided to move my services at home. But instead of just porting services, I'm using this as an opportunity to migrate to a more flexible and robust set up.

I will deploy services on a single mini pc. Since I need to be able to experiment and learn without disrupting my services, I will need to be able to spin up Virtual Machines (VMs). Let's explore how I deployed Proxmox Virtual Environment on a safe host for my specific needs as a homelabber, and how I automated as much of it as possible. In a follow-up post we will explore how to spin-up and configure VMs in a reproducible way on that setup.

[!warning] This guide has been updated

I initially wrote this guide for Debian Bookworm, but have now updated it for Debian Trixie. The major change lies in how to install the Proxmox repository and GPG key, that have change in Debian Trixie.

What I want to do and avoid

Objectives

After realizing that my good old Raspberry Pi 4 was too slow to let me backup or restore on an encrypted disk, I bought a Minisforum UM880 Plus. At €600 it was not extremely expensive, but I don't intend to spend more on hardware in the foreseeable future and I want to make the most of what I have right now.

I love to experiment and would like to do it safely without putting my production set-up at risk. Those are self-hosted services mostly for my personal usage, so I can afford occasional downtime, but I don't want to have to rebuild everything if my experiments go wrong. I also don't want to experiment by spinning up VMs at a cloud provider, because I will not know what I'm doing while learning, and cloud providers can get expensive very quickly.

One of my main objectives as I write these lines is to get up to speed with Kubernetes. I want to stay on a single-node k3s deployment while I get comfortable with operating services on a Kubernetes cluster, but I know I will want to explore deployments with several nodes, and eventually create a full blown k8s cluster based on Talos Linux.

Threat model

My server is in my living room. The most prominent threat in my model is a burglary. If my server gets stolen I will lose access to my infrastructure and my data. I also don't want my data to leak in the wild if the burglars put their hands on the disk in my server.

[!info] I need to have disk encryption and solid backups to keep my data safe

The second biggest threat is hardware failure. All devices can fail, but I'm fairly certain this is particularly true of a €600 mini pc that was not necessarily designed to serve as a home server.

[!info] I need to have a setup that can be automatically installed and configured

I am also a team of only one, I am fallible, and I don't have peers to review my exact set-up. To mitigate this risk I have a group of friends called the Infra Nerds Club, whom I regularly ask for advice.

[!info] I need to have a versioned set-up that can easily be rolled back

My ISP-provided router supports wireguard. Even when I'm out, I can join the local network of my server. But my server could be shut down because of a power outage or another reason. I might be at work or even on holidays when it happens, and even wireguard can't solve this.

[!info] I need a KVM on my local network so I can send Wake on LAN packages to my server

Unsurprisingly with my objectives and hardware constraints, I need to be able to spin up VMs to play with. The only realistic option on the table for a hobby homelabber is Proxmox Virtual Environment.

It is important to also highlight that if my server gets stolen or fails, I will not be able to spin up a hypervisor on a baremetal server again, and VPS providers would likely not let me configure a bridged network like I will do below.

The hypervisor and virtual machines I will deploy are just meant to give me flexibility. I consider the hypervisor and Virtual Machines as disposable, so I will not perform backups of the VMs themselves. I will however perform backups of the data and configuration of the services running on it.

One of my goals is to be able to quickly move my infrastructure to a cloud provider if something happened to my baremetal server, and back to a new baremetal server after it's been delivered.

Implementing it

I will deploy a Proxmox hypervisor on the physical server in my living room. On that hypervisor, I want to be able to statically declare what VMs must be spun up, how they should be configured, how the services (e.g. k3s) are deployed on those VMs, and what DNS records must be set to reach those services. There isn't a single unified tool to do so, and I will have to rely on opentofu, cloud init and ansible.

In this post I will only focus on deploying a rock stable Proxmox hypervisor on my server, but it's worth having a glimpse at how I will manage it.

Opentofu, and the mother project Terraform it originated from, are often described as Infra as Code (IaC). In other words, it lets you describe in a text file what VMs you want to create on your infrastructure. With cloud-init, you can add a basic configuration for your VM, such as the users credentials, ssh keys to trust, and network configuration.

A typical opentofu snippet to spin up a VM with a Debian OS pre-configured with cloud-init looks like this. We will explain how to actually use opentofu and cloud-init to spin up VMs in a further blog post.

resource "proxmox_virtual_environment_vm" "k3s-main" {
  name        = "k3s-main"
  description = "Production k3s' main VM"
  tags        = ["production", "k3s", "debian"]
  node_name   = "proximighty"

  cpu {
    cores = 4
    type  = "x86-64-v4"
  }

  memory {
    dedicated = 4096
    floating  = 4096
  }

  disk {
    datastore_id = "local"
    interface    = "virtio0"
    iothread     = true
    size         = 50
    file_id      = proxmox_virtual_environment_download_file.debian_cloud_image.id
  }

  [...]
}

Opentofu runs from my laptop. It reads the .tf files and will talk to the Proxmox host to spin up VMs and their basic configuration. It can also talk to my registrar (Cloudflare for now) to add new DNS records if I ask it to.

Having a VM pre-configured with network, users and trusted ssh keys is very useful to hook in the second configuration tool: ansible.

An ansible playbook is a text file describing the desired state of a server, often without describing how it must be achieved. For example, instead of describing "Open the file /etc/hosts and add the line 192.168.1.200 myhost.example.com", you describe "the line 192.168.1.200 myhost.example.com must be present in the file /etc/hosts".

It can sound like the same, but it's not: running the first description twice would result in the same line being added twice to the /etc/hosts file. Running the second description would result in having the desired lined only once. A typical playbook will look like this.

---
- name: Set the timezone to UTC
  community.general.timezone:
    name: UTC

- name: Install bridge utils
  ansible.builtin.apt:
    name: bridge-utils
    state: present

- name: Override Debian's default network configuration
  ansible.builtin.copy:
    src: interfaces
    dest: /etc/network/interfaces
    mode: "0644"

- name: Create a bridge interface vrm0 and give it a static IP
  ansible.builtin.copy:
    src: vmbr0
    dest: /etc/network/interfaces.d/vrm0
    mode: "0644"

- name: Ensure enp2s0 doesn't have an IP
  ansible.builtin.copy:
    src: enp2s0
    dest: /etc/network/interfaces.d/enp2s0
    mode: "0644"

- name: Add local IP to the hosts filename
  ansible.builtin.lineinfile:
    path: /etc/hosts
    line: 192.168.1.200 proximighty.ergaster.org proximighty
    create: true
    mode: "0644"

Ansible also runs from my laptop. It reads the playbook's .yaml files, and uses ssh to log into the target machine and apply the playbook configuration.

Setting up the Proxmox host

Installing an encrypted Debian

Proxmox is based on Debian and can be installed in 3 different way:

Via the official installer
By creating an automated installer
By installing it on top of an existing Debian install

The second option sounds very appealing, but there is a major issue: the Proxmox (automated) installer doesn't support setting up disk encryption. The simplest way to have disk encryption on the host is to install Debian first, and to install Proxmox on top.

I could automate the Debian install using preseeding. Preseed files contain the answers to the questions asked by the Debian installer. A colleague who wrote a preseed file for Debian 8 told me he didn't have to update it since. After writing my own preseed, I could [add it to the Debian netinst usb disk](preseed on usb disk) and Debian would be installed automatically without human intervention.

But preseed files can only customize the basic install of Debian. To install additional packages (like Proxmox) and configure my machine I need to rely on an ansible playbook.

It is also worth noting that if my server got stolen or if its hardware failed, I wouldn't be able to replace it with a baremetal server right away. I would have to choose a cloud provider and spin up VMs that roughly correspond to the ones I had running on my Proxmox.

[!info] Using a preseed would be a case of XKCD 1205

I shouldn't have to perform regular reinstalls of Debian for the Proxmox host, and I need to write an ansible playbook to configure it properly anyway. I would spend a lot of time automating the Debian install, but I wouldn't save a lot of time in doing so.

I grabbed a Debian netinstall and performed a regular install with disk encryption, with the following specificities:

I used the full disk with LVM, and set up disk encryption.
I didn't let the installer fill my disk with random data because it takes a lot of time and doesn't match my threat model.
I did set a root password. Proxmox is very root centric, and while there are workarounds to use a non-root user, it gets tedious very fast for little extra security.
At the package selection step, I disabled everything but SSH Server and standard system utilities. I need both to be able to ssh into my server and let ansible control my Proxmox host.

[!warning] Keyboard required

The disk is encrypted by a password. The server will prompt me for the password when it (re)starts and will not be able to boot if I don't type the password.

My server is connected to a KVM, so I can enter the disk encryption password when the server reboots. If you don't, you can install and configure Dropbear to do it over ssh or create a magic usb stick that LUKS will read to decrypt the disk.

Now, I need to interact with my . After installing Debian and unlocking the disk, I need to configure the ssh server to let me temporarily log in as root to copy my public key. Via my KVM, I update the /etc/ssh/sshd_config as follows

PermitRootLogin prohibit-password
PermitRootLogin yes

And I restart the sshd so I can log in as root

# systemctl restart sshd

On my laptop, I copy my public key to the server with

$ ssh-copy-id root@192.168.1.200

And finally I reverse the change on my server by editing /etc/ssh/sshd_config again so I can only log in by ssh key

PermitRootLogin yes
PermitRootLogin prohibit-password

One restart of the sshd later, my server is safe again

# systemctl restart sshd

Installing and configuring Proxmox

Installing Proxmox

Installing Proxmox on top of an existing Debian is a well supported and documented process. I followed these steps on my encrypted Debian until the Proxmox VE package install and... my machine didn't boot anymore. It was very confusing at first sight, because there was no error. I was prompted for my disk encryption password, the disk was successfully unlocked, and then nothing. The system just didn't boot, was unreachable via SSH, and didn't display anything via the KVM.

Figuring out why installing Proxmox bricks my Debian

I was extremely surprised that installing a vanilla Proxmox on a freshly installed, pristine Debian would completely brick the system!

I initially thought that the issue was the Proxmox kernel that didn't support disk encryption, or that didn't support my hardware well. After a few reinstalls and rebooting between the install steps, I figured out that booting on the Proxmox kernel without Promox VE installed worked perfectly fine. Even from an encrypted disk.

So I installed Proxmox, and asked the computer to tell me what it does when it boots. To do so, I wait for the GRUB screen to appear, and pressed <kbd>e</kbd> to get access to the boot command editor.

I replaced the quiet boot parameter by noquiet, and pressed <kbd>Ctrl</kbd> + <kbd>x</kbd> to save my changes and boot with this altered command. I could see that the machine was stuck on Job networking.service/start running.

Looking up Proxmox job networking start running yielded good results on the Proxmox forums. In this thread and that one users say ntp is causing issues. But I didn't have ntp, ntpsec-ntpdate or any related package installed!

After a few reinstalls and a bit of trial and error, I could figure out that my machine wouldn't boot after installing Proxmox VE if I didn't set up a static IP configuration for it. Configuring a static IP for the machine after a fresh reinstall fixed the issue.

Setting up a bridge network

I only have a single physical enp2s0 network interface card on my host, but I will have several guest VMs. Each VM needs to be able to use my host's network card and make it "impersonate" its virtual card. I'm writing a more detailed post about how this works, but the gist of it is that you need to create a virtual network interface vrmb0 called a bridge. The bridge will be connected both to the host's physical network card, and to the VMs network interface.

The physical network card no longer operates at the IP level: it merely serves as a packet sender and receiver. So I need to remove the default IP configuration on enp2s0, and configure vrmb0 to have an IP the host will be able to use instead.

Since I installed a minimal Debian, I need to install the required tools to create bridged networks

# apt install bridge-utils

Then, let's clean up the default network configuration in /etc/network/interfaces to only keep the loopback interface

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source /etc/network/interfaces.d/*

# The loopback network interface
auto lo
iface lo inet loopback

Let's add a file in /etc/network/interfaces.d/ for enp2s0 to be brought up but not try to get an IP

auto enp2s0
iface enp2s0 inet manual

And now let's create and configure vrmb0 to have a static IP

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.200
        gateway 192.168.1.254
        bridge_ports enp2s0
        bridge_stp off
        bridge_fd 0

I can finally restart the network to ensure everything is configured properly

# systemctl restart networking

That was a lot of work, and I'm not sure I will remember how to perform all of these steps if I need to rebuild a Proxmox host. Let's use ansible to automate everything I did after installing a clean Debian!

Automating the Proxmox install with ansible

Installing ansible

The ansible documentation lists several ways to install ansible. I didn't find anything related to homebrew, but the package is still present and seems up to date. Since I regularly upgrade the packages installed with homebrew, I decided it was the simplest way to keep an up to date ansible on my laptop and installed with

$ brew install ansible

Writing the playbook

I created a ~/Projects/infra folder that will contain everything related to my homelab. In this directory, I created two subdirectories: one called opentofu that we will use later to spin up VMs, and one called ansible that will contain my playbooks.

$ cd ~/Projects/infra
$ tree -L 1
.
├── ansible
└── tofu

I want to keep the ansible playbook for my infrastructure in a single place. At the root of my ansible repository, I have created two folders: inventory and proximighty (the name of the Proxmox host).

In the inventory folder I can list all my hosts and organize how I want. I created a production file that contains the following

[proximighty]
192.168.1.200 ansible_ssh_user=root

Since I don't have a local DNS set-up and I'm not too keen on using my public domain name for my internal network, I'll stick to the host IP. I've put it under the [proximighty] group so I can easily refer to it later in ansible, and specified that ansible must ssh as root into the machine to perform operations.

I then create a proximighty folder under ansible where I will describe everything that must be done on a fresh Debian to get it to the desired state. I create a configure.yaml that will be the root of my playbook.

$ cd ~/Projects/infra/ansible
$ tree -L 2
.
├── inventory
│   └── production
└── proximighty
    └── configure.yaml

In the configure.yaml file I describe the rough steps. In my case, I want to do two things:

Configure the host. That means setting the timezone to UTC, and installing the kitty-terminfo package so I can use kitty with my server.
Install Proxmox.

The basic structure looks like this

---
- name: Configure the host
  hosts: proximighty
  tasks:
    - name: Set timezone to UTC
      community.general.timezone:
        name: UTC

    - name: Install kitty files
      ansible.builtin.apt:
        name: kitty-terminfo
        state: present

- name: Install Proxmox
  hosts: proximighty
  tasks:
    - name: Install Proxmox
      ???

I left question marks at the end of the file, because there are quite a few steps to install Proxmox, including reboots. To keep the playbook readable, I will isolate these steps into their own module. Ansible calls this module a role. Let's go to the proximighty folder and create a role folder in it. Inside it we can create a proxmox folder that will contain all the instructions to install proxmox.

$ cd ~/Projects/infra/ansible
$ tree -L 3
.
├── inventory
│   └── production
└── proximighty
    ├── configure.yaml
    └── roles
        └── proxmox

The entry point of a task is a main.yaml file nested inside a tasks folder, so let's create the relevant file structure

$ cd ~/Projects/infra/ansible/proximighty
$ tree -L 4
.
├── configure.yaml
└── roles
    └── proxmox
        └── tasks
            └── main.yaml

Finally we can open open the main.yaml file and start describing the steps necessary to install Proxmox! The file starts with --- and will then contain the various steps. Let's start by ensuring that the bridge-utils package is present, so we can set-up a bridged network

---
- name: Install bridge utils
  ansible.builtin.apt:
    name: bridge-utils
    state: present

Then we will fiddle with the network files. We want to

Override the default configuration in /etc/network/interfaces
Create a bridge interface vmbr0 described by a file in /etc/network/interfaces.d/vmbr0
Configure enp2s0 with a file in /etc/network/interfaces.d/enp2s0 so it doesn't try to get its own IP
Add a local IP into /etc/hosts
Restart the network

Let's describe that in ansible terms

[...]

- name: Remove Debian's default network configuration
  ansible.builtin.copy:
    src: interfaces
    dest: /etc/network/interfaces
    mode: "0644"

- name: Create a bridge interface vrm0 and give it a static IP
  ansible.builtin.copy:
    src: vmbr0
    dest: /etc/network/interfaces.d/vrm0
    mode: "0644"

- name: Ensure enp2s0 doesn't have an IP
  ansible.builtin.copy:
    src: enp2s0
    dest: /etc/network/interfaces.d/enp2s0
    mode: "0644"

- name: Add local IP to the hosts filename
  ansible.builtin.lineinfile:
    path: /etc/hosts
    line: 192.168.1.200 proximighty.ergaster.org  proximighty
    create: true
    mode: "0644"

- name: Restart the networking service
  ansible.builtin.systemd_service:
    name: networking
    state: restarted

We're asking ansible to copy files over to the server, but we didn't tell it where to take the source files. By default, ansible looks up file in a files folder at the root of the role. Let's create the relevant files then:

$ cd ~/Projects/infra/ansible/proximighty
$ tree -L 4
.
├── configure.yaml
└── roles
    └── proxmox
        ├── files
        │   ├── enp2s0
        │   ├── interfaces
        │   └── vmbr0
        └── tasks
            └── main.yaml

The content of the files is the same as in the previous section. Now, to install Proxmox we need to add the Proxmox apt repositories to our apt list. For apt to trust it, we need to add the Proxmox signing key. We need the gpg package to be able to manipulate it. So let's add those steps in our proxmox/tasks/main.yaml file

[...]

- name: Ensure gpg is installed
  ansible.builtin.apt:
    name: gpg
    state: present

- name: Add the Proxmox key
  ansible.builtin.get_url:
    url: https://enterprise.proxmox.com/debian/proxmox-archive-keyring-trixie.gpg
    dest: /usr/share/keyrings/proxmox-archive-keyring.gpg
    checksum: sha256:136673be77aba35dcce385b28737689ad64fd785a797e57897589aed08db6e45
    mode: "0644"

- name: Add pve-no-subscription repository
  ansible.builtin.copy:
    src: pve-install-repo.sources
    dest: /etc/apt/sources.list.d/pve-install-repo.sources
    mode: "0644"

I'm also adding the repository file to the list of the role files in roles/proxmox/files/pve-install-repo.sources

Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
Architechtures: amd64

Finally we can update all the packages, ensure no package related to ntp is present, and reboot. Let's add those instructions to proxmox/tasks/main.yaml

[...]

- name: Update all packages
  ansible.builtin.apt:
    upgrade: full
    update_cache: true
  notify: Reboot

- name: Ensure ntp and related packages are absent
  ansible.builtin.apt:
    name:
      - ntp
      - ntpsec
      - ntpsec-ntpdate
    state: absent
  notify: Reboot

- name: Reboot after upgrading packages
  ansible.builtin.meta: flush_handlers

You probably noticed the notify: Reboot that appears twice. It could look like the machine is going to reboot twice, but this is not the case. It means each step will notify the Reboot handler, but handlers are only called at the end of a task... unless they are flushed before the end of the task. We explicitly flush the handlers with ansible.builtin.meta: flush_handlers, so the reboot will only happen here.

We called a handler, but we didn't define it anywhere. Like for files, ansible has a default place to look up for handlers: the handlers directory at the root of the module. Let's create the relevant files.

$ cd ~/Projects/infra/ansible/proximighty
$ tree -L 4
.
├── configure.yaml
└── roles
    └── proxmox
        ├── files
        │   ├── enp2s0
        │   ├── interfaces
        │   ├── storage.cfg
        │   └── vmbr0
        ├── handlers
        │   └── main.yaml
        └── tasks
            └── main.yaml

And let's add the Reboot handler in there

---
- name: Reboot
  ansible.builtin.reboot:

We can now finalize the install by

Installing the Proxmox kernel and rebooting
Installing Proxmox VE and dependancies
Removing the Debian kernel and os-prober
Removing the pve-enterprise repository that Proxmox automatically installed
Rebooting one last time

Let's append those steps to the proxmox/tasks/main.yaml file

[...]

- name: Install Proxmox VE Kernel
  ansible.builtin.apt:
    name: "proxmox-default-kernel"
    state: present
  notify: Reboot

- name: Reboot after installing Proxmox VE Kernel
  ansible.builtin.meta: flush_handlers

- name: Install Proxmox VE and dependencies
  ansible.builtin.apt:
    name:
      - proxmox-ve
      - postfix
      - open-iscsi
      - chrony
    state: present

- name: Remove the Debian kernel and os-prober
  ansible.builtin.apt:
    name:
      - linux-image-amd64
      - os-prober
    state: absent
  notify:
    - Update GRUB
    - Reboot

- name: Remove pve-enterprise repository
  ansible.builtin.apt_repository:
    repo: deb https://enterprise.proxmox.com/debian/pve {{ debian_version }} pve-enterprise
    state: absent
    update_cache: true
    filename: pve-enterprise

- name: Reboot after installing Proxmox VE and removing old kernels
  ansible.builtin.meta: flush_handlers

You might notice the extra Update GRUB handler, that we need to also add to our handlers

[...]

- name: Update GRUB
  ansible.builtin.command: update-grub
  changed_when: true
  notify: Reboot

Finally, we can wrap it all together by calling this proxmox role from our main configure.yaml file

---
- name: Configure the host
  hosts: proximighty
  tasks:
    - name: Set timezone to UTC
      community.general.timezone:
        name: UTC

    - name: Install kitty files
      ansible.builtin.apt:
        name: kitty-terminfo
        state: present

- name: Install Proxmox
  hosts: proximighty
  tasks:
    - name: Install Proxmox
      ansible.builtin.import_role:
        name: proxmox

It's now time to execute that playbook!

Executing the playbook

After writing this playbook, it's now time to execute it! To be able to execute this playbook, we need to be able to ssh as root on the Debian host that will get Proxmox installed, with a ssh key and not a password.

As a quick test, running ssh root@192.168.1.200 should log me in without prompting me for a password or a fingerprint verification.

From my laptop, I go to the ansible directory, from which I can run a command to invoke the configure.yaml playbook with the production inventory like so

$ cd ~/Projects/infra/ansible
$ ansible-playbook -i inventory/production proximighty/configure.yaml

Ansible will install everything and occasionally reboot the server when needed. Since my server has an encrypted disk, I need to monitor what's happening on my KVM and unlock the disk with my encryption passphrase when prompted to.

I now have an ansible playbook I can use to quickly spin up a new Proxmox host on a fresh Debian with an encrypted disk! This is a solid foundation for a flexible homelab. I will be able to spin up a long-lived VM for my main k3s node. I will be able to spin up additional k3s workers if need be, or an entirely different cluster to play with, all while keeping my production reasonably isolated and stable.

We'll see in another blog post how to use opentofu, cloud-init and ansible to spin up new VMs on that Proxmox host!

Massive thanks to my colleagues and friends Half-Shot, Davide, and Ark for their insights!