Configuring Gitea/Forgejo to Sign Merge Commits

2024-05-05T00:00:00+00:00

Gitea supports automatically signing commits</a> it generates (such as when merging pull requests, or editing files through the web editor). Sadly there is no documentation on how to actually configure this, besides vague references that it is left up to the server administrator to achieve.

Secure key management is a topic fraught with complexity and trade off decisions, and the Gitea development team holds a (sensible) position that it is preferable to give no advice than it is to give bad advice.

A position that I, as an internet rando, am absolutely not bound by, so here's what I did!

Edit 2024-08-19: these exact steps also work for Forgejo! Just use /var/lib/forgejo</code> instead of /var/lib/gitea</code> for all instructions below 

Seriously though, take a close look at your own threat model and security posture before you apply any of the following instructions, which are, frankly, provided for free, as in used mattress. These are educational instructions, and you bear all responsibility for the consequences of following them. </blockquote>

Overview</h1>
Under the hood, Gitea basically has its own .gitconfig</code> which is applied to all git operations that it performs. At the end of the day, all we need to do is ensure a GPG signing key is available inside Gitea's "home" directory and make sure that Gitea can interact with it unattended.
Which is easier said than done as GPG strongly insists on passphrase prompts and is otherwise hostile to unattended interaction. Perhaps it is possible to achieve this by starting and priming a gpg-agent</code> instance to be used by Gitea, but I did not bother investigating it as it adds more complexity, and still requires automating entering the passphrase to it. At that rate, a secret of one form or another needs to be available. I also have not (yet) tried using SSH signing. Since git already supports it with some config changes, it might be possible that it "just works", but I don't know if there might be other GPG-assumptions baked into Gitea (like the /api/v1/signing-key.gpg</code> endpoint. </blockquote>
Generating a new key</h1> First we want to create a brand new GPG key for this Gitea instance. Note that this key only needs to be configured for signing as we'll only use it to generate a second signing subkey that will be used by Gitea.
gpg --full-generate-key </code></pre> Follow the wizard and fill in the values as appropriate. Note that GPG will ask for a passphrase. Generate a strong password and save it in your password manager (as we'll need it in a bit).
gpg (GnuPG) 2.4.5; Copyright (C) 2024 g10 Code GmbH This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Please select what kind of key you want: (1) RSA and RSA (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) (9) ECC (sign and encrypt) *default* (10) ECC (sign only) (14) Existing key from card Your selection? 10 Please select which elliptic curve you want: (1) Curve 25519 *default* (4) NIST P-384 (6) Brainpool P-256 Your selection? 1 Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) 0 Key does not expire at all Is this correct? (y/N) y GnuPG needs to construct a user ID to identify your key. Real name: Example Gitea Email address: gitea@git.example.com Comment: You selected this USER-ID: "Example Gitea <gitea@git.example.com>" Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. gpg: revocation certificate stored as '/home/ivan/.gnupg/openpgp-revocs.d/F41E36D6735FD9BE1B53518EA1596EF2CE56B350.rev' public and secret key created and signed. pub ed25519/0xA1596EF2CE56B350 2024-05-05 [SC] Key fingerprint = F41E 36D6 735F D9BE 1B53 518E A159 6EF2 CE56 B350 uid Example Gitea <gitea@git.example.com> </code></pre> Note the line about gpg: revocation certificate stored as '/home/ivan/.gnupg/openpgp-revocs.d/F41E36D6735FD9BE1B53518EA1596EF2CE56B350.rev'</code>. This is a revocation certificate for the new key, generated here in case the key needs to be revoked in the future (but the secret part of the key is somehow lost or destroyed). Save this somewhere safe (like a password manager) and destroy this copy when done (and optionally ensure it is scrubbed from your ZFS snapshots if you care to).
Generating a signing subkey</h1> Now it's time to generate a signing subkey that will be used by Gitea directly. The idea is that this subkey can be rotated at will while the main key remains offline/in cold storage (so it can be trusted for longer periods of time). Note that GPG will ask for the password we set earlier.
gpg --edit-key 0xA1596EF2CE56B350 </code></pre>
gpg (GnuPG) 2.4.5; Copyright (C) 2024 g10 Code GmbH This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Secret key is available. sec ed25519/0xA1596EF2CE56B350 created: 2024-05-05 expires: never usage: SC trust: ultimate validity: ultimate [ultimate] (1). Example Gitea <gitea@git.example.com> gpg> addkey Please select what kind of key you want: (3) DSA (sign only) (4) RSA (sign only) (5) Elgamal (encrypt only) (6) RSA (encrypt only) (10) ECC (sign only) (12) ECC (encrypt only) (14) Existing key from card Your selection? 10 Please select which elliptic curve you want: (1) Curve 25519 *default* (4) NIST P-384 (6) Brainpool P-256 Your selection? 1 Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) 0 Key does not expire at all Is this correct? (y/N) y Really create? (y/N) y We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. sec ed25519/0xA1596EF2CE56B350 created: 2024-05-05 expires: never usage: SC trust: ultimate validity: ultimate ssb ed25519/0xA36F2B11E12C310D created: 2024-05-05 expires: never usage: S [ultimate] (1). Example Gitea <gitea@git.example.com> gpg> save </code></pre> Backing up the newly generated keys</h1> Now is a good time to store a secure copy of the key's we've just generated (e.g. storing the output somewhere in your password manager). Once again GPG will ask for the password we created earlier. gpg --export-secret-keys --armor --export-options export-backup 0xA1596EF2CE56B350 </code></pre> Transferring the signing subkey</h1> Next we'll need to transfer the signing subkey to the Gitea host itself. Running the following will print out the secret portion which we'll copy/paste afterwards. There are other ways to achieve this without leaving traces (either on disk or through the system clipboard) but that is left as an exercise to the (sufficiently paranoid) reader. gpg --export-secret-subkeys --armor 0xA36F2B11E12C310D </code></pre> Then, we need to ssh to the Gitea host and temporarily take on the gitea</code> user. ssh giteahost.example.com sudo su gitea </code></pre> On NixOS, Gitea will use /var/lib/gitea</code> as its root directory, and data/home</code> for it's $HOME</code> equivalent (though both of these are configurable so consult your distro docs and replace the values as appropriate). # NB: using --batch here will suppress password prompts, meaning the key will # be imported without any (password) protection. Make sure Gitea's root # directory is fully sandboxed away from other users gpg --homedir /var/lib/gitea/data/home/.gnupg --batch --import # Paste the output from the previous command here. # Hit Enter then Ctrl+D to finish # # Note: if importing fails, kill all instances of `gpg-agent` and try again. </code></pre> Now that the (subkey) is imported, we need to update its trust level: gpg --homedir /var/lib/gitea/data/home/.gnupg --edit-key 0xA36F2B11E12C310D </code></pre> gpg (GnuPG) 2.4.5; Copyright (C) 2024 g10 Code GmbH This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Secret subkeys are available. gpg: /var/lib/gitea/data/home/.gnupg/trustdb.gpg: trustdb created pub ed25519/A1596EF2CE56B350 created: 2024-05-05 expires: never usage: SC trust: unknown validity: unknown ssb ed25519/A36F2B11E12C310D created: 2024-05-05 expires: never usage: S [ unknown] (1). Example Gitea <gitea@git.example.com> gpg> trust pub ed25519/A1596EF2CE56B350 created: 2024-05-05 expires: never usage: SC trust: unknown validity: unknown ssb ed25519/A36F2B11E12C310D created: 2024-05-05 expires: never usage: S [ unknown] (1). Example Gitea <gitea@git.example.com> Please decide how far you trust this user to correctly verify other users' keys (by looking at passports, checking fingerprints from different sources, etc.) 1 = I don't know or won't say 2 = I do NOT trust 3 = I trust marginally 4 = I trust fully 5 = I trust ultimately m = back to the main menu Your decision? 5 Do you really want to set this key to ultimate trust? (y/N) y pub ed25519/A1596EF2CE56B350 created: 2024-05-05 expires: never usage: SC trust: ultimate validity: unknown ssb ed25519/A36F2B11E12C310D created: 2024-05-05 expires: never usage: S [ unknown] (1). Example Gitea <gitea@git.example.com> Please note that the shown key validity is not necessarily correct unless you restart the program. gpg> quit </code></pre> Lastly, we need to configure Gitea to not even attempt to open a pinentry prompt since there isn't going to be anyone around to answer it. We can achieve this by configuring git to call a wrapper program which invokes GPG with --batch</code> (which will disable these prompts) # On NixOS, system-wide packages show up in /run/current-system/sw/bin/ # swap it out with the appropriate location on your system echo >/var/lib/gitea/data/home/gpg-nopinentry <<'EOF' #!/usr/bin/env bash exec /run/current-system/sw/bin/gpg --batch "$@" EOF chmod +x /var/lib/gitea/data/home/gpg-nopinentry echo >>/var/lib/gitea/data/home/.gitconfig <<EOF [gpg] program = "/var/lib/gitea/data/home/gpg-nopinentry" EOF </code></pre> Telling Gitea about the new key</h1> Gitea still needs to be instructed to use the key. There are a number of configuration options</a> to consider but here are the most important ones to set. Make sure they match the values that were originally set when generating the root key. [repository.signing] SIGNING_KEY=0xA36F2B11E12C310D SIGNING_NAME=Example Gitea SIGNING_EMAIL=gitea@git.example.com </code></pre> Clean up</h1> Back on the original host we can now purge the secret keys (assuming they have been backed up securely) to minimize them getting compromised: gpg --delete-secret-keys 0xA1596EF2CE56B350 </code></pre> Happy self-hosting!

NixOS on the PiBox

2023-02-19T00:00:00+00:00

The PiBox</a> is a small personal server powered by a Raspberry Pi CM4. It comes in a nice enclosure which has a fan, an LCD screen, and has two bays for SATA SSDs. KubeSail</a>, the company behind it, offers backup storage and proxy traffic as a service. They also support a bunch of templates for easily self hosting apps like Jellyfin or NextCloud.

But, since I'm already very comfortable with NixOS and deploy a bunch of custom workloads with it, I wanted to try using it instead of the OS that ships with the PiBox.

Here's how I went about it, and how you can too!

Preparing the CM4 for boot</h1>

Although the CM4 (technically) supports booting from a variety of different targets besides its EEPROM (like an NVMe drive, a USB stick, and even the network!), it doesn't actually support booting from a SATA drive. I guess most people either boot from a USB drive or go all in with an NVMe drive, so for whatever reason direct boot from SATA was never designed into the firmware. The good thing is we can still boot from the EEPROM itself but keep the OS and all other data on the SSD drive, so that's what we'll do here.

The first step is to optionally update the board's firmware and change the boot order. The default configuration tries a whole bunch of boot options which likely will never be used which means it takes it a while to actually boot.

We'll start by opening up the PiBox and carefully separating the carrier board from the backplane</a>, then flip the "boot mode" switch to rpiboot</code> and connect the board with a USB-C cable to a PC. To change the boot config we'll need to use the raspberrypi usbboot</code> toolkit:

git</span> clone https://github.com/raspberrypi/usbboot </span>~</span>/usbboot
</span>cd </span>~</span>/usbboot/recovery
</span>
</span># Edit the `boot.conf` file and set the `BOOT_ORDER` variable
</span># I noticed that if the "SD" (same as the EEPROM on CM4) option (1) is set to
</span># run first the board doesn't actually boot from it but tries everything else
</span># first. Maybe there's some kind of delay the hardware needs to warm up but I
</span># found if I set `BOOT_ORDER=0xf514` then things work pretty smoothly. This
</span># configuration basically tries things in the following order:
</span># - 4: USB mass storage device: don't care about this, try anything first
</span># - 1: SD card (or the EEPROM for the CM4), where we would normally boot from
</span># - 5: BCM-USB: boot from the board hardware headers or something idk
</span># - f: Restart if all else fails
</span>#
</span># Also note that you can keep `ENABLE_SELF_UPDATE=1` to allow updating the
</span># firmware directly from a booted OS in the future without having to reconnect
</span># to the board like we are doing here
</span>vim</span> boot.conf
</span>
</span># Then apply the changes and flash the firmware, if it went well the lights
</span># should start blinking red and green
</span>./update-pieeprom.sh
</span>nix</span> shell nixpkgs#rpiboot</span> --command</span> sudo rpiboot</span> -d</span> .
</span></code></pre>
Disconnect and reconnect the board to the PC again. This time we'll mount the
CM4's EEPROM storage as a generic device we can manipulate from the PC.</p>
nix</span> shell nixpkgs#rpiboot</span> --command</span> sudo rpiboot</span> -d </span>~</span>/usbboot/mass-storage-gadget
</span></code></pre>
# The disk should show up as something like /dev/sda, though if the system
</span># already has other disks present it might show up as /dev/sdb or /dev/sdc, etc.
</span># so DOUBLE CHECK THIS!
</span>SD_DISK</span>=</span>"/dev/sd..."
</span></code></pre>

Note: I noticed that sometimes the mount would randomly disconnect and
reconnect on its own (usually showing up as a new</em> device like /dev/sdb</code> if
it was previously /dev/sda</code>). Not sure if this was due to my cable connection
or something else, so I highly recommend double checking things before doing
any operations on the EEPROM.</p>
</blockquote>
Next we'll partition the EEPROM and initialize a file system on it:</p>
nix</span> shell nixpkgs#parted</span> --command</span> sudo parted</span> --align</span> optimal </span>"$</span>{SD_DISK}</span>" </span><<</span>EOF
</span>    mklabel gpt \
</span>    mkpart primary fat32 1MiB 100% \
</span>    name 1 'EFI system partition' \
</span>    set 1 esp on \
</span>    print \
</span>    quit
</span>EOF
</span>
</span>nix</span> shell nixpkgs#dosfstools</span> --command</span> sudo mkfs.vfat </span>"$</span>{SD_DISK}</span>1"
</span>
</span># Write down UUIDs for what will become our boot partition
</span># and add it to the NixOS config
</span>ls</span> -l</span> /dev/disk/by-uuid
</span></code></pre>
Finally, disconnect the CM4, but don't reassemble it quite yet as we'll write
some more data to it later!</p>
Preparing a builder VM</h1>
I had originally set out to plug the new SSD into my desktop and do the
partitioning and installation from there, in the hope that it would be faster
and save me various reboot and debug cycles. Even though I have binfmt enabled
(let's me run aarch64 binaries via QEMU) and I can successfully do remote
aarch64 deployments from my x86 desktop via nixos-rebuild switch</code>, I was not
able to get nixos-install</code> to work despite my best efforts.</p>
I was also unsuccessful in booting the PiBox from a USB stick (though maybe I
made a mistake with the firmware config or my USB image). I briefly considered
flashing the NixOS installer directly on the EEPROM and doing the installation
from there, but I wasn't sure if it could cope with doing the installation
directly on the partition it was booted from.</p>
Instead I decided to spin up a quick QEMU VM and trick nixos-install</code> into
working. To save you some effort, copy the config below a flake.nix</code> in a new
directory, then nix build -L && ./result</code>.</p>
{
</span>  </span>inputs</span>.</span>nixpkgs</span>.</span>url </span>= </span>"github:NixOS/nixpkgs/nixos-unstable"</span>;
</span>
</span>  </span>outputs </span>= </span>{ </span>self</span>, </span>nixpkgs </span>}:
</span>    </span>let
</span>      </span>DISK </span>= </span>throw </span>"REPLACE ME: /dev/disk/by-id/ata-..."</span>;
</span>      </span>sshKey </span>= </span>throw </span>"REPLACE ME: ssh-ed25519 ..."</span>;
</span>
</span>      </span>pkgs </span>= </span>import </span>nixpkgs </span>{
</span>        </span>system </span>= </span>"x86_64-linux"</span>;
</span>      }</span>;
</span>      </span>pkgsAarch64 </span>= </span>import </span>nixpkgs </span>{
</span>        </span>system </span>= </span>"aarch64-linux"</span>;
</span>      }</span>;
</span>
</span>      </span>isoConfiguration </span>= </span>nixpkgs</span>.</span>lib</span>.</span>nixosSystem </span>{
</span>        </span>system </span>= </span>"aarch64-linux"</span>;
</span>        </span>modules </span>= </span>[
</span>          ({ </span>modulesPath</span>, ... </span>}: {
</span>            </span>imports </span>= </span>[
</span>              </span>"</span>${</span>modulesPath</span>}</span>/installer/cd-dvd/iso-image.nix"
</span>            ]</span>;
</span>            </span>isoImage</span>.</span>makeEfiBootable </span>= </span>true</span>;
</span>
</span>            </span>boot</span>.</span>supportedFilesystems </span>= </span>[ </span>"zfs" </span>]</span>;
</span>
</span>            </span>networking</span>.</span>hostId </span>= </span>"a7dbd851"</span>; </span># random value
</span>
</span>            </span>environment</span>.</span>systemPackages </span>= </span>with </span>pkgsAarch64</span>; [
</span>              </span>coreutils
</span>              </span>cryptsetup
</span>              </span>dosfstools
</span>              </span>gitMinimal
</span>              </span>htop
</span>              </span>openssh
</span>              </span>parted
</span>              </span>smartmontools
</span>              </span>unzip
</span>              </span>util-linux
</span>              </span>wget
</span>              </span>zfs
</span>            ]</span>;
</span>
</span>            </span>services</span>.</span>openssh</span>.</span>enable </span>= </span>true</span>;
</span>            </span>users</span>.</span>users</span>.</span>root</span>.</span>openssh</span>.</span>authorizedKeys</span>.</span>keys </span>= </span>[ </span>sshKey </span>]</span>;
</span>          })
</span>        ]</span>;
</span>      }</span>;
</span>
</span>      </span>iso </span>= </span>isoConfiguration</span>.</span>config</span>.</span>system</span>.</span>build</span>.</span>isoImage</span>;
</span>      </span>vmScript </span>= </span>pkgs</span>.</span>writeScript </span>"run-nixos-vm" ''
</span>        #!</span>${</span>pkgs</span>.</span>runtimeShell</span>}
</span>        </span>${</span>pkgs</span>.</span>qemu</span>}</span>/bin/qemu-system-aarch64 \
</span>          -machine virt,gic-version=max \
</span>          -cpu max \
</span>          -m 4G \
</span>          -smp 4 \
</span>          -drive file=$(echo </span>${</span>iso</span>}</span>/iso/*.iso),format=raw,readonly=on \
</span>          -drive file=</span>${</span>DISK</span>}</span>,format=raw,readonly=off \
</span>          -nic user,hostfwd=tcp::3333-:22 \
</span>          -nographic \
</span>          -bios </span>${</span>pkgsAarch64</span>.</span>OVMF</span>.</span>fd</span>}</span>/FV/QEMU_EFI.fd
</span>      ''</span>;
</span>    </span>in
</span>    {
</span>      </span>packages</span>.</span>x86_64-linux</span>.</span>default </span>= </span>vmScript</span>;
</span>    }</span>;
</span>}
</span></code></pre>
Now that we have a VM running we can connect to it using</p>
ssh</span> -p</span> 3333 root@localhost </span>\
</span>  -o </span>"UserKnownHostsFile=/dev/null" </span>\
</span>  -o </span>"StrictHostKeyChecking=no"
</span></code></pre>
Unless specified otherwise, the below commands should be run within this
session.</p>
Partitioning the SSD</h2>
I like to partition my drives as follows:</p>

1MiB for the GPT partition metadata</li>
1023MiB for the boot partition

Although we're going to be booting from the EEPROM, I'm still going to
reserve a boot partition in case the drive needs to be salvaged and booted
elsewhere. That way it won't be necessary to add a partition later and risk
destroying the existing data.</li>
</ul>
</li>
32MiB for storing LUKS keys and metadata</li>
8GiB for (encrypted) swap (same size as the RAM on the device)</li>
The remainder of the space for the (encrypted) data</li>
</ol>
# The disk should show up at /dev/vdb on the VM, change this if it doesn't
</span>VDISK</span>=</span>/dev/vdb
</span>
</span># Confirm this we've chosen the right disk
</span>smartctl</span> -a </span>${VDISK}
</span></code></pre>
# Do the actual partition as described above
</span>parted</span> --align</span> optimal ${VDISK}</span> -- </span>\
</span>    mklabel gpt </span>\
</span>    mkpart primary fat32 1MiB 1024MiB </span>\
</span>    name 1 </span>'EFI system partition' </span>\
</span>    set 1 esp on </span>\
</span>    mkpart primary 1024MiB 1056MiB </span>\
</span>    name 2 </span>'luks key' </span>\
</span>    mkpart primary 1056MiB 9248MiB </span>\
</span>    name 3 </span>'swap' </span>\
</span>    mkpart primary 9248MiB 100% </span>\
</span>    name 4 </span>'root' </span>\
</span>    print </span>\
</span>    quit
</span>
</span># Lastly, write down the UUIDs for the LUKS, swap, and root partitions
</span># and add them to the NixOS config
</span>ls</span> -l</span> /dev/disk/by-uuid
</span></code></pre>
LUKS Configuration</h2>
Encrypting the SSD('s non-boot partitions) works just like on any other machine.
To give a concrete example, we're going to set up things in the following manner:</p>

cryptkey</code> - this is the second partition we created above, and will be
unlocked with a password we remember. Its contents will be a bunch of random
data which will be used for unlocking the rest of the encrypted partitions</li>
cryptswap</code> - this is the third partition we created above, and will be
unlocked using the decrypted cryptkey</code> partition. It will be used for the
system's swap.</li>
cryptroot</code> - this is the fourth partition we created above, and will be
unlocked using the decrypted cryptkey</code> partition; it will also have a
backup</em> password (which we need not remember, but should store a copy in a
safe place) in case cryptkey</code> is corrupted. It will contain the actual root
filesystem for the device.</li>
</ul>

Note: I choose to enable --allow-discards</code> which instructs the mapper to
propagate trim commands issued by the underlying filesystem, allowing the SSD
to better perform wear leveling. This option is disabled by default since
there are some theoretical attack vectors from having it enabled and allowing
an attacker to physically access the disk (namely leaking which blocks are
trimmed, an some potential oracle attacks if the attacker can influence what
data is written to the disk). Consult your threat model if this option is
appropriate for you.</p>
</blockquote>
# Note set the password you will use for "day-to-day" unlocking of the system
</span># at boot, and make it a good one!
</span>cryptsetup</span> luksFormat</span> --type</span> luks1 </span>"$</span>{VDISK}</span>2"
</span>cryptsetup</span> open</span> --type</span> luks1 </span>"$</span>{VDISK}</span>2"</span> cryptkey
</span># Fill the (decrypted) cryptkey partition full of random data
</span># This invocation will fail with a "device out of space" error which is expected
</span>dd</span> if=/dev/urandom of=/dev/mapper/cryptkey bs=1024 status=progress
</span>
</span># Create and mount the encrypted swap partition
</span>cryptsetup</span> luksFormat </span>\
</span>  --type</span> luks1 </span>\
</span>  --keyfile-size</span> 8192 </span>\
</span>  --key-file</span> /dev/mapper/cryptkey </span>\
</span>  </span>"$</span>{VDISK}</span>3"
</span>cryptsetup</span> open </span>\
</span>  --type</span> luks1 </span>\
</span>  --keyfile-size</span> 8192 </span>\
</span>  --key-file</span> /dev/mapper/cryptkey </span>\
</span>  </span>"$</span>{VDISK}</span>3" </span>\
</span>  cryptswap
</span>
</span># Create and mount the data partition.
</span># Use a strong backup unlock phrase (e.g. dice ware) and write this down someplace safe!
</span>cryptsetup</span> luksFormat</span> --type</span> luks1 </span>"$</span>{VDISK}</span>4"
</span>cryptsetup</span> luksAddKey </span>\
</span>  --new-keyfile-size</span> 8192 </span>\
</span>  </span>"$</span>{VDISK}</span>4" </span>\
</span>  /dev/mapper/cryptkey
</span>cryptsetup</span> open </span>\
</span>  --type</span> luks1 </span>\
</span>  --keyfile-size</span> 8192 </span>\
</span>  --key-file</span> /dev/mapper/cryptkey </span>\
</span>  --allow-discards </span>\
</span>  </span>"$</span>{VDISK}</span>4" </span>\
</span>  cryptroot
</span></code></pre>
Finally, we initialize the (unencrypted) boot partition with an empty FAT32
partition, and initialize and enable the (decrypted) swap partition.</p>
mkfs.vfat </span>"$</span>{VDISK}</span>1"
</span>mkswap</span> /dev/mapper/cryptswap
</span>swapon</span> /dev/mapper/cryptswap
</span></code></pre>

Remember to update the configuration with a postDeviceCommand</code> to cryptsetup close cryptkey</code> so that the contents of the cryptkey</code> partition don't remain
accessible after booting!</p>
</blockquote>
ZFS Configuration</h2>
There's nothing specific to this setup which dictates how ZFS must be
configured, but as a complete example I want to describe my approach with the
following dataset hierarchy:</p>

local</code> - for data which is either ephemeral or does not need to be backed up

root</code> - mounted as the system root, reverted to a blank snapshot on every
boot so I can Erase My Darlings</a></li>
nix</code> - mounted as /nix/store</code>, no need to snapshot as it can be trivially
rebuilt if necessary</li>
</ul>
</li>
persist</code> - for all data that should be persisted across reboots, snapshotted
by default

journal</code> - systemd logs, mounted at /var/log/journal</code></li>
lib</code> - catchall for application state, mounted at /var/lib</code>. Normally I
like to split out services into their own datasets (for independent
snapshotting and rollback), though this acts as a good safety to avoid
forgetting to persist a particular service's state directories and losing them
on reboot.</li>
system</code> - miscellaneous system-specific files which should be persisted
across reboots, mounted at /persist</code></li>
user</code> - parent dataset for users' data/home directories</li>
</ul>
</li>
reserved</code> - used for over-provisioning the disk (i.e. no data will be written
here to allow the SSD to move blocks and maintain the health of the
flash storage)</li>
</ul>
# Configure the pool and user names we want to use
</span>POOL</span>=</span>phlegethon
</span>MY_USER</span>=</span>ivan
</span></code></pre>
# Note: when creating the pool, lower case `-o` is used to configure properties at
</span># the _pool_ level, while upper case `-O` is used to configure properties at the
</span># _dataset_ level
</span>#
</span># Note: ashift MUST BE SET or there will be horrible horrible write performance
</span># using the default value ZFS selects. If you aren't sure what to pick and
</span># have a modern drive, just take my word for it and set it to 13.
</span>zpool</span> create </span>\
</span>  -o</span> ashift=13 </span>\
</span>  -o</span> autotrim=on </span>\
</span>  -O</span> acltype=posixacl </span>\
</span>  -O</span> atime=off </span>\
</span>  -O</span> canmount=off </span>\
</span>  -O</span> compression=lz4 </span>\
</span>  -O</span> xattr=sa </span>\
</span>  -m</span> legacy </span>\
</span>  ${POOL} /dev/mapper/cryptroot
</span>
</span># Reserve (i.e. overprovision) ~10% of the disk (assuming 1TB disk)
</span>zfs</span> create </span>\
</span>    -o</span> reservation=200G </span>\
</span>    -o</span> quota=200G </span>\
</span>    -o</span> canmount=off </span>\
</span>    ${POOL}/reserved
</span>
</span># systemd-remount-fs.service complains if mountpoint not set
</span>zfs</span> create</span> -o</span> canmount=off</span> -o</span> mountpoint=/ ${POOL}/local
</span>zfs</span> create ${POOL}/local/nix
</span>zfs</span> create ${POOL}/local/root
</span>
</span># Snapshot the root while still empty so we can easily revert it back
</span>zfs</span> snapshot ${POOL}/local/root@blank
</span>
</span>zfs</span> create</span> -o</span> com.sun:auto-snapshot=true</span> -o</span> canmount=off ${POOL}/persist
</span>zfs</span> create ${POOL}/persist/system
</span>zfs</span> create ${POOL}/persist/lib
</span>zfs</span> create ${POOL}/persist/journal
</span>zfs</span> create</span> -o</span> canmount=off ${POOL}/persist/user
</span>zfs</span> create ${POOL}/persist/user/${MY_USER}
</span></code></pre>
Finally, mount all datasets at their appropriate paths before</em> doing the
installation (lest the data be written in the wrong spot and missing during
boot):</p>
mkdir</span> /mnt
</span>mount</span> -t</span> zfs </span>"$</span>{POOL}</span>/local/root"</span> /mnt
</span>
</span>mkdir</span> -p</span> /mnt/{boot</span>,</span>home/${MY_USER}</span>,</span>nix</span>,</span>persist</span>,</span>var/{lib</span>,</span>log/journal}}
</span>
</span>mount </span>"$</span>{VDISK}</span>1"</span> /mnt/boot
</span>mount</span> -t</span> zfs </span>"$</span>{POOL}</span>/local/nix"</span> /mnt/nix
</span>mount</span> -t</span> zfs </span>"$</span>{POOL}</span>/persist/user/$</span>{MY_USER}</span>"</span> /mnt/home/${MY_USER}
</span>mount</span> -t</span> zfs </span>"$</span>{POOL}</span>/persist/system"</span> /mnt/persist
</span>mount</span> -t</span> zfs </span>"$</span>{POOL}</span>/persist/lib"</span> /mnt/var/lib
</span>mount</span> -t</span> zfs </span>"$</span>{POOL}</span>/persist/journal"</span> /mnt/var/log/journal
</span></code></pre>
(Optional) Remote LUKS Unlock</h2>
Having to plug in a keyboard and monitor to the PiBox to unlock after a restart
can be a chore, so being able to unlock the drive remotely via SSH can be much
more convenient. First we need to generate a unique host key for the boot</em>
stage.</p>

Note that although the key itself will be stored on the encrypted partition,
it will be copied to the initrd stored on the unecrypted</em> boot partition
since the CM4 has no TPM that can be used to further encrypt secrets.
Therefore, this needs to be a unique host key that is only</em> used for remote
unlocking and not shared with other hosts. Once the root drive is unlocked,
the machine will use another host key which is protected by the disk
encryption.</p>
</blockquote>
# This can be stored anywhere on the host, so long as the path is accessible
</span># when the system activation script is run. When doing remote deployments it
</span># _need not_ be accessible by the deploying host
</span>mkdir</span> -p</span> /mnt/persist/etc/ssh
</span>ssh-keygen</span> -t</span> ed25519</span> -N </span>""</span> -f</span> /mnt/persist/etc/ssh/initrd_ssh_host_ed25519_key
</span></code></pre>

Note the generated fingerprint here and add it to ~/.ssh/known_hosts</code> (for the
correct address and port) so you can be (more) sure you are connecting to the
correct host before entering the unlock password.</p>
</blockquote>
Next, pick a port and update the NixOS configuration with the following:</p>
boot</span>.</span>network </span>= </span>{
</span>  </span>enable </span>= </span>true</span>;
</span>  </span>ssh </span>= </span>{
</span>    </span>enable </span>= </span>true</span>;
</span>    </span>port </span>= </span>9999</span>;
</span>    </span>authorizedKeys </span>= </span>[ (</span>throw </span>"add an authorized key here"</span>) ]</span>;
</span>    </span>hostKeys </span>= </span>[
</span>      </span># Note this file lives on the host itself,
</span>      </span># and isn't passed in by the deployer
</span>      </span>"/persist/etc/ssh/initrd_ssh_host_ed25519_key"
</span>    ]</span>;
</span>  }</span>;
</span>}</span>;
</span></code></pre>
Also note that when connecting over SSH during boot you'll need to use the port
defined above (to avoid ambiguity with connecting to the default port (22) after
unlocking) and you will need to set the user as root</code>. These can be configured
with a host alias in ~/.ssh/config</code>:</p>
Host boot-unlock
</span>  Hostname = REPLACE_WITH_IP_FOR_THE_HOST
</span>  User root
</span>  Port 9999
</span></code></pre>
Then, to unlock the host, simply run ssh boot-unlock</code> and execute
cryptsetup-askpass</code> to enter the password. You might get a warning about
"Passphrase is not requested now" after 10 seconds, but this is completely
normal. I've noticed that it takes about 15 seconds before the disk and CPU LEDs
start blinking, and about 45 more seconds until the actual boot sequence starts.</p>

Also note that it is probably worth making sure the PiBox has an ethernet
cable plugged in, as it will not be able to connect to WiFi during boot,
unless the SSID password is also stored (unprotected) on the initrd</p>
</blockquote>
NixOS Installation</h2>
To set a password for the root user which persists across unlocks we'll need to
use a password file:</p>
sudo</span> mkpasswd</span> -m</span> sha-512 </span>></span> /persist/root/passwordfile
</span></code></pre>
And update the configuration with:</p>
users</span>.</span>users</span>.</span>root</span>.</span>passwordFile </span>= </span>"/persist/root/passwordfile"</span>;
</span>fileSystems</span>.</span>"/persist"</span>.</span>neededForBoot </span>= </span>true</span>;
</span></code></pre>
Next we need to get the configuration on to the VM. A quick and dirty solution
is to scp</code> it over:</p>
# In a fresh terminal
</span>scp</span> -P</span> 3333</span> -r </span>~</span>/dotfiles root@localhost:/root </span>\
</span>  -o </span>"UserKnownHostsFile=/dev/null" </span>\
</span>  -o </span>"StrictHostKeyChecking=no"
</span></code></pre>
Then, back in the VM session we can finally install NixOS on the SSD</p>
HOST</span>=</span>asphodel
</span></code></pre>
cd </span>~</span>/dotfiles
</span>nixos-install</span> --root</span> /mnt</span> --flake</span> .#${HOST}</span> --no-channel-copy
</span>
</span># Gracefully detach the disks and exit
</span>umount</span> /mnt/boot
</span>zpool</span> export </span>"$</span>{POOL}</span>"
</span>halt
</span># Afterwards, hit "Ctrl-a", then "x" on the QEMU window to terminate the image
</span></code></pre>
Firmware and Bootloader Installation</h1>
The last few things we need to do are installing the firmware needed to boot the
CM4 (as well as inform the kernel about the fan and LCD peripherals) and the
EFI/bootloader files generated by the NixOS installation.</p>
I've already done the hard part of writing a flake which can prepare the
firmware, as well as enable the fan and display services bundled with the
PiBox OS so checkout the repo</a> for more info!</p>

2023-06-04: Hello from the future! If you are updating from 22.11 to 23.05
you may need to build and copy the firmware before updating (but take a backup
of your existing /boot</code> directory first)! There was a kernel update (from
5.15 to 6.1) between the two releases and the device tree definitions need to
be updated or the new kernel may not recognize the fan and display hardware!</p>
</blockquote>
# Build and prepare the raspberrypi firmware and apply the device tree
</span># overrides to make the PWM fan and LCD discoverable by the kernel.
</span># Results will be found in `./result`
</span>nix</span> build github:ipetkov/nixos-pibox#packages.aarch64-linux.firmware</span> -L
</span>
</span># Prepare mount paths for the disks
</span>sudo</span> mkdir</span> -p</span> /mnt /mnt_ssd
</span>
</span># Reconnect the CM4 as a mass storage device
</span>nix</span> shell nixpkgs#rpiboot</span> --command</span> sudo rpiboot</span> -d </span>~</span>/usbboot/mass-storage-gadget
</span></code></pre>
Double check the disks paths here again:</p>
DISK</span>=</span>"/dev/disk/by-id/ata-..."
</span>SD_DISK</span>=</span>"/dev/sd..."
</span></code></pre>
# Mount the "boot" partition of the SSD
</span>sudo</span> mount </span>"$</span>{DISK}</span>-part1"</span> /mnt_ssd
</span># Mount the CM4's EEPROM
</span>sudo</span> mount </span>"$</span>{SD_DISK}</span>1"</span> /mnt
</span>
</span># Copy the firmware and EFI/bootloader
</span>sudo</span> cp</span> -r</span> ./result/</span>*</span> /mnt_ssd/</span>*</span> /mnt
</span>
</span># Cleanup
</span>sudo</span> umount /mnt
</span>sudo</span> umount /mnt_ssd
</span>sudo</span> rmdir /mnt /mnt_ssd
</span></code></pre>
Et voilà, the installation is complete! Flip the "boot mode" switch back to
normal</code>, put the SSD in, and reassemble the PiBox case. If all else has gone
well you should be able to power on, unlock the disk, and boot and into NixOS.</p>
Happy hacking!</p>

A quick aside on installing the firmware directly instead of using something
like the boot.loader.raspberryPi</code> NixOS module: the plans for NixOS on ARM</a>
highlight that this module largely exists for legacy reasons and should not be
used going forward. Even though the raspberrypi was designed to have this
firmware written to its boot partition, it's more akin to the BIOS of a PC
motherboard: it's not something NixOS should be attempting to manage as
without it the hardware can't even boot</em>, so getting something wrong means we
can't even use a "previous generation". Hence these files should be installed
once and not touched further (unless you have a backup ready and are willing
to physically restore the files if something goes wrong).</p>
</blockquote>
Appendix</h1>
References</h2>

https://github.com/ipetkov/nixos-pibox</a></li>
https://github.com/kubesail/pibox-os/tree/main/lcd-display</a></li>
https://github.com/raspberrypi/usbboot</a></li>
https://carjorvaz.com/posts/nixos-on-raspberry-pi-4-with-uefi-and-zfs</a></li>
https://mth.st/blog/nixos-initrd-ssh</a></li>
https://mgdm.net/weblog/nixos-on-raspberry-pi-4</a></li>
https://www.jeffgeerling.com/blog/2022/how-update-raspberry-pi-compute-module-4-bootloader-eeprom</a></li>
https://discourse.nixos.org/t/planning-for-a-better-nixos-on-arm/15346</a></li>
https://grahamc.com/blog/erase-your-darlings</a></li>
https://jrs-s.net/2018/08/17/zfs-tuning-cheat-sheet</a></li>
https://docs.kubesail.com/guides/pibox/os</a></li>
</ul>
Sample Configuration</h2>
Here's a sample NixOS configuration to get you started with everything we've set
up above. This also includes the systemd services required for controlling the
PWM fan and LCD on the PiBox!</p>
{
</span>  </span>inputs </span>= </span>{
</span>    </span>nixpkgs</span>.</span>url </span>= </span>"nixpkgs/nixos-unstable"</span>;
</span>    </span>nixos-hardware</span>.</span>url </span>= </span>"github:NixOS/nixos-hardware"</span>;
</span>    </span>nixos-pibox </span>= </span>{
</span>      </span>url </span>= </span>"github:ipetkov/nixos-pibox"</span>;
</span>      </span>inputs</span>.</span>nixpkgs</span>.</span>follows </span>= </span>"nixpkgs"</span>;
</span>    }</span>;
</span>  }</span>;
</span>
</span>  </span>outputs </span>= </span>inputs</span>@{ </span>self</span>, </span>nixpkgs</span>, ... </span>}:
</span>    </span>let
</span>      </span># Replace all these placeholders with your actual values!
</span>      </span>deviceBoot </span>= </span>throw </span>"REPLACE ME: /dev/disk/by-uuid/..."</span>;
</span>      </span>deviceCryptKey </span>= </span>throw </span>"REPLACE ME: /dev/disk/by-uuid/..."</span>;
</span>      </span>deviceCryptRoot </span>= </span>throw </span>"REPLACE ME: /dev/disk/by-uuid/..."</span>;
</span>      </span>deviceCryptSwap </span>= </span>throw </span>"REPLACE ME: /dev/disk/by-uuid/..."</span>;
</span>      </span>hostId </span>= </span>throw </span>"REPLACE ME: deadbeef"</span>;
</span>      </span>hostName </span>= </span>throw </span>"REPLACE ME"</span>;
</span>      </span>userSshKey </span>= </span>throw </span>"REPLACE ME: ssh-ed25519 ..."</span>;
</span>      </span>user </span>= </span>throw </span>"REPLACE ME"</span>;
</span>      </span>zpool </span>= </span>throw </span>"REPLACE ME"</span>;
</span>    </span>in
</span>    {
</span>      </span>nixosConfigurations</span>.</span>${</span>hostName</span>} </span>= </span>nixpkgs</span>.</span>lib</span>.</span>nixosSystem </span>{
</span>        </span>modules </span>= </span>[
</span>          ({ </span>config</span>, </span>lib</span>, </span>pkgs</span>, ... </span>}: {
</span>            </span>imports </span>= </span>[
</span>              </span>inputs</span>.</span>nixos-hardware</span>.</span>nixosModules</span>.</span>raspberry-pi-4
</span>              </span>inputs</span>.</span>nixos-pibox</span>.</span>nixosModules</span>.</span>default
</span>            ]</span>;
</span>
</span>            </span>nixpkgs</span>.</span>overlays </span>= </span>[
</span>              </span>inputs</span>.</span>nixos-pibox</span>.</span>overlays</span>.</span>default
</span>            ]</span>;
</span>
</span>            </span>boot </span>= </span>{
</span>              </span>extraModulePackages </span>= </span>[ ]</span>;
</span>
</span>              </span># </span>!!!</span> cryptkey must be done first, and the list seems to be
</span>              </span># alphabetically sorted, so take care that cryptroot / cryptswap,
</span>              </span># whatever you name them, come after cryptkey.
</span>              </span>initrd </span>= </span>{
</span>                </span>luks</span>.</span>devices </span>= </span>{
</span>                  </span>cryptkey </span>= </span>{
</span>                    </span>device </span>= </span>deviceCryptKey</span>;
</span>                  }</span>;
</span>
</span>                  </span>cryptroot </span>= </span>{
</span>                    </span>allowDiscards </span>= </span>true</span>;
</span>                    </span>device </span>= </span>deviceCryptRoot</span>;
</span>                    </span>keyFile </span>= </span>"/dev/mapper/cryptkey"</span>;
</span>                    </span>keyFileSize </span>= </span>8192</span>;
</span>                  }</span>;
</span>
</span>                  </span>cryptswap </span>= </span>{
</span>                    </span>allowDiscards </span>= </span>true</span>;
</span>                    </span>device </span>= </span>deviceCryptSwap</span>;
</span>                    </span>keyFile </span>= </span>"/dev/mapper/cryptkey"</span>;
</span>                    </span>keyFileSize </span>= </span>8192</span>;
</span>                  }</span>;
</span>                }</span>;
</span>
</span>                </span>postDeviceCommands </span>= </span>lib</span>.</span>mkAfter </span>''
</span>                  cryptsetup close cryptkey
</span>                  zfs rollback -r </span>${</span>zpool</span>}</span>/local/root@blank && echo blanked out root
</span>                ''</span>;
</span>
</span>                </span># Support remote unlock. Run `cryptsetup-askpass` to unlock
</span>                </span>network </span>= </span>{
</span>                  </span>enable </span>= </span>true</span>;
</span>                  </span>ssh </span>= </span>{
</span>                    </span>enable </span>= </span>true</span>;
</span>                    </span>authorizedKeys </span>= </span>config</span>.</span>users</span>.</span>users</span>.</span>${</span>user</span>}</span>.</span>openssh</span>.</span>authorizedKeys</span>.</span>keys</span>;
</span>                    </span>port </span>= </span>9999</span>;
</span>                    </span>hostKeys </span>= </span>[
</span>                      </span># Note this file lives on the host itself, and isn't passed in by the deployer
</span>                      </span>"/persist/etc/ssh/initrd_ssh_host_ed25519_key"
</span>                    ]</span>;
</span>                  }</span>;
</span>                }</span>;
</span>              }</span>;
</span>
</span>              </span>loader </span>= </span>{
</span>                </span>efi</span>.</span>canTouchEfiVariables </span>= </span>true</span>;
</span>                </span>generic-extlinux-compatible</span>.</span>enable </span>= </span>false</span>;
</span>                </span>systemd-boot</span>.</span>enable </span>= </span>true</span>;
</span>                </span>timeout </span>= </span>10</span>; </span># seconds
</span>              }</span>;
</span>
</span>              </span>kernelParams </span>= </span>[
</span>                </span>"8250.nr_uarts=1"
</span>                </span>"console=ttyAMA0,115200"
</span>                </span>"console=tty1"
</span>              ]</span>;
</span>
</span>              </span>supportedFilesystems </span>= </span>[ </span>"zfs" </span>]</span>;
</span>            }</span>;
</span>
</span>            </span>fileSystems </span>= </span>{
</span>              </span>"/" </span>= </span>{
</span>                </span>device </span>= </span>"</span>${</span>zpool</span>}</span>/local/root"</span>;
</span>                </span>fsType </span>= </span>"zfs"</span>;
</span>                </span>options </span>= </span>[ </span>"zfsutil" </span>]</span>;
</span>              }</span>;
</span>
</span>              </span>"/boot" </span>= </span>{
</span>                </span>device </span>= </span>deviceBoot</span>;
</span>                </span>fsType </span>= </span>"vfat"</span>;
</span>                </span>options </span>= </span>[ </span>"noatime" </span>]</span>;
</span>              }</span>;
</span>
</span>              </span>"/home/</span>${</span>user</span>}</span>" </span>= </span>{
</span>                </span>device </span>= </span>"</span>${</span>zpool</span>}</span>/persist/user/</span>${</span>user</span>}</span>"</span>;
</span>                </span>fsType </span>= </span>"zfs"</span>;
</span>              }</span>;
</span>
</span>              </span>"/nix" </span>= </span>{
</span>                </span>device </span>= </span>"</span>${</span>zpool</span>}</span>/local/nix"</span>;
</span>                </span>fsType </span>= </span>"zfs"</span>;
</span>              }</span>;
</span>
</span>              </span>"/persist" </span>= </span>{
</span>                </span>device </span>= </span>"</span>${</span>zpool</span>}</span>/persist/system"</span>;
</span>                </span>fsType </span>= </span>"zfs"</span>;
</span>                </span>neededForBoot </span>= </span>true</span>;
</span>              }</span>;
</span>
</span>              </span>"/var/lib" </span>= </span>{
</span>                </span>device </span>= </span>"</span>${</span>zpool</span>}</span>/persist/lib"</span>;
</span>                </span>fsType </span>= </span>"zfs"</span>;
</span>              }</span>;
</span>
</span>              </span>"/var/log/journal" </span>= </span>{
</span>                </span>device </span>= </span>"</span>${</span>zpool</span>}</span>/persist/journal"</span>;
</span>                </span>fsType </span>= </span>"zfs"</span>;
</span>                </span>neededForBoot </span>= </span>true</span>;
</span>              }</span>;
</span>            }</span>;
</span>
</span>            </span>swapDevices </span>= </span>[
</span>              {
</span>                </span>device </span>= </span>"/dev/mapper/cryptswap"</span>;
</span>              }
</span>            ]</span>;
</span>
</span>            </span>powerManagement</span>.</span>cpuFreqGovernor </span>= </span>lib</span>.</span>mkDefault </span>"ondemand"</span>;
</span>
</span>            </span>networking </span>= </span>{
</span>              </span>inherit </span>hostId hostName</span>;
</span>              </span>useDHCP </span>= </span>false</span>;
</span>              </span>interfaces</span>.</span>eth0</span>.</span>useDHCP </span>= </span>true</span>;
</span>            }</span>;
</span>
</span>            </span>services </span>= </span>{
</span>              </span>openssh </span>= </span>{
</span>                </span>enable </span>= </span>true</span>;
</span>                </span>hostKeys </span>= </span>[
</span>                  {
</span>                    </span>path </span>= </span>"/persist/etc/ssh/ssh_host_ed25519_key"</span>;
</span>                    </span>type </span>= </span>"ed25519"</span>;
</span>                  }
</span>                ]</span>;
</span>              }</span>;
</span>              </span>piboxPwmFan</span>.</span>enable </span>= </span>true</span>;
</span>              </span>piboxFramebuffer</span>.</span>enable </span>= </span>true</span>;
</span>              </span>zfs </span>= </span>{
</span>                </span>autoScrub </span>= </span>{
</span>                  </span>enable </span>= </span>true</span>;
</span>                  </span>interval </span>= </span>"monthly"</span>;
</span>                }</span>;
</span>                </span>autoSnapshot</span>.</span>enable </span>= </span>true</span>;
</span>                </span>trim</span>.</span>enable </span>= </span>true</span>;
</span>              }</span>;
</span>            }</span>;
</span>
</span>            </span>users</span>.</span>users</span>.</span>root</span>.</span>passwordFile </span>= </span>"/persist/root/passwordfile"</span>;
</span>            </span>users</span>.</span>users</span>.</span>${</span>user</span>} </span>= </span>{
</span>              </span>isNormalUser </span>= </span>true</span>;
</span>              </span>home </span>= </span>"/home/</span>${</span>user</span>}</span>"</span>;
</span>              </span>openssh</span>.</span>authorizedKeys</span>.</span>keys </span>= </span>[
</span>                </span>userSshKey
</span>              ]</span>;
</span>            }</span>;
</span>
</span>            </span># Other files to persist, e.g. NetworkManager
</span>            </span># environment.etc."NetworkManager/system-connections" = {
</span>            </span>#   source = "/persist/etc/NetworkManager/system-connections/";
</span>            </span># };
</span>          })
</span>        ]</span>;
</span>      }</span>;
</span>    }</span>;
</span>}
</span></code></pre>

Transparently Exposing Services Over Tailscale and the LAN

2023-02-07T00:00:00+00:00

Suppose we have control of our own domain and a set of services we want to share with (only) our friends and family. Here's how we can make them accessible over both Tailscale or when connected to the same physical network while using the exact same domain in each case. 

I personally love Tailscale and it truly makes securely connecting devices incredibly easy. That said, it's not always practical to expect all friends and family to have Tailscale set up and always connected, so there's definitely a value to being able to access services on the local network directly. </blockquote>
Prerequisites</h1>
Things we'll need and their example values. Remember to substitute these with your own!

A domain we control: example.com</code></li>
A subdomain we would like to use for the service: rainbows.example.com</code></li>
A tailnet name: cat-crocodile.ts.net</code></li>
A host (on the local network) running the rainbows</code> service

A machine name assigned to this host in Tailscale: fido</code> (meaning the host can be reached via fido.cat-crocodile.ts.net</code> over Tailscale)</li>
A stable local IP address for fido</code>: likely something like 192.168.0.42</code></li> </ul> </li>
A(n always reachable) host to run a DNS server: dennis</code>

A stable local IP address for dennis</code>: likely something like 192.168.0.42</code></li>
The local network's DHCP server (i.e. your router) should assign this IP as the primary DNS server for the rest of the local network</li> </ul> </li>
MagicDNS enabled on the tailnet</li>
The appropriate Tailscale ACLs such that dennis</code>' port 53 is accessible (and whatever other ports the rainbows</code> service will use on fido</code>, e.g. 443 for HTTPS)</li>
A "base" DNS server: This can be a Pi-hole</a>, though if it is going to be running on dennis</code> you will need to configure it to listen to any port besides 53 (the default DNS port).</li> Or this can be any public DNS provider like 9.9.9.9</code></li> </ul> </li> </ol> The Approach</h1> First we need to create a CNAME record pointing rainbows.example.com</code> to fido.cat-crocodile.ts.net</code>; this should be done on the nameserver for example.com</code>, likely your domain registrar. Next, we could configure our local DNS server (i.e. our Pi-hole instance running on dennis</code>) to hard-code an address record pointing fido.cat-crocodile.ts.net</code> to 192.168.0.42</code> except this will not work if dennis</code> is set as the global nameserver for your tailnet: if you ever leave the local network dennis</code> will recursively resolve rainbows.example.net</code> to the "wrong" address. Instead dennis</code> needs to run a recursive DNS server which can tailor results based on the requester's address. Namely, if a request comes from a local address for rainbows.example.com</code> or fido.cat-crocodile.ts.net</code> it would need to respond with 192.168.0.42</code> directly; otherwise it should use MagicDNS to resolve to the Tailscale IP. BIND's named</code> fulfills our use-case perfectly: it can present different views of DNS records based on the requester's address (among other things). Configuration</h1> Below is the minimal configuration necessary to get things working with named</code>. Remember to change the placeholder values with your own, but otherwise feel free to tailor it further to your needs: acl tailnet { 100.64.0.0/10; }; acl mynet { localnets; # bind builtin, automatically represents all interfaces on the device tailnet; # also include tailscale's range (which is the Carrier Grade NAT range) }; options { directory "/run/named"; querylog no; # Change to `yes` to debug queries listen-on { any; }; listen-on-v6 { any; }; # Keep this set to `only` if using Pi-hole. Setting it to `first` # will result in `named` trying to resolve results on its own which # would defeat Pi-hole's filtering forward only; # Assuming there is a Pi-hole instance running on this host on port 9053, # forward requests for any zones not configured here. If you aren't using # Pi-hole this can be replaced with any other public DNS (e.g. `9.9.9.9`). forwarders { 127.0.0.1 port 9053; }; # Allow recursive resolution for any LAN/Tailscale queries recursion yes; allow-query { mynet; }; allow-recursion { mynet; }; allow-query-cache { mynet; }; }; # NB: view order is significant here: views are considered one by one and only # the first one to match is used. Specifically, the `catchall` view will # effectively be used for non-tailnet clients as they would have otherwise been # matched by the `tsnet` view view tsnet { match-clients { tailnet; }; # We forward any queries for our tailnet directly to the MagicDNS server # since it should have the results for any hosts on the tailnet (which the # upstream DNS likely won't). # NB: be _very_ careful that the tailnet name does not change here # and also be careful to ensure that this host was initialized with # `tailscale up --accept-dns=false` otherwise we could end up recursively # ourselves if the MagicDNS forwards any non-tailnet queries back to us zone "cat-crocodile.ts.net" IN { type forward; forward only; forwarders { 100.100.100.100; }; # Tailscale's MagicDNS IP }; }; # The "catchall" view which will match all clients. Remember # that the earlier view will filter out any tailnet clients # maning this view represents clients directly on the local network view catchall { match-clients { any; }; zone "cat-crocodile.ts.net" IN { type primary; file "/path/to/file/for/lan/zone/cat-crocodile.ts.net"; }; }; </code></pre> Where the contents of /path/to/file/for/lan/zone/cat-crocodile.ts.net</code> are: $TTL 2d $ORIGIN cat-crocodile.ts.net. @ IN SOA dns.example.com. hostmaster.example.com. ( 1 ; serial number 12h ; refresh 15m ; update retry 3w ; expiry 2h ; minimum ) @ IN NS dns.example.com. @ IN A 192.168.0.42 </code></pre>

Crane Support for Alternative Registries and Git Dependencies

2022-02-13T00:00:00+00:00

Since the initial release of Crane</a>, I've been busy hacking on adding support for building projects which may pull in dependencies from alternative registries as well as git repositories. I wanted to share how it works, so let's dive right in!

Alternative Registries</h1>
Although crates.io is the default registry for the majority of (public) Rust projects, cargo does allow for configuring any other crate registry/index to be used for dependencies. So far, the main use-case for alternative registries seems to be for privately publishing crates (e.g. on an enterprise network) judging by the lack of other public registries present in the ecosystem (well, except for Alexandrie</a> which was very useful for my testing!).
The workflow for vendoring crates is pretty much the same, regardless if the crates come from crates.io or some other index:

We check the project's `.cargo/config.toml</code> file (if it exists) to see what registries are defined (specifically their unique index URL and the name used to link them to dependency definitions in Cargo.toml</code>)</li>`
`We then crawl the Cargo.lock</code> file to find out the name, version, checksum, and (registry) source (i.e. the index URL) for each dependency package</li>`
`Using this information we can construct the download URL for the crate and pull down the source to the Nix store. We'll come back to this step in a bit.</li>`
`The sources are then grouped by the registry they come from, and are unpacked in a format that cargo can understand`The sources are basically tarballs which are extracted into directories named after the crate's name and version, along with some checksum metadata used by cargo to validate the sources are as expected.</li> </ul> </li> Finally, we write some configuration which can instruct cargo to look at the directories we've prepared when building the project, instead of trying to access the network itself (which would fail if running inside of a sandboxed build).</li> </ol> So how does cargo figure out what the download URL is for a particular crate? The specification</a> requires that the index contain a config.json</code> file at its root which defines the endpoint and path that should be used for downloading crate sources. This definition can also contain placeholders (like the crate's name and version among other things) which need to be substituted to create the final URL used for fetching the source. How does Crane figure out the download configuration for each registry? Unfortunately, we have to tell it. We can take a look at a full example</a> in context, but in summary, we have two options to take: The first option is to straight up copy the configuration out of the index's config.json</code> file and tell Crane about it. This is the simplest and most lightweight option we can employ, especially if the download endpoint and path virtually never change.craneLibOrig = crane.lib.${system}; craneLib = craneLibOrig.appendCrateRegistries [ (craneLibOrig.registryFromDownloadUrl { indexUrl = "https://github.com/Hirevo/alexandrie-index"; dl = "https://crates.polomack.eu/api/v1/crates/{crate}/{version}/download"; }) ]; </code></pre> </li> The second option is to tell Crane about a particular revision of the index and let it figure out the download template on its own. This option has the benefit of having a single canonical source of truth (without copying URLs around by hand), and, if the download endpoint or path changes from time to time, it can easily be remedied by updating the index snapshot to a newer revision (which is especially nice if it needs to be automated). The cost of this option, however, is needing to check out the entire index at that revision and put a copy of it in the store before we can evaluate the derivation. Note that this revision only needs to be updated if the config.json</code> file changes, so it is safe to pin to a version for as long as that takes.craneLibOrig = crane.lib.${system}; craneLib = craneLibOrig.appendCrateRegistries [ (craneLibOrig.registryFromGitIndex { indexUrl = "https://github.com/Hirevo/alexandrie-index"; rev = "90df25daf291d402d1ded8c32c23d5e1498c6725"; }) ]; </code></pre> </li> </ol> Technically, we could try to automate this away completely by always fetching the latest version of the index and looking at the configuration before downloading any crate sources. Even ignoring issues with requiring impure evaluations to make this work, this would make for a really bad default from a performance standpoint. You see, cargo keeps a checked out version of every registry index that has ever been used (usually somewhere in your home directory). It used during dependency resolution (such as what are the latest published/yanked version, etc.) and incrementally fetched as needed. When we build with Nix, however, we don't care about re-resolving dependencies since the Cargo.lock file already pins everything into place; checking out the entire index to the store, just to peek at a small configuration file and throw the results away seems wasteful. Even the store is regularly cleaned out, we would still need to fetch the index again and again any time the derivation is evaluated. That's a lot of wasted bandwidth, especially as the index accrues newly published crates. And lets not forget that Nix doesn't keep the repository around such that it can be incrementally fetched, either. I really wish this paper cut experience of having to manually specify alternative registries can be improved in the future, but for now, it seems like the best choice available. Git Dependencies</h1> Vendoring crate sources from git repositories is roughly the same as vendoring from registries: We crawl the Cargo.lock</code> file to look for any packages originating from git sources, and find out the repository's URL as well as the revision that has been locked</li> We then pass the git URL and revision to Nix which will pull down the source for us Note: this does not pull down the entire repository, we only get a checkout of the revision.</li> Fetching a git repository is not reproducible, as any new commit or branch would add new data which would result in invalidating all of our build caches</li> </ul> </li> The one main difference between a git dependency and a registry tarball is that the tarball always contains a single crate. The git repository could contain an entire workspace of crates. To handle this, we crawl the source looking for Cargo.toml</code> files as a proxy for identifying what crates are present. Looking for Cargo.toml</code> is a simple heuristic which goes a long way. Although there can be a false-positive (we vendor a crate not part of the actual workspace), we cannot have a false-negative (accidentally ignore a real crate) since you cannot define a crate without a Cargo.toml</code> file.</li> Ultimately, cargo will ignore the crates it does not care about which gives us some flexibility here.</li> Why not ask cargo metadata</code> to tell us about the workspace members? Doing so will make cargo try to pull down the sources from the network to tell us about them. Since this whole exercise is to pull the sources down for cargo, we need to avoid this chicken-egg problem somehow.</li> </ul> </li> Why not look at the [workspace]</code> definition in the Cargo.toml</code> file if it exists? Cargo supports glob patterns both for including and excluding members. Re-implementing this logic ourselves is way too overkill when a simple search can get us where we need.</li> </ul> </li> </ul> </li> We then transform the crates into the same vendor directory structure as for registries (i.e. each crate goes into its own sub-directory using the crate's name and version).</li> And finally, we generate some configuration that can instruct cargo to look at these vendored directories as is appropriate.</li> </ol> One interesting thing to note that whereas the "unique unit of vendoring" for a registry is the index itself, for git dependencies it is the specific revision of a particular repository. In other words, all crates coming from the same registry/index are vendored in one directory which is registered as a single source replacement with cargo. All crates coming from the same git repository and revision are also vendored in one directory and registered as a single source replacement with cargo, but more git revisions in the dependency closure will result in more cargo sources behind the scenes. There's several benefits to this approach: First and foremost we don't have to care (or worry) about whether there is a name/version collision between crates coming from a registry or from a git repository. Each will get their own unique "vendor space" for which we know is impossible to have collisions!</li> Even if we do get a collision, we can avoid the risks of having to establish which source would take precedence! We simply make the sources available to cargo, and it is free to use (or ignore them) based on how the project authors' have dictated via the Cargo.toml</code> and Cargo.lock</code> files To illustrate this point a bit further, consider the following: you may have a workspace which may contain an auxillary crate used for running tests. Perhaps this crate pins to some ancient git revision of a dependency crate to perform some compatibility testing.</li> We wouldn't want this dependency to get selected when building our production binaries as we would likely want to use the latest and greatest version of that dependency as pinned by the Cargo.lock</code> file</li> At the same time, we wouldn't want to ignore the pinned git version as that could break the tests that we thought were running</li> All in all, this means that cargo will behave the same when running under Nix as it does outside of it, without any unexpected surprises!</li> </ul> </li> </ol> Oh and the other cool thing about this implementation is there is nothing to configure! Everything should Just Work™ out of the box :) Feedback</h1> As always, if something doesn't seem quite right or you have any feedback, feel free to let me know on the project repo</a>!

Introducing Crane: Composable and Cacheable Builds with Cargo and Nix

2022-01-22T00:00:00+00:00

I'm pleased to announce the initial release of Crane</a>: a Nix library for building cargo projects!

In a nutshell it offers:

Source fetching: automatically done using a Cargo.lock file</li>

Incremental: build your workspace dependencies just once, then quickly lint, build, and test changes to your project without slowing down</li>

Composable configuration: split builds and tests into granular steps. Gate CI without burdening downstream consumers building from source.</li> </ul> Acknowledgements</h1>
I want to take a moment to acknowledge and give credit to the impressive work behind Naersk</a>. It has been a huge source of inspiration, especially since it absolutely nails a number of features like:

automatic dependency handling with out needing to mess with SHAs yourself</li>
dependency artifacts can be built, cached, and reused independently of the project's source, making it a great fit for running in CI</li>
the defaults "just work" out of the box, regardless if a project is building applications, libraries, or an entire workspace of crates</li> </ul>
Motivation</h2>
My biggest motivation for writing Crane was wanting to use an API that allows for build configurations which feel intuitive to write and easy to reconfigure without sacrificing the performance of cacheable artifacts.
I find Naersk's configuration options work best when you are intimately familiar with its internals. And if you aren't, even a goal like "run clippy on the project" can leave you having to face evaluation errors, build errors, accidental cache invalidation, or the acceptance that a fraction of your build dependencies just have to be built twice...
Philosophy</h1>
Crane is designed around the idea of composing cargo invocations such that they can take advantage of the artifacts generated in previous invocations. This allows for both flexible configurations and great caching (à la Cachix) in CI and local development builds.
Here's how it works at a high level: when a cargo workspace is built its source is first transformed such that only the dependencies listed by the Cargo.toml</code> and Cargo.lock</code> files are built, and none of the crate's real source is included. This allows cargo to build all dependency crates and prevents Nix from invalidating the derivation whenever the source files are updated. Then, a second derivation is built, this time using the real source files, which also imports the cargo artifacts generated in the first step.
This pattern can be used with any arbitrary sequence of commands, regardless of whether those commands are running additional lints, performing code coverage analysis, or even generating types from a model schema. Let's take a look at two examples at how very similar configurations can give us very different behavior!
Example One</h2> Suppose we are developing a crate and want to run our CI assurance checks via nix flake check</code>. Perhaps we want the CI gate to be very strict and block any changes which raise warnings when run with cargo clippy</code>. Oh, and we want to enforce some code coverage too! Except we do not want to push our strict guidelines on any downstream consumers who may want to build our crate. Suppose they need to build the crate with a different compiler version (for one reason or another) which comes with a new lint whose warnings we have not yet addressed. We don't want to make their life harder, so we want to make sure we do not run cargo clippy</code> as part of the crate's actual derivation, but at the same time, we don't want to have to rebuild dependencies from scratch. Here's how we can set up our flake to achieve our goals:
{ inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable"; crane.url = "github:ipetkov/crane"; crane.inputs.nixpkgs.follows = "nixpkgs"; flake-utils.url = "github:numtide/flake-utils"; flake-utils.inputs.nixpkgs.follows = "nixpkgs"; }; outputs = { self, nixpkgs, crane, flake-utils, ... }: flake-utils.lib.eachDefaultSystem (system: let pkgs = import nixpkgs { inherit system; }; craneLib = crane.lib.${system}; src = ./.; # Build *just* the cargo dependencies, so we can reuse # all of that work (e.g. via cachix) when running in CI cargoArtifacts = craneLib.buildDepsOnly { inherit src; }; # Run clippy (and deny all warnings) on the crate source, # resuing the dependency artifacts (e.g. from build scripts or # proc-macros) from above. # # Note that this is done as a separate derivation so it # does not impact building just the crate by itself. my-crate-clippy = craneLib.cargoClippy { inherit cargoArtifacts src; cargoClippyExtraArgs = "-- --deny warnings"; }; # Build the actual crate itself, reusing the dependency # artifacts from above. my-crate = craneLib.buildPackage { inherit cargoArtifacts src; }; # Also run the crate tests under cargo-tarpaulin so that we can keep # track of code coverage my-crate-coverage = craneLib.cargoTarpaulin { inherit cargoArtifacts src; }; in { defaultPackage = my-crate; checks = { inherit # Build the crate as part of `nix flake check` for convenience my-crate my-crate-clippy my-crate-coverage; }; }); } </code></pre> When we run nix flake check</code> the following will happen:
The sources for any dependency crates will be fetched</li> They will be built without our crate's code and the artifacts propagated</li>Our crate, the clippy checks, and code coverage collection will be built, each reusing the same set of artifacts from the initial source-free build. If enough cores are available to Nix it may build all three derivations completely in parallel, or schedule them in some arbitrary order.</li> </ol> Splitting up our builds like this also gives us the benefit of granular control over what is rebuilt. Suppose we change our mind and decide to adjust the clippy flags (e.g. to allow certain lints or forbid others). Doing so will only rebuild the clippy derivation, without having to rebuild and rerun any of our other tests! The arrows show which way the results "flow", from sources (represented in boxes) to intermediate and final derivations (represented as circles). </figcaption> </figure> Example Two</h2> Let's take an alternative approach to the example above. Suppose instead that we care more about not wasting any resources building certain tests (even if they would succeed!) if another particular test fails. Perhaps binary substitutes are readily available so that we do not mind if anyone building from source is bound by our rules, and we can be sure that all tests have passed as part of the build. { inputs = { nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable"; crane.url = "github:ipetkov/crane"; crane.inputs.nixpkgs.follows = "nixpkgs"; flake-utils.url = "github:numtide/flake-utils"; flake-utils.inputs.nixpkgs.follows = "nixpkgs"; }; outputs = { self, nixpkgs, crane, flake-utils, ... }: flake-utils.lib.eachDefaultSystem (system: let pkgs = import nixpkgs { inherit system; }; craneLib = crane.lib.${system}; src = ./.; # Build *just* the cargo dependencies, so we can reuse # all of that work (e.g. via cachix) when running in CI cargoArtifacts = craneLib.buildDepsOnly { inherit src; }; # First, run clippy (and deny all warnings) on the crate source. my-crate-clippy = craneLib.cargoClippy { inherit cargoArtifacts src; cargoClippyExtraArgs = "-- --deny warnings"; }; # Next, we want to run the tests and collect code-coverage, _but only if # the clippy checks pass_ so we do not waste any extra cycles. my-crate-coverage = craneLib.cargoTarpaulin { inherit src; cargoArtifacts = my-crate-clippy; }; # Build the actual crate itself, _but only if the previous tests pass_. my-crate = craneLib.buildPackage { cargoArtifacts = my-crate-coverage; inherit src; }; in { defaultPackage = my-crate; checks = { inherit # Build the crate as part of `nix flake check` for convenience my-crate my-crate-coverage; }; }); } </code></pre> When we run nix flake check</code> the following will happen: The sources for any dependency crates will be fetched</li> They will be built without our crate's code and the artifacts propagated</li> Next the clippy checks will run, reusing the dependency artifacts above.</li> Next the code coverage tests will run, reusing the artifacts from the clippy run</li> Finally the actual crate itself is built</li> </ol> In this case we lose the ability to build derivations independently, but we gain the ability to enforce a strict build order. However, we can easily change our mind, which would be much more difficult if we had written everything as one giant derivation. The arrows show which way the results "flow", from sources (represented in boxes) to intermediate and final derivations (represented as circles). </figcaption> </figure> Try it out</h1> I wanted to get an initial version out in the open, but there is, of course, more work to be done. Support for git dependencies and private registries is missing, but something I'd like to add in the near future.</del> In the meantime, feel free to check it out for yourself and kick the tires. The getting started</a> guide, examples</a>, and API</a> documentation will be useful resources. I'd love to know what you think! Edit 2022-01-30: alternate cargo registry support has landed in Crane v0.2.0! Edit 2022-02-11: git dependency support has landed in Crane v0.3.0!

Installing NixOS and ZFS on my Desktop

2021-01-24T00:00:00+00:00

This write up was originally published within my dotfiles</a>.

NixOS is a great tool for declaratively managing system configurations (namely what packages and other config files are available), but there are still a number of imperative steps to run when setting up a brand new machine before installation. (Okay, there are some projects which seek to automate this process as well, but unless you are constantly (re-)provisioning new machines, it is probably overkill...). There's a number of available tutorials and guides on how to prepare a machine for NixOS installation, but I found all of them to be incredibly basic (e.g. set up single ext4 partition and move on), or didn't quite fit the requirements I was imposing.

Since this was also my first "real" Linux install, there were a number of things I was unsure of and had to research on my own to figure out whether it was relevant or what configurations to choose. Here I document all the steps I went through, along with (an attempt!) to capture my assumptions and deliberate decisions so that whether I got something wrong, it stops being correct in the future, or you simply disagree with my choices, it becomes easy to spot where to stray and where to follow this guide.

Acknowledgements</h2> Huge thanks to Graham Christensen</a> whose blog posts were my main inspiration and guide for installation:

NixOS on a Dell 9560</a></li>
ZFS Datasets for NixOS</a></li>
Erase your darlings</a></li> </ul>
Assumptions and Requirements</h2>
Below is a quick summary of the assumptions and requirements I had for the system to provide some historical context and keep the rest of the guide focused on running each step.
The installation was done in early December of 2020 on a brand new 1 TiB SSD. The disk has four partitions, and here is the final result:

A 1 GiB unencrypted boot partition (more on this later). I chose 1 GiB because I've been bitten by having too small of a boot partition in the past, and it will be a headache to try to resize the partition if it turns out too small. Most people recommend using 300-500 MiB, but since I have plenty of storage to spare I decided to use 1 GiB and forget about it (note that every NixOS generation adds links in this partition, so having too many generations laying around can fill it up as well).</li>
A 32 MiB LUKS encrypted partition which contains the key for the remaining partitions. I forget exactly why I went with 32 MiB exactly, but I wanted to make this partition large enough to handle any LUKS upgrades or extra key configurations, and I have space to spare (I think the LUKS2 max header size is 4 MiB, and most people recommend having a partition about this big plus some space for the actual key).</li>
A 32 GiB swap partition because this machine has 16 GiB of RAM and I left some headroom in case I upgrade it. I don't plan on enabling swap (to avoid wearing out my SSD), but I made the partition anyway in case I change my mind.</li>
The remainder of the disk is managed by ZFS split into the following datasets:

`/local</code> - dataset for mounting /nix/store</code>. It is not snapshotted since the nix store can be trivially repopulated/rebuilt if the data is lost or corrupted</li>`
`/reserved</code> - A 100 GiB reserved partition to act as an over-provisioning guard and preserve the SSD performance (SSDs avoid wearing out individual blocks by moving writes around. But if the drive fills up, the speed and health of the drive will decrease. By never mounting this dataset, and asking ZFS to ensure there is always 100 GiB available for it, I'm effecitvely capping the disk at 90%).</li>`
`/system</code> - dataset for mounting /root</code> and /var</code>. This dataset is regularly snapshotted so I can rollback in case something catastrophic happens. I have not yet decided toerase my darlings</a> but if I do I would move my /root</code> mount...</li>`
/user</code> - dataset for storing user home directories, regularly snapshotted</li> </ul> </li> </ol> Why ZFS?</h3> I had generally heard good things about it, namely that it's a stable file system implementation which supports efficient snapshots, rollbacks, and data exports. I briefly looked into btrfs which also supports very similar features, but the NixOS support (at the time) was lacking, and since I had not previously used neither btrfs nor zfs, I went with the latter since I expected the experience to be smoother. What about SSD over-provisioning?</h3> SSDs are made up of flash storage which supports a (large, but) finite number of write operations before the medium begins to degrade. The SSD controller performs wear leveling by effectively writing new data in a new location by transparently remapping the block identifier to the new location. This requires having some free space on the disk to "move" the blocks around. If the disk fills up, the controller will be forced to do subsequent writes in the same spot. Over-provisioning is a name that storage vendors give to the concept of reserving some storage capacity to avoid accidentally filling it up and degrading performance. Some vendors state that their modern products automatically achieve this in their firmware without any manual intervention (maybe the drive itself has more storage than advertised?). Other vendors peddle special tools like Samsung Data Magician (which simply creates an empty partition) to achieve the task. Since I have lots of storage to spare on my 1 TiB drive, I decided to over-provision 10% of its capacity by creating a reserved ZFS pool which I will never mount. I can easily remove or shrink that reservation if needed, so this seemed like a sensible choice. Why an unencrypted boot partition?</h3> Ultimately, you have to trust some piece of software somewhere to take your keyboard input and unlock the disk without leaking the key somehow. An unencrypted boot partition means someone who gets access to the disk can put a compromised boot loader that can steal the key. Using an encrypted boot partition avoids this risk, but that means that the UEFI implementation needs to do the decryption, and someone who can access it could flash a compromised implementation which also steals the key. A solution to that can be to use Secure boot/Trusted boot but now we have to trust that the hardware itself isn't compromised with some other back-door... It's turtles all the way down. My threat model does not include someone physically accessing my machine, so an unencrypted boot partition works fine for me. Setting up a trusted boot sequence sounds interesting, but it's a project for another time. Why LUKS and not native-ZFS encryption?</h3> I chose to go with using LUKS to encrypt the entire disk and run ZFS from within it. LUKS has been around for a while and there is plenty of tooling/documentation/guides around it, so it seemed like a safe approach. ZFS apparently supports natively encrypting the disk, which avoids some double indirection when trimming SSD blocks (and then having the decryption mapper propagate those to the device). There are some potential security concerns (like leaking dataset names/sizes and dedup tables), but none of them are within my threat model. What really convinced me against using native-ZFS encryption was the impression that the feature was somewhat newer, and I didn't want to risk having it eat my laundry... Why use the allowDiscards</code> flag with LUKS?</h3> The allowDiscards</code> option instructs the mapper to propagate trim commands issued by the underlying filesystem, which allows the SSD to better perform wear leveling. This option is disabled by default since there are some theoretical attack vectors from having it enabled (namely leaking which blocks are trimmed, an some potential oracle attacks if the attacker can influence what data is written to the disk). Since this doesn't fit my threat model (namely someone gaining physical access to my disk) and since I am more worried about maintaining my SSD performance, I decided to enable this option. Installation</h2> On to the good stuff, actual installation steps start here! Note all commands should be run as root. Installer preparation</h3> Download an installer</a> and burn it to a bootable USB drive. Worth noting that if you already have an existing nix install you can create your own custom installer (e.g. if you're a power user or you need specific tools available during installation), but the base installer should cover all the bases.</li> If there is an existing Windows installation on this machine (even if it is on an entirely separate drive), consider turning off the "Enable fast startup" option</a> and rebooting before continuing. I had to do this to get my bluetooth/wifi (AX200) adapter to work (yay Windows hacks to gain speedup!).</li> (Optional) if you have other drives in the machine, consider unplugging them to avoid accidentally overwriting the wrong disk due to a typo...</li> Plug in the USB, reboot the computer, hit the appropriate keys during the BIOS, and boot into the USB</li> </ol> Partitioning the Disk</h3> If a graphical installer image was used, it should drop us in a desktop environment which should set up some basic stuff like networking. The rest of the commands all need root privileges, so open a terminal and switch to root to avoid having to prefix everything with sudo</code>. sudo su </code></pre> </li> Next, we need to figure out which disk we want to use. ls /dev </code></pre> </li> In my case this is my second NVMe in this machine so I will be using nvme1n1</code>, but you may see a different number based on what is connected. We'll store this in a variable to make it easier to copy-paste other commands, so make sure to replace the ...</code> with your selected drive: DISK=/dev/... </code></pre> </li> Next, it's time to partition the actual disk. I'm going to be creating the following partitions: 1 GiB (unencrypted) boot partition - for storing the initial boot files</li> 32 MiB LUKS key partition - the key for the rest of the disk. This will be encrypted with a password that we remember (and type in during boot)</li> 32 GiB swap partition - for enabling system swap</li> The remainder of the drive will be our actual, usable, partition</li> </ul> We're going to be using gdisk</code> below, but if you know how to use another disk partition program, feel free to use it instead. gdisk "${DISK}" </code></pre> Click to expand!</summary> GPT fdisk (gdisk) version 1.0.5 Partition table scan: MBR: not present BSD: not present APM: not present GPT: not present Creating new GPT entries in memory. Command (? for help): o This option deletes all partitions and creates a new protective MBR. Proceed? (Y/N): Y Command (? for help): n Partition number (1-128, default 1): First sector (34-1953525134, default = 2048) or {+-}size{KMGTP}: Last sector (2048-1953525134, default = 1953525134) or {+-}size{KMGTP}: +1G Current type is 8300 (Linux filesystem) Hex code or GUID (L to show codes, Enter = 8300): EF00 Changed type of partition to 'EFI system partition' Command (? for help): n Partition number (2-128, default 2): 2 First sector (34-1953525134, default = 2099200) or {+-}size{KMGTP}: Last sector (2099200-1953525134, default = 1953525134) or {+-}size{KMGTP}: +32M Current type is 8300 (Linux filesystem) Hex code or GUID (L to show codes, Enter = 8300): Changed type of partition to 'Linux filesystem' Command (? for help): c Partition number (1-2): 2 Enter name: luks key Command (? for help): n Partition number (3-128, default 3): First sector (34-1953525134, default = 2164736) or {+-}size{KMGTP}: Last sector (2164736-1953525134, default = 1953525134) or {+-}size{KMGTP}: +32G Current type is 8300 (Linux filesystem) Hex code or GUID (L to show codes, Enter = 8300): Changed type of partition to 'Linux filesystem' Command (? for help): c Partition number (1-3): 3 Enter name: swap Command (? for help): n Partition number (4-128, default 4): First sector (34-1953525134, default = 69273600) or {+-}size{KMGTP}: Last sector (69273600-1953525134, default = 1953525134) or {+-}size{KMGTP}: Current type is 8300 (Linux filesystem) Hex code or GUID (L to show codes, Enter = 8300): Changed type of partition to 'Linux filesystem' Command (? for help): c Partition number (1-4): 4 Enter name: root Command (? for help): p Disk /dev/nvme1n1: 1953525168 sectors, 931.5 GiB Model: Samsung SSD 970 EVO Plus 1TB Sector size (logical/physical): 512/512 bytes Disk identifier (GUID): 22610B10-DB5F-467D-8B9E-ECD88878ABA5 Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 1953525134 Partitions will be aligned on 2048-sector boundaries Total free space is 2014 sectors (1007.0 KiB) Number Start (sector) End (sector) Size Code Name 1 2048 2099199 1024.0 MiB EF00 EFI system partition 2 2099200 2164735 32.0 MiB 8300 luks key 3 2164736 69273599 32.0 GiB 8300 swap 4 69273600 1953525134 898.5 GiB 8300 root Command (? for help): w Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING PARTITIONS!! Do you want to proceed? (Y/N): Y OK; writing new GUID partition table (GPT) to /dev/nvme1n1. The operation has completed successfully. </code></pre> </details> </li> </ol> LUKS Setup</h3> Now that the disk is partitioned, it's time to turn on encryption! First we'll initialize our cryptkey</code> partition and fill it with random data. This will eventually become the key to decrypt our actual drive. Note that this is the day-to-day password used to unlock the computer at boot. cryptsetup luksFormat --type luks1 "${DISK}p2" cryptsetup luksOpen "${DISK}p2" cryptkey dd if=/dev/urandom of=/dev/mapper/cryptkey bs=1024 status=progress </code></pre> </li> Next, we initialize the swap partition (which will share the same key written to our cryptkey</code> partition along with the rest of the drive). Note that there is no backup key set for this partition, but there should never be any reason to try to recover any data written in the swap in case the cryptkey</code> partition is damaged. cryptsetup luksFormat --type luks1 --keyfile-size 8192 --key-file /dev/mapper/cryptkey "${DISK}p3" # Mount the partition after creation cryptsetup luksOpen --keyfile-size 8192 --key-file /dev/mapper/cryptkey "${DISK}p3" cryptswap </code></pre> </li> Now it's time to encrypt the rest of the drive. Note that first we'll initialize the drive with a backup passphrase. Make this a strong password (e.g. diceware) and write it down and store is someplace safe. If the cryptkey</code> partition becomes damaged, this will be the only way to recover the data on the drive! # Initialize with a passphrase cryptsetup luksFormat --type luks1 "${DISK}p4" # Add the cryptkey partition as a keyfile for unlocking during boot cryptsetup luksAddKey --new-keyfile-size 8192 "${DISK}p4" /dev/mapper/cryptkey </code></pre> </li> Finally, mount the root partition. Note the use of --allow-discards</code> which may be a security risk. Read about the choices and assumptions above as to why I have chosen to use this flag, but feel free to omit it if desired. cryptsetup luksOpen --keyfile-size 8192 --key-file /dev/mapper/cryptkey --allow-discards "${DISK}p4" cryptroot </code></pre> </li> Note that some guides recommend filling the drive with random data before doing the encryption to avoid leaking information about how big the drive is and which blocks are encrypted, etc. I am going to omit this step since I want to avoid wearing out my SSD, </li> </ol> Filesystem setup</h3> Initialize the boot partition as vfat mkfs.vfat "${DISK}p1" </code></pre> </li> Initialize the swap partition mkswap /dev/mapper/cryptswap </code></pre> </li> Next, it's time to initialize zfs. Feel free to pick any pool name you want, but consider keeping it unique if you manage other zfs pools elsewhere POOL=nvme-pool # change as desired zpool create "${POOL}" /dev/mapper/cryptroot # autotrim enabled to maintain SSD performance zpool set autotrim=on "${POOL}" </code></pre> </li> Create the desired root datasets (or mounts) in the pool. Note that at the time of writing, using mountpoint=legacy</code> is required for correct NixOS interoperation. zfs create -o compression=on -o mountpoint=legacy "${POOL}/local" zfs create -o compression=on -o mountpoint=legacy "${POOL}/local/nix" zfs create -o compression=on -o mountpoint=legacy "${POOL}/system" zfs create -o compression=on -o mountpoint=legacy "${POOL}/user" zfs create -o compression=on -o mountpoint=legacy "${POOL}/reserved" </code></pre> </li> Set a variable with the default username you wish to use which we'll use for creating a home directory later MY_USER=... </code></pre> </li> Next, we create any child datasets. Note that the acltype=posixacl</code> flag is required wherever /var</code> will be mounted, so that users can access their own journal logs zfs create -o xattr=sa -o acltype=posixacl "${POOL}/system/var" zfs create "${POOL}/system/root" zfs create "${POOL}/user/home" zfs create "${POOL}/user/home/${MY_USER}" </code></pre> </li> Set a quota and reservation on the reserved</code> data set. This will ensure that the disk always has the specified amount of space available, and since we will never mount this partition, we're effectively saving some space from never being written (i.e. over-provisioning the SSD to maintain its performance) zfs set reservation=100G "${POOL}/reserved" zfs set quota=100G "${POOL}/reserved" # ensure we can't accidentally write more than 100G to this partition </code></pre> </li> Next, we enable local snapshotting so that we can quickly recover past state if something goes wrong. Note that we only need to snapshot user data and the system root. Other easily-rebuilt partitions (like local</code>) don't need snapshotting enabled. Also note that the actual snapshot frequency will be managed by our NixOS configuration zfs set com.sun:auto-snapshot=true "${POOL}/system" zfs set com.sun:auto-snapshot=true "${POOL}/user" </code></pre> </li> Lastly, mount everything! If you forget to mount a zfs dataset to the right place, then data may get written in the wrong place and fail during boot! I made the mistake of forgetting to mount the /nix/store</code> path on the new drive. The installer happily filled the root partition with the data, but when my configuration correctly mounted the right dataset during boot, suddenly all the packages were missing!</li> </ul> # Mount the root partition itself mount -t zfs "${POOL}/system/root" /mnt # Make directory entries for the subsequent mounts mkdir -p /mnt/boot mkdir -p /mnt/nix mkdir -p /mnt/var mkdir -p "/mnt/home/${MY_USER}" # Mount the boot partition mount "${DISK}p1" /mnt/boot # Mount the rest of our zfs datasets mount -t zfs "${POOL}/local/nix" /mnt/nix mount -t zfs "${POOL}/system/var" /mnt/var mount -t zfs "${POOL}/user/home/${MY_USER}" "/mnt/home/${MY_USER}" </code></pre> </li> </ol> NixOS installation</h3> Finally it's time to get nix involved! Run the generation command below and it should do a good job at auto-detecting any hardware and filesystem configurations nixos-generate-config --root /mnt </code></pre> </li> Edit the generated config in /mnt/etc/nixos/configuration.nix</code>. If you're new to NixOS, or missing your favorite editor/environment setup, consider lightly tweaking the default config (e.g. turning on ssh, setting up networking, etc.) to get things going and come back to flesh it out later. But before we continue there are a few more things to double check: Make sure that the initrd.luks.devices</code> are correctly configured. If anything is missing, or the disk uuid is incorrect, carefully update the config and double check everything</li> Also carefully note that the cryptkey</code> declaration shows up before any other partions which are unlocked by it!</li> Make sure to update the keyFileSize</code> parameter to whatever was used during initialization</li> Also make sure to set the allowDiscards</code> flag if used above (noting the security caveats from before)</li> Make sure that all filesystems are correctly mapped to their zfs data sets.</li> Add any missing boot.initrd.availableKernelModules</code>. For example, I had to add "amdgpu"</code> to fix some screen resolution issues during early boot.</li> Add your default user and set their home directory</li> </ol> </li> Time to actually install NixOS now! After the initial install is done, reboot and hope everything went well... nixos-install reboot </code></pre> </li> If you got this far and were able to log in, congrats, you did it! A few more things to consider doing: Change the password for your default user</li> Change the root password, or even better, lock the root account so no one can log into it directly</li> </ul> </li> </ol>

Clean up</h1>
Back on the original host we can now purge the secret keys (assuming they have been backed up securely) to minimize them getting compromised:</p>
`gpg</span> --delete-secret-keys</span> 0xA1596EF2CE56B350 </span></code></pre> Happy self-hosting!</p>`

Feedback</h1>
As always, if something doesn't seem quite right or you have any feedback, feel free to let me know on the project repo</a>!</p>

Ivan Petkov

The Markdown to PDF pipeline I wish someone told me about

Configuring Gitea/Forgejo to Sign Merge Commits

Open Source is a Gift, Relicensing is a Grift

NixOS on the PiBox

Transparently Exposing Services Over Tailscale and the LAN

Happy birthday Crane!

Bisecting Nix Configurations

Steam Deck First Impressions

Crane Support for Alternative Registries and Git Dependencies

Introducing Crane: Composable and Cacheable Builds with Cargo and Nix

Tips and Tricks for Nix Flakes

Building with SQLx on Nix

Installing NixOS and ZFS on my Desktop

Ivan Petkov

The Markdown to PDF pipeline I wish someone told me about

Configuring Gitea/Forgejo to Sign Merge Commits

Clean up</h1> Back on the original host we can now purge the secret keys (assuming they have been backed up securely) to minimize them getting compromised:</p> gpg</span> --delete-secret-keys</span> 0xA1596EF2CE56B350 </span></code></pre> Happy self-hosting!</p>

Open Source is a Gift, Relicensing is a Grift

NixOS on the PiBox

(Optional) Remote LUKS Unlock</h2> Having to plug in a keyboard and monitor to the PiBox to unlock after a restart can be a chore, so being able to unlock the drive remotely via SSH can be much more convenient. First we need to generate a unique host key for the boot</em> stage.</p>

Transparently Exposing Services Over Tailscale and the LAN

Prerequisites</h1> Things we'll need and their example values. Remember to substitute these with your own!</p> A domain we control: example.com</code></li> A subdomain we would like to use for the service: rainbows.example.com</code></li> A tailnet name: cat-crocodile.ts.net</code></li>

Happy birthday Crane!

Bisecting Nix Configurations

Bisecting Neovim</h1> Knowing the exact range which includes the behavior change means we can run another git bisect. In this case I directly checked out the Neovim repo since I could test more quickly that way without having to redeploy my NixOS configuration.</p>

Steam Deck First Impressions

Crane Support for Alternative Registries and Git Dependencies

Git Dependencies</h1> Vendoring crate sources from git repositories is roughly the same as vendoring from registries:</p>

Feedback</h1> As always, if something doesn't seem quite right or you have any feedback, feel free to let me know on the project repo</a>!</p>

Introducing Crane: Composable and Cacheable Builds with Cargo and Nix

Tips and Tricks for Nix Flakes

Wrangling Flake Inputs</h1> The flake input schema allows for:</p> Having your flake pull in another flake</em> as an input</li> Having your flake pull in an input that is specified by another flake</em></li> Forcing another flake</em> to use an input specified in your flake</li>

Building with SQLx on Nix

Installing NixOS and ZFS on my Desktop

Clean up</h1>
Back on the original host we can now purge the secret keys (assuming they have been backed up securely) to minimize them getting compromised:</p>
`gpg</span> --delete-secret-keys</span> 0xA1596EF2CE56B350 </span></code></pre> Happy self-hosting!</p>`

Feedback</h1>
As always, if something doesn't seem quite right or you have any feedback, feel free to let me know on the project repo</a>!</p>