what this system has been working on recently
▌▖▌▖▖▘ [2026JUN24] ▘▌
it is a DNS server backed by S3-like storage, allowing for incredibly clever geo-distributed DNS! garage as an S3 backend can enable this quite easily, as well as distributing this over any number of S3 endpoints via tools like rclone.
this will be the authoritative DNS server for all of its domains so it really wants to overbuild it smile
^aki
▌▖▘▘▌▘ [2026JUN12] ▘▘
missed a few things here, currently working on learning BGP/inter-VPS networking and its weirdly fun! ^aki
made an openscad preprocessor that lets us STL export multiple parts from one openscad file, keeping the transforms and all; for the goal of easy multi-material 3D printing in orca slicer
we've been making parts for our MSA Millenium so this has been a no brainer ^junko
▘▌▌▘▖▖ [2026APR12] ▘▖
it has some shortcomings, but we'll address them in time.
one of the shortfalls is that it doesn't keep directory structure very well, so the next step is automatic library sorting!
(if for some reason that one wants all of our music, we're 41666 on slsk!)
▘▌▌▖▘▌ [2026APR08] ▖▌
this one wanted automatic registration of nginx and dns endpoints...
so it started on this!
currently, it only supports setting up nginx, but it also wants it to set up dns records on various providers; our domains are heavily spread across cloudflare, porkbun, and other internal dns servers that may or may not exist yet
but for now, lets talk nginx.
and specifically, it'll just focus on its recent hydra server
it wanted literally.. this:
internal.endpoints."hydra.dolly.sh" = {port = 3000;};and we got that :>
behind the scenes, there's a lot of automatic configuration happening, with the entire
internal.endpoints. block becoming this (json encoded from nix eval)"hydra.dolly.sh": {
"acme": true,
"acmeProvider": "letsencrypt",
"dns": true,
"dnsProvider": "porkbun",
"dnsTrackInterface": null,
"extraProxyConfig": {},
"hostname": "hydra.dolly.sh",
"https": true,
"port": 3000,
"proxy": true,
"proxyCache": null,
"proxyTag": "dead",
"upstream": "ci.hoki-porgy.ts.net"
}the
acmeProvider and dnsProvider are picked by matching patterns against hostname, so hydra.dolly.sh matches dolly.sh which is on porkbun dns, and would want public certs from letsencrypt.proxy* ones are a little more complex, these actually translate into services.nginx.* config. so, when the proxy named dead is building, it will filter down every system's internal.endpoints. such as nixosConfigurations.ci.config.internal.endpoints. in our case, and derive the result...services.nginx = {
upstreams = { "hydra_dolly_sh__3000".servers."http://ci.hoki-porgy.ts.net:3000"; };
virtualHosts = {
"hydra.dolly.sh" = {
location."/" = {
proxyPass = "http://hydra_dolly_sh__3000";
};
};
};
};this is actually a lot of code so
▘▌▘▌▘▘ [2026MAR29] ▖▘
hydra! hydra.dolly.sh
!!!
wow wow wow
it build every machine at the same time and it can keep everything in cache..
hydra is so cool
compiling 3 different linux kernels at the same time is fun!!!! woof!!
^#6 trying to type like noe
what we did
we decided our CI server that runs woodpecker could also run hydra, but then also be our binary cache, so we moved harmonia to our CI server!!!.. which is subby.blood.pet
which also, our ingress nginx server was given a quite big cache to work with ... so hopefully things are just nicer all around.
hydra also keeps gcroots around, so we can keep our most recent builds for as long as they need to exist. things stay cached because we are referencing it somewhere, e.g. any nixos machine on our network, or just blood.pet's dev dependencies...
this integrates a lot of stuff together and solves its ever-present authentication problem it hadd with the separate nix cache...
so now, woodpecker agents, hydra, and its nix binary cache all share the same exact nix store.
lovely (:
^noe
▘▌▘▌▖▘ [2026MAR26] ▖▖
two fun things this time.
our goal was to get full-disk encryption onto a netcup root server!
we were trying for a few days but finally found the right mix of approaches.
disko
disko lets us define our storage layout declaratively, both being a tool that can automatically partition but also automatically configure our fstab!!
some notes:
- the hardcoded paths are a necessary evil for disko LUKS setups; we'll talk about what's going on here later. good news, its pretty idempotent
- this is a pretty basic LUKS container with a btrfs inside.
it tried to only do disko because the nixos install process is tedious...
but it was doing so much trial and error stuff trying to get remote unlocking working that it had to find a better way...
nixos-anywhere
- setup SSH keys for itself
- copy files that disko needs (like passphrase or keys)
- apply disko partitions
- copy files that nixos needs (like initrd SSH host keys...)
- optionally copy the nixos installer's SSH host key into the new install (useful!!)
- run nixos-install with flake context
- reboot
and there should be a working system on the other end...
but not just a working system, one that looks like all of the other machines in our flake. it has the right passwords, SSH keys, auth tokens, everything. this would normally be a 2 step process of getting a bare minimum system to deploy to, and nixos-anywhere helped do both at once.
it remembers a similar tool for getting nixos onto hetzner, nixos-infect which did something super similar... like it can kexec into a nixos installer if it isn't already in one!! but nixos-anywhere works there too.
remote unlocking
so now we're at the part things got less documented...
so.. this server wants a password before it boots fully..
in the file we shared earlier, the
boot.initrd block is where our magic happens; this starts an SSH server with a specific host key (which is not the normal host key!!!) and gives us some tools to let us unlock the disks.the main one it sees documented is
cryptsetup-askpass but... what is this?? searching in general LUKS setups doesn't get us very far. this is obviously some bespoke tool. oh yeah it is.looking around that file, and we see a lovely thing we've been looking for.
a lot of the docs around this mention
/lib/cryptsetup/passfifo but for whatever reason, this doesn't apply to nixos.instead, we write our password onto the machine at
/crypt-ramfs/passphrase which this script is always looping over reading from, mostly in service to a weird feature that allows that file to be pre-populated after use. we don't want that, but we will use the hack (:so our unlock script is now really boiling down to this:
sops decrypt --extract '["crypt_pass"]' machines/still/secrets/default.yaml \ | ssh -p 2222 root@still cat > /crypt-ramfs/passphrase
and et voila, the SSH server dies, and boot occurs.
... now how could it make this faster without being too silly ...
^noe
▘▌▘▘▘▖ [2026MAR19] ▌
its pretty cool, and is a huge improvement over its current tooling,,,
altho it needs to find a way to make its sudo auth work.... not really sure how to approach yet. thinking about
NH_SSH_ASKPASS...
yay watching kernel compile!!
^noe
▘▌▘▘▖▌ [2026MAR18] ▘
just starting this /now page,,, we hope we can talk more about our little things we're working on here
noe added some stuff it did recently to retroactively fill this page
also the ▘▌▘▘▖▌ part is days since 2024JAN01; the 41666 epoch <3
^光R
▘▌▘▖▘▘ [2026MAR11] ▖
its made in crystal, which it has mixed feelings about. it wouldn't use it again.
it decided to progressively support different types of clients, so
curl vs w3m/lynx vs fetch() vs firefox should all get equally okay experiences!
the cool parts in CSS it learned about is color-mix which lets us take a base color and blend it with another, like white, to replicate the color patterns in the original site.
--tag-accent: #f36aff; --blend: #ffffff; color: color-mix(in srgb, var(--tag-accent), var(--blend) 66%);
becomes roughly
#fbccff^noe
