Terin Stock

Netboot Windows 11 with iSCSI and iPXE

terin@terinstock.com (Terin Stock) — Wed, 26 Feb 2025 03:28:00 +0000

Purposefully ambiguous and fictious permanent ban.

(created with @foone’s The Death Generator)

My primary operating system is Linux: I have it installed on my laptop and desktop. Thanks to the amazing work of the WINE, CodeWeavers, and Valve developers, it’s also where I do PC gaming. I can spin up Windows in a virtual machine for the rare times I need to use it, and even pass through a GPU if I want to do gaming.

There is one pretty big exception: playing the AAA game ████████████████ with friends. Unfortunately, the developer only allows Windows. If you attempt to run the game on Linux or they detect you’re running in a virtual machine, your device and account are permanently banned. I would prefer not to be permanently banned.

For the past several years my desktop has also had a disk dedicated to maintaining a Windows install. I’d prefer to use the space in my PC case¹ for disks for Linux. Since I already run a home NAS, and my Windows usage is infrequent, I wondered if I could offload the Windows install to my NAS instead. This lead me down the course of netbooting Windows 11 and writing up these notes on how to do a simplified “modern” version.

iPXE and iSCSI

My first task was determining how to get a computer to boot from a NAS. My experience with network block devices is with Ceph RBD, where a device is mounted into an already running operating system. For booting over an Ethernet IP network the standard is iSCSI. A great way to boot from an iSCSI disk is with iPXE. To avoid any mistakes during this process, I removed all local drives from the system.²

I didn’t want to run a TFTP server on my home network, or reconfigure DHCP to provide TFTP configuration. Even if I did, the firmware for my motherboard is designed for “gamers”, there’s no PXE ROM. I can enable UEFI networking and a network boot option appears in the boot menu, but no DHCP requests are made³. Fortunately, iPXE is available as bootable USB image, which loaded and started trying to fetch configuration from the network.

Hitting ctrl-b as directed on screen to drop to the iPXE shell, I could verify basic functionality was working.

iPXE 1.21.1+ (e7585fe) -- Open Source Network Boot Firmware -- https://ipxe.org
Features: DNS FTP HTTP HTTPS iSCSI NFS TFTP VLAN SRP AoE EFI Menu
iPXE> dhcp
Configuring (net0 04:20:69:91:C8:DD)...... ok
iPXE> show ${net0/ip}
192.0.2.3

I decided to use tgt as the iSCSI target daemon on my NAS⁴ as the configuration seemed the least complicated. In /etc/tgt/targets.conf I configured it with two targets: one as the block device I wanted to install Windows onto and the other being the installation ISO.

<target iqn.2025-02.com.example:win-gaming>
backing-store /dev/zvol/zroot/sans/win-gaming
params thin-provisioning=1
</target>
<target iqn.2025-02.com.example:win11.iso>
backing-store /opt/isos/Win11_24H2_English_x64.iso
device-type cd
readonly 1
</target>

Back on the PC, I could tell iPXE to use these iSCSI disks, then boot onto the DVD. As multiple network drives are being added, each must be given a different drive ID starting from 0x80.

iPXE> sanhook --drive 0x80 iscsi:nas.example.com:::1:iqn.2025-02.com.example:win-gaming
Registered SAN device 0x80
iPXE> sanhook --drive 0x81 iscsi:nas.example.com:::1:iqn.2025-02.com.example:win11.iso
Registered SAN device 0x81
iPXE> sanboot --drive 0x81
Booting from SAN device 0x81

After a minute of the Windows 11 logo and a spinner, the Windows 11 setup appears. In an ideal situation, I could immediately start installing. Unfortunately, the Windows 11 DVD does not ship drivers for my network card, and the iSCSI connection information passed to the booted system from iPXE couldn’t be used. I’m a bit impressed the GUI loaded at all, instead of just crashing.

Chainloading WinPE

To rectify this, I would need to build a Windows PE image that included my networking drivers. WinPE is the minimal environment used when installing Windows. Fortunately, Microsoft has made this pretty easy nowadays. I downloaded and installed the Windows Assessment and Deployment Kit and the Windows PE add-on. After running “Deployment and Imaging Tools Environment” as an administrator, I could make a folder containing a base WinPE image.

> mkdir C:\winpe
> copype amd64 C:\winpe\amd64

After mounting the image, I was able to slipstream the Intel drivers. I searched through the inf files to find the folder that supported my network card.

> imagex /mountrw C:\winpe\amd64\media\sources\boot.wim C:\winpe\amd64\mount
> dism /image:C:\winpe\amd64\mount /add-driver /driver:C:\temp\intel\PRO1000\Winx64\W11\
> imagex /unmount /commit C:\winpe\amd64\mount

This new image is what we need to boot into to install Windows. As my NAS is also running an HTTP server, I copied over the files relevant to netbooting: from “C:‍\winpe\amd64\media” I copied “boot/BCD”, “boot/boot.sdi”, and “sources/boot.wim”, preserving the folders. I also downloaded wimboot to the same directory.

You can use iPXE to execute a script fetched with HTTP, which I took advantage of to reduce the amount of typing I’ll need to do at the shell. I saved the following script as “install.ipxe” in the same HTTP directory.

#!ipxe
sanhook --drive 0x80 iscsi:nas.example.com:::1:iqn.2025-02.com.example:win-gaming
sanhook --drive 0x81 iscsi:nas.example.com:::1:iqn.2025-02.com.example:win11.iso
kernel wimboot
initrd boot/BCD BCD
initrd boot/boot.sdi boot.sdi
initrd sources/boot.wim boot.wim
boot

Rebooting back to the iPXE prompt I could then boot using this script.

iPXE> dhcp
iPXE> chain http://nas.example.com/ipxe/install.ipxe

After a few seconds I was booted into WinPE with a Command Prompt. The command “wpeinit” ran automatically, configuring the network card and mounting the iSCSI disks. I found the DVD had been mounted as drive “D”, and could start the Windows Setup with “D:‍\setup.exe”.

However, after reaching the “Searching for Disks” screen the installer closed itself without any error. This seems to be a bug with the new version of setup, as restarting it and selecting the “Previous Version of Setup” on an earlier page used a version of the installer that worked.

The installation was spread across several restarts. Fortunately, once the installation files are copied over, nothing but the main disk image is required, reducing what I needed to type in the iPXE shell. The HTTP server could also be cleaned up at this point.

iPXE> dhcp
iPXE> sanboot iscsi:nas.example.com:::1:iqn.2025-02.com.example:win-gaming

After several more minutes, and a forced installation of a Windows zero-day patch, I was greeted by a Windows 11 desktop, booted over iSCSI. Task Manager even reports the C drive as being “SDD (iSCSI)”.

Automating Boots

Booting from a USB stick and typing into an iPXE prompt every time I want to boot into Windows isn’t a great user experience. Fortunately, iPXE is also available as an EFI application which can be installed to the local EFI System Partition. I also discovered that iPXE will execute commands provided on the command line.

I reinstalled the disks used for Linux, copied over ipxe.efi to the EFI System Partition, and added a new entry to systemd-boot by creating “$ESP/loader/entries/win11.conf”

title Windows 11 (iPXE)
efi /ipxe/ipxe.efi
options prompt && dhcp && sanboot iscsi:nas.example.com:::1:iqn.2025-02.com.example:win-gaming

There seems to be a bug where the first word in the options field is ignored.⁵ I used a valid iPXE command prompt, which also provides a clear signal should it ever start being interpreted in the future version.

Let’s Game

After a little bit of extra setup (installing Firefox and switching to dark mode), I was able to install Steam and the game. The game took a little bit longer to install due the slower disk speed over my network (time to upgrade to 10GbE?), but there was no noticeable delay during normal gameplay. I didn’t see any network saturation or high disk latencies in Task Manager during loading.

My computer is built into the Cooler Master NR200P mini-ITX case, which only has space for two 2.5" SATA drives. ↩︎
You probably want to remove the disks anyways so Windows doesn’t install its boot manager to an EFI System Partition on a local disk. To boot with iPXE later, you’ll want Windows to create an EFI System Partition on the iSCSI disk. ↩︎
The UEFI configuration doesn’t have any options for manually specifying networking parameters, either. ↩︎
Long time iSCSI on Linux users may find the choice of STGT peculiar, it having been replaced with the kernel-space target LIO as the iSCSI target daemon back in 2010. My limited usage at home isn’t going to run into performance issues with tgt, and I prefer being able to start and stop a user-space daemon. ↩︎
I’m not sure if this is a bug in systemd-boot or in iPXE. ↩︎

Assembling a Game Boy Game with Meson

terin@terinstock.com (Terin Stock) — Mon, 28 Oct 2024 02:31:00 +0000

I’ve been working on a Game Boy game off-and-on for the last few months (hopefully more details about this soon!). Up until recently I was building the game with GNU Make, but I was frustrated with configuring Make’s prerequisites for accurate dependency tracking. Often I’d change a file included by my target, but Make wasn’t rebuilding appropriately.

I started looking at other build systems to solve this problem. I really liked how approachable Meson’s build rules where, and also that it supported projects composed of multiple languages. This is useful to me as parts of my toolchain for handling assets, and converting them to a format usable on the Game Boy, are written in higher level languages. If I change the code of part of that toolchain, the build system knows what assets need to processed and what parts of the game need to be relinked, automatically.

There’s just one problem: Meson doesn’t support the Game Boy. However, that’s never stopped me before!

Technology is incredible!

I’ve created a fork of Meson that adds support for using RGBDS to assemble and link Game Boy games. This was surprisingly easy to do, which is a testament for both the Meson and RGBDS projects. In Meson I added a new “language” rgbds which sets up rgbasm as the compiler/assembler and rgblink as the linker. A ROM can be built just by calling the executable function.

project('rgbds-test', 'rgbds')

executable('rgbdstest.gb', 'src/main.asm',
 include_directories: ['include'],
 link_language: 'rgbds')

Then Meson can be initialized to assemble to the Game Boy’s CPU and the game built.¹ You can notice that Meson configures rgbasm to output dependency tracking metadata, which ninja uses for fast rebuilds.

$ ../../../meson.py setup --cross-file crossfile.ini --wipe build
The Meson build system
Version: 1.6.0
Source dir: /home/terin/Development/github.com/mesonbuild/meson/test cases/rgbds/1 basic
Build dir: /home/terin/Development/github.com/mesonbuild/meson/test cases/rgbds/1 basic/build
Build type: cross build
Project name: rgbds-test
Project version: undefined
Rgbds compiler for the host machine: rgbasm (rgbds 0.8.0)
Rgbds linker for the host machine: rgblink rgblink 0.8.0
Compiler for language rgbds for the build machine not found.
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: sm83
Host machine cpu: sm8320
Target machine cpu family: sm83
Target machine cpu: sm8320
Build targets in project: 1

rgbds-test undefined

 User defined options
 Cross files: crossfile.ini

Found ninja-1.12.1 at /usr/bin/ninja

$ ninja -C build -v
ninja: Entering directory `build'
[1/2] rgbasm -I../include -I.. -I. -Irgbdstest.gb.p -M rgbdstest.gb.p/src_main.asm.o.d -MQ rgbdstest.gb.p/src_main.asm.o -o rgbdstest.gb.p/src_main.asm.o ../src/main.asm
[2/2] rgblink -o rgbdstest.gb rgbdstest.gb.p/src_main.asm.o

$ ninja -C build -t deps
rgbdstest.gb.p/src_main.asm.o: #deps 2, deps mtime 1730079958938870943 (VALID)
 ../src/main.asm
 ../include/hardware.inc

Game Boy ROM files have a header that needs to be set correctly so real hardware will play the cartridge, and emulators know what cartridge type to emulate. Since this header contains two checksums, usually it’s configured using the rgbfix tool rather than being hardcoded in assembly. Unfortunately, I haven’t figured out a good way for executable to automatically call this tool with the correct parameters after the rom is linked, but the tool can be invoked as a custom_target.

project('rgbds-test', 'rgbds')

rgbfix = find_program('rgbfix', required: true)

rom = executable('rgbdstest', 'src/main.asm',
 include_directories: ['include'],
 link_language: 'rgbds')

custom_target(output: 'rgbdstest.gb',
 input: rom,
 command: [
 rgbfix,
 '--title', 'EXAMPLE',
 '--old-licensee', '0x33',
 '--mbc-type', 'ROM',
 '--rom-version', '0',
 '--non-japanese',
 '--validate',
 '-',
 ],
 build_by_default: true,
 capture: true,
 feed: true)

This example also showcases using custom_target to interact with tools that need to be “feed” via stdin and where the results need to be “captured” on stdout, which is convenient as otherwise rgbfix edits the file in-place, which could cause issues later.

Since I’m not ready to release my game just yet, I’ve modified pokered to use my fork of Meson. This builds the two image converters written in C, converts all the images to a format suitable for the Game Boy, and assembles and links with RGBDS. The resulting game matches the expected checksum.²

$ ninja -C build
ninja: Entering directory `build'
[1374/1374] Generating pokered.gbc with a custom command (wrapped by meson to capture output, to feed input)

$ shasum -c --ignore-missing ../roms.sha1
pokered.gbc: OK

$ jollygood -c sameboy build/pokered.gbc
i: Core: sameboy (SameBoy 0.16.5)
i: Video: OpenGL OpenGL ES 3.2 Mesa 24.1.7
i: Audio: 48000Hz Stereo, Speex 3
i: Emulated Input 1: gameboy1, Game Boy, 0 axes, 10 buttons

Pokémon Red's start screen

I’ll be submitting these changes to the upstream Meson project, in the off chance they’re fans of the Game Boy.

Edit 29 Oct: I’ve opened a PR to submit my changes to upstream Meson.

Meson has a concept of module, which are in-tree extensions to the core language to help handle common build tasks with large libraries, such as compiling moc files in Qt projects. I’ve created a “rgbds” module which has a function run rgbfix to patch the ROM header, instead of needing to implement the above custom_target yourself.

rgbds = import('rgbds')
rgbds.fix('rgbdstest.gb', rom,
 title: 'EXAMPLE',
 mbc_type: 'ROM',
 fix_spec: 'lhg')

This module also adds a function with a barebones implementation of rgbgfx, the graphics converter from the RGBDS project. This was previously implemented as a custom_target in the pokered fork above.

pngs = [
 'bug',
 'plant',
 'snake',
 'quadruped',
]
foreach f : pngs
 gen = custom_target(output: '@0@_conv.png'.format(f),
 input: '@0@.png'.format(f),
 command: [rgbgfx, '-o', '@OUTPUT@', '@INPUT@'])
 gfx += custom_target(output: '@0@.2bpp'.format(f),
 input: gen,
 command: [tools_gfx, '-o', '@OUTPUT@', '@INPUT@'])
endforeach

The inner for loop can now use the rgbds module.

pngs = [
 'bug',
 'plant',
 'snake',
 'quadruped',
]
foreach f : pngs
 gen = rgbds.gfx('@0@_conv.2bpp'.format(f), '@0@.png'.format(f))
 gfx += custom_target(output: '@0@.2bpp'.format(f),
 input: gen,
 command: [tools_gfx, '-o', '@OUTPUT@', '@INPUT@'])
endforeach

The meson branch of my pokered fork has been updated with these changes.

While this is the technically correct thing to do, it seems a bit much here as the only system RGBDS can compile to is the Sharp SM8320. ↩︎
Adding support for compiling Pokémon Blue, as well as the debug and Virtual Console patches, have been left as an exercise for the reader. ↩︎

Portage + ORAS: Using a Docker Registry for Gentoo Packages

terin@terinstock.com (Terin Stock) — Wed, 04 Sep 2024 22:19:27 +0000

You didn't think I had a Type R or '86 did you?

Portage is Gentoo’s source-based package manager, commonly invoked with the emerge tool. The Prefix project allows it to be used by users of other distributions and Unix-like operating systems in a “virtualenv”-type fashion. In Portage, packages are described in ebuilds, which are composed of shell functions understandable by many Linux users. This makes it a good fit for sharing packages across distributions and users working on a project.

To compile packages from source, Portage requires being able to fetch archives of the software, and requires those archives to have stable hashes. This works fine for public source code: projects often provide archives for releases, and the Gentoo project can mirror archives generated automatically by public forges. This requirement poses problems for private packages, however, as the archives generated by forges are not guaranteed to be public.

After being stuck manually copying archives between machines for a year, I finally decided to tackle this problem this past weekend. I could mirror the archives to a web server, like the Gentoo project does. However, there was no web server already available, and I was not interested in setting up infrastructure just for Portage, sorting out the permissions to allow multiple people to upload archives, or managing backups of the archives.

There was one bit of infrastructure that was already setup with these attributes: a Docker (or OCI) registry. While we tend to think of these registries as a place to distribute container images and their layers, and they are, beneath the surface they are content-addressable stores that can be used to distribute other types artifacts too. The Notary and cosign projects distribute signatures of containers by storing them in the registry, and Helm charts can be distributed via registries as well.

Could I use a registry as a source archive mirror? Yes! The ORAS¹ project provides a CLI, and libraries, for uploading and downloading artifacts to these registries. For example to upload and download the file “hello-world.txt”, we can can use the oras command and “push” and “pull” with an image tag, very similar to using docker or podman.

oras push example.com/hello:v1 hello-world.txt
oras pull example.com/hello:v1

Using the ebuild for direnv v2.34.0 for example, there are two source archives: one is an archive of the source code, and the other is an archive of the Go module dependencies, following the current recommendations from the Gentoo Wiki. After we have download and generated these two archives, we can upload them to a registry with a variation on the above “push” command.

oras push --artifact-type application/vnd.example.archives.v1 \
 example.com/distfiles/direnv:2.34.0 \
 direnv-2.34.0.tar.gz:application/x-compressed-tar \
 direnv-2.34.0-vendor.tar.xz:application/x-compressed-tar

This creates one tag, “example.com/distfiles/direnv:2.34.0”, with two layers for the two archives we uploaded. We can later pull these archives by pulling the tag.

oras pull example.com/distfiles/direnv:2.34.0

At this point you might be thinking to yourself how this is helpful for ebuilds? As each tarball is now in the content-addressable store, we can use this location in the ebuild. To get those locations we can fetch the manifest at the tag.

$ oras manifest fetch example.com/distfiles/direnv:2.34.0 --pretty

{
 "schemaVersion": 2,
 "mediaType": "application/vnd.oci.image.manifest.v1+json",
 "artifactType": "application/vnd.example.archives.v1",
 "config": {
 "mediaType": "application/vnd.oci.empty.v1+json",
 "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
 "size": 2,
 "data": "e30="
 },
 "layers": [
 {
 "mediaType": "application/x-compressed-tar",
 "digest": "sha256:3d7067e71500e95d69eac86a271a6b6fc3f2f2817ba0e9a589524bf3e73e007c",
 "size": 94449,
 "annotations": {
 "org.opencontainers.image.title": "direnv-2.34.0.tar.gz"
 }
 },
 {
 "mediaType": "application/x-compressed-tar",
 "digest": "sha256:b86942d442f7e2a92a86dcad9b36d9316948e990592f65314137235df6c43293",
 "size": 327916,
 "annotations": {
 "org.opencontainers.image.title": "direnv-2.34.0-vendor.tar.xz"
 }
 }
 ],
 "annotations": {
 "org.opencontainers.image.created": "2024-09-04T21:08:22Z"
 }
}

Each source artifact was uploaded as a layer, and can be identified with the “org.opencontainers.image.title” annotation automatically added by ORAS during our push. We can then use the digest as part of the URL in the ebuild.

SRC_URI="https://example.com/v2/distfiles/direnv/blobs/sha256:3d7067e71500e95d69eac86a271a6b6fc3f2f2817ba0e9a589524bf3e73e007c -> ${P}.tar.gz"
SRC_URI+=" https://example.com/v2/distfiles/direnv/blobs/sha256:b86942d442f7e2a92a86dcad9b36d9316948e990592f65314137235df6c43293 -> ${P}-vendor.tar.xz"

While this would work, it’s not very convenient to update for new versions. We can do better by leveraging Portage’s support for customizing fetch commands with FETCHCOMMAND. Portage ships with commands for http(s), FTP, SSH, SFTP, and rsync.

$ portageq envvar FETCHCOMMAND_RSYNC
rsync -LtvP "${URI}" "${DISTDIR}/${FILE}"

By following the format of FETCHCOMMAND_${protocol}, we can add support for our own protocols by setting a command in Portage’s configuration.

I’ve a new fetch command FETCHCOMMAND_ORAS defining a custom “oras” protocol. This uses the ORAS CLI to fetch the manifest referenced by a tag, selects the digest of the requested file, and downloads it to the prescribed location.²

FETCHCOMMAND_ORAS="bash -c \"x=\\\${0#oras://}; image=\\\${x%/*}; blob=\\\${x##*/}; oras blob fetch \\\$image@\\\$(oras manifest fetch \\\$image | jq -r --arg blob \\\$blob '.layers|map(select(.annotations[\\\"org.opencontainers.image.title\\\"]==\\\$blob))[0].digest') -o \\\$1 \" \${URI} \${DISTDIR}/\${FILE}"

Then we can reference this new protocol in our ebuild, reducing the amount of toil needed to update or add a new package.

SRC_URI="oras://example.com/distfiles/${PN}:${PV}/${P}.tar.gz"
SRC_URI+=" oras://example.com/distfiles/${PN}:${PV}/${P}-vendor.tar.xz"

As an end user, once configured fetching artifacts with ORAS is then transparently used by Portage.

$ emerge -1v apps-shells/direnv
>>> Emerging (1 of 1) app-shells/direnv-2.34.0::terinjokes
>>> Downloading 'oras://example.com/distfiles/direnv:2.34.0/direnv-2.34.0.tar.gz'
✓ Downloaded application/octet-stream 92.2/92.2 kB 100.00% 0s
 └─ sha256:3d7067e71500e95d69eac86a271a6b6fc3f2f2817ba0e9a589524bf3e73e007c
 * direnv-2.34.0.tar.gz BLAKE2B SHA512 size ;-) ... [ ok ]
>>> Downloading 'oras://example.com/distfiles/direnv:2.34.0/direnv-2.34.0-vendor.tar.xz'
✓ Downloaded application/octet-stream 320/320 kB 100.00% 0s
 └─ sha256:b86942d442f7e2a92a86dcad9b36d9316948e990592f65314137235df6c43293
 * direnv-2.34.0-vendor.tar.xz BLAKE2B SHA512 size ;-) ... [ ok ]

OCI Registry As Storage ↩︎
Hopefully in the future oras pull will support selecting specific file names, avoiding the subshell to fetch the manifest and parse with jq. ↩︎

When Serial Isn't RS-232, and Geocaching with the Garmin GPS 95

terin@terinstock.com (Terin Stock) — Sun, 11 Aug 2024 18:35:11 +0000

Found the first Dutch Geocache, Amsterdam Urban 1.

Recently I picked up a box of early 1990s Garmin GPS receivers along with an array of accessories. I cleaned up one receiver, a GPS 95, installed 4 new AA batteries, and positioned it with a clear view of the sky. After an agonizing long time searching for satellites it eventually received a full almanac and made a GPS lock. It was pretty cool that it still works!

However there were two problems discovered while using the receiver. First, while the longitude and latitude were accurate, the altitude was completely wrong. Second, the receiver still thought it was in the 1990s. While the first problem is inherent to GPS, we can fix the second issue!

The receiver has no control for changing the date or time: it relies almost entirely on the GPS signal. Surely, the date being sent by GPS wasn’t wrong. The GPSJAM map did not show any interference anywhere near the Netherlands.

No GPS jamming detected in the Netherlands.

The “legacy” LNAV signal transmitted by GPS doesn’t actually contain the date. That is instead computed by the receiver based on two other signals transmitted: a 10-bit week counter that is incremented once a week, and a 19-bit time of week signal incremented every 1.5 seconds. This week counter rolls over every 1024 weeks (or approximately 19.6 years).¹² So a GPS receiver requires some other way to know which week counter epoch to use.

The first GPS rollover occurred at midnight between 21-22 August 1999. This was at the tail end of support for the GPS 95 receivers. Fortunately, Garmin released a tool at the time “GPS EOW”³ that existed entirely to adjust the time of the reciver’s clock. This would allow the built-in logic to track the date afterwards. The GPS 95 has now joined us in the 21st century.⁴

GPS EOW, a very minimalist application for Windows 98.

I started my Windows 98SE virtual machine⁵ and attached /dev/ttyUSB0 as COM1 and started “GPS EOW”. Except it complained that it couldn’t communicate with the receiver.

GPS EOW was unable to connect to the GPS 95.

After some fiddling, I switched to a different USB adapter and that worked successfully.

Later, I wanted to try out GPS for Flipper Zero, an application for the Flipper Zero that decodes NMEA 0183 messages, with the GPS 95. I connected the serial connection from the GPS receiver to the Flipper Zero as described, but the application never saw any data.

GPS for Flipper Zero reporting receiving no fix.

Since I knew NMEA 0183 messages are ASCII, I switched to the UART Terminal application. Instead of the expected NMEA, I saw complete gibberish.

The UART Terminal application displaying garbage.

Those that have been around serial for a while are undoubtedly rushing to a comment section to complain about the bone-headed move I just described. I may have just destroyed the pins on my Flipper Zero. As we’ll see in a bit, I lucked out and did not.

The serial port on the PC platform uses RS-232, a loose standard going back to the 1960s describing the electrical and timing of the signals, but not the encoding of data. This is a bipolar signal ±25V where the data signal are “inverted”, that is a signal ≤-3V is a logical “1” and a signal ≥3V is a logical “0”. The range between -3V and 3V is undefined. The pins of a Flipper Zero are at 3.3V, but is tolerant of signals up to 5V. If the GPS 95 was indeed sending RS-232 levels that could have gone very wrong.

To better understand what was going on, I used the oscilloscope at the Technologia Incognita hackerspace. Fortunately, the GPS 95’s serial was only going up to 5V, but also not using negative voltages at all. This reminded me of “TTL serial”, the type of serial we use with microcontrollers and what the original USB serial adapter and the Flipper Zero expect. However, like the UART Terminal app, the UART decoder on the oscilloscope was also having difficulties decoding something useful from what should be NMEA.

The Garmin GPS 95's serial data displayed on the Hantek DSO2D15 oscilloscope. Notice how it starts to decode randomly in the middle of the data; it has interpreted something else as the start bit.

Besides the voltages used, there is another major difference between RS-232 and “TTL serial”. The latter signal is not inverted. Instead a zero level is a logical “0”, which a high level is a logical “1”.

If you search online these are the two types of serial you’ll see described again and again. You might find the occasional reference to a third type that was popular in the early 1990s. This serial type operates between 0-5V like “TTL serial”, but with inverted data lines like RS-232, allowing it to be connected directly to a PC’s serial port.⁶

While I’m not aware of a name used at the time for this type of serial, this is the type of serial used by the GPS 95. It is my understanding that portable device manufacturers did not want to add an extra voltage rail just for the serial port, but RS-232 drivers with integrated charge pumps weren’t yet cheap enough.

To be able to receive the NMEA data on my Flipper Zero I’d need to convert this to “TTL serial”; that is invert the signal again so that a zero voltage is a logical “0” and a positive voltage is a logical “1”. I choose to use an SN74HC04N inverter, a “jelly bean” part I already had on my bench from another project.

Flipper Zero connected to the Garmin GPS 95 through the inverter chip.

With the circuit put together on a prototyping board, both the UART Terminal and GPS for Flipper Zero applications worked correctly! Now I can connect the GPS 95 to any device expecting “TTL serial” without problems, which opens up some cool ideas for future projects.

The UART Terminal application with NMEA sentences.

GPS for Flipper Zero is able to decode the data too!

Today I set out to find the oldest hidden Geocache in the Netherlands, Amsterdam Urban 1. While the Garmin GPS 95 is primarily for aviation, complete with a database of airports, navigation waypoints, and frequencies last updated in 1999, adding custom waypoints is well supported.

The GPS 95 navigating me across Vondelpark to the cache. Just about a quarter of a kilometer to go!

As the image at the top of this post shows, I was able to find the cache. Since user waypoints can be synchronized with a PC, it would be interesting in the future to write a program that can push a list of geocaches to the Garmin GPS 95. I’ll be typing them in manually in the meantime!

The week counter was expanded to 13-bits when using the newer CNAV signal, rolling over every 157 years. ↩︎
GPS signals#Time ↩︎
The most common version seems to be v1.20 (local archive) (Internet Archive) ↩︎
I’ve noticed the altitude is more accurate after the date was corrected, but I’m not sure why. ↩︎
The application should also work under WINE. ↩︎
While undefined in the RS-232 standard, most recievers on the PC platform considered zero voltage to be a logical “1”, allowing this to work. ↩︎

A modern, universal, Dreamcast power supply

terin@terinstock.com (Terin Stock) — Sun, 09 Jun 2024 16:46:00 +0000

I have fond memories of playing the Dreamcast growing up, of many hours trying to figure out Ecco the Dolphin in the days before ubiquitous home internet. A few years ago I picked up a Dreamcast and installed an optical disc emulator (ODE), and had a good time revisiting old games from my childhood. As a huge fan of Jet Set Radio Future on the original Xbox, it was fun to play Jet Grind Radio for the first time.

When I moved to the Netherlands I shipped over my consoles, including the Dreamcast. Unlike most of my other consoles, the Dreamcast used an internal power supply that was region-specific. As I had a North American console, it was expecting 120V while the power system in the Netherlands is 240V.

For convenience I didn’t want the bulk of a step-down converter whenever and wherever I wanted to play Dreamcast games. So I set out to replace the power supply instead.

Replacement Power Supplies

At first I looked at replacing the power supply with a SEGA OEM power supply for the European region. Unfortunately, I was unable to find a power supply on its own on eBay or the Dutch auction website Marktplaats, nor a “for parts” Dreamcast. Most of my options were system bundles, which made this approach uneconomical since I’d still need to replace the capacitors on the power supply.

I also looked at the PicoDreamcast, which uses the PicoPSU and an external 12V DC power brick. However, I wanted to avoid the extra bulk of an external power brick. This also ruled out the similar DreamPSU.

Mean Well RPT-6003

Discouraged by my options, I happened across 3DprintRC’s /r/Dreamcast post discussing the near perfect fit of Mean Well’s RPT-6003 power supply in the Dreamcast. This is a modern universal power supply designed by a company well-known for making high quality power supplies, and it provides all the voltages needed by the Dreamcast.

3DprintRC later posted a follow-up showing the RPT-6003 being used by cutting up the original power supply. Unfortunately, my Dreamcast had a different revision of the power supply which did not allow for such a clean cut.

Dreamcast RPT-6003

A populated Dreamcast RPT board.

Dreamcast RPT installed alongside the RPT-6003 in a Dreamcast

I designed the Dreamcast RPT, a small board that interfaces the RPT-6003 with the Dreamcast’s AC connector and power button. In my design I focused on being able to reuse many components already found on the OEM power supply, as well as documenting widely available modern alternatives.¹

I had boards manufactured here in Europe with AISLER and they arrived a few days later, looking great! It slotted into the Dreamcast case with no issue and worked the first time. 🥳

I’ve released the board as open hardware under the CERN-OHL-W 2.0 license. I designed it in Horizon EDA, an open source PCB design suite. I’d love to see others take and adapt the project. If you do, let me know what you build!

I’m still hoping to find a drop-in replacement for the AC connector. The connector interlocks with the case, which rules out most without a case modification. ↩︎

Creating Self-Hosted Tile Maps from OpenStreetMap Data

terin@terinstock.com (Terin Stock) — Mon, 19 Jun 2023 21:30:55 +0000

The work-in-progess homepage of NLAlerts, showing the most recent alerts.

I’ve been working on NLAlerts, a project to archive and display the mobile emergency alerts sent by the Netherlands government. The data provided by the government includes the text of the alert, along with targetted geographical area. With JavaScript libraries such as Leaflet or OpenLayers it is easy to turn this geographical area onto an interactive map.

These libraries can be used with external tile servers, such as the one provided by OpenStreetMap or one of the many commercial offerings. However, I wanted self-host all the data required for the website, which includes the map tiles. Rendered tiles can be stored in a SQLite database following the MBTiles Specification. Since the NLAlerts site already uses SQLite by way of being powered by data exploration framework Datasette, using MBTiles would be fairly complimentary. In fact, there’s already a plugin datasette-tiles to have Datasette act as a tileserver.

This just left one remaining problems: How do I get OpenStreetMap data into MBTiles? I found many guides online, but they seemed to fall into one of two camps: they were written many years ago and they suffer from linkrot or bitrot, or they ripped tiles from other servers. The former problem makes it hard to follow these guides in 2023, the latter is often against the terms of service (or rude to the remaining operators of non-profit tileservers).

This past weekend I was able to cobble together a working pipeline to “pre-render” tiles from OpenStreetMap extracts to MBTiles.

I first started by downloading the “osm.pbf” file for the Netherlands from Geofabrik. This file is the OpenStreetMap data in Protobuf format.

I then used the openstreetmap-tile-server container to import the “osm.pbf” file into PostGIS. I created volumes to store the database (osm-data) as well as where the tiles would later be rendered to (osm-tiles).

podman volume create osm-data
podman volume create osm-tiles
podman run -v $PWD/netherlands-latest.osm.pbf \
-v osm-data:/data/database \
-v osm-tiles:/data/tiles \
overv/openstreetmap-tile-server:2.3.0 \
import

After several minutes, the container should exit successfully and all the data will be imported. The same image could be used to run the renderer. I had to increase the shared memory configured in the container to avoid rendering errors later.

podman run -v osm-data:/data/database \
-v osm-tiles:/data/tiles \
-shm-size=1G \
overv/openstreetmap-tile-server:2.3.0 \
run

In another terminal we could exec into this container to start the pre-rendering.

podman exec -it --latest bash

Within the container, we can fetch the render_list_geo.pl script, which wraps the render_list command to render within a bounding box at different zoom levels. I then ran the script with the bounding box determined with the Bounding Box Tool for zoom levels 8 through 13.

wget https://raw.githubusercontent.com/alx77/render_list_geo.pl/master/render_list_geo.pl
perl ./render_list_geo.pl -z 8 -Z 13 \
-x 3.3316001 -X 7.2275102 \
-y 50.7503838 -Y 53.6316

If all went well, we can exit and stop the container, now having completed the hardest part. We have the map rendered in the osm-tiles, unfortunately not in a format we can directly use, but instead in meta tile, which can be efficiently used by mod_tile.

Fortunately, Geofabrik has a tool to help us out, meta2tile, which can be used to generate MBTiles databases. After cloning the repository, I built the tool in the most barebones configuration, only enabling MBTiles support.

gcc -DWITH_MBTILES meta2tile.c -o meta2tile -lsqlite3 -lcrypto -lm -O3

It’s worth noting that meta2tile uses the deprecated MD5 function from OpenSSL, which may stop being provided in a future version of OpenSSL. Fortunately, it’s also not the most difficult function to replace.

On my system Podman runs in rootless mode, with volumes being plain directories on disks. Thus I was able to run meta2tile directly against this directory. You may need to mount the volume or copy the data out first.

meta2tile --mbtiles \
--meta name="Netherlands" \
--meta type=raster \
--meta format=png \
--meta version=1.0 \
--meta bounds="3.3316001,50.7503838,7.2275102,53.6316" \
--meta description="OpenStreetMap tiles for Netherlands"
/home/terin/.local/share/containers/storage/volumes/osm-tiles/_data/default/ \
nl.db

I was then able to start Datasette with the datasette-tiles plugin installed and browse to http://localhost:8001/-/tiles/nl/ and browse my tiles.

The alert page showing a test alert sent, rendering the targetted area with OpenLayers.

When creating my template for the custom alert pages, I was able to reference "/-/tiles/nl/{z}/{x}/{y}.png" as the location of the tileserver. The result can be seen on this test alert.

Controlling the XM PCR receiver from Linux

terin@terinstock.com (Terin Stock) — Mon, 08 May 2023 20:04:12 +0000

The XM PCR in question, complete with mysterious screw holes.

The XM PCR was a satellite radio receiver sold to consumers in 2003 that was controllable by a PC running Windows over USB. Other than satellite, audio and USB connectivity, the receiver does not have any form of user interactivity, everything is done over USB. XM Radio discontinued it in 2004 due to software to easily rip songs, which was discussed on The Tech Guy #71.

It’s been nearly 20 years since the device was released, and the SiriusXM satellites are still broadcasting audio in a format this receiver can understand. I was shocked when the activation signal went through and the radio sprang to life. With the radio working, audio compression artifacts and all, I sought to control it from Linux.

The receiver uses an FTDI component to expose a serial port over USB. This is supported by modern Linux, and after attaching the receiver to the computer, a serial device was exposed under /dev. This was off to a promising start.

After a little bit of investigation I found that the protocol had already been reverse engineered and Michael Minn had released a GUI program MMXMPCR in 2003. It was last updated in 2005 which was a long time ago in the realm of Linux desktop software. I wasn’t even sure if modern distributions still shipped the required libraries, and if they did, if the program would even compile. Fortunately, Michael still hosts the source tarballs on their website; I downloaded the latest release to find out.

For the moment, Linux distributions still ship the necessary MOTIF and X Toolkit Intrinsics libraries. As desktop Linux continues the migration to Wayland, I wonder how much longer this will hold true. I modified the makefile to specify the correct header location and the linker flag for the Intrinsics library. I ran make and was greeted with a mmxmpcr binary, and a wall of compiler warnings.

Screenshot of mmxmpcr, showing the first 10 channels in the application window.

Upon running mmxmpcr, a window appeared which began populating with channel information from the radio, and I could control the receiver without issue. To remain cool, I tuned into TikTok Radio and attempted to dance.

In order to preserve the project into the future, I’ve used the available release tarballs to create the mmxmpcr repository on GitHub. It would be interesting to refactor the project to use libusb to discover an attached radio, rather than hardcoding a path into the binary.

However, the original Windows software distributed by XM Radio for this receiver seems missing from the Internet Archive. If you have it, please upload it and let me know!

HPSS-Disassembly Progress Report (May 2022)

terin@terinstock.com (Terin Stock) — Tue, 31 May 2022 16:16:16 +0000

Introduction

Title screen of Harry Potter and the Sorcerer’s Stone for the Game Boy Color

A little over 20 years ago the first movie in the Harry Potter series, Harry Potter and the Sorcerer’s Stone in the United States, was released by Warner Bros. I had been caught up in the Harry Potter mania and received the movie tie-in Game Boy Color game as a Christmas gift. I recall playing it for a while, before my attention returned to the other hype machine of 2001: the Pokémon series.

Harry Potter and the Sorcerer’s Stone on PC. Image credit Federico Dossena.

The PlayStation and PC versions of the tie-in game are 3D action-puzzle games where you play as the titular Harry Potter as he navigates around Hogwarts, the grounds, the Forbidden Forest, and a side quest to Diagon Alley. The PlayStation version is where we get the low-poly meme Hagrid from.

Harry Potter fighting bats in a Final Fantasy-inspired battle system.

The Game Boy Color version was completely different: a top-down Role Playing Game in the style of classic Final Fantasy. While you still play as Harry Potter, it follows the plot of the book much more faithfully than the 3D counterparts. There are some changes to adapt into a game: spells are learned for combat, items can be equipped for buffs, and wizard cards can be collected and used for combat effects.

I had entirely forgotten about this game until YouTube channel Flandrew put out a comparison of every version of Harry Potter and the Philosopher’s Stone¹, jogging back a flood of forgotten memories.

Picking it up again, I was excited by how colorful, expansive, and smooth the game felt, especially compared to Pokémon titles of the time. I decided I would learn more about how the game ticked.

Disassembly Beginnings

Disassembling Game Boy games isn’t new; Pokémon Red and Blue has been completely reverse engineered (this has prompted projects disassembling the rest of the Pokémon series). Other classics, such as Zelda: Link’s Awakening DX, have also been in progress for years.

I’ve started a project to disassemble Harry Potter and the Sorcerer’s Stone for the Game Boy Color. My goal is to document the techniques used to develop Game Boy games during the final years of the hardware lifecycle. I’m also planning on taking techniques learned from this project forward towards disassembling the sequel, Harry Potter and the Chamber of Secrets, which also holds the distinction of being the last Game Boy Color game released in North America.²

As I’ve never worked on disassembling a game before, I had to start from scratch. Fortunately, the Game Boy homebrew development scene is quite active, so modern tools are available to help get started.

mgbdis, the disassembler

To begin understanding what’s happening “behind the curtain” we need to turn the binary code the Game Boy Color CPU executes back into assembly code.

Unfortunately, the labels and symbols are not preserved in the binary code, so we’ll never be able to reproduce the exact structure the original game developers had for the game. However, we can use computers to make best guesses as a starting point, and use logic and reason (and our own guesses) to craft a structure back on top.

I used Matt Currie’s mgbdis disassembler to begin this task. Sorcerer’s Stone is a 4 MiB game, so disassembling took a while, but it ended up creating 256 files each representing 16 KiB of bankable memory. I also attempted to use Matt’s emulator, Beaten Dying Moon, to automate generating a symbol file to aid in separating executing code from data like images, but it did not seem very successful.

RGBDS, the assembler and linker

Once we have disassembled the game (even with our imperfect results) in order to get back to a playable game again we’ll need an assembler and linker. Fortunately we have RGBDS, an open source toolchain dating back to the 1990s.

RGBDS’s assembler, rgbasm, takes the assembly files as inputs and turned them into object files. It’s likely down the road we’ll have object files containing code that are logically related to each other, but for now we assemble each bank into it’s own object file.

RGBDS’s linker, rgblink, collects the object files together and decides how to combine them together into a Game Boy ROM. As the code is “fixed” in memory, the primary responsibility of the linker in this project is to resolve symbol references across object files, so the correct memory locations can be written into the ROM. As we cleanup the assembly files, the linker will likely become more important.

RGBDS includes a header fixer, rgbfix, used to generate a valid Game Boy header. The original hardware uses information in this header as checksums, to setup compatibility modes, and to implement basic DRM with the “Nintendo” logo. When the game was disassembled this header was partially decoded as instructions, and partially as data. It was removed and replaced by padding so rgbfix could be used instead. This allows for greater flexability in generating debug builds later, as the tool can generate the correct checksums later.

One other included tool is RGBDS’s image converter, rgbgfx. This is a tool for storing graphics as PNGs instead of 2bpp, an encoding more suitable for the Game Boy’s hardware. mgbdis did not separate the game’s images into individual files, so we won’t be using it for now, but it will be indespensible later once we’ve extracted the images.

The RGBDS project also documents the object file format, allowing for project specific tools to be written (if that proves to be necessary). Since Sorcerer’s Stone is a JRPG, there is a lot of dialog, menus, and world building, resulting in lots of text. The game also supports 11 different languages, farther multiplying the amount of text. I suspect we might need tooling to easily handle all of it.

gup, the recursive build system

After disassembling a game, mgbdis generates a basic GNU Make-compatible Makefile. This Makefile calls rgbasm over a single assembly file “game.asm” that simply includes all the banks. For a small game this might work, but for a large game like Sorcerer’s Stone re-assembling the entire game each rebuild was actually taking a significant amount of time. It would be far better to only reassemble the files that changed, and the targets that depend on that file.

I’ve switched the project to using gup, a recursive build system inspired by Daniel J. Bernstein’s redo. In gup, targets are executable scripts written in any language, and they can discover and register their own dependencies.

In HPSS-Disassembly, gup assembles banks by calling a script that calls rgbasm and registers each file it lists as a dependency with gup. When a file is changed, gup knows it only needs to rebuild banks that included it the previous build. This makes the testing iteration cycle extremely fast.

Disassembly Progress

I’ve written a lot of words here about the progress made on project infrastructure. What have I actually accomplished on the disassembly side? Since this is my first month working on the project, I’m afraid I haven’t accomplished too much.

Ready? Let’s Start

Error displayed when the game is inserted into a system not compatible with the Game Boy Color.

As Harry Potter and the Sorcerer’s Stone is a Game Boy Color-only game, the very first thing it does is check to see if it’s running on a Game Boy Color-compatible system. When it jumps to Start it compares the value left in the a register by the system boot ROM to $11. If the zero flag is set, it later jumps to code to show an error message.

Start::
 and a ; clear flags
 cp BOOTUP_A_CGB ; is Game Boy Color?

 ld a, $00 ; set a to 0
 jr nz, .notGBC ; if not GBC:
 inc a ; increment a (a=1)

.notGBC:
 ldh [$ef], a ; save GBC value
 ld sp, $cfff ; setup stack pointer
 ldh a, [$ef]
 or a
 call z, Unknown_Non_GBC ; call if not GBC

As this code saves the GBC status into High RAM, instead of directly jumping to code to display the error message, I wonder if at some point in development the game was targetting compatibility with the earlier Game Boy systems.

–floop-flatten

What do you do if you want quickly copy contiguous memory from multiple parts of the game, but part of the game needs to copy a different number of bytes, and you also want to keep the number of CPU cycles to a minimum? One approach taken by the Sorcerer’s Stone’s developers is to flatten the loop. You can call the same 3 instructions 32 times, then call into the code at whatever point has the required number of iterations remaining.

Fortunately, rgbasm is a macro assembler, allowing us to refactor this logic to generate the assembly for us, while also generating more readable names.

FOR V, 32, 0, -1 ; loop from 32 to 0, decrementing each time
CopyHL2DE_{d:V}: ; generate a label we can reference from other code
 ld a, [hl+] ; load the byte pointed to by hl into a,
 ; and also increment hl
 ld [de], a ; load a into the byte referenced by de
 inc de ; increment de
ENDR

This reduced over 200 lines of code into a much more manageable 6!

What’s Next

I’ve still have a lot of tasks ahead. Some immediate tasks to start working on:

Begin extracting tiles into bitmaps, and convert and assemble on-the-fly during builds.
Work on extracting text into forms that can be easier to work with, especially for translators.
Label and comment even more code.

In the first blog post for disassembling Zelda: Link’s Awakening DX, Pierre writes:

Reverse-engineering assembly code is quite slow, but I’ll try to post some findings on this blog.

The statement is just as true here as it was all those years ago. We’ll see how this goes!

Flandrew must be from outside the United States of America. ↩︎
According to Wikipedia five games released after Chamber of Secrets in other markets, 1 in Germany, 1 in Korea, and 3 in Japan, including the last licensed game Doraemon no Study Boy: Kanji Yomikaki Master. ↩︎

Systemd's clock-epoch for RTC-less systems

terin@terinstock.com (Terin Stock) — Sun, 26 Dec 2021 06:24:00 +0000

Earlier today I restarted a Raspberry Pi I use on my home network. When the system came up after reboot, many services failed to resume, which I quickly noticed due to the lack of DNS resolution on my network. Upon checking the system logs, the culprit became clear: CoreDNS was failing to verify the certificates of my DNS-over-TLS provider. According to my Raspberry Pi, we were partying like it was 1999. Without DNS resolution the system time would never automatically synchronize with NTP.

On a Raspberry Pi running Raspian or Raspberry Pi OS, the system is automatically configured to use fake-hwclock to save the time to disk from a cron, and restore it at boot. However, this package isn’t widely distributed outside Debian and it’s derivatives.

At boot, systemd compares the system time to a builtin epoch, usually the release or build date of systemd. If it finds the system time is before this epoch, it resets the clock to the epoch. Unfortunately, this is still sometimes too old, especially when it comes to TLS certificates.

We can still utilize this systemd feature however, at least if systemd is version 247 or newer, by utilizing a hook added to aid system administrators and image distributors. If the file /usr/lib/clock-epoch exists, systemd uses the modified time as the systemd epoch.

We can even script periodically updating this file. First, create /etc/systemd/system/set-clock-epoch.service:

[Unit]
Description=Updates the mtime of clock-epoch

[Service]
ExecStart=/bin/touch -m /usr/lib/clock-epoch

Then, create a timer to start it periodically, at /etc/systemd/system/set-clock-epoch.timer:

[Unit]
Description=Timer for updating clock-epoch

[Timer]
OnBootSec=5min
OnUnitInactiveSec=17min

[Install]
WantedBy=timers.target

I’ve arbitrarily delayed activation of this timer by setting OnBootSec, since this is low priority at boot time.

Then reload the daemon, then enable and start the timer.

systemctl daemon-reload
systemctl enable --now set-clock-epoch.timer

Now when the system reboots, systemd will read the modification time of this file during early boot, setting the time to a reasonably close value, allowing certificates to be validated and NTP to take over.

As gioele on Lobste.rs points out, on systemd version 250 and later, if you’re using systemd-timesyncd you can set SaveIntervalSec= to automatically write the time out to the filesystem. This follows a similar mechanism, the file /var/lib/systemd/timesync/clock has it’s modification time set, and is used to restore out reboot. The time is set by the epoch mechanism much earlier in the startup process than the timesyncd mechanism (which first requires service activation to begin). It also doesn’t help if, like me, you’re not using timesyncd.

Clearing Custom Search Engines in Chrome

terin@terinstock.com (Terin Stock) — Wed, 18 Aug 2021 09:20:00 +0000

In addition to your search engine in Google Chrome, the browser tries to be helpful by remembering custom search engines for websites you visit in the course of navigating the web. As some websites can become registered simply by navigating to their homepage, this can result in a very long list of custom search engines.

Unfortunately, the page to manage custom search engines in Google Chrome (currently located at chrome://settings/searchEngines) leaves a lot to be desired. This page has no mechanism for bulk editing. With over two hundred engines registered, I did not want to click my mouse to manually manage them.

Fortunately, we can activate super developer powers!

const nl = document.querySelector("settings-ui").shadowRoot
 .querySelector("#main").shadowRoot
 .querySelector("[role=main]").shadowRoot
 .querySelector("settings-search-page").shadowRoot
 .querySelector("settings-search-engines-page").shadowRoot
 .querySelector("#otherEngines").shadowRoot
 .querySelectorAll("settings-search-engine-entry");

Ah, right. So, the settings pages uses Shadow DOM, which makes getting to the actual list of custom search engines a bit of a chore. Query through a bunch of document fragments to get to the actual list.

const ops = Array.from(nl).map((elem) => elem.shadowRoot)
 .filter((elem) => {
 return !elem.querySelector("#keyword-column").innerText.startsWith("!");
 })
 .map((elem) => {
 return [elem.querySelector("#keyword-column").innerText, elem.querySelector("#delete")];
 });

console.table(ops);

I use the custom search engine feature to implement DuckDuckGo-style !Bang searches. So before removing engines, I filter out items starting with “!”.

If the table printed to the console looks good, click the virtual mouse.

ops.forEach(entry => entry[1].click());

Setting up a git server on NixOS

terin@terinstock.com (Terin Stock) — Wed, 27 Jan 2021 21:23:25 +0000

Over the years I’ve tried out many VPS providers, which has resulted in personal infrastructure spread far and wide. I’ve decided to start 2021 by consolidating this infrastructure, where reasonable. This also gives me an opportunity to properly document deployments, ideally making them easy to redeploy in the future. NixOS allows me to solve both goals relatively easily.

One piece of personal infrastructure is a git server, git.terinstock.com, which I use for personal configuration or smaller projects. Since I’m the only user, I don’t need the fancy interfaces provided by options like gogs, gitea, or GitLab. I’ve chosen to use cgit, which had a few more features I wanted over gitweb.

It did feel a little strange to deploy a cgi server in 2021. I think we call this “serverless” now?

cgit

NixOS already has cgit packaged, so we can start with configuring it.

let
 cgit = pkgs.cgit;
 cgitConfig = pkgs.writeText "cgitrc" (lib.generators.toKeyValue { } {
 css = "/cgit.css";
 logo = "/cgit.png";
 favicon = "/favicon.ico";
 about-filter = "${cgit}/lib/cgit/filters/about-formatting.sh";
 source-filter = "${cgit}/lib/cgit/filters/syntax-highlighting.py";
 clone-url = (lib.concatStringsSep " " [
 "https://$HTTP_HOST$SCRIPT_NAME/$CGIT_REPO_URL"
 "ssh://git@git.terinstock.com:$CGIT_REPO_URL"
 ]);
 enable-log-filecount = 1;
 enable-log-linecount = 1;
 enable-git-config = 1;
 root-title = "git.terinstock.com";
 root-desc = "Terin's Git Repositories";
 scan-path = "/srv/git";
 });

This assigns a derivation to cgitConfig that when evaluated would create the ini-like configuration file. Most of this comes down to personal preference with a few exceptions:

about-filter and source-filter reference the respective filters wpithin the cgit package. NixOS will expand these to full paths when creating the configuration file.
scan-path is the location on disk I’m using to host the git repositories.

git-shell

To allow for authenticated pushes, I use git-shell provided by the git project. This provides a minimal “shell” that an execute a prescribed list of git commands. I define a system user git and assign it this shell.

{
 users.users.git = {
 isSystemUser = true;
 description = "git user";
 home = "/srv/git";
 shell = "${pkgs.git}/bin/git-shell";
 openssh.authorizedKeys.keys = [
 "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIINNSIl3/j3KqW/x3kFj1ZvZlSxp+MDhk8LAIDlqs/9w"
 ];
 };
}

As I’m the only user that will be pushing to these repositories, I don’t need to configure anything extra for authenticating actions.

Web Server, h2o

Since cgit is a cgi application, it needs a web server to actually handle the HTTP connections. I’ve chosen to use h2o as it supports being an application proxy in addition to serving static files, while still being configurable with YAML/JSON.

let h2oFile = file: {
 "file.file" = "${file}";
 "file.send-compressed" = "ON";
 };
 h2oHeaderList = attrs: (lib.mapAttrsToList (k: v: "${k}: ${v}") attrs);
 h2oConfig = pkgs.writeText "h2o.conf" (lib.generators.toYAML { } {
 hosts."git.terinstock.com" = {
 paths = {
 "/cgit.css" = h2oFile ./cgit.css;
 "/cgit.png" = h2oFile "${cgit}/cgit/cgit.png";
 "/favicon.ico" = h2oFile "${cgit}/cgit/favicon.ico";
 "/robots.txt" = h2oFile "${cgit}/cgit/robots.txt";
 "/" = {
 "fastcgi.spawn" = "${pkgs.h2o}/share/h2o/fastcgi-cgi";
 setenv = {
 SCRIPT_FILENAME = "${cgit}/cgit/cgit.cgi";
 CGIT_CONFIG = "${cgitConfig}";
 };
 compress = "ON";
 };
 };
 "header.set" = {
 header = (h2oHeaderList {
 x-frame-options = "deny";
 x-xss-protection = "1, mode=block";
 x-content-type-options = "nosniff";
 referrer-policy = "no-referrer, strict-origin-when-cross-origin";
 cache-control = "no-transform";
 strict-transport-security = "max-age=63072000";
 content-security-policy = (lib.concatStringsSep "; " [
 "default-src 'none'"
 "style-src 'self' 'unsafe-inline'"
 "img-src 'self' data: https://img.shields.io"
 "script-src-attr 'unsafe-inline'"
 ]);
 expect-ct = "enforce, max-age=30";
 });
 };
 listen = {
 port = 443;
 ssl = {
 certificate-file = "/var/lib/acme/git.terinstock.com/fullchain.pem";
 key-file = "/var/lib/acme/git.terinstock.com/key.pem";
 min-version = "TLSv1.2";
 cipher-preference = "server";
 cipher-suite = (lib.concatStringsSep ":" [
 "TLS_AES_128_GCM_SHA256"
 "TLS_AES_256_GCM_SHA384"
 "TLS_CHACHA20_POLY1305_SHA256"
 "ECDHE-ECDSA-AES128-GCM-SHA256"
 "ECDHE-RSA-AES128-GCM-SHA256"
 "ECDHE-ECDSA-AES256-GCM-SHA384"
 "ECDHE-RSA-AES256-GCM-SHA384"
 "ECDHE-ECDSA-CHACHA20-POLY1305"
 "ECDHE-RSA-CHACHA20-POLY1305"
 "DHE-RSA-AES128-GCM-SHA256"
 "DHE-RSA-AES256-GCM-SHA384"
 ]);
 };
 };
 };
 });

Like with cgit’s configuration, h2oConfig is assigned a derivation that when evaluated will create a configuration file for h2o. In this configuration one host is defined and paths are defined for static assets, A fastcgi handler is configured for the root, which will be used for all paths not matched by a static file path.

h2o does not support cgi directly, but ships with a script to proxy through a fastcgi server. This wrapper is configured with the path to cgit’s and the cgit configuration file defined earlier.

This is more verbose than strictly neccessary, as I wanted to have high marks on Qualys’s SSL Labs report and Security Headers. I imagine in the future I’ll have refactored this into smaller functions and options.

The documented syntax for adding response headers involves setting the headers.set key multiple times. This is not representable in Nix, as Nix will not allow the same key to be set multiple times. An undocumented syntax using a YAML sequence instead has been available since 2.3.0-rc2.

header.set:
 header:
 - "x-frame-options: \"deny\""
 - "x-xss-protection: \"1, mode=block\""

Finally, I we need to start the h2o web server. At the time of writing, h2o does not have a NixOS module, but we can use lower-level modules ourselves.

{
 systemd.services.h2o = {
 description = "H2O web server";
 after = [ "network-online.target" "acme-git.terinstock.com.service" ];
 wantedBy = [ "multi-user.target" ];
 path = [ pkgs.perl pkgs.openssl ];
 serviceConfig = {
 ExecStart = "${pkgs.h2o}/bin/h2o --mode master --conf ${h2oConfig}";
 ExecReload = "${pkgs.coreutils}/bin/kill -s HUP $MAINPID";
 ExecStop = "${pkgs.coreutils}/bin/kill -s QUIT $MAINPID";
 User = "h2o";
 Group = "h2o";
 Type = "simple";
 Restart = "on-failure";
 AmbientCapabilities = "cap_net_bind_service";
 CapabilitiesBoundingSet = "cap_net_bind_service";
 NoNewPrivileges = true;
 LimitNPROC = 64;
 LimitNOFILE = 1048576;
 PrivateDevices = true;
 PrivetTmp = true;
 ProtectHome = true;
 ProtectSystem = "full";
 ReadOnlyPaths = "/var/lib/acme/";
 ReadWriteDirectories = "/var/lib/h2o";
 };
 };
}

This configures a systemd unit that will be enabled at startup. The ExecStart will start h2o without daemonizing, and provides the path to the evaluated h2o configuration. The PATH environment variable for the service will be updated to include Perl (for the fastcgi proxy) and OpenSSL (for OCSP stapling). I may submit a pull request to nixpkgs to fix the h2o derivation to reference these directly.

Additional settings are provided to the unit to configure systemd’s sandbox for the service.

{
 users.users.h2o = {
 group = "h2o";
 home = "/var/lib/h2o";
 createHome = true;
 isSystemUser = true;
 };

 users.groups.h2o = { };
}

Another system user is created, this time for h2o, along with a correspondng group used shortly.

ACME Certificates

NixOS ships with great support for using ACME to create TLS certificates, such as from Let’s Encrypt or your own certificate authority.

{
 security.acme = {
 # Set to true if you agree to your ACME server's Terms of Service.
 # For Let's Encrypt: https://letsencrypt.org/repository/
 acceptTerms = false;
 email = "terinjokes@gmail.com";
 certs = {
 "git.terinstock.com" = {
 dnsProvider = "cloudflare";
 credentialsFile = "/var/lib/secrets/cloudflare.secret";
 group = "h2o";
 };
 };
 };
}

To avoiding needing to configure h2o to proxy to lego for HTTP challenges, I prefer to use the DNS ACME challenges with lego’s support for Cloudflare DNS. Configuring lego to do DNS challenges is outside the scope of this post.

Housekeeping

I configure a few more NixOS options for general housekeeping that don’t fit in the above sections.

{
 environment.systemPackages = [ pkgs.git ];
}

I install git as a system package so it’s avialable should I need to log into the box to handle something.

{
 services.openssh.passwordAuthentication = false;
 services.sshguard.enable = true;
}

I disable OpenSSH’s password authentication mechanisms, as I have a strong preference to using more secure options. I also enable sshguard to block connections that try to log in with a password anyways.

{
 nix.gc = {
 automatic = true;
 options = "--delete-older-than 30d";
 };
 nix.optimise.automatic = true;
 system.autoUpgrade = {
 enable = true;
 allowReboot = true;
 };
}

Configures NixOS to perform routine maintaince in the background. This allows for NixOS to update at a regular interval, rebooting if needed to install kernel updates, as well as optimizing and garbage collecting the nix store.

While the goal in 2021 is to have less machines to manage, this ensures I don’t forget to install security patches because I haven’t logged in for a while.

Pebble and lego to test ACME with NixOS

terin@terinstock.com (Terin Stock) — Thu, 14 Jan 2021 22:04:00 +0000

When configuring a new service I want to run on NixOS, I often use nixos-shell to quickly standup a temporary VM locally with my new configuration. When this configuration is a service, I often want to ensure I have TLS configured properly, including the correct permissions on the certificate and key files managed by lego.

However, I don’t neccessarily want to send unacceptable amounts of traffic to production ACME servers, or deal with proper validation with their staging services. Fortunately, we can configure the NixOS VM to start a testing ACME server, Pebble, and configure Lego to use it.

Pebble

Pebble is a small, single-binary ACME server intended for testing. Keys and certificates are randomnized between calls, but this is fine for an emphermial VM.

First, we’ll want to configure Pebble to start, which we can do with the systemd.service NixOS option. I use the toJSON builtin function to create a JSON configuration file for Pebble from a Nix attribute set. I also reference the default certificate and key from the source package, as they are not yet copied to the output package.

{ pkgs }:

let
 pebbleConfig = pkgs.writeText "pebble.json" (builtins.toJSON {
 pebble = {
 listenAddress = "0.0.0.0:14000";
 managementListenAddress = "0.0.0.0:15000";
 certificate = "${pkgs.pebble.src}/test/certs/localhost/cert.pem";
 privateKey = "${pkgs.pebble.src}/test/certs/localhost/key.pem";
 httpPort = 5002;
 tlsPort = 5001;
 ocspResponderURL = "";
 externalAccountBindingRequired = false;
 };
 });
in {
 systemd.services.pebble = {
 description = "Pebble ACME Test Server";
 after = [ "network-online.target" ];
 wantedBy = [ "multi-user.target" ];
 serviceConfig = {
 Environment = [ "PEBBLE_VA_NOSLEEP=1" "PEBBLE_VA_ALWAYS_VALID=1" ];
 ExecStart = "${pkgs.pebble}/bin/pebble -config ${pebbleConfig}";
 DynamicUser = true;
 };
 };
}

You can modify some behavior of Pebble through environment variables. I set two for better behavior in a development VM:

PEBBLE_VA_NOSLEEP disables any artifical sleeps in the issuance path, as we’re not interested in testing lego’s validationg polling.
PEBBLE_VA_ALWAYS_VALID disables all validation methods, and assumes domains have already been successfully validated.

One warning from the Pebble documentation bares repeating here:

Pebble is NOT INTENDED FOR PRODUCTION USE. Pebble is for testing only.

lego

lego is a widely used ACME client that implements all of the ACME challenges, bindings for major DNS providers, and support for bundling certificates. It’s used as the implementation to NixOS’s security.acme options. This means most of the configuration is already done for us.

{
 security.acme = {
 server = "https://localhost:14000/dir";
 acceptTerms = true;
 email = "webmaster@example.com";
 certs = {
 "example.com" = {
 webroot = "/var/www/example.com";
 group = "prosody";
 };
 };
 };
}

Despite not needing to implement any challenges, as we’ve disabled them in Pebble, we still need to provide webroot configuration. The group attribute is used to set the group permission on the generated certificates and keys, and should be set to the same user your server is running as.

There’s one farther complication: lego does not trust the certificate authority used by the ACME server, and thus it won’t issue requests out of the box. We can configure the lego to trust these certificates by setting the environment of the generated service unit.

{
 systemd.services."acme-example.com".serviceConfig.Environment = [
 "LEGO_CA_CERTIFICATES=${pkgs.pebble.src}/test/certs/pebble.minica.pem"
 "LEGO_CA_SERVER_NAME=localhost"
 ];
}

Usage

The certificates and keys created by security.acme are stored underneath /var/lib/acme/. This can be provided to your server’s configuration.

{
 services.prosody = {
 enable = true;
 virtualHosts."example.com" = {
 enabled = true;
 ssl.cert = "/var/lib/acme/git.terinstock.com/fullchain.pem";
 ssl.key = "/var/lib/acme/git.terinstock.com/key.pem";
 };
 };
}

K9P: Kubernetes as 9P Files

terin@terinstock.com (Terin Stock) — Wed, 20 Nov 2019 11:55:00 -0800

K9P is a virtual filesystem that allows you to interact with Kubernetes clusters via a 9P filesystem, allowing us to continue to believe that everything is a file. Presented at KubeCon + CloudNativeCon North America 2019.

Building a Hardware MIDI Player

terin@terinstock.com (Terin Stock) — Sat, 02 Feb 2019 15:30:00 +0100

Building a MIDI player using Go and various Linux system interfaces, presented at FOSDEM 2019.

memfd_create: Temporary in-memory files with Go and Linux

terin@terinstock.com (Terin Stock) — Sat, 20 Oct 2018 23:21:27 +0000

My MIDI Player project requires interfacing with libraries and system calls that don’t yet have Go equivalents. A few of these C libraries expect to be invoked with file paths or file descriptors.

If the data is already available as a file on a mounted disk, this is fine, I can pass the path or descriptor that already exists. I run into a problem, however, as soon as I want to work with data that isn’t directly backed by a file, such as after being manipulated by the program, or having been unpacked from a container format.

There’s a simple way to get a file path: create a temporary file somewhere. This requires having a writable filesystem mounted, and cleaning up the files when no longer required. On a device that’s intended to be used as an appliance, I don’t have anywhere writable.

In Linux 3.17 and later, I have another option memfd_create(2):

memfd_create() creates an anonymous file and returns a file descriptor that refers to it. The file behaves like a regular file, and so can be modified, truncated, memory-mapped, and so on. However, unlike a regular file, it lives in RAM and has a volatile backing storage. Once all references to the file are dropped, it is automatically released.

This checks the boxes for most of my requirements: a file descriptor to something that acts like a normal file, backed by memory, doesn’t require any writable mount points, and automatically cleans up files when closed.

The system call is available in the Go x/sys/unix package, along with a few companion calls.

package main

import (
 "fmt"

 "golang.org/x/sys/unix"
)

// memfile takes a file name used, and the byte slice
// containing data the file should contain.
//
// name does not need to be unique, as it's used only
// for debugging purposes.
//
// It is up to the caller to close the returned descriptor.
func memfile(name string, b []byte) (int, error) {
 fd, err := unix.MemfdCreate(name, 0)
 if err != nil {
 return 0, fmt.Errorf("MemfdCreate: %v", err)
 }

 err = unix.Ftruncate(fd, int64(len(b)))
 if err != nil {
 return 0, fmt.Errorf("Ftruncate: %v", err)
 }

 data, err := unix.Mmap(fd, 0, len(b), unix.PROT_READ|unix.PROT_WRITE, unix.MAP_SHARED)
 if err != nil {
 return 0, fmt.Errorf("Mmap: %v", err)
 }

 copy(data, b)

 err = unix.Munmap(data)
 if err != nil {
 return 0, fmt.Errorf("Munmap: %v", err)
 }

 return fd, nil
}

Some libraries require a file path instead of a file descriptor. Fortunately, we can turn a file descriptor into a path by way of the proc filesystem using the symlinks in the /proc/self/fd directory.

package main

import (
 "fmt"
 "log"
 "os"
)

func main() {
 fd, err != memfile("hello", []byte("hello world!"))
 if err != nil {
 log.Fatalf("memfile: %v", err)
 }

 // filepath to our newly created in-memory file descriptor
 fp := fmt.Sprintf("/proc/self/fd/%d", fd)

 // create an *os.File, should you need it
 // alternatively, pass fd or fp as input to a library.
 f := os.NewFile(uintptr(fd), fp)
 defer f.Close()
}

Eventually, I’ll replace these C libraries with Go packages that operate on io.Reader, and this workaround will become unnecessary. Until then, I’m glad the option is available.

Hardware MIDI Player (Part One)

terin@terinstock.com (Terin Stock) — Mon, 17 Sep 2018 00:35:55 +0000

Overview

Raspberry Pi MIDI Player

Last December, I watched Techmoan’s YouTube video on the Roland MT-80S, a player built to help teach people how to play songs, that used HD 3.5" floppy disks as storage medium for MIDI files. Since then AkBKukU has done a video on the Yamaha Disk Orchestra DOU-10, that used DD disks instead.

As I enjoy listening to chiptune and DOS-era video game music, I was keen on having a player in my collection. Cursory searches for players sold online resulted in typically high eBay prices. Considering my growth in ability with electronics in the past year, I decided to forgo the market and build one myself.

The vision in my mind belongs in the mid-90s. My completed player shouldn’t feel out of place alongside a tape deck and CD changer. The insides, however, should be built from modern components.¹

Hardware

I began with the internal hardware, which is somewhere underneath all those wires.

I’ve built the project using Raspberry Pi 3B. This is significantly more powerful than required for playing back MIDI files.

In fact, I had originally planned to use an Arduino Uno as a rudimentary central processor and floppy drive controller. However, I was unable to reliably control any of the floppy drives shown here—they might not even work properly—and decided to switch to using a USB floppy drive that I could more easily verify. I switched to the Raspberry Pi because it is a better USB host than an Arduino.

Maccally USB Floppy Disk Drive for Mac

I settled on a rather generic USB floppy drive. Well, not just any USB floppy drive, but one “for Mac”. Based on the stripe design, I’m guessing it was meant to pair (or, pear) with the fruit-colored iBook G3 clamshells. Fortunately, the plastic case comes off simply enough.

This choice of drive proved to be troublesome as I would soon find myself on a pretty significant detour as a result.

I used the Protoboard hat from MakerSpot for the remainder of my hardware work. The Pi’s serial data and power lines are connected through to a 90s-era WaveBlaster-compatible 2x13² header. I’m used an absolutely tiny Dreamblaster S2 as a modern General MIDI synth. The output from the synthesis chip, a SAM2695, was routed to the Pi’s 3.5mm AV jack. A proper quarter-inch jack has been ordered.

Software

As a software guy, I expected the hardware to be the hard part. As Caesar would have said, “The fault, dear Brutus, is not in our stars, but in our software.”³

So far I’ve only written a rather trivial MIDI player. The path to an SMF or RIFF encoded MIDI file is provided as the command line argument, the software decodes and at the appropriate time sends the MIDI event to the S2 daughterboard for synthesis. I also confirmed manual playback of files on a floppy disk worked.

Shortly thereafter, I switched from Raspbian to a stripped down Linux distribution. Much to my horror, I found the floppy drive no longer worked. Any attempt to mount or otherwise interact with the floppy drive would completely freeze the running program, and no amount of SIGINT or SIGTERM would regain control. The floppy drive had to be disconnected from the Pi.

I spent the next several weeks debugging why the floppy drive stopped working. I started with debugging the floppy drive itself, then confirmed the power supply was outputting the correct voltages and was clean. I even recompiled the Linux kernel with different options!

Eventually, I discovered the drive worked with the 4.14 Linux kernel, but stopped working some point thereafter. Facing an 8-month range of changes to the kernel, I turned to git-bisect and started a week-long hunt for the breaking change. Since this range was so large, and the underlying files so different, setting up a compilation cache did not decrease the rebuild durations.

After at least 16 bisection points, I was lead to a single commit in the kernel driver for the USB controller used by the Raspberry Pi, the DesignWare USB2. In this change, a communication delay was added after multiple successive NAKs in the USB protocol, to allow low-speed Chromebooks (and other ARM-based devices) to do other work, instead of being tied up in an interrupt loop.

The commit requests a 1 millisecond delay, but uses the kernel’s low-resolution timer API. On devices where this timer is coarse, like the Raspberry Pi, this delay might actually be closer to 10-20 milliseconds. Fortunately, most USB devices tolerate delays in the communication. Unfortunately, this drive is not one that does so. This drive only tolerates delays of up to 5 milliseconds.

With some help from some of the maintainers of the module, I was able to use the kernel’s Ftrace functionality to be able to debug without losing the use of my serial TTY. I was then able to rapidly iterate with changes to the kernel module, and load the rebuilt module into the running kernel using rmmod and insmod.

I have submitted my first ever kernel patch⁴, to convert the delay to the high resolution API. This API has a finer timer, allowing the delay to operate much closer to the requested 1 millisecond on retries.

Next Steps

Complete Player Software

With my kernel patch in hand, I can now return to work on the MIDI player software. I want to be able to insert a floppy disk and, with one button, load all the MIDI files stored there into a playlist and began playback. Then I will add buttons for the other basic playback controls.

I’ve been happy with using go-krazy as the Linux distribution. Writing the MIDI player in Go has been pretty nice, and it makes the whole project feel fast.

I’m now battle testing the MIDI parser to make sure my player can withstand whatever is thrown at it. I hope to make this available as open source software by the time I publish the next part of this blog.

Thinking farther into the future, the following project would add a display for album and track metadata.

Custom PCB and Case

I also aim to design a custom PCB for the synthesizer, buttons, display, and RTC, as well as integration with a power supply that can signal the Pi to shutdown before removing power.

I haven’t yet decided if I’ll do this PCB as a custom hat, or, since I’m not using the majority of the hardware functions, as a board housing the Raspberry Pi Compute Module. The decision here affects the case design.

Like the software, I plan on releasing both the PCB and the case designs as

MIDI Albums

To really sell the illusion that, on an alternate timeline for the 90s, we bought our pre-recorded music as MIDI on floppies, I’d love to commission musicians to compose some awesome tracks that can be packaged onto a floppy. Let me know via email if this appeals to you.

Except, perhaps, the floppy drive itself. ↩︎
There are no 12V rails in the current design, as the WaveBlaster-compatible, which we’re able to talk about, doesn’t require it. For compatibility, I’m wanting to include them in the final design. ↩︎
If not Caesar, then perhaps Peter van Houten. ↩︎
At this time, it is still pending review for a future kernel release. ↩︎

Black-box Serial Testing

terin@terinstock.com (Terin Stock) — Fri, 27 Jul 2018 17:43:56 +0000

Occasionally, we all end up working on a program that could, at best, be described as a ball of mud. We want to refactor it into smaller, more testable components, but we don’t want to risk breaking it in the process.

Enter the concept of “black-box testing”. In black-box testing, the functionality of a program is tested by a tester, without the ability to directly leverage implementation details of the program.

For some programs, all we need to do is give it input, and observe the output. As the software grows in complexity, we may have to compare created files, setup mock web servers, and test databases. Many the great author and many the great blog have written on these problems.

The “ball of mud” side project I’m currently working on communicates over a serial port. This presents a larger challenge for a tester: how can we observe behavior of the application with the serial port from our test? Pointing the program at a file won’t work, as it wants to connect to a remote device and configure options, such as baud rate and parity bits.

The long evolutionary history of UNIX provides us with one solution. In the early days of UNIX, the output from the computer was printed onto serially connected teleprinter. When technology marched on, terminal emulators replaced the physical teleprinter, and the functionality needed to create these pseudo-teleprinters was added to the kernel.

As these teleprinters were originally serially connected, their modern virtual forms retain their serial nature.

In our test harness, we can create a new pseudo-teleprinter then pass the emulated end to our program as the serial port. This example uses the Go test harness, but equivalent functionality is available in most programming and scripting languages, so you can use your favorite.

package main

import (
 "io/ioutil"
 "os/exec"
 "testing"

 "github.com/google/go-cmp/cmp"
 "github.com/pkg/term/termios"
)

func TestCLI(t *testing.T) {
 // create and open the pseudo tty master, and the linked
 // child tty.
 pty, tty, err := termios.Pty()
 if err != nil {
 t.Errorf("unable to create pty: %s\n", err)
 }

 // execute our program, passing in the child tty name (eg, /dev/pts/1)
 // as our serial port paramater.
 cmd := exec.Command("ball-of-mud", "-serialport", tty.Name())
 output, err := cmd.Output()
 if err != nil {
 t.Errorf("error running ball-of-mud: %s\n", err)
 }

 // close the child tty. if we forget to do this the
 // master pseudo tty will wait for more data forever.
 tty.Close()

 // read all the written serial data from the master
 // pseudo tty. ReadAll reads until EOF or an error is
 // returned, but Linux sends EIO when attempting to read
 // from a master with no open children. So let's just
 // look for the error string.
 bytes, err := ioutil.ReadAll(pty)
 if err != nil && err.Error() != "read ptm: input/output error" {
 t.Errorf("error reading pty: %s\n", err)
 }

 // confirm we saw the expected text from standard out
 if diff := cmd.Diff("hello", string(output)); diff != "" {
 t.Errorf("unexpected output (-want +got):\n%s", diff)
 }

 // confirm we saw the expected bytes written to the serial device
 if diff := cmd.Diff([]byte{"world"}, bytes); diff != "" {
 t.Errorf("unexpected serial (-want +got):\n%s", diff)
 }
}

As the two parts of the pseudo-teleprinter are connected to each other, writes to one will be readable from the other side. Thus duplex communication is possible for more complex black-box tests.

This method isn’t entirely perfect. For example, on Linux, it’s not always possible to observe the configured baud rate. Even with these limitations, it’s still a helpful tool in our toolkit.

Controller: Extending your K8s cluster

terin@terinstock.com (Terin Stock) — Fri, 04 May 2018 14:45:00 +0200

Building your own Kubernetes Controllers to extend the functionality of your cluster, presented at KubeCon + CloudNativeCon Europe 2018.

Go Internalization and Localization

terin@terinstock.com (Terin Stock) — Wed, 02 May 2018 19:00:00 +0200

An introduction to Go’s text package, which allows developers to build localized applications and services. Presented at Go Cph on May 2, 2018.

Extending Kubernetes

terin@terinstock.com (Terin Stock) — Tue, 01 May 2018 18:50:00 +0200

Talk on extending Kubernetes Dyanmic Admission control with webhooks, presented at GOTO Cph in Copenhagen. This talk was given as being related to our talk at KubeCon + CloudNativeCon Europe.

Game Boy Collage #1

terin@terinstock.com (Terin Stock) — Mon, 13 Nov 2017 05:34:31 +0000

For a few months now, I’ve wanted to work on some new old-school artwork: creating a collage from the internals of a classic Nintendo Game Boy. Similar to the How Things Work series of books, each component of the Game Boy would have annotations and detailed schematics. Unlike those books, the Game Boy would continue to work. After all, if it doesn’t work, it’s not very fun.

8-bit handhelds

Although I’ll only be taking apart one, I have three 8-bit handhelds.

Game Boy (DMG) playing Smurfs

On the left is the original gray Game Boy. It sports a super reflective black and green screen, guzzling 4 AA batteries, and a big boxy shape.

Game Boy Color (GBC) playing Mario Tennis

To the right in the center is a lime green Game Boy Color, capable of up to 32K colors, and backward compatible with the original Game Boy. I have fond memories playing on my lime green Game Boy Color growing up. I can’t even guess how many hours I put on it.

GB Boy Colour playing Mario Tennis

Finally, on the rightmost side, is the GB Boy Colour, a 2010s clone of the Color. The screen is the wrong aspect ratio and the wrong resolution. The custom processor runs the games at the wrong clock speed. The audio hardware, while loud, is a poor emulation of the original. It does, and this is the saving grace, have a backlight LCD screen.

My understanding is that this uses a custom System on Chip design to emulate the Game Boy Color. I haven’t taken it apart, but I think it will be interesting to do so in a future post.

Backside of DMG. Serial Number G27198639

Backside of DMG with the battery compartment visible

Moving back to the original, gray, Game Boy unit, before we take it apart, I should take some pictures of the outside. The serial number and service stickers aren’t in pristine condition, but they’re still in pretty decent shape! I’d like to repair both labels; I’m looking into the best repair methods.

Left side of the DMG

Right side of the DMG

Bottom side of the DMG

The sides look great, too. I’m surprised that the condition is this good. It includes a 3.5mm headphone jack, as electronics should.

Backside of the LCD board in the case

Backside of the main board in the case

After popping it open, I find that it’s relatively dust free. The Game Boy separates into two main pieces, connected by a ribbon cable. The top board contains the LCD screen, input controls, and contrast knob. The bottom board houses the main CPU, coprocessors, audio amplifiers, volume knob, and cartridge loader. Two daughterboards are attached to the bottom board, one for power regulation, the other for the 3.5mm jack.

Front of the main board, along with the power and audio jack boards

The manufacturing date of both boards is July 1992, based on the date codes stamped. That means these capacitors are at least 25 years old. Since most manufacturers rate their capacitors for 15-20 years, I should replace them before this project gets to far along. The boards are otherwise in great shape.

Inside of the front case

Unfortunately, on closer inspection, this unit has two broken standoffs on the front case. I don’t think it’s a problem: you won’t be able to see them in the final design.

DMG's battery contacts are covered in acid

Battery acid left behind in DMG's battery compartment

The biggest problem is the corrosion in the battery compartment. All the terminals were pretty corroded. They were pretty easy to pop out, but lots of battery acid stayed behind in the case.

A quick aside to the kitchen. For Science!

Silly me, I had no terminal cleaner available, nor any other acidic household cleaner. So I improvised a bit: I had an extra lime in the kitchen from a previous meal. Lime juice is a citric acid with a pH about the same as lemon juice and vinegar, so I figured it would work just as well. It’s also fitting, because, you know, both color handhelds are lime green.

No more acid remains

After cleaning, it appears the battery acid etched some of the shininess away. But it otherwise cleaned up nicely. I cleaned up the battery acid in the case using a cotton swab along with the lime juice.

That’s it for now. In the next installment, I will share the plan for how breaking up the Game Boy and wiring it together. I also have ideas for future enhancements I think would be cool. I’ve also ordered a set of replacement capacitors; I’ll do a post on replacing them once they arrive.

Aliasing JavaScript Modules

terin@terinstock.com (Terin Stock) — Tue, 20 Jun 2017 17:45:55 +0000

A number of years ago I wrote a blog post detailing how to replace modules in a Browserify bundle. In the interceding years, the JavaScript ecosystem has had at least three lifetimes, and we’re now due for an update.

Browserify

As the last post focused entirely on Browserify, I’ll first revisit it first. As noted in the previous post, the browserify-swap transform is designed to swap individual files, not modules. We were able to work around it by defining the configuration as a RegExp, but it’s easy to get wrong. You can now use the aliasify transform, which does support replacing modules.

After installing, enable the transform, and add a new section to your package.json.

{
 "browserify": {
 "transform": [
 "aliasify"
 ]
 },
 "aliasify": {
 "aliases": {
 "underscore": "lodash"
 }
 }
}

In addition to aliasing modules, aliasify supports replacments that are relative to the file calling require, as well as using RegExps to define replacement.

Webpack

Webpack supports aliasing modules right out of the box, no plugins needed. Just add an additional section to your Webpack configuration.

{
 "resolve": {
 "alias": {
 "underscore": "lodash"
 }
 }
}

Babel

You can also setup aliases in Babel with the babel-plugin-module-resolver plugin. Like the above, just add a section to your Babel configuration.

{
 "plugins": [
 ["module-resolver", {
 "root": ["."],
 "alias": {
 "underscore": "lodash"
 }
 }]
 ]
}

Yarn

Finally, you can alias before dependencies are even written to disk if you use Yarn¹. Since it breaks compatibility with npm, it’s best if you only use this method in private applications, not public applications or in libraries.

In this method you combine the alias with defining your dependencies in package.json.

{
 "dependencies": {
 "underscore": "npm:lodash"
 }
}

Unlike the other methods in this guide, using the Yarn method also allows you to install the same dependency at different versions by giving them different names.

{
 "dependencies": {
 "lodash3": "npm:lodash@^3",
 "lodash4": "npm:lodash@^4"
 }
}

Unfortunately, it seems this isn’t yet in the documentation. It has, however, been a “Yarn tip”. ↩︎

A rose by any other name…

terin@terinstock.com (Terin Stock) — Wed, 15 Mar 2017 06:10:00 -0700

I gave the following Browserify talk at the Webpack edition of NodeSource’s monthly meetup. Webpack’s Sean Larkin had given his “Core Concepts” talk before I went on, and I made a comparison to Browserify’s concepts.

While my talk is chiefly satirical, I get serious towards the end: Webpack is powerful because of how configurable it is. On the other hand, people like Browserify because it’s opinionated, so you can focus on building your project. I hope to see improvements to Webpack in 2017 that make it more accessible.

TLS with Erlang

terin@terinstock.com (Terin Stock) — Wed, 02 Jul 2014 00:00:00 +0000

I recently configured an HTTP server written in Erlang for secure communication with Transport Layer Security (TLS), successor to Secure Sockets Layer (SSL). Unfortunately, my attempt resulted in TLS errors from both browsers and command line tools. Determined to find a solution, I dug into the HTTP server and Erlang’s SSL library to resolve these TLS connection failures.

In the process I uncovered issues with intermediate bundles and elliptic curve selections, as well as a configuration optimization.

Bundling Intermediate Certificates

When you buy a certificate for your site, you’re likely to also receive an intermediate bundle. This intermediate bundle is a chain of certificates that bind the certificate issued for your site to a trusted certificate located on the user’s computer or browser (a “root certificate”). If this bundle is excluded, the browser won’t trust the connection, since it can’t verify each link of the chain.

The issuer reminds you to add this bundle to your server’s configuration—it is important not to forget this step! Erlang’s SSL library can be configured to include the intermediate bundle by providing the path to the cacertfile option.

ssl:start().
{ok, ListenSocket} = ssl:listen(443, [
 {certfile, "/full/path/to/server_cert.pem"},
 {keyfile, "/full/path/to/server_key.pem"},
 {cacertfile, "/full/path/to/bundle.pem"}
 ]).
ssl:transport_accept(ListenSocket).

Erlang will now send the entire certificate chain to the browser during the connection, and the browser can trust the connection.

Ode to Debugging TLS

At this point, I expected to be done. Unfortunately, while OpenSSL and TLS scanning tools such as sslyze would connect fine, my copies of Chrome, Firefox, and curl refused to connect with cryptic SSL errors such as ERR_SSL_CLIENT_AUTH_SIGNATURE_FAILED and sec_error_invalid_key. Chrome’s debugging logs provided no additional information.

TLS Handshakes: A Primer

To start a connection to a secure site, the client and browser must first configure the secure connection in a process commonly called a “TLS Handshake”.

In a full handshake, two roundtrips between the browser and the server are required:

The browser sends a ClientHello message to the server.

This message includes the highest TLS version supported, lists of supported cipher suites and compression algorithms, and other TLS extensions, including a list of known named elliptic curves.
The server responds with ServerHello, Certificate, an optional ServerKeyExchange message, and ServerHelloDone messages.

The ServerHello message details specific options used for the connections including the TLS version, the cipher suite, the compression algorithm, and any additional TLS extensions. The TLS version should be the highest version support by both client and server. The server maintains a priority list for cipher suites and compression algorithms, and it selects the highest priority supported by the browser.

The server then attaches the entire certificate chain in a Certificate message, so the browser can verify the authenticity of the connection.

If the Certificate message doesn’t contain enough information to allow a client to exchange a session key, such as with a Diffie–Hellman key exchange, then a ServerKeyExchange message is included.

The server then ends with ServerHelloDone.
The client responds with a ClientKeyExchange, ChangeCipherSpec and Finished messages.

The ClientKeyExchange message contains a secret key determined by both sides using public key encryption. The specifics of how the session key is generated is outside the scope of this primer.

With the ChangeCipherSpec message, the browser tells the server to switch to encrypted communication for the rest of the communication.

To verify the integrity of the communications up to this point, a hash of the previous messages is taken and sent as part of the Finished message. The server will also compute a hash over the same messages and compare.
Finally, the server sends ChangeCipherSpec and Finished messages.

The server verifies the hash sent in the client’s Finished message, then acknowledges the encryption communications and finishes, sending a similar hash to the client.

The Failed TLS Handshake

Interested in understanding the exact exchange between my browser and the server, I logged the TLS traffic with the network protocol analyzer Wireshark. The browser and the server successfully exchanged ClientHello, ServerHello, Certificate, ServerKeyExchange, and ServerHelloDone messages before the browser unexpectedly closed the connection.

Inspecting the ServerHello message informed me the server agreed on using TLS 1.2 and choose the TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA cipher suite. Both supported by the browser.

From RFC4492, when the ECDHE_ECDSA, ECDHE_RSA, or ECDH_anon ciphers are chosen, a ServerKeyExchange message is sent containing the ECDHE public key used to derive the shared key.

The ServerKeyExchange portion of the TLS handshake from the server to the client is replicated below.

0000 0c 00 01 49 03 00 16 41 04 b2 33 23 71 c9 da 80
0010 94 d3 ec eb 05 9f e5 36 91 a7 e2 e5 40 78 aa 03
0020 38 4f eb 7c 36 1b 92 21 58 cf c3 e5 b7 08 40 5a
0030 6a eb d2 6a 22 90 e0 47 28 ce 70 9b bb 87 17 d3
0040 4a bc 7c 78 14 ef 97 0d 0d 02 01 01 00 91 7e 3c
0050 ce 9f 06 1d 00 47 4f 53 85 df 2e 04 31 9a 14 a3
0060 25 bd 51 b3 1a 0f dd 3b c3 f4 25 b0 23 d5 34 0a
0070 a3 fc 2a e2 08 34 29 87 00 91 0e 10 6a 40 b3 b5
0080 61 0c 77 9a 8a 0c 50 dc 78 57 ab 2a 51 66 d0 0d
0090 b7 3c 4d c0 28 b9 06 b4 f5 f6 48 f6 5a 02 c2 7e
00a0 8f b2 ac 4b 03 3a 40 c0 e2 c6 2f 77 61 58 ea 0d
00b0 ab 6c 7f 57 be e1 03 0b c6 1e 2a b0 67 ab c2 db
00c0 0a 5b c4 ab 51 9a 76 e6 75 2d e6 ca ce 06 4b f5
00d0 8f dc f0 c1 42 65 14 c0 79 80 51 f2 68 3b 4a 51
00e0 0d 50 5a 01 32 e3 5c 8d cd 8c ec c1 c4 fa 84 3a
00f0 33 37 4c 9d d5 54 f9 6c aa b8 27 27 7b 4a 7c 33
0100 27 8e 48 48 33 87 73 11 9b 92 0b e3 99 49 23 7b
0110 c5 ab 53 ef f2 86 df 56 e5 97 6b 2d 93 5f c0 8a
0120 e6 68 4f 6b 3a 1b 55 26 08 aa c0 36 74 21 ed cc
0130 0e c9 22 0b 97 51 c1 01 48 3f 01 d2 74 fe 36 18
0140 5f 5c 91 47 b3 19 1c 00 69 7f 17 1b c3

0c indicates this message is a ServerKeyExchange message
00 01 49 is the length of 0x000149 (in decimal, 329) bytes
03 is the elliptic curve type, in this case “named_curve”
00 16 is the named curve, secp256k1
The ECDHE public key and signature follows.

At this point the browser verifies the signature and retrieves the elliptic curve parameters and ECDHE public key from the ServerKeyExchange message.

Section 5.4 of RFC4492 ends with the following note:

A possible reason for a fatal handshake failure is that the client’s capabilities for handling elliptic curves and point formats are exceeded

While Erlang supports all 25 elliptic curves named in RFC4492, common browsers only support a smaller subset of two or three.¹ In the above snippet, we see that Erlang chose secp256k1, the elliptic curve used in Bitcoin, and not one supported by my browsers.

Erlang’s early support of elliptic curves are problematic. When picking an elliptic curve, Erlang does not consider the list of supported curves sent by the browser. This has been resolved with the Erlang R16R03-1 release.

Configuration of TLS and Ciphers

Erlang’s SSL library has defaults for the TLS versions, cipher suites and renegotiation behavior. You may want to change these options for client compatibility and for resiliency to TLS attacks.

ssl:start().
{ok, ListenSocket} = ssl:listen(443, [
 {server_renegotiate, true},
 {versions: [ "tlsv1.1", "tlsv1.2" ]},
 {ciphers: [ "ECDHE-ECDSA-AES128-SHA256", "ECDHE-ECDSA-AES128-SHA" ]}
 ]).
ssl:transport_accept(ListenSocket).

CloudFlare publishes the cipher suites they use with nginx. You can check the ciphers supported by your Erlang installation by running the following in a erl session.

rp(ssl:cipher_suites(openssl)).

I created a patch for CouchDB that adds the configuration options secure_renegotiate, ciphers, and tls_versions to the SSL section:²

[ssl]
certfile = /full/path/to/server_cert.pem
keyfile = /full/path/to/server_key.pem
cacertfile = /full/path/tp/bundle.pem
secure_renegotiate = true
tls_versions = [ "tlsv1.1", "tlsv1.2" ]
ciphers = [ "ECDHE-ECDSA-AES128-SHA256", "ECDHE-ECDSA-AES128-SHA" ]

While presented through the lens of an HTTP server in Erlang, the same basic steps could be extrapolated to any secure server written in any language. Recapping my debugging process: First, I ensured the server is configured to send the entire certificate chain to the client. Then, I tested the connection with a TLS scanner or a network protocol analyzer. Finally, once the server is properly communicating, I took a look at my server’s TLS configuration to ensure it is secure and reflect current best practices.

Edit: An earlier version of this post claimed browsers only supported three elliptic curves: secp192r1, secp224r1, and secp256r1. This information was incorrectly included from an earlier draft. ↩︎
Edit: My patch did not land in CouchDB 1.6.0. CouchDB has since been reorganized into multiple repos, but my patch continues to be in place for a future stable release. ↩︎

Replacing packages in a Browserify bundle

terin@terinstock.com (Terin Stock) — Thu, 27 Feb 2014 00:00:00 +0000

As a developer on a large Backbone application built with Browserify, there are a number of occasions where I want to replace one dependency with another. In this specific case, I wanted to swap underscore for lodash.

Browserify already supports this with the “browser field” in package.json.

There is a special “browser” field you can set in your package.json on a per-module basis to override file resolution for browser-specific versions of files.

This only works for resolution within your package, if any of your dependency packages require Underscore they’ll get Underscore. This is to help ensure your replacements don’t break your dependencies.

However, it’s suboptimal to ship both Lo-Dash and Underscore, as is maintaining a fork simply to replace the dependency. In these edge cases, it’s useful to replace files or packages even within dependencies.

Luckily, the Browserify transform browserify-swap allows you swap dependencies in certain packages, as defined via the @packages key, while generating the output bundle.

As I want to replace Underscore in Backbone, Marionnette and related packages, the configuration seemed pretty straight-forward.

/* package.json */
{
 "browserify": {
 "transform": [
 "browserify-swap"
 ]
 },
 "browserify-swap": {
 "@packages": [
 "backbone",
 "marionette",
 "backbone.babysitter",
 "backbone.wreqr"
 ],
 "all": {
 "underscore.js$": "lodash"
 }
 }
}

I was a bit discouraged to find that Underscore was still present in the output bundle. After triple-checking that my configuration was valid, I broke out the node debugger to find what was wrong.

I believed browserify-swap to swap files while resolving the require calls. The transform actually checks if the current file path matches a RegEx defined in the package.json file and replaces the contents to require the swapped in file.

The Solution

With this information in hand, it became clear that we needed to swap in the underscore package.

/* package.json */
{
 "browserify": {
 "transform": [
 "browserify-swap"
 ]
 },
 "browserify-swap": {
 "@packages": [
 "underscore"
 ],
 "all": {
 "underscore.js$": "lodash"
 }
 }
}

This causes the swap to happen for each instance of Underscore in the bundle, but only the one instance of Lo-Dash would be included.

I believe that the transform should also allow swapping packages for other packages, as the folder structure of a package is usually not part of the public API. I’ve opened an issue against the project.