Adventures with my old iPod Touch

The iPod Touch (4th Gen) in the car could no longer be detected, so we pulled it out of the console to find it in DFU mode.  Yikes.

I had extremely little hope, but I took it and its USB-A to 30-pin dock cable (the only extant cable of this type in my collection) inside, plugged it into a USB-C to USB-A adapter, and plugged it into the Mac.  It… er… worked. Sure, it appeared to open in Finder and not iTunes (rip), but it actually worked (on the second try; the advice for “unknown error 9” is to try again.  What are we doing as a profession.)  I was able to restore iOS 6.1.6 to it, although I did not have the option to keep my data.

But, I never moved my music onto the Mac.  I figured, with a hardware device that is fifteen years old, and back in factory state, surely, Linux should be able to sync to it?

The first problem was even getting it connected, because Amarok threw an error from ifuse.  Copying the command out of the error message and running it in a terminal worked totally fine.  (I didn’t think of this at the time, but… Amarok logs in the systemd journal.  Maybe its permissions have been stripped down too far.)

Once that was up and running, I restarted Amarok a couple of times, before I found out where it had hidden the iPod.  It’s under “Local Collection.”

I then waited a long time for things to sync.  I waited so long that I wandered off and forgot to set “Don’t Sleep”, so the computer suspended.  The iPod made its ancient, discordant glissando when the computer woke up, and then Amarok—and any process trying to stat() the ifuse mount point—froze.  ifuse sat there burning 100% CPU for a couple of minutes, and then I restarted.

(Apparently the sleep interval was fifteen minutes, the longest time that doesn’t make KDE System Settings complain about ’using more energy.'  Well… I paid for it, one way or the other.)

I got it going again.  Amarok carefully loaded gigabytes of tracks onto the iPod Touch, then started complaining about checksum errors for the database.  That’s the part that makes them useful, instead of having the iPod show “No content” and a button for the iTunes Store.  That ended up being the final boss that I couldn’t beat.  The tracks are still there, apparently, showing up as “Other” data on the Mac.

Yeah.

I plugged the backup drive into the Mac, imported everything, and exported it to the iPod Touch.  The double copy was orders of magnitude faster than Amarok’s unidirectional efforts.  I should never have been so lazy.

• Free Software: 0
• Proprietary OS: 2

I don’t know how we got here.

Podman Desktop Isn’t Great

Since buying a new computer, my primary desktop is no longer Linux, and containers are no longer native.  I decided that the path of least resistance would be to try Podman Desktop, but it leaves a lot to be desired.

The Ruthless Elimination of Differences

I am excited for image-based Linux.  Yes, I usually complain about people upending things just when they get stable, but I think there’s a real benefit here: the elimination of differences.

Why, exactly, does installing Ubuntu have to unpack a bunch of .deb files inside a system? Thousands or millions of machines will go consume CPU to run maintainer scripts, to hopefully produce identical output, when most of the desired result should have been possible to save as an image in the first place.  Upstream should know what’s in ubuntu-minimal!  Looking through a different lens, Gentoo distributes a stage2 image.

In theory, an installation CD could carry the minimal image, the installer overlay, and the flavor’s overlay.  The installer’s boot loader would bring up the kernel, use the minimal+installer pair as root file system, and the installer would unpack the minimal+flavor images into the new disk partition.

“Image-based Linux” more or less takes this one more step, running the entire system directly from the images (or a singular combined image.)  Everyone gets to use the same pre-made images, and bugs become less dependent on the history of package operations.

If any of this sounds like Puppy Linux, that’s not entirely accidental.

This is also the space where things like ABRoot are being introduced.  Image-based Linux lends itself well to having an integrated rollback/recovery pathway. Even on my non-image systems, having “a recovery partition” has been more valuable than I ever anticipated.  It let me test backups without having to work very hard about simulating a disaster. I also created my own recovery partition when I was still using a RealTek USB WiFi device, to avoid being stranded without internet.  (Word to the wise: use Mediatek instead, or an Intel PCIe card is a good non-USB option.)

Image-based Linux and the tools around it are poised to make real improvements to the repeatability and reliability of the systems.  I don’t know when I, personally, might benefit (as my daily driver is macOS now), but I am very excited about the progress being made here.

Reminiscing about fvwm

Inspired by Chris Siebenmann talking about his setup and reminiscing about MGR, let’s jump in the Epoch and set the dial for 2002.

I was in college, absolutely blown away by the customizability of X11.  You could have a “full desktop” like CDE on the Suns, or KDE/Gnome on my home computer (variously Linux and FreeBSD); or else you could choose to run with only a window manager.  Those ranged from quasi-desktops like WindowMaker and Fluxbox, down to minimalist options like Ratpoison (so named because it was intended to be used without the mouse), with what I can only describe as “normal” options in between.

In this milieu, I found fvwm2 and really dug into that.  The configuration makes it more of a “window manager construction kit” than a fully-defined window manager.  (In contrast, one could barely do anything but theme Metacity.)  I put my window title bars down the left side, because it let me have ten more pixels of vertical space!  And, let’s be honest, just because I could.

I had keyboard shortcuts for everything.  Nobody in Unix space used Macs in those days; we were busy calling the iMac a “lampshade.” Therefore, Linux applications all used ctrl/alt/shift for shortcuts, and left the Windows key (Super) free for my window management shortcuts.  Oh, and I think I drew my own icons, so I could have Amiga-style raise/lower buttons in the title bars.

I also found out I had limits.  I configured a 3x3 grid of virtual desktops each with 3x3 pages on them, but even “9 places to look for windows” was absurdly oversized.  In practice, I only ever used 3 pages on 1 desktop.  By the time I wrote Layer Juggling, I had forgotten about fvwm’s “pages” layer entirely.

I had a great run with fvwm and WindowMaker for 2–4 years there, but it soon became clear I was trading features (like having a volume control, or instantly applying themes) for an environment that nobody else could use.  Meanwhile, I was losing familiarity with Windows, which would slow me down if I were using anyone else’s computer.  I switched to KDE by 2004, and I would eventually, relucantly, capitulate and try Gnome again in 2008.  More or less just in time for Gnome and Canonical to blow everything up again!

Ironically, over time, there has been less need to use other peoples’ computers.  Besides which, I have kept using Dvorak for 20+ years now, despite that being a much bigger issue when switching systems.

Residual Config Without Config Files

apt makes a distinction between “removed” and “purged.” In both, the packages are uninstalled; in the former state, config files remain, and in the latter, those are also removed.  Actually, that’s not quite the whole story.

A package can have no configuration files, yet still be in ”residual config” state when removed.  This happens if a package defines a postrm maintainer script. These can have basically any shell commands in them, so their actions aren’t visible in any list-of-files.

The specific package I was looking into was a library, with a postrm script that ran ldconfig… during removal.  The package was being shown in residual-config state because it had a script.  Although that script would do nothing during purge, apt (and dpkg) can’t know that.

How to list residual-config packages: apt list 2>/dev/null | grep residual-config or dpkg -l | grep ^rc.

Listing configuration files: try one of these answers as this gets real complex, real fast.

Reading a postrm script: look at /var/lib/dpkg/info/{PACKAGE}[:{ARCH}].postrm (the ARCH component may not be present.)

What I Learned Trying to Install Kubuntu (alongside Pop!_OS)

First and foremost, once again, this is clearly not a supported configuration that I tried to make.  I'm sure that if I wiped the drive and started afresh, things would have gone much better.  I just… wanted to push the envelope a bit.

Pop!_OS installs (with encryption) with the physical partition as a LUKS container, holding an LVM volume group, and the root filesystem is on a logical volume within.  The plan was hatched:

• Create a logical volume for /home and move those files over to it
• Create a logical volume for Kubuntu’s root filesystem
• Install Kubuntu into the new volume, and share /home for easy switching (either direction)

Things immediately got weird.  The Kubuntu installer (calamares) knows how to install into a logical volume, but it doesn’t know how to open the LUKS container.  I quit the installer, unlocked the thing, and restarted the installer.  This let the installation proceed, up to the point where it failed to install grub.

Although that problem can be fixed, the whole installation ended up being irretrievably broken, all because booting Linux is clearly not important enough to get standardized. Oh well!

Ubuntu 24.10 First Impressions

I hit the button to upgrade Ubuntu Studio 24.04 to 24.10.  First impressions:

1. The upgrade process was seriously broken.  Not sure if my fault.
2. Sticky Keys is still not correct on Wayland.
3. Orchis has major problems on Ubuntu Studio.

Pulling at threads: File Capabilities

For unimportant reasons, on my Ubuntu 24.04 installation, I went looking for things that set file capabilities in /usr/bin and /usr/sbin.  There were three:

• ping: cap_net_raw=ep
• mtr-packet: cap_net_raw=ep
• kwin_wayland: cap_sys_resource=ep

The =ep notation means that only the listed capabilities are set to “effective” and “permitted”, but not “inheritable.”  Processes can and do receive the capability, but cannot pass it to child processes.

ping and mtr-packet are “as expected.”  They want to send unusual network packets, so they need that right.  (This is the sort of thing I would also expect to see on nmap, if it were installed.)

kwin_wayland was a bit more surprising to see.  Why does it want that?  Through reading capabilities(7) and running strings /usr/bin/kwin_xwayland, my best guess is that kwin needs to raise its RLIMIT_NOFILE (max number of open files.)

There’s a separate kwin_wayland_wrapper file.  A quick check showed that it was not a shell script (a common form of wrapper), but an actual executable.  Could it have had the capability, set the limits, and launched the main process?  For that matter, could this whole startup sequence have been structured through systemd, so that none of kwin’s components needed elevated capabilities?

The latter question is easily answered: no.  This clearly isn’t a system service, and if it were run from the user instance, that never had any elevated privileges.  (The goal, as I understand it, is that a systemd user-session bug doesn’t permit privilege escalation, and “not having those privileges” is the surest way to avoid malicious use of them.)

If kwin actually adjusts the limit dynamically, in response to the actual number of clients seen, then the former answer would also be “no.”  To exercise the capability at any time, kwin itself must retain it.

I haven’t read the code to confirm any of this.  Really, it seems like this situation is exactly what capabilities are for; to allow limited actions like raising resource limits, without giving away broad access to the entire system.  Even if I were to engineer a less-privileged alternative, it doesn’t seem like it will measurably improve the “security,” especially not for cap_sys_resource.  It was just a fun little thought experiment.

Scattered Thoughts on Distrobox

Distrobox’s aim is to integrate a container with the host OS’ data, to the extent possible.  By default, everything (process, network, filesystem) are shared, and in particular, the home directory is mounted into the container as well.  It is not even trying to be “a sandbox,” even when using the --unshare-all option.

I also found out the hard way that Distrobox integrates devices explicitly. If some USB devices are unplugged, the container will not start.  This happened because I pulled my laptop away from its normal dock area (with USB hub, keyboard, and fancy mouse) and tried to use a distrobox.  Thankfully, I wasn’t fully offline, so I was able to rebuild the container.  [Updated 2024-11-28: This danger is persistent.  Creating a container without the USB devices connected, then running it later with the devices, will make it fail to start if the devices are unplugged.  This ended up being impossible to live with.]

Before it stopped performing properly in Ubuntu Studio 23.10, I used distrobox to build ares and install it into my home directory.  This process yielded a ~/.local/share/applications/ares.desktop file, which my host desktop picked up, but which would not actually work from the host.  I always needed to be careful to click the “ares (on ares)“ in the menu after exporting, to start it on the ares container.

I have observed that distroboxes must be stopped to be deleted, but then distrobox will want to start them to un-export the apps.  Very recent distrobox versions ask whether to start the container and do the un-exporting, but there’s still a base assumption that you wanted the distrobox specifically to export GUI apps from it.  It clearly doesn’t track whether any apps are exported, because it always asks, even if none are.

The Wrong Terminal

Somewhere in my Pop!_OS 22.04 settings, I set Tilix as the preferred terminal emulator. When I use the Super+T* keyboard shortcut, I get a Tilix window.  However, when I use a launcher that Distrobox has created for a container from the Super+A (for Applications) UI, the command-line doesn’t come up in Tilix… it comes up in gnome-terminal instead. Why is that, and can I fix it?

AIUI, all the Application Launcher UI does is ask the system to open the .desktop file that Distrobox added.  That file has the “run in terminal” option, but lacks the ability to request some specific terminal. That gets chosen, eventually, by the GIO library.

GIO isn’t desktop-specific, so it doesn’t read the desktop settings.  It actually builds in a hard-coded list of terminals that it knows how to work with, like gnome-terminal, konsole, and eventually (I am assuming) ye olde xterm.  It walks down the list and runs the first one that exists on the system, which happens to be gnome-terminal.  AFAIK, there is no configuration for this, at any level.

It is also possible that one of the distributions in the chain (Debian, Ubuntu, or Pop!_OS) patched GIO to try x-terminal-emulator first.  If so, it would go through the alternatives system, which would send it directly to gnome-terminal, since between that and Tilix, gnome-terminal has priority.  We are deep into speculative territory, but if all of that were the case, I could “make it work” by making a system-level change resulting in all users now preferring Tilix… but only for cases where x-terminal-emulator is involved, specifically.

I want the admin user to have all the defaults, like gnome-terminal, because the less deviation made in that account, the less likely I am to configure a weird problem for myself.** (Especially for Gnome, which has a configuration system, but they don’t want anyone to use it.  For simplicity.) Changing the alternatives globally is in direct contradiction to that goal.

It seems that the “simplest” solution is to change the .desktop file to avoid launching in a terminal, and then update the command to include the desired terminal.  It would work in the short term, but fall “out of sync” if I ever changed away from Tilix as default in the desktop settings, or uninstalled Tilix.  It’s not robust.

It seems like there’s some sort of desktop-environment standard missing here.  If we don’t want to invoke threads or communication inside GIO, then there would need to be a way for the Gnome libraries to pass an “XDG Configuration” or something, to allow settings like “current terminal app” to be passed in.

If we relax the constraints, then a D-Bus call would be reasonable… but in that case, maybe GIO could be removed from the sequence entirely.  The Applications UI would make the D-Bus call to “launch the thing,” and a desktop-specific implementation would pick it up and do it properly.

It seems like there should be solutions, but from searching the web, it looks like the UX has “just done that” for years.

* Super is the |□| key, because it's a System76 laptop, a Darter Pro 8 in particular.

** As a side effect, this makes the admin account feel like “someone else’s computer,” which makes me take more care with it.  I may not want to break my own things, exactly, but I would feel even worse about breaking other people’s stuff.

Reverting Flatpaks

Today, I learned a lot about Flatpak, motivated by Thunderbird giving me an error at startup.  The error message said that I had used the profile with “a newer version of Thunderbird,” and now it was not compatible.  It gave me the choice of creating a new profile, or quitting.  This was incredibly confusing, since I simply ran flatpak update as usual this morning, and took whatever was there.

That turned out to have been 115.13.0.  By the end of the day, it was “resolved” in the sense that a new version was published on Flathub (128.0esr) and that version was capable of opening my profile.

In the meantime, I learned more commands:

flatpak history

flatpak history produces a list of install/update activity, but it does not have version information at all. We can at least get the commit by asking for it specifically:

flatpak history --columns time,change,app,commit

Or with a sufficiently large terminal window (1920+ pixels wide), try using --columns all.

flatpak remote-info --log

It turns out that the remote can—and Flathub in particular does—keep older versions around for “a while.”  We can get these older versions, with their full commit IDs, by using flatpak remote-info with the repository and the package name.

flatpak remote-info --log flathub \
    org.mozilla.Thunderbird | less

(Line wrapped for readability on mobile; remove the backslash and put it all on one line.)

This prints out some nice header information, then the commit, followed by History.  For Thunderbird in particular, as I write this, the top 3 are:

• Commit: 2131b9d618… on 2024-07-17 16:26:34 +0000
• Commit: c2e09fc595… on 2024-07-16 18:58:52 +0000 (this is the one I installed this morning, about 2024-07-17 13:00:00 +0000.)
• Commit: 2151b1e101… on 2024-07-11 18:18:41 +0000 (which I installed 2024-07-12)

I opened a second terminal window, so I could copy and paste the full commit IDs between them, while experimenting.

sudo flatpak update --commit

Now that we have our old version, how do we install it?  Let’s assume the most-recent version wasn’t published yet, and I just wanted to roll back to my version from 2024-07-12.  We’ll pass its hash to the update command, and run it with root privileges:

sudo flatpak update --commit=2151b1e101… \
    org.mozilla.Thunderbird

Of course I did it without sudo at first, but after confirming, it failed, stating I needed root privileges.  I guess it makes sense (they don’t want someone who doesn’t know my password to downgrade to a known-vulnerable app and then exploit it) but I’m also miffed that it couldn’t tell me this before confirmation.

Anyway, one quick test later, I had my email again.

Versions in Flatpak

After the rollback, I checked the Thunderbird version through Help → About: it was “128.0esr, Mozilla Flatpak version 1.0”.

I followed up with a plain flatpak update org.mozilla.Thunderbird to get the latest 128.0esr build, and verified that was able to access my email as well.  I checked the version again in Help → About: it was “128.0esr, Mozilla Flatpak version 1.0”.

That’s why flatpak update and flatpak history don’t have version numbers at all.  They don’t have any guarantees of clarity or accuracy.

What I didn’t learn

I might have been able to give the short commit ID (from flatpak history) directly to flatpak update without going through flatpak remote-info --log in between.  I didn’t actually try it.

I kept trying to find information about branches, to see if there was a Thunderbird beta branch I could try since stable was broken, but I never did find any information about that.  There’s some build-related documentation about how to specify the branch during build, but absolutely nothing about listing available branches.

I also didn’t find anything about this situation in web searches.  How did version 115.x get pushed after 128.x?  Why did it take 21 hours to get it fixed?  Where would I find out whether Mozilla even knew about the problem?  I discovered it around 15:00, and couldn’t tell if anyone else was having the issue!

There’s a “Subject” in the flatpak remote-info --log data for each commit, but it is invariably “Export org.mozilla.Thunderbird”, so that didn’t add any signal, either.

Revisiting Backups

Since comparing DejaDup and Pika Backup for work, I’ve also used KDE backup/kup at home, and gotten a little more experience with both systems.  How are things going?

Pika Backup

Pika has a firm internal idea of the schedule.  At the end of a vacation, where the normal weekly backup was skipped, I manually asked it to run the backup “6 days late / 1 day early”, hoping I could skip it the next day.  No such luck: it promptly asked for the drive the next morning.  No big deal; I just didn’t know the details of the system.

One small file recovery was quick and effective through the GUI.  It looked like a special folder in the normal file explorer, which allowed for drag-and-drop copying from the backup to the desired location in another file window.  As usual when coding, I had done something, changed my mind, deleted it, and then changed my mind back later.  And probably changed a couple more times.

KDE Backup

KDE Backup had been set up as a redundant system behind my handcrafted (smaller, faster, non-versioned) script, the latter of which was—er, intended—to store only my critical data.

A few months after a trial-by-fire backup test, I discovered that ~/.gnupg was not in my manual backup.  This could have been a critical fault… but KDE Backup had the data.  Clicking the “Restore” button opened a “File Digger” window, and from there it was just like using Pika Backup.

I got my key back!

I added the missing ~/.gnupg directory to manual backup, then reconfigured KDE Backup to back up into the existing repository.  That went smoothly, too.  It noticed that the directory I gave it already had data, so it verified the integrity, and then backed up into it.

Format-wise, KDE uses kup to do the backup, which is a front-end to bup, storing the data as a bare git repository.  I didn’t need a password to get data out of it, which makes sense, because I didn’t need one to set up the backup.  That’s also great news for actually recovering my data, because I have no idea what I would have chosen for a password when I started KDE Backup in the first place.

Conclusion

Both systems are working great, i.e., better than my manual one.

My ssh/sshd Configurations

Let’s look at my SSH configurations!

File Layout

Starting with Ubuntu 22.04 LTS and Debian 12, the OpenSSH version in the distribution is new enough that the Include directive is supported, and works properly with Match blocks in included files.  Therefore, most of the global stuff ends up in /etc/ssh/sshd_config.d/01-security.conf and further modifications are made at higher numbers.

Core Security

To minimize surface area, I turn off features I don’t use, if possible:

GSSAPIAuthentication no
HostbasedAuthentication no
PasswordAuthentication no
PermitEmptyPasswords no

AllowTcpForwarding no
X11Forwarding no
Compression no

PermitUserRC no
# Debian and derivatives
DebianBanner no

Some of these are defaults, unless the distribution changes them, which means “explicit is better than implicit” is strongly advised.

Next, I use a group to permit access, allowing me to explicitly add the members to the group without needing to edit the ssh config when things change.  Don’t forget to groupadd ssh-users (once) and gpasswd -a USER ssh-users (for each user.) Then, permit only that group:

AllowGroups ssh-users
# extra paranoia
PermitRootLogin no

Note that all of the above may be overridden in Match blocks, where required. TCP forwarding may also be more finely controlled through PermitListen and PermitOpen directives.

Note also that my systems are essentially single-user.  The group doesn't permit any sharing (and doesn't participate in quotas or anything) that would otherwise be forbidden.

Performance

Machines I use for ssh and sshd are all amd64, so for personal usage, I bump the AES algorithms to the front of the list:

Ciphers ^aes256-gcm@openssh.com,aes256-ctr

SFTP

The biggest trouble is the SFTP subsystem.  I comment that out in the main config, then set it in my own:

# /etc/ssh/sshd_config:
#Subsystem sftp ...

# /etc/ssh/sshd_config/02-sftp.conf:
Subsystem sftp internal-sftp
Match group sftp-only
    # ForceCommand, ChrootDirectory, etc.

I forget the details of what goes in that Match block.  It’s work stuff, set up a while ago now.

[Updated 2024-08-15: It seems that Subsystem is permitted inside a Match block as of OpenSSH 9.5, which is included in Ubuntu 24.04 LTS.  My statements above apply to 22.04, which uses version 8.9; or to Debian 12, with 9.2.]

Ongoing Hardening

I occasionally run ssh-audit and check out what else shows up.  Note that you may need to run it with the --skip-rate-test option these days, particularly if you have set up fail2ban (guess how I know.)

There are also other hardening guides on the internet; I have definitely updated my moduli to only include 3072-bit and up options.  Incidentally, if you wonder how that works:

awk '$5 >= 3071' ...

The default action for awk is print, so that command prints lines that fulfill the condition.  The fifth field is the length of the modulus, so that’s what we compare to.  The actual bit count is 3071 instead of 3072, because the first digit must be 1 to make a 3072-bit number, so there are only 3071 bits that aren’t predetermined.

Client Config Sample

Host site-admin
    # [HostName, Port, User undisclosed]
    IdentityFile ~/.ssh/id_admin
    IdentitiesOnly yes

Host 192.168.*
    # Allow talking to Dropbear 2022.83+ on this subnet
    KexAlgorithms +curve25519-sha256,curve25519-sha256@libssh.org
    MACs +hmac-sha2-256

Host *
    Ciphers aes256-gcm@openssh.com,aes256-ctr
    KexAlgorithms sntrup761x25519-sha512@openssh.com
    MACs hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com
    GSSAPIAuthentication no

It’s mostly post-quantum, or assigning a very specific private key to the administrative user on my Web server.

Using tarlz with GNU tar

I have an old trick that looks something like:

$ ssh HOST tar cf - DIR | lzip -9c >dir.tar.lz

The goal here is to pull a tar from the server, compressing it locally, to trade bandwidth and client CPU for reduced server CPU usage.  I keep this handy for when I don’t want to disturb a small AWS instance too much.

Since then, I learned about tarlz, which can compress an existing tar archive with lzip.  That seemed like what I wanted, but naïve usage would result in errors:

$ ssh HOST tar cf - DIR | tarlz -z -o dir.tar.lz
tarlz: (stdin): Corrupt or invalid tar header.

It turned out that tarlz only works on archives in POSIX format, and (modern?) GNU tar produces them in GNU format by default.  Pass it the --posix option to make it all work together:

$ ssh HOST tar cf - --posix DIR | \
    tarlz -z -o dir.tar.lz

(Line broken on my blog for readability.)

Bonus tip: it turns out that GNU tar will auto-detect the compression format on read operations these days.  Running tar xf foo.tar.lz will transparently decompress the archive with lzip.

Things I learned Reinstalling My Ubuntu

I did not want to wait for Ubuntu Studio 24.04 to be offered as an update to 23.10, so I got the installer and tried it.  Also, I thought I would try repartitioning the disk as UEFI.

Brief notes:

• I did not feel in control of manual partitioning
• I found out one of my USB sticks is bad, thanks to F3…
• …and no thanks to the Startup Disk Creator!
• If the X11 window manager crashes/doesn’t start, goofy things happen
• Wayland+KWin still don’t support sticky keys, smh
• snap remove pops up the audio device overlay… sometimes repeatedly
• I depend on a surprising amount of configuration actually

No More Realtek WiFi

The current Debian kernel (based on 6.1.66, after the ext4 corruption mess) seems to be locking up with the Realtek USB wireless drivers I use.  Anything that wants the IP address (like agetty or ip addr) hangs, as does shutdown.  It all works fine on the "old" kernel, which is the last version prior to the ext4 issue.

Meanwhile in Ubuntu 23.10, the in-kernel RTW drivers were flaky and bouncing the connection, so I had returned to morrownr’s driver there, as well.  But now that I don’t trust any version of this driver?  Forget this company.  In the future, I will be using any other option:

1. A Fenvi PCIe WiFi card with an Intel chip on board, or the like
2. Using an extra router as a wireless client/media bridge, with its Ethernet connected to the PC
3. If USB were truly necessary, as opposed to simply “convenient,” a Mediatek adapter

Remember that speed testing and studying dmesg output led me to the conclusion that this chipset comes up in USB 2.0 mode, and even the Windows drivers just use it that way.  While morrownr’s driver offers the ability to switch it to USB 3.0 mode under Linux, this prevents it from being connected properly.  I never researched hard enough to find out if there is a way to make that work, short of warm rebooting again so that it is already in USB 3.0 mode.

It’s clearly deficient by design, and adding injury to insult, the drivers aren’t even stable.  Awful experience, one star ★☆☆☆☆, would not recommend. Intel or Mediatek are much better choices.

Addendum, 2024-01-13: I purchased an AX200-based Fenvi card, the FV-AXE3000Pro.  It seemed not to work at all.  In Windows it would fail to start with error code 10, and in Linux it would fail to load RT ucode with error -110.  And then, Linux would report hangs for thermald, and systemd would wait forever for it to shut down.  When the timer ran out at 1m30s, it would just kick up to 3m.

Embarrassingly enough, all problems were solved by plugging it into the correct PCIe slot.  Apparently, despite being physically compatible, graphics card slots (which already had the punch-outs on my case, um, punched out) are for graphics cards only.  (My desktop is sufficiently vintage that it has two PCIe 3.0 x16 slots, one with 16 lanes and one with 4 lanes, and two classic PCI slots between them.)

Result: my WiFi is 93% faster, matching the WAN rate as seen on the Ethernet side of the router.  Good riddance, Realtek!

Update on earlyoom

Back in Linux Behavior Without Swap, I noted that the modern Linux kernel will still let the system thrash.  The OOM killer does not come out until it is extremely desperate, long after responsiveness is near zero.

It has been long enough since installing earlyoom on our clouds that I did another stupid thing.  earlyoom was able to terminate the script, and the instance stayed responsive.

I also mentioned “swap on zram” in that post.  It turns out, the ideal use case for zram is when there is no other swap device. When there’s a disk-based swap area (file or partition), one should activate zswap instead.  zswap acts as a front-end buffer to swap, storing the compressible pages, or letting others go to the swap device.

One other note, zswap is compiled into the default Ubuntu kernels, but zram is part of the rather large linux-modules-extra package set.  If there’s no other need for the extra modules, uninstalling them saves a good amount of disk space.

Upgrading a debootstrap'ped Debian Installation

For context, last year, I created a Debian 11 recovery partition using debootstrap for recovering from issues with the 88x2bu driver I was using.

This year, I realized while reading that I have never used anything but Ubuntu’s do-release-upgrade tool to upgrade a non-rolling-release system.  My Debian 12 desktop felt far less polished than Ubuntu Studio, so I reinstalled the latter, and that means I once again don’t need the 88x2bu driver.

Therefore, if I trashed the Debian partition, it wouldn’t be a major loss.  It was time to experiment!

Doing the upgrade

The update process was straightforward, if more low-level than do-release-upgrade.  There are a couple of different procedures online that vary in their details, so I ended up winging it combining them:

1. apt update
2. apt upgrade
3. apt full-upgrade
4. apt autoremove --purge
5. [edit my sources from bullseye to bookworm]
6. apt clean
7. apt update
8. apt upgrade --without-new-pkgs
9. apt full-upgrade
10. reboot
11. apt autoremove

DKMS built the 88x2bu driver for the new kernel, and userspace appeared to be fine.

Fixing the Network

The link came up with an IP, but the internet didn’t work: there was no DNS.  I didn’t have systemd-resolved, named, dnsmasq, nor nscd.  Now, to rescue the rescue partition, I rebooted into Ubuntu, chroot’ed to Debian, and installed systemd-resolved.

Fixing Half of the Races

One of the Debian boots left me confused.  Output to the console appeared to have stopped after some USB devices were initialized.  I thought it had crashed.  I unplugged a keyboard and plugged it in, generating more USB messages on screen, so I experimentally pressed Enter.  What I got was an (initramfs) prompt!  The previous one had been lost in the USB messages printed after it had appeared.

It seems that the kernel had done something different in probing the SATA bus vs. USB this time, and /dev/sdb3 didn’t have the root partition on it.  I ended up rebooting (I don’t know how to get the boot to resume properly if I had managed to mount /root by hand.)

When that worked, I updated the Ubuntu partition’s /boot/grub/custom.cfg to use the UUID instead of the device path for Debian.

It seems that the kernel itself only supports partition UUIDs, but Debian and Ubuntu use initrds (initial RAM disks) that contain the code needed to find the filesystem UUID.  That’s why root=UUID={fs-uuid} has always worked for me!  Including this time.

os-prober (the original source of this entry) has to be more conservative, though, so it put root=/dev/sdb3 on the kernel command line instead.

The Unfixed Race

Sometimes, the wlan0 interface can’t be renamed to wlx{MAC_ADDRESS} because the device is busy.  I tried letting wlan0 be an alias in the configuration for the interface (using systemd-networkd) but it doesn’t seem to take.

I resort to simply rebooting if the login prompt doesn’t reset itself and show a DHCP IP address in the banner within a few seconds.

You have to admire the kernel and systemd teams’ dedication to taking a stable, functional process and replacing it with a complex and fragile mess.

A Brief Flirtation with X11

I installed Cinnamon.  It ran out of space; I ran apt clean and then continued, successfully.  This is my fault; I had “only” given the partition 8 GiB, because I expected it to be a CLI-only environment.

Cinnamon, however, is insistent on NetworkManager, and I was already using systemd-networkd.  It’s very weird to have the desktop showing that there is “no connection” while the internet is actually working fine.

Due to the space issue, I decided to uninstall everything and go back to a minimal CLI.  I would definitely not be able to perform another upgrade to Debian 13, for instance, and it was unclear if I would even be able to do normal updates.

In Conclusion

The Debian upgrade went surprisingly well, considering it was initially installed with debootstrap, and is therefore an unusual configuration.

Losing DNS might have been recoverable by editing /etc/resolv.conf instead, but I wasn’t really in a “fixing this from here is my only option” space.  Actually, one might wonder what happened to the DHCP-provided DNS server?  I don’t know, either.

Trying to add X11 to a partition never designed for it did not work out, but it was largely a whim anyway.

Sound for Firefox Flatpak on Ubuntu 23.04 with PipeWire

I reinstalled Ubuntu Studio recently, excised all Snaps, and installed Firefox from Flatpak.  Afterward, I didn’t have any audio output in Firefox.  Videos would just freeze.

I don’t know how much of this is fully necessary, but I quit Firefox, installed more PipeWire pieces, and maybe signed out and/or rebooted.

sudo apt install pipewire-audio pipewire-alsa

The pipewire-jack and pipewire-pulse packages were already installed.

AIUI, this means that “PipeWire exclusively owns the audio hardware” and provides ALSA, JACK, Pulse, and PipeWire interfaces into it.

It’s not perfect.  Thunderbird (also flatpak; I’d rather have “some” sandbox than “none”) makes a bit of cacophony when emails come in, but at least there’s sound for Firefox.

Installing Debian 12 and morrownr's Realtek driver (2023-07-03 edit)

Due to deepening dissatisfaction with Canonical, I replaced my Ubuntu Studio installation with Debian 12 “bookworm” recently.

tl;dr:

1. My backups, including the driver source, were compressed with lzip for reasons, but I fell back on a previously-built rescue partition to get the system online.
2. I ended up with an improper grub installation, that couldn’t find /boot/grub/i386-pc/normal.mod. I rebooted the install media in rescue mode, got a shell in the environment, verified the disk structure with fdisk -l, and then ran grub-install /dev/___ to fix it. Replace the blank with your device, but beware: using the wrong device may make the OS on it unbootable.
3. The USB doesn’t work directly with apt-cdrom to install more packages offline. I “got the ISO back” from dd if=/dev/sdd of=./bookworm.iso bs=1M count=4096 status=progress conv=fsync (1M * 4096 = 4G total, which is big enough for the 3.7G bookworm image; you may need to adjust to suit), then made it available with mount -o loop,ro ~+/bookworm.iso /media/cdrom0 (the mount point is the target of /media/cdrom.)
4. Once finished, I found out the DVD had plzip, and if I’d searched for it (lzip), I could have used it (plzip). I didn’t actually need the rescue partition.
5. Once finished, I realized I hadn’t needed to dd the ISO back from the USB stick. The downloaded ISO was on my external drive all along, and I could have loop-mounted that.
6. [Added 2023-07-02]: Letting the swap partition get formatted gave it a new UUID. Ultimately, I would need to update the recovery partition’s /etc/fstab with the new UUID, and follow up with update-initramfs -u to get the recovery partition working smoothly again.

Full, detailed rambling with too much context (as usual) below.