So I used graphviz to knock out a state diagram showing the number of clicks to switch between modes:

This is missing that the ‘Regular’ mode will start in the mode that it last spent 3 minutes in. I couldn’t think of a way to represent that graphically that would be clear at a glance, so if I had to use words to explain the graphic I might as well just have written out that previous sentence.

While this diagram may be useful in the unlikely case you buy the exact same flashlight, my own familiarity came more from the effortful study of creating the diagram than looking at the diagram. So I’m really sharing this as an example of a useful study technique.

Here’s the source, if you’d like to use it as a starting point for your own diagrams:

digraph FlashlightStateMachine {
    rankdir=TB;
    node [shape=rectangle, style=rounded];

    // Main states
    locked [label="Locked Off\n(blinks twice)" style="rounded,dashed"];
    unlocked [label="Unlocked Off\n(blinks three times)" style="rounded,dashed"];
    momentary [label="Momentary On", style="rounded,bold"];
    regular [label="Regular On", style="rounded,bold"];
    sidelight [label="Sidelight On", style="rounded,bold"];

    // Bidirectional edge between locked and unlocked
    locked -> unlocked [label=" 5", dir="both"];

    // Momentary transitions
    unlocked -> momentary [xlabel="hold", dir=both];

    // Regular on transitions
    unlocked -> regular [label="2"];
    regular -> unlocked [label=" long"];

    // Sidelight transitions
    unlocked -> sidelight [label="3"];
    regular -> sidelight [label="3"];
    sidelight -> unlocked [label=" long"];

    // Subgraph for Regular On brightness levels
    subgraph cluster_regular {
        label="Regular Modes";
        style=dashed;

        low [label="Low"];
        med [label="Medium"];
        high [label="High"];
        moon [label="Moon"];

        low -> med;
        med -> high;
        high -> moon;
        moon -> low;
    }

    // Subgraph for Sidelight modes
    subgraph cluster_sidelight {
        label="Sidelight Modes";
        style=dashed;

        white_low [label="White Low"];
        white_high [label="White High"];
        red [label="Red"];
        red_flash [label="Red Flash"];

        white_low -> white_high;
        white_high -> red;
        red -> red_flash;
        red_flash -> white_low;
    }

    // Connect main states to substates
    regular -> low [style=dotted, arrowhead=none];
    sidelight -> white_low [style=dotted, arrowhead=none];

    // Layout
    {rank = same; locked; unlocked;}
    {rank = same; momentary; regular; sidelight;}
}

https://github.com/pushcx/typeid-lua

TypeID is a nice standard for creating unique id tokens with a Stripe-like ⊕An aside to Stripe from an Xtripe: Please write a eng blog post about the features and history of tokens. It would be pillar content that would be enormously popular for meaningfully advancing the state of the art to a new standard (Not the post about using them.) hungarian notation:

TypeIDs are a modern, type-safe extension of UUIDv7. Inspired by a similar use of prefixes in Stripe’s APIs.

TypeIDs are canonically encoded as lowercase strings consisting of three parts:

1. A type prefix (at most 63 characters in all lowercase snake_case ASCII [a-z_]).
2. An underscore ‘_’ separator
3. A 128-bit UUIDv7 encoded as a 26-character string using a modified base32 encoding.

Here’s an example of a TypeID of type user:

user_2x4y6z8a0b1c2d3e4f5g6h7j8k
└──┘ └────────────────────────┘
type    uuid suffix (base32)

A formal specification defines the encoding in more detail.

Cleverly, the spec comes with a suite of labeled test cases of valid and invalid examples. I wish more specs did this!

I’m happy with the functionality my library offers, and there was the familiar delight of making things the first time I round-tripped a TypeID.

TypeID = require("typeid")
-- or in dev: TypeID = require("./typeid")

t = TypeID.generate("comment")
-- t = {
--   prefix = "comment",
--   suffix = "01jvbhbbdje07rnyqkvstpvcge"
-- }

-- TypeID tables implement __tostring
print(t) -- "comment_01jvbhbbdje07rnyqkvstpvcge"

-- You can extract a standard UUID string
t:uuid() -- "0196d715-adb2-700f-8afa-f3de756db20e"

-- and round trip that back into a TypeID
TypeID.from_uuid_string("comment", "0196d715-adb2-700f-8afa-f3de756db20e")

-- parse and validate a TypeID from a string
TypeID.parse("comment_01jvbhbbdje07rnyqkvstpvcge")

-- finally, you can generate with a unix timestamp in ms:
TypeID.generate("comment", 1) -- "comment_0000000001e8avt0nh7a68v2jc"

This was a fun practice project for me. I’ve used Lua more and more over the last few years in video game scripts and my window manager, and while 1-based array indexes will always feel odd, I think there’s a lot of potential in the language.

I experimented with style while implementing, and a lot of what I’m taking away from it is idioms I’m ignorant of. The TypeID is more OO style and returns an object with a method; the Base32 and UUID7 modules work on primitives. After implementing, I guess users would probably prefer getting a primitive back, as there doesn’t seem to be an idiomatic way to type-check. A module can export a trusted constructor, but without types there’s no way to use that to prevent instantiating invalid objects; everything is a table anyways. Coming from Ruby and ActiveRecord it’s frustrating to have most of a solution to the pervasive problem of passing around invalid objects but not be able to complete it.

I guess have to read popular libraries to get a feel for style. I don’t really know what level to aim at between “data-hiding high-level interface” and “yolo, all primitives and seams showing for perf”. Maybe it’s different inside and outside of games.

Along those lines, I ported Base32 from the official TypeID Golang implementation and then wrote UUID7 in bytes to match it. But all that intermediate bit twiddling could be simplified by generating a UUID7 directly into the Base32 encoding if I wanted to spend a lot more time on this.

Maybe I’m looking under the wrong name, but it seems odd there isn’t a bitfield type I could use, given Lua’s popularity in games. Some searching turned up a library but the absence of multi-bit operations seems inconvenient. Which points to:

-- typeid.lua
uuid = function(self)
  return UUID7.to_string(Base32.decode(self.suffix))
end

There’s a code smell in TypeID: the nested conversions in the metatable uuid function suggest the internal representation of suffix is wrong. The syntactic distinction between .field and :method() means duplicating the data into two fields, exposing the internal representation by it being a field and the other a method, or getting away from what seems like common struct-y style and making both into methods. I’ve really grown to like the way Ruby’s optional parenthesis blur the line on fields and methods.

I wrote this library because I’d like to add a TypeID request identifier as a trace ID that nginx would generate and log, and pass along through Rails logs to MariaDB logs. It’s overkill for Lobsters but once a year I really wanted the ability to correlate logs like that. While the ROI may not really justify the time, it was a uniquely well-scoped small practice project.

Writing in Lua and adding the Lua module support to nginx seemed an easier path than writing in C and adding that compilation step to the deploy pipeline. Ultimately though, I’m not going to write that wrapper module.

On a parallel track, our ansible setup has slowly been succumbing to bit rot and my inexpert maintenance. I learned that Hatchbox could fill the same role and paying a couple bucks a month means it’s maintained by an expert professional. So 355e3b are going to move our hosting over soon, and it uses Caddy instead of nginx, so I guess I’ll wrap the official TypeID Golang implementation in a Caddy module instead. Still, it’s rewarding to contribute to TypeID’s list of supported languages.

I put together 15 variants, with inspiration and copying from here, here, and especially here. These aren’t intended to be long-term playable games with polished rules, just wacky variations for a silly night.

I wrote these out on a stack of index cards. We played each variant for one round of dealing, so 6-7 hands of each. I only revealed one variant before we played:

I put the variants in order to escalate complexity, to set up the Fuck Yeah/Fuck No joke, to provide a breather after the complexity of Auction House, and to end with Double Trouble because it was the idea that kicked all this off.

1. Bonus Start with 1 extra hole card, face up.
2. Roswell Start with 2 hole cards down and 1 up. After river betting: draw an extra hole card, do an extra betting round.
3. Trashy Start with 3 hole cards. After each betting round: players select 1 hole card and simultaneously pass it to the player on their left.
4. Stairstep Five betting rounds, each round you get 1 hole card and 1 community card.
5. Fuck Yeah After each betting round: draw 1 hole card, then discard 1.
6. Fuck No After each betting round: discard 1 hole card, then draw 1.
7. My Ship Will Come In After the turn betting: reveal 2 cards. Those ranks are wild.
8. Regret Start with 5 hole cards. After the flop, turn, and river betting: discard 1 hole card face up near you.
9. Dead Man’s Chest All folded cards go to the chest, down. If the hand ends early, nothing special happens. After river betting, reveal the chest. If the dead hand wins, the pot rolls over.
10. Swapsies At start, turn up a 5 card sideboard. After each betting round, each player may swap 1 hole card with a sideboard card. Keep it face up.
11. Auction House Deal 3 boards. Before each betting round is bidding. Going around once, each active player may bid chips to any 1 board. The board with the most chips is the real one. All bid chips go to the pot.
12. Death on the Nile After river betting: roll a die. Count from the first turn card to discard and replace 1 board card. (Ignore 6, nothing changes.)
13. Haggle Any 2 players, even folded, may agree to a table stakes price or for free, swap 1 hole card, seen or unseen.
14. River of Blood If the final community card is black, bet and end normally. If it’s red, bet and then repeat until you get a black card.
15. Double Trouble Shuffle together 2 full decks, include the 4 jokers as wilds. A pair/trip/quad with a suited pair outranks one without.

Except where noted otherwise, it’s Texas Hold ‘Em rules, hole cards are face down, and actions are limited to the players who haven’t folded.

Mostly I got it because the only movie theater near me is badly managed, but a new apartment’s layout means that if I’m playing games at my desk I can’t talk to my spouse sitting in the front room.

Privacy goal

I care about personal privacy. Basically every TV manufacturer IDs what you’re watching and sells that to advertisers, who can correlate it to you personally because the TV will have the same IP as your web browsing. Creepy shit.

People occasionally suggest buying “commercial panels” but the ones I found were more expensive, lower-quality, and had weird limitations. The solution is to never connect a TV to the internet. ⊕There’s a persistent conspiracy theory that TVs connect to open wifi. This would be very easy to test but I couldn’t find anyone who claimed to have observed it. Luckily TV surveillance is so inefficient about bandwidth that manufacturers can’t afford to put cellular modems in like car manufacturers.

So what are the options for an external device?

• All the popular open source media servers have clunky UI or were unreliable; mostly both. ⊕This HN thread on Jellyfin hits all of the issues I had with the open source options and also hits all of the “you’re doing it wrong” nitpicking I was not interested in participating in.
• Amazon Fire and Roku are invasive and janky.
• The Google Chromecast was made by Google, and so will be the Google TV Streamer.
• The NVIDIA Shield is not bad.
• The Apple TV 4k is good and has a good ecosystem.

OK!

Choosing a TV model

My last TV was a 15” CRT with a failing vertical hold that took a couple minutes to warm up before the picture would stop rolling. A 4k OLED is a stunning upgrade. I’ve been rewatching all my favorite movies because they look so good.

I don’t have an opinion on manufacturer besides never buying Samsung because of low reliability and poor UX.

Use Rtings to find a good current model. Use their size to distance calculator to pick a size for your room.

Apple TV setup

Mount the Apple TV 4k to the back of the TV with velcro tape. Ethernet is almost mandatory; more on this below.

Conveniently, you can use the terrible remote that comes with your TV to turn off motion smoothing and then put it in a drawer to never use. The Apple TV remote is nice, though once I had to reset it when it forgot how to control the TV volume. There’s a virtual remote that is nice for typing searches but otherwise is a peculiar demonstration that directly translating a physical device to a touchscreen feels clunky.

Settings:

• General
  • Restrictions: This is the one missing stair on Apple TV. It has the parental controls you’d expect except you can’t lock the Settings app, so curious kids can change every setting that isn’t about watching a TV show that might have a curse word in it. Whyyyyyyyyy.
  • Siri: off. It doesn’t run on-device.
• Users and Accounts -> Default User
  • Shared with You: off, distracting.
  • Require password for purchases and free downloads. You can auth with an iOS device, which is clever.
• Video and Audio
  • Format: 4k Dolby Vision
  • Match Content: Range & Frame Rate
  • Check HDMI Connection: run after setting the previous
  • Audio: has several accessibility settings, no idea why they’re here instead
• Notifications -> Search, TV: off
• AirPlay and HomeKit: weird that nobody’s gone after them for antitrust about it, but there’s the usual Apple practice of having a wonderful experience for their hardware like AirPods and meh for competitors.
  • Conference Room Display: on, so guests can show things from their phones
  • Require a PIN for AirPlay: on, unless you live somewhere rural
• Remotes and Devices -> Bluetooth: Apple doesn’t make a video game controller so they’re clunky, but controllers that claim to work on Apple TV/iPad/iPhone work fine, with only the usual Bluetooth pain switching between them between multiple devices.
• Accessibility: There is so much good stuff in here, it’s an incredible amount of thoughtful design.
  • Physical and Motor -> Tap to Navigate: off. Guests and kids struggle with the too-clever touchpad on the remote.

Home Screen:

Apple would really like you to subscribe to their things, so there’s a bunch of shovelware you can’t delete like Music, TV, Arcade, etc. You can at least make a folder named “Shovelware” and move it to be the last thing on the home screen. You can’t set Restrictions to keep curious kids out of these, either.

Apps

Tailscale, on the off chance you don’t already use it for your home network. You don’t have to expose your NAS to the internet. The TV app can act as an exit node, which is convenient for dealing with bank “security” that panics if you log in from a coffeeshop. Beyond TV setup, Tailscale continually impresses me with its features, reliability, and polish. It has solved a ton of hassles that have come with having multiple devices and working remotely.

Infuse for playing any kind of media files from most any storage. Excellent UI with automatic metadata download, ⊕Subtitles are only OK. Infuse pulls from OpenSubtitles but could do a much better job of sorting them based on frame rate, resolution, and keywords in the video filename. The UI for styling subtitles lacks a live preview, and the process of adjusting a delay is downright bad. overall much better experience than the various streaming services. Calmer, too, it’s trivial to turn off “Watch Next” suggestions and autoplay of the next episode.

We made a category for “Watch Together” and then a category each for a personal queue.

If Apple didn’t limit apps ability to cache so strictly, Infuse could start playback in 0.1s instead of 2s. But I have to assume most apps would be terrible at filling and pruning their caches, so I guess it’s fair.

SpeedTest for diagnosing network issues.

Steam Link for playing games from your desktop. ⊕UFO 50 is an incredible love letter to the weirdness of the NES/Genesis era with a few modern genres reimagined. If you played back then, don’t read or watch anything about it, just go get the buddy whose basement couch you used to play on and fire this up. It’s really a general purpose remote desktop tool, you can “Add a Non-Steam Game to My Library” to run anything. I’ve used it to run calibre and YNAB from my iPad. One bug/limitation: many games get letterboxed if the aspect ratio of your monitor doesn’t match the TV. I wrote a small shell script and added it three times for switching ratios to match the device I’m using.

8BitDo makes the best controllers.

While the latency is low enough for action games, it won’t be low enough for competitive shooters, fighting games, or precision platformers. The Apple TV doesn’t have ports for USB and the max cable length is only 2 meters anyways. If you don’t run ethernet to the Apple TV, you’ll get an annoyingly disruptive lag spike when your neighbor turns on their crappy microwave.

I’m surprised to say it, but it’s much more reliable to play games from a Linux desktop than Windows. Windows is swarming with intrusive upsell ads and will randomly reboot, so I found myself having to walk over to the desktop fairly often to unbreak it. Valve has put a ton of work into the Steam Deck Compatibility program and Proton so almost every game I play just works. It’s worth noting I get motion sick from 3d first-person and over-the-shoulder games and those are the AAA blockbusters, so I wouldn’t know if those run worse. ⊕A comment on Lobsters reports Linux is bad at HDR. I’ve never played an HDR game and this falls directly in this blind spot.

RetroArch has an Apple TV app, but tvOS is still trapped in the old iPhone misdesign that users can’t be trusted with files. Loading games or retrieving savegames is a chore. It’s acceptable if you only play on the TV but if you want to carry a game between platforms you’ll have to play via Steam Link. ⊕At least on iPhone/iPad you can use the Möbius Sync app for Syncthing to share your RetroArch ROMs and saves. Use an Ignore Pattern for retroarch.cfg to keep configs separate.

There’s some good settings in this article once you scroll past the affiliate ads.

Syncthing

One small caveat with Tailscale and Syncthing: many packages bind its web interface to the local network interface so it won’t be visible across your intranet. This is annoying for headless devices. Find Syncthing’s config.xml and change the <address> to 0.0.0.0 or, better, the machine’s Tailscale IP (100.x.x.x).

<gui enabled="true" tls="true" debugging="false" sendBasicAuthPrompt="false">
  <address>0.0.0.0:8384</address>
  <user>...</user>
  <password>...</password>
  <apikey>...</apikey>
  <theme>default</theme>
</gui>

Controllers are a cake with too many layers

This is a button on my controller:

Then I can rebind it in the firmware:

Then I can rebind it in the Apple TV Settings:

Then I can rebind it in Steam Link:

Then I can rebind it in Linux:

Then I can rebind it in Steam:

Then I can rebind it in a game (here, RetroArch):

Figuring out which layer misconfigs a button is terrible, and it’s just sloppy that none of them know what the controller looks like or how its buttons are labeled.

Valve, you are the only company with market position to drag manufacturers, OS developers, and game developers into the same room to sort this out. I know several of them take Not Invented Here to the extreme, but it would be great if you would abuse your monopoly power for my convenience by dragging them kicking and screaming into pleasant interoperability. ⊕And while you’re at it, bring back the Steam Controller.

If you still have one in a drawer, dear reader, here’s my notes on flashing the latest firmware. Please and thank you.

On November 19, Google announced the Google App for iOS is injecting unlabeled ads into pages. They look like the author created them, as seen in Google’s own screenshot:

Google claims they are injecting ads live now, so I’d like to quickly turnaround a js snippet that sites can use to detect the tampering. I’m strongly reminded of how a motivation for the big lift to HTTPS was slimy ISPs injecting ads into pages.

This system is nearly identical to Microsoft Smart Tags, a 2001 Internet Explorer feature that injected links to Microsoft sites into pages, but without the ability for site owners to disable them with a meta tag. (Recognized by esprehn.)

It’s also similar to Google AutoLink, a 2005 Google Toolbar feature a user could click to inject links into pages.

Google has published no information on how it targets the ads. It may target sites based on Googlebot crawls or may send the url or text of pages that require a login to Google. (Thanks @tasket@infosec.exchange.)

Initial reporting by SERoundtable and 9to5google.

Do you have the iOS Google App installed? Can you find an inserted ad that looks and acts like the screenshot above?

If you have the app and own a site, please check your site. Maybe start with high-ad-value terms like tourist destinations, insurance, loans, attorneys/lawyers, donations, hosting, trading, consumer electronics, or rehab. I would especially appreciate hearing from you so we could iterate on a couple possibilities for detecting this via the DOM.

(I found the victim in their screenshot and have collected 3 negative reports from people unable to reproduce the ad, so it may have been a one-off demo for the announcement.)

“This isn’t ads”

Oddly, a common response has been that the world’s largest advertising company adding links to keywords directed at their own sites isn’t advertising. Sometimes with the caveat that it’s because they’re currently only house ads and not yet for sale to third parties.

A bit of useful history might be Google’s own press release, “Why we sell advertising, not search results”. This was written when AdWords were placed to the side of search results with a blue background to be unambiguously ads. Despite saying that results wouldn’t be for sale, Google slowly iterated the design to make the ads nearly indistinguishable from results.

Today, these injected links start nearly indistinguishable from the author’s own links, with no “Ad” label or icon, and only a faint pastel background similar to the 2010 AdWords treatment. Like AdWords, every experiment Google runs on engagement will show “improvement” as the ads become harder to distinguish,

If you found that an advertising company was adding links to your site without your knowledge or consent, would you consider it a useful service?

Do you think it acceptable that, to disable the injected links, you have to agree to the ad company’s continually-updated terms of service and ask to individually opt-out each of your sites?

Ongoing

There may be more on Bluesky or Mastodon where I’ve tried to find help.

Here’s the first rough draft of a js snippet that a site owner who sees an ad could add to a page to see some more info. It needs text from the link inserted. There’s some assumptions noted for how the injection might appear in the DOM.

(function() {
  const logDiv = document.createElement('div');
  div.style.cssText = 'position: fixed; top: 0; left: 0; right: 0;' +
                      'background: #fee; color: #000; font-family: monospace; ' +
                      'z-index: 999; padding: 10px; max-height: 50vh; overflow: auto;';
  document.body.prepend(div);

  function log(msg) {
    const entry = document.createElement('div');
    entry.textContent = msg;
    logDiv.appendChild(entry);
  }

  try {
    /* edit 'castle' to include the text of an ad link */
    const text = 'osaka castle';

    /* if it's not an 'a', try 'span' and then 'div' */
    for (const ad of document.querySelectorAll('a');) {
      if (a.textContent.toLowerCase().includes(text)) {
        log(ad.outerHTML);
      }
    }
  } catch(e) {
    if (logDiv) {
      log(`Error: ${e.message}`);
    }
  }
})();

Hey folks,

When this post is two hours old, and indefinitely twice a week, I’m going to stream Lobsters office hours and development on Twitch. The channel is here and more info + an archive is on my blog. (You don’t have to sign up to Twitch to watch.)

The office hours are largely to support iteratively running queries. People don’t take me up on this very often except via IRC because it’s hard to write these things perfectly out of the gate. Having a fast feedback loop is essential. Also, maybe having an open office hours session will be a nicely informal opportunity to answer questions and generally demystify the site.

If nobody has questions I’ll hack on the Lobsters codebase. Hopefully this encourages more activity on the repo. I’m also open to support people using the code to start a sister site, which is another activity that benefits from a tight feedback loop. (example)

I plan to stream weekly on (US Central) Monday afternoons at 2 PM and Thursday mornings at 9 AM in the hopes that this gives reasonable timezone coverage. Sorry for the inconvenience if you are antipodal to Chicago, though the site has always run on Chicago time. Sessions will probably run 1.5 to 3 hours and I’ll put summaries and transcripts in my archive for easy searching. I’ll try to keep the twitch schedule up-to-date and I already know I’ll miss 9/1 for a vacation.

This is pretty experimental! I did it a few streams back in summer of 2020 that went well, but everything was pretty chaotic that year for some reason and I dropped it. I did a technical rehearsal this weekend that went reasonably smoothly. If this continues to be interesting and useful, I’ll keep it up and try new things. Feedback here, in the stream chat, or by email (peter@ my domain) is much appreciated.

Special guest host will be my cat (subject to his schedule and whim). See y’all in our weird parasocial future.

There’s a more thorough description but #lobsters has three big purposes:

1. share a feed of links and have a lighter, ephemeral discussion on them
2. give potential new users a place to connect to existing ones for invites ⊕2024-08-08: To make an implicit value explicit, a big value of the chat is that it keeps our border a little porous, it helps Lobsters avoid becoming stagnant
3. some off-topic chat and community bonding

So we care a lot about text chat with a bit of custom functionality and a great new user experience. IRC is no longer a good experience for new users and a couple times a year #lobsters rehashes a discussion on IRC’s features and prospects. I finally realized I should collect my notes/logs into something linkable even if it’s only a braindump.

• Most of this is me comparing Discord to IRC because it’s the alternative that’s most-used by current chatters, but the Rebuttal section is pretty universal to any discussion of IRC’s shortcomings.
• I’m not seriously considering moving Lobsters chat to Discord, and the possibility is less attractive now that I’ve collected a list of its problems, which has built a compelling case it’s a bad culture fit.
• This is all pretty rough and contains a significant amount of frustrated venting.
• Items are not in priority order.
• Nothing here is urgent.
• We’ll use a chat for decades so any decision is less about current parity and more about trend lines.
• I’m compiling many discussions so we can rehash less in the future, whatever happens.

Desirable Discord Features

• good new user experience
  • GUI
  • familiar signup
  • if user already has an account, joining server is a link to a one-click join dialog
  • scrollback on join so the channel doesn’t look dead unless the new user joins in the ten seconds before an existing user hits enter
  • doesn’t leak IP
• desktop streaming (1080p)
  • I want to resume streaming Lobsters coding + office hours (esp for queries) ⊕2024-10-15: I’ve been streaming as hoped for a few months.
  • My attempts to contact Twitch eng about their login bug failed ⊕2024-07-18: Found it: Twitch’s login is broken if Firefox’s fingerprinting protection is on. I have to make a separate profile to use on-stream anyways, so it’s easy enough to leave off there.
• text formatting: links, bold, underline, italic, code blocks
• security - MFA, alert emails, active login list
• message editing
• emoji reactions (and allergic users can hide them)
• mod tools
  • reasonable banning (+b is garbage; separation of kick is misdesign)
  • no @ on mod names (https://libera.chat/guidelines/#channel-operators-are-users-too)
  • highlight/filter on keywords w great default lists
  • many high-quality 3rd-party tools (eg https://carl.gg/)
  • public modlog (might req carl.gg, I forget)
• mobile app
• good UI for muting channels/groups/users
• threads for breaking out overlapping/side conversations
• good user docs with screenshots
• per-server user profiles w bio, links
• I can pay for it and it improves over time
• can oauth to link account from lobsters profile
• file attachments
• higher discoverability/much better new user onboarding
  • #lobsters gets traffic from being ~25th in libera’s webchat top channel list
  • but discord is staggeringly popular and has 3p public directories

Discord Downsides

big stuff, potentially blockers:

• might not be able to disable donations (reasoning)
• mobile client sends a notification by default for any activity
  • painfully user-hostile and inappropriate for Community Servers
• slow even for a desktop GUI
  • a loading interstitial! takes seconds on a fast connection and powerful desktop
  • changing servers/channels takes x00ms and regularly much more
  • animated placeholders during delays
  • mobile app is noticeably slower at everything
• countless distracting animations, mouseover popouts, and mystery meat
• broken activity UI
  • not clear whether a channel is selected or has activity, both use bold
  • very flaky about marking channels as read (much worse on web than mobile)
  • messages from blocked users still highlight channel as active
• actively hostile to 3rd party clients/general hackery
• a serious culture clash that prompts most of the UI problems
  • Discord is oriented to mass-appeal to passive consumption of games, gossip, and memes
  • Lobsters is about creating, learning, sharing experiences/expertise

smaller stuff, antifeatures:

• no active ruby bot library
  • so almost every integration and workaround requires a 3rd-party service
  • I’m sick of trying to keep a js service running, but maybe I’m just underwhelmed by the IRC library
  • 355e3b notes webhooks get us ~all current functionality so I demoted this out of blockers
• threads/replies are clunky af
  • feels like they were bolted on the UI and never really integrated; even slack is better
• much more spam
• stickers, inline images, animated emoji
• enormous obnoxious link preview cards (no setting; maybe could delete by bot)
• intrusive upsells in clients (these go away on paid servers, right?)
• can’t turn off external link warnings even solely for our own site (sharing links is a core activity; we have sophisticated users)
• big company support (vs a libera admin is a lobsters user and active in the channel)
  • forum is overwhelmed by support requests/mental health crises
  • no chance of reporting bugs or influencing features
• complicated subscription structure with unclear pricing (but probably <$20/m)
• users can’t export logs
• no TUI
• no Reactions page collecting emoji reactions to own posts
• can’t disable X integration on profiles
• it’s so slow I have to list it a second time

IRC Rebuttals

• Fix/script your client
  • This is “baby why do you make me hit you” levels of helpful.
  • I would rather have one reasonable UI than have every user fail to reinvent the wheel.
  • Bad out-of-the-box UI is why IRC has rounds-to-100% new user churn
• So the server/client split…
  • I am a user, not an implementer. This is a technical decision that seems to have permanently hamstrung development.
  • It’s about as interesting and useful as a customer support line that refuses to help because you have the wrong department.
• Nickserv/chanserv/etc
  • Bolting on features/services via chatbot is powerful but bad UI (even when Discord does it!)
  • The big difference is that IRC never integrates these into the core product
• IRC is open source/open standard.
  • It’s nice and I’m willing to take a haircut for it, but I’m ready to take off the hair shirt.
  • But maybe being a protocol instead of a product is why it’s so far behind and not improving.
• IRC is volunteers/PRs welcome
  • IRC is a heroic accomplishment and should not be one
  • My time is more valuable than my money. For our non-toy usage $50/month barely registers as a cost.
  • Especially compared to the cost of joining a multi-stakeholder consensus-mandatory process about the redesign of legacy software.
• IRCv3
  • Mostly technical underpinnings with little addressing the feature gap.
  • IRC is not closing the gap. Discord is very actively widening the gap.
• Discord is a single service run by a VC-funded business
  • Lots of big, familiar privacy/sustainability/control risks here
  • This also prompts lots of the UI problems
  • VC B2C orients to growth metrics, so quality will nosedive hard when growth plateaus and the established processes can’t adapt to that “failure”

Underinformed Pontificating

Libera Chat isn’t at fault, incompetent, foolish, or anything else negative. Neither is the broader IRC community (well, aside from that one infamous guy, who is all of those things and more).

Most of IRC’s problems are structural. Network effects make chat valuable. But feature development has stalled beacuse it’s brutally hard to reach consensus. Maybe email and the web managed it because of competitive commercial use. Maybe the protocol isn’t as extensibile because it’s not as forgiving of unsupported features. There’s probably an amazing book waiting to be written about how open protocols and standards thrive or die.

Text-oriented group chat has products like Discord, Slack, Zulip, WhatsApp, Telegram, Messenger, WeChat, iMessage, GChat, Skype, Teams, Kik, Mattermost, Snapchat, Wickr, and then, you know, some small ones that only have tens of millions of active users. Nearly every human with a phone uses at least one. That’s a lot of room for open source and standards, and IRC seems to have attracted and extinguished potential development. Maybe the pressing thing to design is not a revised protocol but a process for sustaining consensus over revisions.

2024-07-12 Edit: Chat discussion has pointed out to me that Matrix is an open standard. I really only know it from the not-so-pleasant bridge to Libera Chat. I feel pretty good about putting “underinformed” in this section heading.

2024-07-26 Edit: When I published this blog post, the feed-reading bot shared it in the Recurse Center Chat. I had wandered away from it after my batch, mostly because Zulip was unpleasantly slow and janky. That’s been fixed in the years since, it’s a smooth and polished experience. I think it has all of the features listed here and none of the downsides (even an official TUI!), this is a plausible alternative. Zulip 9.0 was released today.

My last job had a codebase large enough (~50M LOC ⊕Edit 2025-02: I recently reread this post. I think this number must be too high, but I can’t remember what the right number was anymore, maybe 7M? But I don’t know how I would’ve made this mistake, it’s not clearly a simple typo. I guess I’ll just reiterate that it was by far the largest codebase I’ve worked in and not alter this post.) that there were always large-scale refactors in flight, which was a new experience for me.

One thing this article doesn’t specifically call out is that any kind of dependency update or replacement, whether an internal or external library, is a large-scale refactor. You benefit enormously if you can do this incrementally as the author describes instead of a flag day or One Giant Merge to update all uses. A counterintuitive result is that replacing one dep with another (foolib -> barlib) is easier than updating one (foolib 1 -> foolib 2)! Most languages do not allow you to depend on multiple versions of a package and have different sections of your codebase call different versions. Sometimes internally-maintained dependencies will rename just to get around this limitation.

There’s a style of managing dependencies that mandates you must wrap usage of libraries or APIs. Rather than calling Foolib::Thing.new, you’ll create your own FooThing (maybe using the decorator or facade patterns) and that class is the only place allowed to import from or call into foolib. With less exposure of foolib, it’s easier to create internal documentation, audit or control usage, or replace foolib with barlib. I don’t find this a cost worth paying in smaller codebases, but easily worth it in large ones.

Part of why it’s worthwhile is that it gives you two new methods for dealing with dependency updates. First (hopefully), you have a single codesite that uses foolib so a single team can make a small change to update foolib. Or second, if there are extensive changes that mandate changes at callsites, you can rename FooThing to FooThing1 (usually an easy, if large diff), introduce FooThing2 with the new API, and then use a process like the one this article describes to make that change incrementally to the entire codebase. Either you update foolib at the start of this process and FooThing1 maps old usage to new, or FooThing2 maps new usage to old and you bump foolib at the end. This process works quite well, whether foolib is an internal or external dependency. Whereas, say, emailing all-dev@example.com a link to the foolib release notes and dictum that on some particular date that all foolib usage must be updated will inevitably produce significant internal discord and never, ever an on-schedule completion. An even worse and more common failure mode for internal libs is to quietly mark foolib deprecated and direct people to rewrite to barlib when they show up with urgent questions about foolib during an outage - but of course good sense and steps 6-8 of the process described in the post would avoid such an outlandish footgunning.

(This post was originally a comment on Lobste.rs but then I realized it’s a nice excuse to break the 5.5 year dry spell here.)

Provision

I went with the lowest tier, currently 1.25 GiB RAM, 15 GiB Disk for $5/month. I’m only running weechat for irc/twitter/fediverse/slack and some miscellaneous small things. For “pre-installed distribution” I chose “None (HVM)”.

Netboot to start install

I ssh’d into the management console, ssh [hostname]@[hostname].console.xen.prgmr.com

• 6 bootloader
• 4 netboot installer, pick nixos
• 0 twice for main menu
• 4 to power off
• 2 to start (see “Booting” below)
• 1 to log in as root with no password (relax, ssh is off)

Partition

Surprisingly, the included 1.25 GB of RAM was not enough to run some nix commands. I had to back up and recreate the box with some swap space. I didn’t think too hard about it, just guessed at 2 GB and it worked OK. ⊕2018-07-09: Vaibhav Sagar suggested this is probably this known bug.

` gdisk /dev/xvda`{lang=”bash”}

` o to create gpt`{lang=”bash”}

n to create swap partition{lang=”bash”}

Command (? for help): n Partition number (1-128, default 1): 1 First sector (34-31457246, default = 2048) or {+-}size{KMGTP}: Last sector (2048-31457246, default = 31457246) or {+-}size{KMGTP}: +32M Current type is 'Linux filesystem' Hex code or GUID (L to show codes, Enter = 8300): EF02 Changed type of partition to 'BIOS boot partition'{lang=”bash”}

Command (? for help): n Partition number (2-128, default 2): First sector (34-31457246, default = 67584) or {+-}size{KMGTP}: Last sector (67584-31457246, default = 31457246) or {+-}size{KMGTP}: -2G Current type is 'Linux filesystem' Hex code or GUID (L to show codes, Enter = 8300): Changed type of partition to 'Linux filesystem'{lang=”bash”}

Command (? for help): n Partition number (3-128, default 3): First sector (34-31457246, default = 27262976) or {+-}size{KMGTP}: Last sector (27262976-31457246, default = 31457246) or {+-}size{KMGTP}: Current type is 'Linux filesystem' Hex code or GUID (L to show codes, Enter = 8300): 8200 Changed type of partition to 'Linux swap'{lang=”bash”}

w to write and exit{lang=”bash”}

mkswap -L swap /dev/xvda3{lang=”bash”}

swapon /dev/xvda3{lang=”bash”}

mkfs.ext4 -L root /dev/xvda2{lang=”bash”}

` {lang="bash"}mount /dev/xvda2 /mnt `{lang=”bash”}

Configuring nix

I generated the initial config and added a few prgmr-specific tweaks:

` nixos-generate-config –root /mnt`{lang=”bash”}

cd /mnt/etc/nixos{lang=”bash”}

` {lang="bash"}vi configuration.nix `{lang=”bash”}

Here’s my tweaks:

` boot.loader.grub.device = “/dev/xvda”`{lang=”nix”}

` # prgmr console config: boot.loader.grub.extraConfig = “serial –unit=0 –speed=115200 ; terminal_input serial console ; terminal_output serial console”; boot.kernelParams = [“console=ttyS0”];`{lang=”nix”}

environment.systemPackages = with pkgs; [ bitlbee tmux weechat wget vim ];{lang=”nix”}

services.openssh.enable = true;{lang=”nix”}

networking.firewall.allowedTCPPorts = [ 22 ];{lang=”nix”}

sound.enable = false; services.xserver.enable = false; services.openssh.enable = true;{lang=”nix”}

` {lang="nix"} users.extraUsers.pushcx = { name = “pushcx”; isNormalUser = true; extraGroups = [ “wheel” “disk” “systemd-journal” ]; uid = 1000; openssh.authorizedKeys.keys = [ “[ssh public key here]” ]; }; `{lang=”nix”}

Then I ran nixos-install to install the system.

Booting

The NixOS manual says you should be able to run reboot to boot to the new system, but something in xen doesn’t reload the new boot code and I got the netboot again rather than the new system. After talking to prgmr I found it worked if I pulled up the management console and did:

• 6 -> 1 boot from disk
• then 4 to fully poweroff
• then 2 to create/start

After this I had a running system that I could ssh into as a regular user.

Prgmr donates hosting to Lobsters, but because Alan configured the hosting, this was my first time really using the system. It was painless and getting support in #prgmr on Freenode was comfortable for me as a longtime IRC user. I liked them before, and now I’m happy to recommend them for no-nonsense VPS hosting.

Nix/NixOS

I did this setup because I’ve been meaning to learn nix (the package manager) and NixOS (the Linux distribution built on nix) for a while. As I commented on Lobsters, they look like they didn’t start from manual configuration and automate that, they started from thinking hard about what system configuration is and encoded that. (The final impetus was that I ran out of stored credit at Digital Ocean, hit several billing bugs trying to pay them, and couldn’t contact support - six tries in four mediums only got roboresponses.)

The NixOS manual is solid and I had little trouble installing the OS. It did a great job of working through a practical installation while explaining the underlying concepts.

I then turned to the Nix manual to learn more about working with and creating packages and failed, even with help from the nixos IRC channel and issue tracker. I think the fundamental cause is that it wasn’t written for newbies to learn nix from; there’s a man-page like approach where it only makes sense if you already understand it.

Ultimately I was stopped because I needed to create a package for bitlbee-mastodon and weeslack. As is normal for a small distro, it hasn’t packaged these kind of uncommon things (or complex desktop stuff like Chrome ⊕2018-07-16: I’ve learned that Nix does have a package for Chrome, but it doesn’t appear in nix-env searches or the official package list because it’s hidden by an option that is not referenced in system config files, user config files, the NixOS Manual, the Nix Manual, the man page for nix-env, the package search site, or the the documentation of any other tool it hides packages from.) but I got the impression the selection grows daily. I didn’t want to install them manually (which I doubt would really work on NixOS), I wanted an exercise to learn packaging so I could package my own software and run NixOS on servers (the recent issues/PRs/commits on lobsters-ansible tell the tale of my escalating frustration at its design limitations).

The manual’s instructions to build and package GNU’s “hello world” binary don’t actually work (gory details there). I got the strong impression that no one has ever sat down to watch a newbie work through this doc and see where they get confused; not only do fundamentals go unexplained and the samples not work, there’s no discussion of common errors. Frustratingly, it also conflates building a package with contributing to nixpkgs, the official NixOS package repository.

Either this is a fundamental confusion in nix documentation or there’s some undocumented assumption about what tools go where that I never understood. As an example, I tried to run nix-shell (which I think is the standard tool for debugging builds but it has expert-only docs) and it was described over in the Nixpkgs Manual even though it’s for all packaging issues. To use the shell I have to understand “phases”, but some of the ones listed simply don’t exist in the shell environment. I can’t guess if this a bug, out-dated docs, or incomplete docs. And that’s before I got to confusing “you just have to know it” issues like the src attribute becoming unpackPhase rather than srcPhase, or “learn from bitter experience” issues like nix-shell polluting the working directory and carrying state between build attempts. (This is where I gave up.)

I don’t know how the NixOS Manual turned out so well; the rest of the docs have this fractal issue where, at every level of detail, every part of the system is incompletely or incorrectly described somewhere other than expected. I backed up and reread the homepages and about pages to make sure I didn’t miss a tutorial or other introduction that might have helped make sense of this, but found nothing besides these manuals. If I sound bewildered and frustrated, then I’ve accurately conveyed the experience. I gave up trying to learn nix, even though it still looks like the only packaging/deployment system with the right perspective on the problems.

I’d chalk it up to nix being young, but there’s some oddities that look like legacy issues. For example, commands vary: it’s nix-env -i to install a package, but nix-channel only has long options like --add, and nix-rebuild switch uses the more modern “subcommand” style. With no coherent style, you have to memorize which commands use which syntax - again, one of those things newbies stumble on but experts don’t notice and may not even recognize as a problem.

Finally, there’s two closely-related issues in nix that look like misdesigns, or at least badly-missed opportunities. I don’t have a lot of confidence in these because, as recounted, I was unable to learn to use nix. Mostly these are based on my 20 years of administrating Linux systems, especially the provisioning and devops work I’ve done with Chef, Puppet, Ansible, Capistrano, and scripting that I’ve done in the last 10. Experience has led me to think that the hard parts of deployment and provisioning boil down to a running system being like a running program making heavy use of mutable global variables (eg. the filesystem): the pain comes from unmanaged changes and surprisingly complex moving parts.

The first issue is that Nix templatizes config files. There’s an example in my configuration.nix notes above: rather than editing the grub config file, the system lifts copies from this config file to paste into a template of of grub’s config file that must be hidden away somewhere. So now instead of just knowing grub’s config, you have to know it plus what interface the packager decided to design on top of it by reading the package source (and I had to google to find that). There’s warts like extraConfig that throw up their hands at the inevitable uncaptured complexity and offer a interface to inject arbitrary text into the config. I hope “inject” puts you in a better frame of mind than “interface”: this is stringly-typed text interpolation and a typo in the value means an error from grub rather than nix. This whole thing must be a ton of extra work for packagers, and if there’s a benefit over vi /etc/default/grub it’s not apparent (maybe in provisioning, though I never got to nixops).

This whole system is both complex and incomplete, and it would evaporate if nix configured packages by providing a default config file in a package with a command to pull it into /etc/nix or /etc/nixos for you to edit and nix to copy back into the running system when you upgrade or switch. This would lend itself very well to keeping the system config under version control, which is never suggested in the manual and doesn’t seem to be integrated at any level of the tooling - itself a puzzling omission, given the emphasis on repeatability.

Second, to support this complexity, they developed their own programming language. (My best guess - I don’t actually know which is the chicken and which is the egg.) A nix config file isn’t data, it’s a turning-complete language with conditionals, loops, closures, scoping, etc. Again, this must have been a ton of work to implement and a young, small-team programming language has all the obvious issues like no debugger, confusing un-googleable error messages that don’t list filenames and line numbers, etc.; and then there’s the learning costs to users. Weirdly for a system inspired by functional programming, it’s dynamically typed, so it feels very much like the featureset and limited tooling/community of JavaScript circa 1998. In contrast to JavaScript, the nix programming language is only used by one project, so it’s unlikely to see anything like the improvements in JS in last 20 years. And while JavaScript would be an improvement over inventing a language, using Racket or Haskell to create a DSL would be a big improvement.

These are two apparent missed opportunities, not fatal flaws. Again, I wasn’t able to learn nix to the level that I understand how and why it was designed this way, so I’m not setting forth a strongly-held opinion. They’re really strange, expensive decisions that I don’t see a compelling reason for, and they look like they’d be difficult to change. Probably they have already been beaten to death on a mailing list somewhere, but I’m too frustrated by how much time I’ve wasted to go looking.

I’ve scheduled a calendar reminder for a year from now to see if the manual improves or if Luc Perkins’s book is out.

2018-08-09: I wasted another two days trying Nix from the other direction. Rather than build up from the basics I tried to start from the top down and create a “Hello World” Rails app. It’s hard to tell around the bugs and docs, but I’m pretty sure it’s not possible to run a Rails app on NixOS.

2019-12-01: New attempt. Got an incorrect error message that /sbin/bash didn’t exist. Paved and reinstalled, then tried to build a Rails demo but I got a useless error message when it failed to install 1password (!?). The two errors came while pasting directly from the install docs and I was told “lmk when you figure out which command you were just calling wrong”. The documentation’s first example is still broken and following install steps invariably leads to errors. I’m sick of being told it’s my fault nix doesn’t work and I’m giving up on it.

1. Yes Take-backsies

We’re playing for the fun of learning new games and competing. If you make a mistake and we can unwind it, you can take it back and do what you meant to do or realize you should have done.

We don’t do take-backsies of things we can’t easily and fairly unwind. For example, in Risk, if you roll the dice to attack another player’s army and get wiped out, well, that’s not fair to unwind. Similarly, if you learn hidden information like turning over the next card on the deck, we can’t make everyone forget that (but mayyyybe if it’s someone’s first time playing a game).

Making mistakes is a normal part of learning. By helping keep them cheap and emphasizing that it’s socially rewarded to admit and correct them, everyone learns and plays better, and gets to relax and enjoy themself more. Games are fun because they’re an exercise in trust as much as exercise in formal systems.

We very often have players with uneven experience and take-backsies helps the newbies get into a game and keep the moderately-experienced competitive with the experts. We encourage suggestions and polite criticism of in-progress mistakes, too. When learning a complex game it’s hard to recognize the legal moves available and what their tradeoffs might be. So when someone looks particularly stumped, it’s normal to hear something like “Sooo... it looks like you have five or maybe six things you could do here, depending on what’s in your hand?” to offer help.

Suggestions show up even with competitive, experienced players, in part because some of us are so competitive we wrap around into helping our opponents make the best moves for their strategies so that we’re ourselves pushed into improving our own play. Suggestions aren’t a rule because some people don’t like hearing them and it’s more common in competitive play to want opponents to make mistakes, but I wanted to go into it because it’s foundational to the Yes Take-Backsies rule.

2. Public Stays Public

Information that’s revealed during gameplay stays public information and can be reviewed at any time unless doing so severely inconveniences the flow of gameplay.

What this means in practice is cards get discarded face-up and spread out for anyone to look through, or other things that have been played can be reviewed. The limit is that we might run out of table space or it would be distinctly un-fun to dig through that many cards, but when there’s that much information laying around it’s probably not particularly important what’s been played.

For example, The Great Dalmuti is a light trick-taking card game that’s a longtime favorite. Players play cards to try to empty their hands before everyone else, and between hands they shift to sit in the order they finished in to get benefits for the next hand. The last player is punished with the chore of clearing away the cards from each trick (and they have some other chores) so we task the next-to-last player. Instead of the game rule that all played cards are flipped face-down, our house rule is to arrange the best six ranks (numbered 1-6 of the deck’s 12 ranks) face-up along the side of the table. We don’t do this for ranks 7+ to balance the work vs how little strategy is at work in high cards (>85% of the time the correct play is “dump any of them at first opportunity”). This has been a big success. Beginners graduate from learning the rules to start exercising strategy in 2-3 hands instead of 5-6 hands, experts can experiment more effectively, and everyone’s happier not trying to remember “wait... did I see all of the threes, or just two of them?”

Information that’s derivable from public information is also public. In Acquire every player starts with the same amount of money and buys and sells stock in public transactions... but the rulebook suggests keeping it secret to make the game “even more challenging”. Arithmetic is not a fun game, it is a chore.

These “memory subgames” crop up all over, are stressful and uninteresting, and seem an unfair advantage for those who are better at this or spend a few days learning mnemonic techniques. When public information stays public, players make fewer uninformed and mistaken decisions.

The common downside is that sometimes players bog down the game while sifting through old cards or falling into analysis paralysis. We solve this through good-natured grumbling and some smoothly-worn old jokes that reference getting on with things before the mountains crumble into the seas, etc. In rare cases [(or rare players)]{.small} we’ll set a turn timer on someone’s phone. Or give suggestions! Once you’ve looked at all the things an opponent might do and thought through how you’d respond to them and gotten bored, there’s no harm in talking to the opponent about their options.

To keep things moving along (especially when playing someone with very good recall of previous game state), it’s common to ask something like, “Wait, have you played all your aces?” rather than spend twenty seconds flipping through their discards. An honesty norm has developed: you can answer honestly, you can fib and say “I’m not sure” (if you think it gives you a competitive advantage and they won’t check), in an fiercely competitive game you can say “count for yourself”, but you can’t lie. If you’ve played all your aces but say you haven’t, or vice-versa, or otherwise deliberately give false information, it’s considered very rude, unsportsmanlike conduct and is treated almost as negatively as cheating. It’s OK to want to win and normal not to want to help your opponent, but we’ve established lying about public information as something that gets you a lot of frowning friends. We’re good friends or becoming them, so social disapproval means this basically never happens. (Exception: Diplomacy is a blood sport.)

Public Stays Public is a much younger and more explicit rule than Yes Take-backsies, which grew out of long habit. We had only very small, limited experiments with it until about six months ago (December 2017), when I heard at Recurse Center that someone knew of a gaming club (maybe at MIT?) that had it as a universal house rule. I decided it was worth trying and we’ve enjoyed it in every game since.

More?

Our process of developing house rules is a very much like developing traditions. We’re doing this slowly and sometimes only recognizing in retrospect that we have one because we learn another group doesn’t. I’m finally getting around to writing this up today because it came up in an online chat and my response kept getting longer and longer, so maybe I’ll add more in the future.

I suppose one thing worth noting is that in the last decade as games have generally gotten much better (and we’ve gotten more patient), we’ve become much more reluctant to add house rules to individual games. Usually something that seems weird and wrong is a corner of gameplay worth exploring and deliberating rather than something we feel we comfortable immediately trying to prune. We have almost no per-game house rules, really.