eta.st

Reversing UK mobile rail tickets

Tue, 31 Jan 2023 00:00:00 +0000

The UK has used small credit-card sized tickets to pay for train travel for years and years, since long before I was born — originally the APTIS ticket¹, which later got replaced by a slightly easier to read version printed onto the same stock.

Nowadays, the industry would very much like you to ditch your paper ticket in favour of a fancy mobile barcode one (or an ITSO smartcard²); not only do they not have to spend money on printing tickets but they also gain the ability to more precisely track the ticket’s usage across the network and minimise fraud.

There are obvious benefits for the user too — I’m willing to bet most people use the mobile tickets anyway, since they’re just easier if you’re booking your train travel in an app like Trainline.

But what data is inside the barcode of a mobile ticket, and how do they work? Could people who aren’t ticket inspectors get the data out of them? It turns out that the answer is a bit more interesting than I initially expected!

Initial explorations

A mobile ticket is just an Aztec barcode, either displayed inside an app or on a PDF you can print out:

Googling around for prior work people had done decoding mobile tickets, I found a bunch of discussion (German-language link) about UIC 918.3, a specification used by the German railway company Deutsche Bahn for their e-tickets. These also use the Aztec barcode format and looked superficially similar — and some people had already written code to read them. Maybe this could work?

(Why would I expect this to work? Well, the UIC is the international standards body for railways — in Europe at least — so it’s reasonable to assume they might’ve used a standard format here.)

However, the formats are sadly nothing alike — decoding our UK barcode using zxing, we get:

06DNQL4XHVK00TTRCGPUQWNTHPGHWBPOUTKRWXAJKGHFBAPBCTOGUZQVTZTKKDEBQXPGRWZJRJBXJZPOHNJGIPDJWEGYWJXLVPGEEZBCUUELIJMOINPRZMSDQCZJGLIZLUTQHXMTPKWCMJISUXQLORAOVYXSOLGXXGMVUDXTMHAYMBLUTKPUPFCRNNTDBBDLNWSBPDUXYKSIMJSBYBURSCPUMFBZPEUTECHTIOXAH

…which looks nothing like you might expect a UIC barcode to look, given the latter are supposed to start with “#UT” (according to the discussion in German earlier).

In fact, this is a custom standard that is only used inside the UK, as the “06” at the start of the data hints at; this is an “RSP-6” ticket (as in RSP for Rail Settlement Plan), which Google doesn’t seem to know much about. It is possible to find a Freedom of Information Act request someone made asking for the spec that never got a response — sadly the Rail Delivery Group (RDG), technically a private company, doesn’t actually have to respond to such requests, so I’d have to figure this out myself.

A friendly wolf comes to help

At this point, I basically had no idea how to continue. Comparing multiple tickets, the data seemed to be mostly random apart from some fixed headers, suggesting that it was probably encrypted in some way — I couldn’t just get lots of tickets and hope to find similarities between them.

image: output of binwalk -W cdq-cys.bin pad-aml.bin, which highlights similarities and differences between two tickets³

My friend Harley (“unlobito”) had noticed me complaining about tickets in a shared group chat and had a clue for me: the word “masabi”, which turned out to be the name of a ticketing company.

Masabi’s website has this lovely page where they explain all about how they invented mobile ticketing in the UK in 2007 and how the RSP6 national standard was actually written by them! They also boast about how their “JustRide Inspect” suite of apps can be used to decode these tickets.

We sadly can’t just get this app off the Play Store. However, after some googling around you can totally get it from one of those less than official APK rehosting websites.

With the APK in hand there are a number of things we can do. We can just install it on an Android device and see whether it’ll give up anything interesting that way; we can also try to “decompile” it, to get a better idea of how the app (and the ticket parser) works.

Running the app

Since I didn’t have any spare throwaway Android phones and that the APK might be malware, my first step was to just run it inside an Android Virtual Device (using the emulator in Android Studio). I reckoned this would be a bit safer than just installing it on my main Android phone.⁴

After a bit of fiddling about, I had it doing something:

Unfortunately, this wasn’t very useful — it wouldn’t scan standard ticket barcodes in this form. The “Login manually” button lets you choose a Train Operating Company (TOC) to sign in as (some of the companies mentioned don’t even exist any more!), which was pretty interesting but not useful for our goals:

I don’t work for a train operating company, so clearly it wasn’t going to be possible for me to proceed⁵. Maybe examining the app another way could get us somewhere?

Decompiling the app

You can point Android Studio at an APK and have it analyse what’s inside. Sort of.

If you try this (via the 3-dot menu in the project chooser → “Profile or Debug APK”), you get something that’s not entirely useful: a bunch of weird looking “smali” files.

This is because the APK only contains compiled bytecode, rather than any useful source code; as the yellow warning banner notes, this is stored inside the APK in .dex format (“Dalvik Executable”, where Dalvik is the name of the Android VM), and smali is just a human-readable representation someone invented for this format (like Assembly).

What we ideally need is something that can turn the bytecode all the way back into Java. Such a tool exists in the form of jadx, a very handy tool that not only does just that, but can also do a bunch of other cool stuff like outputting a project Android Studio can load (and theoretically compile)!

$ jadx --deobf -e -d out ~/Downloads/justride-inspect.apk
INFO  - loading ...
INFO  - processing ...
INFO  - done                                                 
$ ls out/
app  build.gradle  settings.gradle

Making sense of the decompiled output

When I looked in jadx’s output directory I found a perfect copy of the original source code that Masabi had written, and my job was then very easy. Wait, no, that’s a lie.

Before shipping an APK to the end user, Android app developers usually run it through a number of steps to make it smaller including “obfuscation” — turning long class and member names like “mContext” into the smallest string they can get away with, like “a”. This means that the code jadx generated is rather hard to make sense of:

public class TicketInspectActivity extends BaseActivity implements InterfaceC2526a {

   /* renamed from: c */
   private ViewPager f4615c;

   /* JADX INFO: Access modifiers changed from: private */
   /* renamed from: h */
   public C2496p m1419h() {
       return (C2496p) this.f4615c.getAdapter();
   }

   @Override // com.masabi.app.android.ticketcheck.activities.BaseActivity
   /* renamed from: a */
   public final void mo1410a() {
       super.mo1410a();
       if (isFinishing()) {
           return;
       }
       m1419h().m1467a(this.f4615c.getCurrentItem());
   }
   /* ... */
}

Mmm yes, I knew exactly what they meant when they named their class C2496p. Of course! We just need to trace the execution of void mo1410a() and then we’ll figure it all out!⁶

Despite this looking daunting at first it’s actually quite okay with the tools Android Studio gives us. Not everything is completely obfuscated: some class names need to be left deobfuscated, such as activities (like this TicketInspectActivity). That lets us get some idea of where to start. The code also occasionally contains error messages that give away what the classes and methods are supposed to be:

/* renamed from: com.masabi.c.a */
/* loaded from: classes.dex */
public final class C2666a {
   /* renamed from: a */
   public static final int m552a(Calendar calendar) {
       if (calendar == null) {
           System.err.println("DateTimeUtils.packDate() ERROR - Attempt to pack a null date!");
       }
       return (C2668c.m542a(calendar) << 16) | (C2667b.m548a(calendar) & 65535);
   }
   /* ... */
}

In this case, the log line lets us instantly rename C2666a → DateTimeUtils, and m552a → packDate.

Android Studio also has excellent support for doing renames across an entire codebase at once, so after a long afternoon picking things apart it quickly started to take shape and our obfuscated code began to look something like the original source code might have looked⁷.

Interesting uses of RSA

My investigations into the app confirmed my suspicions that the data was indeed encrypted — well, not quite. Technically, the ticket data is actually signed with RSA and PKCS#1 (I think). Ticket issuers generate a payload containing the ticket data, pad it a bit, and then use their RSA private key to create a signed message they put into the barcode. A ticket scanner has a set of the issuers’ public keys on hand to verify the signature and read the original payload.

As a more concrete example, some vague Rust code to do the verification and reading steps looks a bit like this:

// BigUint is an arbitrary size unsigned integer.
// The ticket is base26 encoded, so we need to undo that first:
let ticket: BigUint = base26_decode(&ticket_str[15..]);
// this is doing “S^e mod N”;
// i.e. part of RSA signature verification
let message = ticket.modpow(&key.public_exponent, &key.modulus);
// convert big integer into raw bytes (big-endian)
let message: Vec<u8> = message.to_bytes_be();
// attempt to strip PKCS#1 padding; if it fails, the key is wrong
if let Some(unpadded) = strip_padding(&message) {
   eprintln!("[+] decrypt done: {:?}", unpadded);
}

I’m not a cryptographer, so this was all somewhat new to me! I was used to signatures being a hash of the original message (i.e. you’d send the plaintext, and then sign(hash(plaintext)) along with it), which is usually done so that you can sign messages longer than the size of your keys. In this case, they’ve put the whole message inside the signature to save space on the barcode, meaning you need the public keys to read the message at all.

You also can’t make your own fraudulent tickets using this scheme; you’d need the RSA private key of one of the ticket issuers to do that, or to have a custom public key added to the network of gate readers and ticket inspectors’ apps, neither of which seem easy to do.

Some further details on the cryptography (click to expand)

How can you tell that the ticket payload was unwrapped correctly? The payload is padded in a way that I think corresponds to some of the algorithms in PKCS#1 (see RFC 8017); it'll either be

Scheme 1: padded = [0x00, 0x01, padding-string, 0x00, message]
(where padding-string is a length of 0xFF octets)

Scheme 2: padded = [0x00, 0x02, padding-string, 0x00, message]
(where padding-string is a length of random non zero octets)

If the payload doesn't look like either of these, the RSA operation failed, so you probably have the wrong key and should try another one.

So the public keys are required knowledge for actually being able to decode these tickets. Where do we get those from?

Obtaining the elusive public keys

The public keys aren’t really published anywhere obvious, and reversing the Masabi app seems to indicate that it downloads the keys from a configuration server once you scan the config barcode mentioned earlier.

Global.logger.log(getClass().getSimpleName(), "loadAllKeys() - Fetched " + barcodeKeysList.length + " barcode keys from metadata");
for (int i = 0; i < barcodeKeysList.length; i++) {
   AbstractJSONObject key = (barcodeKeysList[i2];
   if (ExtendedGlobal2.clock.getCurrentTime() < Global.f4949d.mo915a(key.getString("expiryDate")) * 1000 &&
           (decoder = makeDecoder(key.getString("issuerId"), key.getString("ticketType"), key.getString("modulus"), key.getString("exponent"), key.getLong("mQ"))) != null) {
       decoders.addElement(decoder);
   }
}

You’d think this would be a dead end, since we don’t have any login credentials — but they also just left some keys inside the APK as well. As far as I can tell no part of the app actually reads these; maybe it did in the past, or maybe they used them for testing and forgot to take them out of the production version of the app.

$ find . | grep 'keys' | grep rsp6
./app/src/main/assets/keys/rsp6_rsa_ao.dat
./app/src/main/assets/keys/rsp6_rsa_ua.dat
./app/src/main/assets/keys/rsp6_rsa_tt-qa.dat
./app/src/main/assets/keys/rsp6_rsa_tt.dat
./app/src/main/assets/keys/rsp6_rsa_t3.dat
./app/src/main/assets/keys/rsp6_rsa_t2.dat

The keys are split up by ticket issuer, a 2-character code that forms the first part of the ticket ID. This ticket from earlier was issued by Trainline, who have issuer code TT…

…and the keys to decode this ticket are in rsp6_rsa_tt.dat. Nice!⁸

This is only a subset of all the keys that are being used today though, as I quickly discovered when unlobito gave me an Avanti West Coast ticket to decode. The copy of the app I have is only from 2016 and a bunch more TOCs have started issuing mobile tickets since then!

ttkMobile

Around when I was figuring all of this out puck pointed me towards the website for The Ticket Keeper, another firm who makes ticket validation and issuance tooling. They have an iOS app for ticket inspectors called ttkMobile, which you can just download straight off the App Store and use to start validating tickets at home!⁹

Important note if you actually intend to use this app (click to expand)

Be warned!

If you install this app, you can’t ever uninstall it and expect it to work after a reinstall. On first load it registers your device UUID with some server and generates a random password that it stores in local storage. Uninstalling the app removes the password, but your device UUID doesn’t change, so next time you reinstall, it won’t be able to authenticate and it’ll be useless (since it needs to grab keys and stuff to work).

(“But wait,” I hear you cry, “isn’t getting a persistent device ID exactly what Apple don’t want you to do?” And you’d be right! Technically, I believe the device UUID actually does change between installs, but they store a copy in the device keychain, which doesn’t get wiped when you remove the app. This is stupid, and almost certainly a contravention of App Store policy.)

I don’t have an iPhone, but some of my friends do. unlobito and another friend, Eva (“thejsa”), had a poke around and managed to get me a DRM-free¹⁰ .ipa containing the app, which I could unwrap and decompile with the help of Ghidra. This let me figure out some of the pieces of the ticket that the Masabi app didn’t look at.

thejsa also spent some time running the app through a proxy in order to find out how it communicates with the server¹¹, and it turns out there’s just an endpoint where you can get all of the public keys:

$ curl 'https://device.theticketkeeper.com/download_keys?device_name=abc' | jq .
{
  "return_code": "ok",
  "message": null,
  "keys": {
    "AA": [
      {
        "valid_from": "20000101000000",
        "valid_until": "29991231000000",
        "public_exponent_hex": "10001",
        "modulus_hex": "9140AA61F7D9A2E943C0510BACA5FA9CA7D12D78E301A36D640F2D28D8C0AA4D6A7102555CECF138E467730B797509EC1AB5BBA77CA6384BC8F483F609B121E75AE42660EDFE15EF91ADD4DA68C355F830FAAC6FFB25FBCFE1E61C7AF37C4AE8C85E264C151BD9C9AA4DE41D2756A9E260C0CC89AE2ADDD19E452A675E88DA47",
        "public_key_x509": null,
        "test_only": "N",
        "updated": "20200313175331"
      },
[etc]

As I mentioned earlier, this is crucial information to be able to decode tickets at all, so thanks go to The Ticket Keeper developers for making it available so easily!

Brief aside on freedom of information

I don’t know whether people in the industry (e.g. the Rail Delivery Group) will be upset with me publishing this information or not. I hope they won’t be: I really think the public keys should be made available to the public, along with the official specifications for decoding. The tickets are signed, so it’s not as if there’s any practical danger — people can’t use this to start forging tickets en masse, for example — and there are lots of potential innovative uses for this data. Imagine for example a journey logger that used ticket scans to track where you’d been automatically, or an expenses system that used the price information encoded in the ticket to automatically log expense requests!

The railways might be run by a consortium of private companies, but they are in effect a public service owned and controlled by the Government¹² (as of Jan 2023), so they really should be subject to the same Freedom of Information Act provisions as other public bodies.

Some people in the industry already have the right idea; in conversation with one of the Ticket Keeper developers over email, I was made aware that the ttkMobile app being public along with some of this data is actually an intentional choice, which is really nice to see!

Bonus: eTVD logs

The website also tells you how they have an electronic Ticket Validation Database (eTVD) that has a copy of all ticket scans at gatelines and by people using their app. This is the anti-fraud thing I mentioned at the very start; this sort of data is presumably very useful to revenue protection staff trying to figure out systematic fare evasion, like short-faring¹³.

What it doesn’t tell you, though, is that the app will also give you this information unauthenticated, with nothing more than a ticket’s ID (!).

This was reported to the developers as a possible security issue / data leak on 2023-01-19. They confirmed it was intended behaviour, but agreed that it would probably be a good idea to restrict it; I'm told this will happen soon.

This information can be quite disturbingly detailed, even pinpointing the exact username of the inspector who scanned you, where you were scanned, on what exact train service you were scanned, whether it succeeded, and a bunch more stuff. Helpfully it’ll also sometimes give you the entire barcode data, and what the ticket server thinks it decodes as, too!

# getting scan history information for ticket CBCZSCDPVFF
# (this is massively cut down; there are more fields in reality)
$ curl 'https://device.theticketkeeper.com/get_ticket_details?device_name=abc&utn=CBCZSCDPVFF' | jq '.["ticket_detail"]["scans"]'
[
  {
    "event_time_iso": "2022-06-10T18:30:47",
    "created": "2022-06-10T18:30:48",
    "device_type": "ttkMobile",
    "device_id": 1001,
    "device_name": "f95396f5-da22-47f9-8e85-dff4b2294a5d",
    "device_alias": "2021-TK10212",
    "username": "JLazlo01",
    "action_name": "Accepted",
    "rsp_action_code": 4001,
    "event_trigger": "scan",
    "scan_mode": "clip",
    "scan_nlc": "2728",
    "validation_result": "warning",
    "message_displayed": "16-25 Railcard",
    "gate_id": "OPN-3002i[021502]",
    "latitude": 51.7824963,
    "longitude": -0.2141781,
    "device_scan_id": 74402,
    "train_uid": "L77572",
    "departure_date": "2022-06-10",
    "barcode": "06CZSCDPVFF00…",
    "train_info": "Fr1803 KGX-SKI 1D26/GR2600",
    # etc
  },
  # etc
]

So yeah, your ticket barcode — or its ID, which is often written below the code in plain text — might let someone access a surprising amount of detailed tracking information as to where you are and what trains you’re taking! (Rather like the booking reference they send you when you book a flight.)

Trying this out for yourself

With the decompiled Masabi app and the ttkMobile app together, it wasn’t too hard to work out a vague idea of what the ticket format was like. I’ve put together a small repository with a Rust tool to decode a ticket, as well as a small spec with my best guess on what all the fields mean.

There’s also a funky web tool I threw together in an evening or so that’ll let you point your phone at a barcode (or upload a screenshot of one) and give you a relatively nice readout of what data’s inside. Give it a try! (If you need a barcode, feel free to scroll up and use the one from this post!)

Sorry, your browser doesn't support embedded videos.

Do feel free to get in touch if you find anything interesting or need help understanding something about the format!

Acknowledgements

Thanks to unlobito, puck, and thejsa (and assorted others in various chatrooms) for their help with all of this; go check those people out, too!

Footnotes

I still have fond memories of getting these as a child for local journeys to London Waterloo! ↩
This talk is a decent overview of the mess that is ITSO, if you’re interested. ↩
I did try with more than just two tickets, but that’d make for an unwieldy image. ↩
A VM provides a decent level of protection against running untrusted code, but it’s not the end-all and be-all; I’d imagine the Android emulator isn’t exactly hardened against people trying to break out, so this probably isn’t a bulletproof strategy for running shady apps in the general case. ↩
It turns out that the app actually downloads a lot of the keys and data necessary to decode tickets using this login, so even if I could somehow glitch it into starting the main ticket scanner, it wouldn’t have worked. ↩
jadx has made things slightly easier for us by renaming some identifiers (“deobfuscation”): adding some numbers and adding prefixes like “C” for “class” that help us distinguish amongst the multitude of things called “a” or “b”. ↩
It’s impossible to know without the real source what all the classes were actually called, but I can give them names that make sense to me, and that’s all that matters really. puck actually did manage to find an app that used some classes from the Masabi SDK that was not obfuscated later on, which let me compare and see how accurate my invented names were against the real ones! ↩
The format of these .dat files is a bit weird and nonstandard, and you have to reverse the RSA decryption code to figure out what it is. Or, you can be lazy like me, and just paste the decompiled code into a new Java project and treat it as a black box. ↩
It does expect to report back completed ticket scans by default, so you have to futz around and listen to it complain a bit before it’ll work. Also, the “Ticket Issuing” button doesn’t work, if you were wondering; you need a login for that. ↩
As I understand it, you usually need a jailbroken phone to do this. Otherwise, you could just copy paid apps between phones and share them with people who hadn’t paid for them. ↩
This is also how she discovered the reason why the app doesn’t work after being uninstalled. She even went so far as writing a jailbreak tweak to change the device ID to make it work again, which is pretty cool! ↩
Yes, our railway looks privatised on the surface, but COVID actually resulted in so-called Emergency Recovery Measures Agreements that forced the TOCs into effective state ownership and control (they’re now in effect contractors who run the service, rather than taking revenue risk). This means that the Government are actually responsible for things like the current union dispute over jobs and working conditions, despite what some ministers would have you think! ↩
This refers to the practice of buying a ticket that doesn’t actually cover your whole journey (e.g. skipping some stops at the start or end), and gambling that you’ll only get inspected between the stations where it’s actually valid. ↩

Pessimism is a bad defense mechanism

Wed, 30 Jun 2021 00:00:00 +0000

Warning! This post is a mental health / feels post from a long time ago (in fact, one gender ago). It is kept up in case some people find it useful, but I don't necessarily endorse the content in here nowadays.

Why do some people have poor mental health? It’s hard to know – there’s an entire profession centred on answering this question! – but I feel like a part of it has to do with doing things that sabotage oneself, without necessarily being aware of the fact that this is happening. Of course, the next question is inevitably why people even begin to do such things; surely if sabotaging yourself leads to bad mental health, people would just not do it, and therefore lead happier lives.

When you’ve had a history of things not going too well, it can often be difficult to accept that a situation is actually pretty great – because doing so involves accepting the idea that things might turn out to be not so great later on, which is painful, so doing so puts you in a vulnerable position. Some might argue this is silly, and you should just accept the risk and enjoy the benefits of the situation – but this all depends on how things have gone down in the past. If you’ve opened yourself up and had a bad time, you’re probably going to be inclined to avoid showing vulnerability in future.

Taking the avoidance of vulnerability to its extremes often involves a heavy degree of pessimism. If you judge everything to be terrible even when it isn’t, you can conveniently avoid feeling bad about stuff going wrong; you already mentally prepared yourself for this outcome anyway, so there’s no loss. And if it goes well, that’s a bonus! The events in the past that hurt you because you opened yourself up and showed vulnerability can’t any more; there’s a far smaller gap between your expectation and the reality if the reality does turn out to be bad.

As the title of this post says, this is a defence mechanism. I can definitely recall times when something hurt me, often to such an extent that I felt a need to burst out in tears (usually in some inopportune place, like on public transport) – but I didn’t; I clenched my teeth, and muttered something under my breath about the person who had hurt me being terrible all along, or the situation being doomed from the start, or some other overly-negative view that would manage to make the pain hurt less, somehow. Or, if it couldn’t do that, it would at least keep me from visibly crying in front of all the other commuters on the train.

Side note: what even happens if you cry on a train, just in front of people? Do people look at you like some sort of insane person and just let you get on with it in peace? Do complete strangers ask you if you’re alright? Maybe it isn’t actually too terrible – or maybe it’s really rude. Will I ever find out?

For one reason or another¹, I grew up with the idea that showing weakness is bad. To this day, it’s something that I find very difficult to do, and I often criticize myself for having done it. We all need to show weakness eventually, though; sometimes things just get way too much to bear, and it all comes flooding out. Unfortunately, this can perpetuate a vicious cycle: the only occasions where you open yourself up are hysterical disasters where you end up paragraphing one of your friends about all of your problems at once, or having a lengthy phone call where you’re the only one talking for a solid hour. This seems like a really imposing and terrible thing to do to people, so you conclude that showing weakness is bad, so you don’t do it as much… meaning the outburst is even worse next time, because you can’t hold the feelings back forever.

But when you’re trapped in the above cycle, it’s incredibly tempting to just push the inevitable feelings outburst forward by a few days/weeks/months with the liberal application of some pessimism.

Apart from this train-crying-reduction purpose, the other thing pessimism is very good at is making the actual causes of things far more vague and nebulous. You can avoid thinking about why something went wrong if you chalk the whole thing up to it just being a bad situation; this is particularly useful when you’re actually the one doing things to make a situation worse. Here’s an example²: maybe you’re sad about getting out of contact with a friend you haven’t seen in a while. If you take a pessimistic angle, you can usefully assume this is because people just naturally drift apart given space, which is unfortunate but makes the whole thing seem rather impersonal: it’s just a thing which is happening to you which you have no control over.

This isn’t how it works, though – in the above situation, the reason why you don’t talk to the friend could actually be because you don’t make much of an effort any more with them, because of the pessimism (“we’re going to drift apart anyway”). Another example: maybe you’re upset about being fired, apply for a new job, and don’t get it. In a lot of situations, you do indeed have no control over whether or not the company will hire you – but if you thought you’d never get a job again after getting fired and didn’t bother to put in the effort for the application as a result, you’re actually sabotaging yourself here.

This ability of pessimism to confuse situations can start to ruin your life if you’re not careful. If everything is terrible, what’s the point in doing anything? Opportunities to make things better pass you by without you doing anything (because you refuse to be brave enough to actually take things into your own hands and maybe fail), and can then serve as further examples of why everything is terrible when you start feeling bad about what you’ve missed out on, reinforcing the cycle. Because it’s not always obvious that you were the one that made the situation bad, you’re left feeling like you can’t do anything about it. Even when it’s clear that you made a mistake, you might view yourself as incontrovertibly bad – sure, it was your fault, but you’re a terrible person, so you’d never have made the right decision anyway (!).

Thinking like this leads to viewing life as this third-person experience where stuff just happens to you and you have to just sit there and take it – that you have no control over events. (Some people call this “learned helplessness”.) And that’s, to put it mildly, not good; you can get very depressed³ this way!

I would have liked to finish this blog post with some inspiring call to action about how to get out of this cycle and make all the things better in 3 simple steps. Unfortunately, I’m not actually qualified enough to do that.

Knowing what a problem is is often half the battle, however, and it’s definitely possible to use this knowledge for good. The research on learned helplessness actually suggests that it’s not helplessness people learn, it’s actually helpfulness – the default opinion of the brain is (allegedly) that you can’t do things to change your situation, and you have to teach it that you do actually have some power after all.

So, do some things, and pay attention to their effect. I bet at least in some cases, you have more ability to change things than you think.

Is going to a academically selective, all-male private school a reason? No, that couldn’t possibly be related… ↩
Frequent readers of the blog will know that the “examples” are often sanitized versions of things that have recently happened. ↩
I think it’s actually a lot more dangerous when you hold this belief underneath, but otherwise have what seems to be a relatively okay life. Every so often, something will happen that makes you sad for a while, and then you’ll get over it and act like nothing happened without realizing what’s going on – thereby ensuring that you’ll continue to become occasionally sad in perpetuity! ↩

Sending audio to LKV373 HDMI extenders

Sun, 20 Jun 2021 00:00:00 +0000

My current flatmate has a bunch of these HDMI extender devices (also apparently known as “LKV373” / “Lenkeng extenders” / “LK-3080-ACL”). They’re pretty nifty: HDMI goes in one end, they emit UDP over ethernet, and then HDMI comes out on the receiver end with surprisingly minimal latency (around 100ms).

image: the receiver end of a transmitter-receiver pair, taken by said flatmate

A couple months ago we decided it’d be quite nifty to be able to play arbitrary audio out of the devices without having to plug a transmitter into someone’s laptop – so that we could have BBC Radio 4 or something constantly playing whenever the sound system is turned on.

My flatmate’s blog post, “Ludicrously cheap HDMI capture for Linux”, outlined a way to decode the signal the transmitters sent (using his own tool, de-ip-hdmi). But could we do the opposite, and encode signals the receivers would accept?

Testing without the actual devices

I figured I’d start by looking at de-ip-hdmi’s source code to see how it decoded the packets and then use it as a way to test the output of my custom sender until it was able to decode what I was sending. It’s pretty unlikely that I’d just be able to send audio without also sending a picture, so I tried to figure out the picture format first.

The video picture packets have a fairly simple format: two big-endian, 16-bit integers (frame number, and sequence number), and then data, up to 1020 bytes (making UDP packets with a 1024-byte payload, total).

Extracting the output from de-ip-hdmi with a transmitter running confirms that the data is just a JPEG:

$ file test.mjpeg 
test.mjpeg: JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, baseline, precision 8, 1920x1080, components 3

The frame number counts monotonically up from 0, incrementing with each new frame. Since you’re not going to fit a whole JPEG frame in 1020 bytes, the sequence number provides a way of splitting the frame up into multiple chunks; you increment it for each chunk you send within a frame, while keeping the frame number constant.

The de-ip-hdmi source code to decode looks like this:

// taken from the de-ip-hdmi source code, in Go
FrameNumber := uint16(0)
CurrentChunk := uint16(0)

buf := bytes.NewBuffer(ApplicationData[:2])
buf2 := bytes.NewBuffer(ApplicationData[2:4])
binary.Read(buf, binary.BigEndian, &FrameNumber)
binary.Read(buf2, binary.BigEndian, &CurrentChunk)

// does stuff with ApplicationData[4:]

Seems simple enough. I wrote a small Rust struct that could chunk up a slice of bytes into 1024-byte vectors that fit this format:

struct FrameChunker<'a> {
    inner: &'a [u8], // the data to send
    frame_no: u16, // frame number
    seq_no: u16 // sequence number
}

impl<'a> FrameChunker<'a> {
    // constructor emitted for brevity

    // get a new 1024 (or less)-byte chunk to send
    fn next_chunk(&mut self) -> Vec<u8> {
        let mut ret = Vec::with_capacity(1024);
        let (remaining, last) = if self.inner.len() > 1020 {
            (1020, false)
        }
        else {
            (self.inner.len(), true)
        };
        // this is from the `byteorder` crate's `WriteBytesExt` trait
        ret.write_u16::<BigEndian>(self.frame_no).unwrap();
        ret.write_u16::<BigEndian>(self.seq_no).unwrap();
        ret.extend(&self.inner[..remaining]);
        self.inner = &self.inner[remaining..];
        self.seq_no += 1;
        ret
    }

    // returns true if we're done transmitting the current frame;
    // used to know when to send the next one
    fn is_empty(&self) -> bool {
        self.inner.is_empty()
    }
}

I then made a little 1920×1080 all white JPEG in GIMP, wrote it to a file, and wrote some code to send out UDP multicast packets in the format the receiver (and de-ip-hdmi) would expect:

// The magic IPv4 address for the multicast group the transmitter would send
// to.
const MAGIC_ADDRESS: Ipv4Addr = Ipv4Addr::new(226, 2, 2, 2);

fn main() {
    // load the test image data in at compile-time
    let test_data = include_bytes!("../white.mjpeg");

    // make a UDP socket for sending
    // note: this requires creating an interface with this address
    let mjpeg = UdpSocket::bind("192.168.168.55:2068").unwrap();
    // destination address:port for video picture data
    let mjpeg_dest: SocketAddr = "226.2.2.2:2068".parse().unwrap();

    // join the multicast group
    mjpeg.join_multicast_v4(&MAGIC_ADDRESS, &"192.168.168.55".parse().unwrap()).unwrap();

    // current frame number
    let mut frame_no = 0;
    loop {
        let mut chonker = FrameChunker::new(frame_no, image);
        while !chonker.is_empty() {
            let buf = chonker.next_chunk();
            mjpeg.send_to(&buf, mjpeg_dest).unwrap();
        }
        frame_no += 1;

        ::std::thread::sleep_ms(33);
    }
}

I added a suitable network interface for the sender to use (the actual flat networking setup uses VLANs for the HDMI traffic, but I found out that if you just want to test then adding a loopback interface also works):

$ sudo ip link add link enp5s0f3u1u1 name showtime0 type vlan id 1024
$ sudo ip link set dev showtime0 address 00:0b:78:00:60:01
$ sudo ip addr add 192.168.168.55/32 dev showtime0

With much anticipation I then ran the code for the first time, with de-ip-hdmi running in parallel – and got…nothing.

Adventures in bitshifting

Turns out I forgot something else in the packet format: a way to signal the last chunk in a series of chunks for a frame. Going back to de-ip-hdmi:

if uint16(^(CurrentChunk >> 15)) == 65534 {
	// Flush the frame to output
	// [code snipped]

	CurrentPacket = Frame{}
	CurrentPacket.Data = make([]byte, 0)
	CurrentPacket.FrameID = 0
	CurrentPacket.LastChunk = 0
}

What the heck is uint16(^(CurrentChunk >> 15)) == 65534 doing? Let’s break that down:

it shifts CurrentChunk to the right by 15 bits, discarding all bits apart from the most significant bit (MSB), which becomes the new least significant bit (LSB).
unary ^ (caret) is the Go “logical NOT” operator, so we flip all of the bits in the result above
65534 is a 16-bit integer with every bit set except the LSB

So… this is a very long-winded way of checking whether the MSB of CurrentChunk is equal to 1 (in Rust, (number & 0b1000_0000_0000_0000) > 0). Accordingly, we need to set the MSB to 1 to denote the end of a frame, so let’s do that in the FrameChunker:

ret.write_u16::<BigEndian>(self.frame_no).unwrap();
ret.write_u16::<BigEndian>(self.seq_no).unwrap();
if last {
    // it's big-endian, so ret[2] contains the most significant
    // 8 bits of the 16-bit integer
    ret[2] |= 0b1000_0000;
}

Partial victory!

With this change applied de-ip-hdmi would decode our image! Yay!

Sorry, your browser doesn't support embedded videos.

However, the real transmitter wasn’t having any of it; it refused to display a picture (behaving as if it couldn’t see any transmitters; it says “Searching TX” when this happens).

Hmm.

Let’s do the Wireshark again

I connected a real transmitter and dumped some of the packets it sends using Wireshark. Turns out there’s more stuff you need to do.

Mysterious packets on port 2067

Here’s a screenshot of some of the packets the real transmitter sends.

the (source port) 2065 → (destination port) 2066 packets are audio data. We’ll get to those later!
the 2068 → 2068 packets are video picture data
…but what are these 2067 → 2067 packets?

Hmm, it’s all zeroes. What about the next one?

Aha, a byte went from 00 to 01!

Turns out this pattern continues; before each picture frame is sent, an additional packet is sent on port 2067 of length 20, with the 5th and 6th bytes being a 16-bit big-endian integer, which is the same as the frame number of the frame we just sent.

I’m guessing this is a sort of “vsync” / “vblank” packet, used to tell the receiver to get ready for a new frame. We should probably replicate it, so I wrote some code:

fn make_vsync(frame_no: u16) -> Vec<u8> {
    let mut ret = Vec::with_capacity(20);
    ret.write_all(&[0, 0, 0, 0]).unwrap();
    ret.write_u16::<BigEndian>(frame_no).unwrap();
    for _ in 0..14 {
        ret.write_all(&[0]).unwrap();
    }
    ret
}
// code to send this to port 2066 before sending frame packets elided

Alas, this was not enough. The receiver continued to say “Searching TX”.

Heartbeat packets

There are also some weird 512-byte payloads being sent on port 48689. These are easy to miss because they’re only sent once every second (and so are comparatively infrequent relative to the firehose of data the transmitters send for pictures and video). However, you can notice them by looking at the start of a packet capture since they get sent even when no picture data is being sent.

Let’s look at a dump of some of these packet payloads over time (using “[right click] Follow → UDP Stream” in Wireshark):

After squinting at it for a while, you notice a few things:

The first three packets are identical, apart from a small 2-byte region that carries on going up with each new packet (marked in red), and a sequence number incremented with each new packet (marked in pink).
Between the third and fourth packets, a large chunk of data (marked in blue) changes to something else (marked in green), and then stays that way.
This change is also correlated with the video and audio starting.
After the change happens the packets stay identical apart from the counter.

This must be some kind of heartbeat packet sent by the transmitter to tell the receiver that it’s there. Let’s try and figure out the various bits in the payload in more detail.

Incrementing counter

How is this counter being generated? Well, let’s assume that it’s a 16-bit big-endian integer, since everything else in this blog post has been that, and interpret the values accordingly.

0x0458 is 1112, sent at time t = 0.0
0x0842 is 2114 (delta = 1002), sent at t = 1.001
0x0c2b is 3115 (delta = 1001), sent at t = 2.002
0x0ee5 is 3813 (delta = 698), sent at t = 2.700

Hmm…

I think it’s pretty reasonable to assume this counter just tracks time since the transmitter was powered on. Why the receiver needs that information isn’t exactly clear though (maybe for some synchronization purposes?).

Anyway, we should probably do the same as the transmitter when sending our own packets here so we’ll add a timer to the code.

Other gubbins

We have two sets of weird data:

00 03 07 80 04 38 02 57 07 80 04 38 00 78 03 (when the HDMI port is active)
00 10 00 00 00 00 00 00 00 00 00 00 00 78 00 (when it isn’t)

07 80 and 04 38 are repeated multiple times. Let’s try decoding those as 16-bit big-endian integers:

0x0780 is 1920
0x0438 is 1080

That seems familiar; isn’t 1920×1080 the resolution for 1080p? This data is presumably used to initialize the HDMI port on the other end.

What about when the HDMI is off? Well, it looks like most of the fields are just zeroed out, which makes sense since it doesn’t have a resolution or anything to send yet. (This causes the message “Check Tx’s input signal” to appear on the receiver, which is useful for debugging why the receiver isn’t displaying anything.)

In the end, I just encoded all of this data as constants (interpreting it as a set of 16-bit big endian integers), sending one set (the one with 1920s in it) if we’re ready to send data, and another set (the zeroed out set) if we aren’t.

Putting it all together

The code to make a heartbeat packet ended up looking like this:

fn make_heartbeat(seq_no: u16, cur_ts: u16, init: bool) -> Vec<u8> {
    let mut ret = Vec::with_capacity(512);
    ret.write_all(&[0x54, 0x46, 0x36, 0x7a, 0x63, 0x01, 0x00]).unwrap();
    ret.write_u16::<BigEndian>(seq_no).unwrap();
    ret.write_all(&[0x00, 0x00, 0x03, 0x03, 0x03, 0x00, 0x24, 0x00, 0x00]).unwrap();
    for _ in 0..8 {
        ret.write_all(&[0]).unwrap();
    }
    // woohoo, magic numbers
    let (mode, a, b, c, d, e, f, other) = if init {
        (3, 1920, 1080, 599, 1920, 1080, 120, 3)
    }
    else {
        (16, 0, 0, 0, 0, 0, 120, 0)
    };
    for thing in &[mode, a, b, c, d, e, f] {
        ret.write_u16::<BigEndian>(*thing).unwrap();
    }
    ret.write_all(&[0, 0]).unwrap();
    ret.write_u16::<BigEndian>(cur_ts).unwrap();
    ret.write_all(&[0, 1, 0, 0, 0, 0, other, 0x0a]).unwrap();
    for _ in ret.len()..512 {
        ret.write_all(&[0]).unwrap();
    }
    assert_eq!(ret.len(), 512);
    ret
}

Putting it all together and sending heartbeat packets, vsync packets, and the actual picture data… got the receiver to initialize, but output a black rectangle instead of the test image.

Sorry, your browser doesn't support embedded videos.

After trying many other things, I eventually tried sending the original test.mjpeg file I’d dumped from the receiver when first trying to get this to work, instead of my own JPEG that I made in GIMP. That actually produced some output!

Sorry, your browser doesn't support embedded videos.

Yay!

(the test image was produced by writing some text on a laptop, making it full-screen, and connecting it to the transmitter)

But what about audio?

While this is (somewhat) useful, it doesn’t do what we originally wanted; what about actually sending audio?

How does digital audio work, anyway?

Digital audio is represented as a series of samples – individual measurements of what the voltage of your speaker should be at a given point in time (the “signal”), taken at a given sampling rate that’s high enough to provide a decent approximation of what the speaker should be doing to reproduce your favourite tunes.

image: sourced from the above linked Wikipedia article, CC0

There are a number of important variables involved in this process that we need to know in order to send and receive audio in a given format:

what the sample rate was (measured in Hertz, aka “times per second”)
how the samples are stored (as floating-point numbers? as signed integers? how many bits?)
what the “maximum” and “minimum” values are (for floats, we typically use 1.0 and -1.0; integer formats use INT_MAX and INT_MIN)

What format do the extenders use?

de-ip-hdmi has already figured this out for us:

if audio {
    ffmpeg = exec.Command("ffmpeg", "-f", "mjpeg", "-i", uuidpath, "-f", "s32be", "-ac", "2", "-ar", "44100", "-i", "pipe:0", "-f", "matroska", "-codec", "copy", "pipe:1")
}

This is a ffmpeg command line that de-ip-hdmi uses to wrap the data from the extenders in a Matroska container (a way of combining audio and images together into one video file). We can tell a few things from this:

-f s32be: the samples are stored as signed 32-bit big-endian integers
- this uses i32::MIN and i32::MAX as minimum/maximum (i.e. -2147483648 and 2147483647)
-ac 2: there are 2 audio channels
- the samples from each channel will be interleaved, since we just have one big audio stream
-ar 44100: the sample rate is 44100 Hz

It looks like de-ip-hdmi just treats everything after the 16th byte as a set of samples in the above format:

// Maybe there is some audio data on port 2066?
if pkt.Data[36] == 0x08 && pkt.Data[37] == 0x12 && *output_mkv && *audio {
    select {
    case audiodis <- ApplicationData[16:]:
    default:
    }
    
    continue
}

Going back to our Wireshark dump, the first 16 bytes of every audio packet seem to be identical (and the packets are always 1008 bytes long):

So, to send audio data we should in theory be able to write the header and then 992 (1008 - 16) bytes worth of samples into a packet and send it as UDP from port 2065 to port 2066.

However, how frequently should we send the packets?

Timing is everything

We need to provide 44100 × 2 = 88200 samples each second, since we have 2 audio channels at 44100 Hz sample rate. Each packet of audio contains 996 bytes of samples at 32 bits (= 4 bytes) per sample, so 992 / 4 = 248 samples fit in a packet.

That means we need to send a packet every 248 / 88200 seconds, or roughly 2.812 milliseconds (to 3 decimal places) – which seems to line up with the frequency observed in the Wireshark dump:

(e.g. 2.9222 s - 2.9195 s = 2.7ms, 2.9251s - 2.9222s = 2.9ms, etc)

I wrote up some sample code that read samples from stdin (fed in using ffmpeg) and sent a packet, then slept for 2812 nanoseconds, and then repeated. This resulted in…complete silence!

That is, until we changed the device connected to the HDMI receiver unit (a hifi system with HDMI passthrough) to another device (a projector). The projector happily played the audio we were sending… but it sounded distorted (kinda “crunchy” sounding). What’s going on?

Jitter is unacceptable

A while ago, I wrote some audio code for a project called “SQA”. This taught me one of the ‘cardinal rules’ of audio programming: jitter is unacceptable. There’s a great blog post called “Real-time audio programming 101: time waits for nothing” that explains this in more detail but to sum up: when supplying audio samples for playback, you have a fixed amount of time to fill a buffer of a certain size, and you must always fill the buffer in the time given to you (or less).

If you leave the buffer half-filled the audio will “glitch”: it’ll have pops and clicks in it and/or sound distorted, because the new samples you’ve only half written will stop halfway through the playback buffer and jump suddenly to a bunch of zeroes instead (or some other garbage data). This is called a buffer under/overrun, or an xrun for short.

So, how do you make sure you never glitch? Well, doing anything that could block or take lots of time to complete (reading from a file, waiting on a mutex, etc.) in the thread that handles audio is strictly forbidden; you’ve basically got to have something else put the audio samples in a ringbuffer for you to copy out in the audio thread.

Final audio thread code

Armed with the above knowledge, I completely rewrote the audio thread – and it actually works!

It now uses the excellent bounded_spsc_queue ringbuffer library to ship samples from the rest of the program and the audio thread, and spin-waits (loops) until it needs to send the next packet. (Doing the “proper” thing and using something like nanosleep(2) would be more efficient – but was unreliable compared to just spinning in my tests.)

The full audio thread code is below:

fn audio_thread(udp: UdpSocket, dest: SocketAddr, queue: Consumer<u8>) {
    let mut pkt = [0u8; 1008];
    // Pre-write the header on to the packet.
    for (i, b) in [0, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0x55, 0, 0, 0].iter().enumerate() {
        pkt[i] = *b;
    }
    let now = Instant::now();
    // Monotonic nanosecond counter.
    let cur_nanos = || {
        Instant::now().duration_since(now).as_micros()
    };
    // 2.812 milliseconds!
    const INTERVAL: u128 = 2812;
    // Next time (on the counter) we need to send a packet.
    let mut next = INTERVAL;
    // Current buffer pointer.
    let mut ptr = 16;
    // Consecutive Xrun counter.
    let mut num_xruns = 0;
    loop {
        let cur = cur_nanos();
        if cur >= next {
            // Send the packet.
            udp.send_to(&pkt, &dest).unwrap();
            // Check if we managed to fill the buf; if not, we've xrun'd.
            if ptr != 1008 {
                num_xruns += 1;
            }
            else {
                // Reset the pointer.
                ptr = 16;
                num_xruns = 0;
            }
            // Advance the deadline so it's 2.8ms after the previous one.
            // We skip over if we're already going to miss it.
            while cur >= next {
                next += INTERVAL;
            }
        }
        // Okay, now time to fill the buffer a bit.
        if ptr < 1008 {
            if num_xruns > 5 {
                // Let's just fill it up with silence instead.
                // (otherwise you get a buzzing effect that's highly irritating)
                pkt[ptr] = 0;
                ptr += 1;
            }
            else {
                if let Some(smpl) = queue.try_pop() {
                    pkt[ptr] = smpl;
                    ptr += 1;
                }
            }
        }
        else {
            ::std::thread::yield_now();
        }
    }
}

Conclusions

The code is here. Don’t expect miracles. (If you actually plan to use this, you might want to get in touch.)

There are a number of ways to feed data into it, which you choose at compile time using Cargo features. You can either feed in raw s32be samples from either stdin or TCP (it’ll make a TCP server and you can connect and send stuff, but not with more than one client at the same time…). I also ended up adding a JACK backend (using my own sqa-jack library), if that’s your thing.

Why asynchronous Rust doesn't work

Mon, 08 Mar 2021 00:00:00 +0000

In 2017, I said that “asynchronous Rust programming is a disaster and a mess”. In 2021 a lot more of the Rust ecosystem has become asynchronous – such that it might be appropriate to just say that Rust programming is now a disaster and a mess. As someone who used to really love Rust, this makes me quite sad.

I’ve had a think about this, and I’m going to attempt to explain how we got here. Many people have explained the problems with asynchronous programming – the famous what colour is your function essay, for example.¹ However, I think there are a number of things specific to the design of Rust that make asynchronous Rust particularly messy, on top of the problems inherent to doing any sort of asynchronous programming.

In particular, I actually think the design of Rust is almost fundamentally incompatible with a lot of asynchronous paradigms. It’s not that the people designing async were incompetent or bad at their jobs – they actually did a surprisingly good job given the circumstances!² I just don’t think it was ever going to work out cleanly – and to see why, you’re going to have to read a somewhat long blog post!

A study in async

I’d like to make a simple function that does some work in the background, and lets us know when it’s done by running another function with the results of said background work.

use std::thread;

/// Does some strenuous work "asynchronously", and calls `func` with the
/// result of the work when done.
fn do_work_and_then(func: fn(i32)) {
    thread::spawn(move || {
        // Figuring out the meaning of life...
        thread::sleep_ms(1000); // gee, this takes time to do...
        // ah, that's it!
        let result: i32 = 42;
        // let's call the `func` and tell it the good news...
        func(result)
    });
}

There’s this idea called “first-class functions” which says you can pass around functions as if they were objects. This would be great to have in Rust, right?

See that func: fn(i32)? fn(i32) is the type of a function that takes in one singular i32 and returns nothing. Thanks to first-class functions, I can pass a function to do_work_and_then specifying what should happen next after I’m done with my work – like this:

fn main() {
    do_work_and_then(|meaning_of_life| {
        println!("oh man, I found it: {}", meaning_of_life);
    });
    // do other stuff
    thread::sleep_ms(2000);
}

Because do_work_and_then is asynchronous, it returns immediately and does its thing in the background, so the control flow of main isn’t disrupted. I could do some other form of work, which would be nice (but here I just wait for 2 seconds, because there’s nothing better to do). Meanwhile, when we do figure out the meaning of life, it gets printed out. Indeed, if you run this program, you get:

oh man, I found it: 42

This is really exciting; we could build whole web servers and network stuff and whatever out of this! Let’s try a more advanced example: I have a database I want to store the meaning of life in when I find it, and then I can run a web server in the foreground that enables people to get it once I’m done (and returns some error if I’m not done yet).

struct Database {
    data: Vec<i32>
}
impl Database {
    fn store(&mut self, data: i32) {
        self.data.push(data);
    }
}

fn main() {
    let mut db = Database { data: vec![] };
    do_work_and_then(|meaning_of_life| {
        println!("oh man, I found it: {}", meaning_of_life);
        db.store(meaning_of_life);
    });
    // I'd read from `db` here if I really were making a web server.
    // But that's beside the point, so I'm not going to.
    // (also `db` would have to be wrapped in an `Arc<Mutex<T>>`)
    thread::sleep_ms(2000);
}

Let’s run this…oh.

error[E0308]: mismatched types
  --> src/main.rs:27:22
   |
27 |       do_work_and_then(|meaning_of_life| {
   |  ______________________^
28 | |         println!("oh man, I found it: {}", meaning_of_life);
29 | |         db.store(meaning_of_life);
30 | |     });
   | |_____^ expected fn pointer, found closure
   |
   = note: expected fn pointer `fn(i32)`
                 found closure `[closure@src/main.rs:27:22: 30:6]`

I see.

Hang on a minute…

So, this is actually quite complicated. Before, the function we passed to do_work_and_then was pure: it didn’t have any associated data, so you could just pass it around as a function pointer (fn(i32)) and all was grand. However, this new function in that last example is a closure: a function object with a bit of data (a &mut Database) tacked onto it.

Closures are kind of magic. We can’t actually name their type – as seen above, the Rust compiler called it a [closure@src/main.rs:27:22: 30:6], but we can’t actually write that in valid Rust code. If you were to write it out explicitly, a closure would look something like this:

struct Closure<'a> {
    data: &'a mut Database,
    func: fn(i32, &mut Database)
}

impl<'a> Closure<'a> {
    fn call(&mut self, arg: i32) {
        (self.func)(arg, self.data)
    }
}

There are a number of things to unpack here.

An aside on naming types

Being able to name types in Rust is quite important. With a regular old type, like u8, life is easy. I can write a function fn add_one(in: u8) -> u8 that takes one and returns one without any hassle.

If you can’t actually name a type, working with it becomes somewhat cumbersome. What you end up having to do instead is refer to it using generics – for example, closures’ types can’t be named directly, but since they implement one of the Fn family of traits, I can write functions like:

fn closure_accepting_function<F>(func: F)
where
    F: Fn(i32), // <-- look!
{
    /* do stuff */
}

If I want to store them in a struct or something, I’ll also need to do this dance with the where clause every time they’re used. This is annoying and makes things harder for me, but it’s still vaguely workable. For now.

[image: from the msql-srv crate, showing an example of many where clauses as a result of using closures]

An aside on ‘radioactive’ types

The way Rust is designed tends to encourage certain patterns while discouraging others. Because of ownership and lifetimes, having pieces of data that hold references to other pieces of data becomes a bit of a problem. If my type has a & or a &mut reference to something, Rust makes me ensure that:

the something in question outlives my type; you can’t go and drop the thing I’m referring to if I still have a reference to it, otherwise my reference will become invalid
the something in question doesn’t move while I have the reference
my reference to the something doesn’t conflict with other references to the something (e.g. I can’t have my & reference if something else has a &mut reference)

So types with references in them are almost ‘radioactive’; you can keep them around for a bit (e.g. inside one particular function), but attempting to make them long-lived is usually a bit of an issue (requiring advanced tricks such as the Pin<T> type which didn’t even exist until a few Rust versions ago). Generally Rust doesn’t really like it when you use radioactive types for too long – they make the borrow checker uneasy, because you’re borrowing something for an extended period of time.

[image: from the RustViz paper, showing borrowing semantics]

Closures can be pretty radioactive. Look at the thing we just wrote out: it has a &'a mut Database reference in it! That means while we’re passing our Closure object around, we have to be mindful of the three rules (outlives, doesn’t move, no conflicting) – which makes things pretty hard. I can’t just hand off the Closure to another function (for example, the do_work_and_then function), because then I have to make all of those rules work, and that’s not necessarily easy all the time.

(Not all closures are radioactive: if you make them move closures, they’ll take everything by value instead, and create closure objects that own data instead of having radioactive references to data. Slightly more of a pain to deal with, but you lose the blue radiation glow the things give out when you look at them.)

Also, remember what I said about being able to name types? We’re not actually dealing with a nice, written-out Closure object here; we’re dealing with something the compiler generated for us that we can’t name, which is annoying. I also lied when I said that it was as simple as making all of your functions take F, where F: Fn(i32) or something – there are actually three different Fn-style traits, Fn, FnMut, and FnOnce. Do you know the difference between them?

So. A closure is this magical, un-nameable type that the compiler makes for us whenever we use || {...} syntax, which implements one of three traits (and it’s not immediately obvious which), and it also might be radioactive. Try and use one of these, and the Rust compiler is probably going to be watching you very carefully.

The thing I really want to try and get across here is that Rust is not a language where first-class functions are ergonomic. It’s a lot easier to make some data (a struct) with some functions attached (methods) than it is to make some functions with some data attached (closures).

Trying to use ordinary structs is downright easy:

they’re explicitly written out by the programmer with no funky business
you choose what traits and methods to implement on them and how to set them out / implement them
the struct can actually be referred to by other parts of the code by its type

Trying to use closures is hard:

the compiler does some magic to make a closure type for you
it implements some obscure Fn trait (and it’s not immediately obvious which)
it might be radioactive (or force you to use move and maybe insert a bunch of clone() calls)³
you can’t actually name their type anywhere or do things like return them from a function

Importantly, the restrictions applied to using closures infect types that contain them – if you’re writing a type that contains a closure, you’ll have to make it generic over some Fn-trait-implementing type parameter, and it’s going to be impossible for people to name your type as a result.

(Other languages, like Haskell, flip this upside down: functions are everywhere, you can pass them around with reckless abandon, etc. Of course, these other languages usually have garbage collection to make it all work…)

Bearing this in mind, it is really quite hard to make a lot of asynchronous paradigms (like async/await) work well in Rust. As the what colour is your function post says, async/await (as well as things like promises, callbacks, and futures) are really a big abstraction over continuation-passing style – an idea closely related to the Scheme programming language. Basically, the idea is you take your normal, garden-variety function and smear it out into a bunch of closures. (Well, not quite. You can read the blue links for more; I’m not going to explain CPS here for the sake of brevity.)

Hopefully by now you can see that making a bunch of closures is really not going to be a good idea (!)

wibbly wobbly scene transition

And then fast forward a few years and you have an entire language ecosystem built on top of the idea of making these Future objects that actually have a load of closures inside⁴, and all of the problems listed above (hard to name, can contain references which make them radioactive, usually require using a generic where clause, etc) apply to them because of how “infectious” closures are.

The language people have actually been hard at work to solve some (some!) of these problems by introducing features like impl Trait and async fn that make dealing with these not immediately totally terrible, but trying to use other language features (like traits) soon makes it clear that the problems aren’t really gone; just hidden.

Oh, and all the problems from what colour is your function are still there too, by the way – on top of the Rust-specific ones.

Beginner (and experienced) Rust programmers look at the state of the world as it is and try and build things on top of these shaky abstractions, and end up running into obscure compiler errors, and using hacks like the async_trait crate to glue things together, and end up with projects that depend on like 3 different versions of tokio and futures (perhaps some async-std in there if you’re feeling spicy) because people have differing opinions on how to try and avoid the fundamentally unavoidable problems, and it’s all a bit frustrating, and ultimately, all a bit sad.

Did it really have to end this way? Was spinning up a bunch of OS threads not an acceptable solution for the majority of situations? Could we have explored solutions more like Go, where a language-provided runtime makes blocking more of an acceptable thing to do?

Maybe we could just have kept Rust as it was circa 2016, and let the crazy non-blocking folks⁵ write hand-crafted epoll() loops like they do in C++. I honestly don’t know, and think it’s a difficult problem to solve.

But as far as my money goes, I’m finding it difficult to justify starting new projects in Rust when the ecosystem is like this. And, as I said at the start, that makes me kinda sad, because I do actually like Rust.

(Common Lisp is pretty nice, though. We have crazy macros and parentheses and a language ecosystem that is older than I am and isn’t showing any signs of changing…)

This is really recommended reading if you aren’t already familiar with it (as you’ll soon see) ↩
Seriously – when I put out the last blog post, the actual async core team members commented saying how much they appreciated the feedback, and then they actually went and made futures 1.0 better as a result. Kudos! ↩
You might be thinking “well, why don’t you just only use move closures then?” – but that’s beside the point; it’s often a lot harder to do so, because now you might have to wrap your data in an Arc or something, by which point the ergonomic gains of using the closure are outweighed by the borrow checker-induced pain. ↩
You can actually manually implement Future on a regular old struct. If you do this, things suddenly become a lot simpler, but also you can’t easily perform more async operations inside that struct’s methods. ↩
(sorry, I mean, the esteemed companies that deign to use Rust for their low-latency production services) ↩

Getting PIV-based SSH working on a YubiKey

Sat, 06 Mar 2021 00:00:00 +0000

I bought a YubiKey 5C Nano recently. These devices are great – I’ve built a lot of my (metaphorical) empire on top of them, seeing as they’re capable of acting as an SSH agent (store your SSH keys on them, securely!), an OpenPGP smartcard (do encryption and decryption on the key!), FIDO U2F ‘security keys’ (use them as a 2-factor authentication method!), and probably more.

Getting the thing to work as an SSH agent was, however, not the easiest thing I’ve ever done. There are multiple options here – you can use the OpenPGP applet and then configure GnuPG to work as an SSH agent, but that’s a brittle solution in my experience (gpg-agent is quite flaky, and often requires restarting when it forgets about the YubiKey). Instead, I wanted to see whether I could use the YubiKey’s PIV (Personal Identity Verification) applet to get this working.

Procedure

For the SSH agent part, we’re going to use Filippo Valsorda’s yubikey-agent, so you’ll want to have that installed. In my testing, yubikey-agent’s built-in yubikey-agent -setup command errored out, so we’ll configure the PIV applet of the YubiKey manually. (This’ll also leave the door open for you to do other things with the PIV applet later if you like.)

You’ll also need Yubico’s own yubikey-manager (the ykman cli tool) installed.

Updated on 2023-11-23 to reflect the command names being moved around.

Okay, here goes:

Ensure pcscd (the PC/SC smartcard daemon) is installed and running. This might already have been done for you by your Linux distribution but, if not:
- $ sudo systemctl enable --now pcscd
- (This is safe to run if it’s already been set up; the command will just do nothing).
This exercise assumes a fresh YubiKey (i.e. one where you haven’t touched the PIV applet yet). If that’s not the case, and you want to erase all of the PIV data and start afresh (losing all data encrypted with the PIV keys!), use the ykman piv reset command.
First, change the PIN from the default one (123456).
- $ ykman piv access change-pin
- As the command’s help says: The PIN must be between 6 and 8 characters long, and supports any type of alphanumeric characters. For cross-platform compatibility, numeric digits are recommended.
Then, change the PUK (‘personal unblocking key’). This is used to reset the PIN if you ever forget it.
- $ ykman piv access change-puk
- The PUK has the same entry requirements as the PIN (i.e. also 6-8 ASCII characters).
- It might be prudent to generate a random PUK and keep it safe (e.g. by writing it down and locking the paper away). If you lose both the PIN and PUK, you will need to reset the PIV applet, losing all data encrypted with the PIV keys (and SSH access to hosts you don’t otherwise have access to).
The YubiKey PIV applet by default has a well-known management key used to make changes to the PIV keys (etc.). It’s best practice to change this to something else. We’ll use the option to generate a random one, store it in the YubiKey, and secure it with the PIN.
- $ ykman piv access change-management-key -pt
- -p: Store new management key on your YubiKey, protected by PIN. A random key will be used if no key is provided.
- -t: Require touch on YubiKey when prompted for management key.
- If you’re more paranoid than me, you can use the -g option instead, which will generate a key for you to note down and give back later. However, the extent to which you can cause damage with a management key is limited (you can delete and regenerate keys, but not decrypt data or anything), so this is arguably not worth the hassle – especially since forgetting the management key means you’d have to reset the PIV applet to make changes.
Generate a public/private keypair on the key, using the 9a (“PIV Authentication”) key slot. We’ll use ECC (elliptic-curve cryptography) because it’s fast, secure, and has vastly smaller key sizes, and configure the key to always require touches and require the PIN on first use. You might want to use different settings here.
- $ ykman piv keys generate -a ECCP256 --touch-policy ALWAYS --pin-policy ONCE 9a ./yubikey-public.pem
- Note: Some outdated SSH servers/implementations only support RSA keys. If this applies to you, leave off the -a option to use RSA 2048 instead.
- See ykman piv keys generate -h for a full description of all available options.
Use the newly generated key to make a self-signed PKCS#11 certificate to act as our SSH identity.
- $ ykman piv certificates generate -s 'my-yubikey-ssh' -d 365 9a ./yubikey-public.pem
- Modify the -s parameter to include a human-readable description of the key or the machine the key is installed in.
- Modify the -d parameter to set how many days the key will be valid for. The value above specifices 1 year.
- You can get rid of yubikey-public.pem after this step.
Enable yubikey-agent.
- $ systemctl --user enable --now yubikey-agent
- Note: You might need to kill any running instances of gpg-agent if you had that running (and it decided to try and use your YubiKey), and potentially restart pcscd after doing so.
- Note: Check that yubikey-agent started properly with systemctl --user status yubikey-agent.
  - On my machine, it failed to start because of namespacing issues.
  - If this happens, edit the unit file with systemctl --user edit --full yubikey-agent, and remove all lines under [Service] apart from ExecStart and ExecReload.
Update SSH_AUTH_SOCK to point to the running instance of yubikey-agent.
- $ export SSH_AUTH_SOCK=/run/user/1000/yubikey-agent/yubikey-agent.sock
- The above command assumes bash. Other shells may vary.
- If your user ID isn’t 1000, you’ll need to change the above command (you can find the right path by ps aux | grep yubikey-agent).
- Note: You’ll want to put the above command in .bashrc or similar.
Get the key fingerprint for your newly generated SSH identity.
- $ ssh-add -L
Copy the key fingerprint to your remote host(s), and put it in ~/.ssh/authorized_keys.
You should now be able to SSH using the YubiKey!
- $ ssh <host>
- The first time you try this, it should pop up a window asking for the PIN, after which you’ll need to touch the flashing YubiKey.
- Subsequent attempts will only require a touch.
- If it didn’t work, check the notes under step 8. In particular, using OpenPGP and SSH simultaneously with the YubiKey usually requires you to kill either pcscd or gpg-agent, depending on what you want to do.
Ensure you have a backup plan in place for when this YubiKey fails, or is lost.
- The key we generated is local to the YubiKey, and cannot be exported. It can very easily be wiped by anyone running ykman piv reset (which requires no authentication!), or you could lose the key.
- Make sure you have another way of getting into all hosts you use this YubiKey with (encrypted regular SSH key, out-of-band console, etc.).
You’re done!

Hopefully that worked. If it didn’t, well, there’s a comments thing below if you can be bothered, I guess?

Setting the tone in a group is very important

Fri, 15 Jan 2021 00:00:00 +0000

Various people have various different opinions on how they should present themselves. Personally, I’d consider myself quite a sensitive person; I’d like to think I try quite hard to take the feelings and circumstances of other people into account when talking to them, although doing so is by no means easy or automatic (and I often fail at this, sometimes quite badly). Partially, this is because I often have a lot of feelings and circumstances going in my life myself which I’d like other people to attempt to take into account – in fact, especially nowadays, I’d argue that the overwhelming majority of people have some topics or areas that make them uncomfortable, or that it’d be possible to upset them with a correctly targeted remark.

It’s very hard to judge what might upset or offend someone, simply because there can be a lot of stuff going on behind the scenes that people just don’t tell others about (it’s almost as if having an entire country where not talking about stuff is the norm could lead to significant problems!). That said, you can at least attempt to be reactive – it’s sometimes possible to detect that something you’ve said or done wasn’t received well, and try to do less of that thing in future.

There are, of course, cultures and groups where this sort of thing is very much not the norm. Some people – and groups of people – attempt to act in a sort of “macho” / “tough guy” sort of way, where one is supposed to pretend that one doesn’t really have feelings, or that one is immune to whatever life might throw in one’s way. This, of course, is obviously false – everyone has feelings! – but it suits them to conduct themselves in this manner, because, at the end of the day, talking about one’s feelings can be very hard.

Maybe you weren’t brought up in an environment where people do that; maybe you never saw your parents, or your school friends, cry, or be angry, or show feelings toward things, because they thought they had to be ‘strong’). Maybe you had people express too much of their feelings in your past and really got put off by it (as in, maybe you were distressed by other people having issues and now have decided that ‘burdening’ others with your feelings isn’t a good idea). Maybe you don’t have any friends you really trust enough to be able to confide in, perhaps because they’re all tough guys, or perhaps because you have issues trusting people for some other reason – especially in our modern society, that reason can often be loneliness, an epidemic that people don’t actually realise the severity of.

But anyway, you don’t want to talk about your feelings. I get that, because I don’t really want to either¹. Being in that position, while it’s not really a good thing for you long-term, isn’t really wrong; you aren’t hurting anyone except yourself there (although if you read the previous paragraph and found it hit a bit too close to home, you should probably watch this video).

It’s not uncommon for large-ish (i.e. more than 3 or 4) groups of people to have leaders, whether explicitly or implicitly allocated. Usually there are one or two people who do a lot of the talking – who appear to set the tone and the rules of engagement for the rest of the group. This doesn’t always have to happen, of course, but what I’m saying is it probably happens more than you realise. (It can also become painfully obvious when the leaders step out to go and do something else for a bit and you end up with a bunch of people who don’t really know what they should be doing with themselves, which is always a fun scenario!)

As you probably gathered from the title, what I really want to emphasise is the whole “setting the tone” aspect of being a leader. This turns out to be important in a bunch of ways that aren’t immediately obvious. I’d hypothesize that a large part of the issues people who aren’t cishet white males face in STEM fields, and especially programming / IT, are down to this factor; the groups people tend to hang out in are implicitly using a bunch of norms that probably aren’t very inclusive (think people making slightly inappropriate cheeky comments about women they fancy, but also more subtle mannerisms and ways of communicating that tend to only be shared by people from a certain background that make it harder for people not from that background to communicate). A lot of ink has been spilled about this² in terms of how companies should try to avoid it when creating teams at work, because it’s a real problem.

The macho people from earlier can face real issues with this sort of thing. Often, their tough-guy behaviour implicitly sets the tone for the groups of people they find themselves in (which usually end up being filled with people who don’t really care, or are also trying to be macho). To avoid opening themselves up to the potential of insecurity, they often tend to do this more forcefully – terms like “pussy”, “wimp”, et al. are often employed for this purpose, wherein such people attempt to claim that people who do have feelings, are afraid of things, etc. are somehow ‘weaker’ than them.

The astute reader will think that I’m writing this because someone did that to me and I’m angry about it, and this blog post is my way of rationalizing their behaviour and asserting that I’m actually a better person than them. And they’d be partially right!³

It’s more nuanced than that, however. The thing that’s actually really sad about these sorts of situations is that the people responsible for creating the harmful no-feelings-allowed environment are often the people most in need of a way to express their feelings (as implied earlier). And what they’ve managed to do by creating such an environment is ensure they most likely won’t be able to do that thing with those people – if they try, it could get awkward (since the others aren’t really happy having a more ‘deep’ conversation, and that’s why they’re in the group), or they might find themselves met with a surprising lack of sympathy (because others actually did have problems and got humiliated for them).

I don’t even think you can blame these people, either. They’ve just found themselves in a situation that most likely isn’t even their fault, and they don’t really know what they should do to cope with it. If anything, it’s probably society that teaches them to behave in this way – and that’s just a sad, sad situation that’s not exactly easy to fix⁴.

No, it suits me to write pseudo-intellectual blog posts that nobody reads that vaguely hint at a whole bunch of screwed up stuff going on. ↩
Like a lot of the claims on this blog, this one is unsubstantiated. I think it’s true though… ↩
I mean I really don’t post often, so someone has to have annoyed me for things to get this bad. ↩
That said, I’ve seen advertising campaigns that try! I think some biscuit company teamed up with some mental health charity to promote the idea of having a cuppa and a chat about your problems with your friends, which is absolutely a good thing (even if Big Biscuit ends up profiting from it) ↩

Strict COVID-19 restrictions in universities are irresponsible

Sun, 11 Oct 2020 00:00:00 +0000

This post is about mostly personal circumstances / issues, as well as current affairs. If that’s not what you want, turn back now.

At the start of the current coronavirus disease 2019 (COVID-19) pandemic, we were told that “flattening the curve” was a good idea – i.e. attempting to limit the spread of the disease by staying at home, wearing face coverings, etc. was a necessary step we should all take in order to prevent the national health services from getting overwhelmed (leading to an excess of deaths of people who could otherwise be helped).

A significant number of months have passed since March, and a new wave of unsuspecting secondary school graduates have descended on the UK’s universities¹ – but, obviously, since there’s still a pandemic going on, things are different from the way they used to be. Pretty much all universities have new precautions to limit the spread of the disease, including things like

grouping students into (logical) “households”, and restricting interaction between said households
enforcing social distancing requirements
enforcing face covering usage
limiting the number of students that can be in the same place at one time (in line with the nationwide “rule of six”)
getting rid of all face-to-face tuition, and moving everything online
adding a curfew to, or closing, pubs and social spaces

Some of these precautions involve more sacrifices on the part of the students than others; wearing face coverings is relatively zero-cost, and has been shown to limit the spread of the disease quite significantly². However, the goal of the overwhelming majority of the restrictions is clear: limit social interaction as far as practicable. (This ‘makes sense’, because social interaction is how the virus is spread.)

The point I want to express here is that having that as a goal in the context of universities is somewhat irresponsible, and seems to completely ignores the mental health concerns of an entire year’s worth of students at university right now³. Most students have left the (hopefully relatively comfortable) environment of secondary school to come to university – sometimes in an entirely new city, or indeed country. These students typically don’t have many people they can talk to once they arrive, having left the vast majority of their friends behind from school; instead, they must somehow discover new people, usually by having a lot of spontaneous interactions until they’re able to bed in and start to establish some friendships.

It doesn’t take a genius to realise that this process is not compatible with the above stated goal of not having much social interaction.

However, what I think is particularly irresponsible is the lack of discussion surrounding the consequences of not letting this process play out as normal. The need for students to socialize and make friends is invariant; the feeling of loneliness is inherent to being human and isn’t going away any time soon, so people will (attempt to) socialize to feel less lonely, especially when placed in an unfriendly new environment. Examples of consequences arising from a lack of social interactions among students include

greater incidences of mental health problems, as loneliness creates new or exacerbates existing issues
a reduced ability to even notice and help with such problems, as remote learning can mask all sorts of issues that are more easily recognizable in person
reduced academic performance and ability, due to previously mentioned mental health problems
a greater dropout rate, leading to reduced income for universities (some of which are already struggling to stay afloat)
in the extreme case, greater incidences of suicides

It’s also the case that not everyone is perfectly rule-abiding. While more meek students might follow restrictions and suffer the associated consequences, others will flagrantly disobey them, a fact which has consequences of its own:

instead of socializing in ‘controlled’ environments, under the purview of (e.g.) student wellbeing officers, students will socialize elsewhere (e.g. a random park)
- in these ‘uncontrolled’ environments, a greater prevalence of dangerous behaviours (excessive drinking, drug use, etc.) would be expected
- …but since these are the only opportunities available to undersocialized students, more students might end up taking unwise risks than would otherwise
- there is already evidence to suggest more students are taking drugs and dying from it through precisely this mechanism⁴
from the perspective of the virus, the replacements for the now-banned opportunities to socialize are likely a lot worse, increasing net transmission
- think people drinking from the same bottle, cramming themselves into places, not using any face coverings, … (because there’s no reason for them to)
- also see e.g. Manchester students organizing COVID-19 transmission parties

A lot of the problems here tie into greater issues with the discussion of the pandemic in the media and elsewhere; a lot of people seem to think that the worrying graph of growing cases is unquestionably something that must be dealt with immediately (perhaps with a lockdown, which is even worse for students). Don’t get me wrong – COVID-19 is a deadly disease, and must not be underestimated. Letting the disease run completely unchecked throughout the population, without any restrictions whatsoever, is a terrible idea and would kill many people unnecessarily; a very contested document called the Great Barrington declaration calls for something akin to that (albeit with protections in place for vulnerable members of society).

The reality is that it’s very difficult to come to a decision, and neither extremist view is correct; making everyone sit on their hands until a vaccine is available is stupid, but so is letting the disease run wild. There’s much we don’t know about the impacts of the virus, including whether or not it has long-term health implications for certain groups (and the conditions under which such long-term complications might arise) – but sensationalizing (e.g. evocative news headlines that attempt to instil fear as to the deadliness of the disease) does not help us come to a reasoned conclusion about risk.

To conclude, then, I believe the evidence to support strict COVID-19 restrictions in UK universities is questionable, and a re-think about the rationale for, and the consequences of, such strict restrictions is sorely required. It’s really unclear whether the benefits conferred by severely limiting social interaction (at least, imposing rules that attempt to achieve such) are worth the consequences of doing so – heck, it’s even unclear whether people even follow the rules enough to limit transmission at all (and the recent outbreaks in universities across the nation confirm that).

A lack of humane thinking seems to be the case amongst those who impose said restrictions; the problem cannot be viewed as a simple mathematical calculation of how to reduce cases (if reducing cases is even something worth attempting to do!), but one that leads to significant human suffering for those affected. With the world being more divided and polarised than ever, it’s worth trying to be empathetic – to both see the fear on the part of those pushing for a lockdown and limitation of cases, and to recognize the crushing impact restrictions have on the restricted.

I’m one of these, of course, which is why I’m writing this. ↩
Even if you disagree with the evidence here, face coverings are still basically zero-cost – you really don’t sacrifice much by wearing one! ↩
If you disagree with me, please read the whole article first before getting angry. ↩
I can’t find a citation for this, so take this claim with a pinch of salt. ↩

Writing as a form of relief

Thu, 21 May 2020 00:00:00 +0000

The content on this blog does not get updated very frequently.

This is largely because, as a somewhat permanent and public sort of thing, I have to be quite careful about what stuff I write on here, since it could come back to bite me later, right? We’ve all heard the stories about people getting turned down from job offers due to some embarrassing stuff they posted on the social network du jour¹², and I generally get the impression that being careful with what you choose to write into your permanent online record is generally a good thing. (Well, when you phrase it like that…)

What are blogs for, though? As far as I can tell, almost nobody reads this one; there are a few stragglers who come here through Google because I posted about something like microG or Rust programming (even though there are now much better resources to learn Rust out there, given the language has changed a whole load since I started learning it…).

Furthermore, I don’t consider myself the kind of person who’s happy to go and do lots of writing about technical topics (for the moment, at least). Some people can sustain an entire blogging habit by packing things full of interesting technical content / deep dives / whatever. This is great, because then the content is at least ‘useful’ to the person reading it (in some sense³), instead of being some poor sap whining on about random other things happening in their life.

Unfortunately, though, I’m not one of these people. So, either I go about my life and don’t write any part of it up on the blog, or I do the converse, and end up spewing things I’ll probably regret later out into the web at large.

There’s a pandemic on. Everyone’s feeling miserable, a lot of people have very tragically lost their lives to the COVID-19 virus, and people are beginning to question all sorts of things about the existence we led before this all started.

So, what the hell, let’s just get on with it then.

The utility of personal content

Some people might be of the opinion that personal content (like somewhat soppy blog posts) is not worth reading, and should perhaps be gotten rid of entirely. However – and obviously I’m going to be biased here! – I don’t really think so. Okay, I think the sort of content where people talk about whatever mundane things they’ve been getting up to (“so, this week, I washed my bike, went out for a run, tinkered with Node.js a bit…”) is perhaps a bit of a waste of time – I’d call that ‘oversharing’, perhaps. But I do think it’s possible to read stuff about someone else’s problems and gain some insight into how you might be able to solve your own, so I don’t really want to dismiss personal content entirely.

I guess there’s a distinction between content that is purely descriptive – explaining how much you hate yourself, or how annoying thing X is, or whatever – and content that has an analytical or empathetic component as well – trying to figure out the reasons why this is the case and provide some advice to people feeling the same way, or otherwise attempting to connect your own personal experience to what others may feel. The former has no value to the reader, really – oh, poor random internet commentator. How sad. Imagine an agony aunt column without the agony aunt’s responses. How awful would that be? But the latter kind of stuff can definitely be of some value; I’ve read things on the web that have influenced the way I look at the world and respond to things – most people probably have. So it’s not entirely worthless!

The title of this post

In fact, I think writing about things is a great way to process and deal with said things. I’m not just talking about personal or emotional matters – way back in 2016 when I wrote a short Rust tutorial series, the aim was as much to inform others as to force me to be honest about my own Rust abilities; writing something up gets you to specify what exactly you mean in plain English, which can be great for identifying gaps in your knowledge, or areas of flawed thinking.

This is partially why, as it says in the title, writing can be a form of relief; there’s something about putting pen to paper that makes you feel just a bit better about whatever it is you’re writing about, be that your frustrations learning a new programming language or something more personal.

It’s also, in some ways, a lot lower friction than talking to someone about something. If you start calling up your friends and ranting to them about how much asynchronous programming paradigms suck, you eventually lose most of your friends – whereas you aren’t going to annoy anyone, or take up anyone’s time, by writing about things⁴. (Unless you do something crazy like start sending your friends letters in the post containing your rants. This is also a good way to lose most of your friends.)

‘Mental health awareness’

Now, of course, you don’t actually have to publish anything to get these benefits; simply writing something up should be enough. (This is the idea behind journaling, I think.) In fact, as I discussed at the start, publishing things can be harmful to your career.

However, I think it’s still worth doing: I’m a human being, and you are too. There are thousands of tech blogs that just talk about tech and don’t talk about anything personal or human; there are thousands of people who only talk about technical topics on their website and never mention a thing about their private lives. I’m not saying they should – but I do tend to think that seeing other people talk about their problems publicly can be a great motivator for you to do the same (for example, I’m a big fan of rachelbythebay.com and her occasional post about toxic Silicon Valley culture). To me, that’s what this concept of ‘mental health awareness’ is about (at least in part): recognizing that other people are people too, and trying to get people to talk more openly about their thoughts and feelings, instead of just keeping them to themselves.

So, yeah. Write (somewhat critically) about things that bother you, even if they aren’t technical. It’s helpful for you, and you never know what impact it’ll have on somebody else!

Or, you know, just don’t, if you’re not into that sort of thing. But I’m going to give it a try.

Also, the hope is that just trying to get into a semi-regular pattern of writing about /anything/ without much of a filter will mean that more technical stuff seeps out as well. We’ll see what happens!

Well, people tell me this happens. I’m not, ehm, experienced enough to actually have heard of this happening first-hand. ↩
On a related note, if you’re someone who might have the capability to make me a job offer, just… do me a solid and don’t read the blog, okay? :p ↩
More on this later. ↩
You also are probably not going to annoy your friends by talking about personal issues if you really feel the need to talk to someone about them, since that’s what friends are for! However, it doesn’t feel too great having to do this a lot (where ‘a lot’ is subjectively defined) – in other words, even though your friends might not actually get annoyed, your fear of them getting annoyed (and perhaps not telling you) might be enough to make you not want to talk to them. ↩

Somewhat contrived schema designing for a new chat system

Thu, 26 Dec 2019 00:00:00 +0000

[This is post 5 about designing a new chat system. Have a look at the first post in the series for more context!]

This post follows on from the previous one in the series¹, wherein I had a shot at designing / specifying what state – persistent information, shared amongst all servers in a federated system – in group chats should look like. To summarize, we ended up with the group chat state containing three important things:

a set of roles, which are a way of grouping together capabilities available to users with said roles
- Remember, capabilities are simple keywords like speak or change-topic that represent actions users can take
a list of memberships (users in the chat), together with the role for each chat member
non-user-related state, like the chatroom topic, which sort of follows a key-value store
- We figured out that allowing arbitrary stuff to be stored in a room’s state was a bad idea, so this just contains…some random fields we’ll specify more formally later².

In this post, we’ll look into how this state will be represented in server databases, and spec out a database schema for our server implementations to use³.

Unpacking our group chat state object

We could just store the group chat state as a big JSON blob in the database (indeed, if we were using something like MongoDB, that would be commonplace). However, this probably isn’t a good idea – for a number of reasons:

we’d have to retrieve the whole thing every time we wanted to access information about it, which is suboptimal performance-wise
things in the blob could quietly become inconsistent with the rest of the database if we didn’t check it all the time
the database wouldn’t be able to enforce any schemas; we’d have to do that in our application code
unless we (ab)use something like PostgreSQL’s native JSON support, our group chat state would be completely opaque from the database’s point of view – meaning it’d be hard to draw links between things in there (e.g. user IDs) and the rest of the database

These concerns are similar to the concerns third-normal form (3NF), a way of structuring database schemas from 1971, hopes to address. Under 3NF, you store your data in a set of database tables representing various objects, with each table having a primary key (an identifier or set of identifiers uniquely identifying that object). 3NF then states that other information in each table must only tell you something about the primary key, and nothing else; they aren’t allowed to depend on anything other than the value of the primary key.

As a more concrete example, let’s say we have a User table representing a user of our chat system, where the primary key is a combination of a username and the user’s home server. If I wanted to add a column describing what channels they’re in, for example, that would be fine – but if I wanted to also add the most recent messages for each channel, say (let’s imagine you’re designing a UI like WhatsApp’s, with a homescreen that shows this information), that wouldn’t be valid under 3NF, because the most recent messages are a property of the channel, not the user. We could end up having two users in the same channel, and forget to keep this ‘recent messages sent’ property consistent, which would lead to confusion!

So, using 3NF seems like a pretty good idea – and that’s exactly what we’re going to do! Our group chat state object doesn’t fit 3NF as one large blob, so we’re going to need to decompose it into a set of database tables that store the same information.

Let’s do some schema designing!

`groupchats`: our starting point

Of course, we need some object to represent a group chat. Since group chats are going to be shared across different servers, we have to choose some identifier that’s always going to be unique – we can’t just give them textual names, otherwise they’d be a possibility of them clashing. Instead, we’ll use a universally unique identifier (UUID) – it does what it says on the tin!

Our set of group chat state (for now) is in the list below. I’ve also put 🚫 next to things we can’t put in the groupchats table directly due to 3NF, and explained why.

topic / subject
- This is purely a property of the group chat itself, and it doesn’t depend on anything else.
list of users 🚫
- This one technically could be a property of the group chat, but making it one isn’t a great idea.
- Firstly, that’d mean we’d have to use an array, which is generally frowned upon; it makes it harder to do things like database table JOINs when the users are stuck in an array attribute.
- Also, we probably want to associate some information with a user’s membership, like their role. Doing that in the groupchats table would be a big 3NF violation.
list of defined roles 🚫
- Roles have capabilities associated with them, so they should be their own thing.
- Said otherwise, our primary key is the channel’s UUID, not (channel UUID, role), so storing capabilities (which depend on those two things) would be a 3NF violation.
mapping of users to what roles they have 🚫
- Similarly to the last item, this mapping introduces a 3NF violation.
- We’ll probably end up doing this one in a separate object, as discussed above.
list of servers involved in this group chat, as well as whether they’re sponsoring or not 🚫
- Ditto, really.
current state version
- We need to keep track of what state version we’re on (remember, the state version is a monotonically incrementing integer), for the purposes of our consensus algorithm.

So, now that that’s all clear, we’re left with group chat UUID, subject, and current state version. Here’s the SQL DDL:

CREATE TABLE groupchats (
    uuid UUID PRIMARY KEY,
    state_ver INT NOT NULL,
    subject VARCHAR -- can be null, if a group chat is unnamed.
);

(We’ll include the DDL for each table in our schema.)

`groupchat_roles` and `groupchat_role_capabilities`: storing group chat role information

Before we can actually express user memberships, we need something to store group chat role information; what roles exist, and what capabilities are associated with them. Behold:

CREATE TABLE groupchat_roles (
    role_id SERIAL PRIMARY KEY,
    groupchat_uuid UUID NOT NULL REFERENCES groupchats,
    role_name VARCHAR NOT NULL,
    UNIQUE(group chat_uuid, role_name)
);

CREATE TABLE groupchat_role_capabilities (
    role_id INT NOT NULL REFERENCES groupchat_roles,
    capability VARCHAR NOT NULL,
    UNIQUE(role_id, capability)
);

A row in the groupchat_roles table represents a role name in a group chat. Role names are unique per group chat, so the 2-tuple (groupchat_uuid, role_name) is unique; the only bit of information associated with a role is a list of capabilities, but we aren’t going to use arrays (q.v.), so the separate groupchat_role_capabilities table represents capabilities granted to users with a given role.

We’ve given roles an internal integer ID (role_id) just to make the primary key less annoying; the ‘real’ primary key should be (groupchat_uuid, role_name), but that’d be a real pain to refer to in the groupchat_role_capabilities table (we’d have to store both the UUID and the role name! So much wasted space!⁴), so we just use an integer instead.

`groupchat_memberships`: associating users with group chats

Now that we’ve got a group chat table, and a way of expressing user roles, we want a way to express the set of users that are in said group chats, along with the role they have. This is very simple – we have a (groupchat, user) primary key, and a role foreign key.

CREATE TABLE groupchat_memberships (
    groupchat_uuid UUID NOT NULL REFERENCES groupchats,
    user_id INT NOT NULL REFERENCES users,
    role_id INT NOT NULL REFERENCES groupchat_roles,
    PRIMARY KEY(groupchat_uuid, user_id)
);

I’m deliberately not going to mention what’s in the users table yet; we’re going to discuss that in another blog post.

`groupchat_sponsoring_servers`: associating sponsoring servers with group chats

We also need to associate servers with groups somehow. This is done for us partially, in that users reside on servers, and so naturally the set of servers associated with a given group chat are just the servers on which the members reside – but that doesn’t take into account the fact that some of these servers might be sponsoring servers for the purposes of federation.

Enter the groupchat_sponsoring_servers table:

CREATE TABLE groupchat_sponsoring_servers (
    groupchat_uuid UUID NOT NULL REFERENCES groupchats,
    server_id INT NOT NULL REFERENCES servers,
    PRIMARY KEY(groupchat_uuid, server_id)
)

Again, there’s this mysterious servers table we haven’t got to.

Next steps

We’ve now established what principles we’re going to use to design our schema (3NF), and we’ve got 5 lovely tables that express all the group chat state stuff we’ve been jabbering on about for the last two blog posts in proper SQL DDL that we could actually use!

We now need to tackle the other two important parts of our schema; we’ve done group chats, but messages and users are yet to be specified. We’ll need to discuss a few important points about how we design our protocol to fit both of these, as there’s more to it than you might think! (For example, a user might seem simple – just someone on a server somewhere, right? – but what about them having a profile picture, or a bit of ‘status’ text describing what they’re up to, or things like that?)

All of that will come in the next blog post in the series, coming (hopefully quite) soon to a website near you!

Hey, those deadlines really are quite scary. Wouldn’t it be lovely if we had a roadmap, with nice intermediate dates on it, so we could actually plan stuff?

Reading this post might be somewhat confusing, if you haven’t read that one! ↩
This blog series is more “broad brushstrokes” than “exhaustive details” – because otherwise both you and I would get horrifically bored. Don’t worry, though – the exhaustive details will turn up somewhere and be featured in the final thing… ↩
Of course, this schema isn’t part of the specification. Server implementors can do whatever they want; there does, however, have to be a reference implementation out there… ↩
If we’re really wanted to save space, of course, we’d refer to the group chats table itself using an integer instead of a UUID (because a UUID is a blob of four integers, I think). I definitely draw the line at foreign composite primary keys, though… ↩

Designing group chat state for a new chat system

Wed, 18 Dec 2019 00:00:00 +0000

[This is post 4 about designing a new chat system. Have a look at the first post in the series for more context!]

This is potentially the point at which the previous few blog posts, which have talked in vague, vacuous terms about the possibility of designing some new utopian chat system, start to gradually become more about actually getting the work done – purely due to the fact that, as previously mentioned, I have a deadline! It’s been a while¹ since I last said anything about the chat systems project – so, let’s get things rolling again with a practical post about our database schema, shall we?

The importance of a good schema

“Bad programmers worry about the code. Good programmers worry about data structures and their relationships.”

~ Linus Torvalds

It’s been said, as you can see in the quote above, that thinking hard about your data structures – the way you choose to store whatever data it is that you’re processing – is vitally important to the success of whatever it is that you’re building – so it’s worth having good ones, or else you’ll end up with hacky code that doesn’t quite work right². If you have the right data structures, the code usually sort of flows in to fill the gaps and make everything work right – whereas if you do the code first, it doesn’t usually work as nicely.

Now, data structures come in all shapes and sizes – there are trees, hash tables, association lists, binary heaps, and all sorts of other fun stuff that you’d probably find in some CS textbook somewhere. We could use some of these for our new chat system – and, in fact, we probably will use the odd hash table or two. However, given I want to keep things nice and boring, sticking to proven, reliable technologies for this project³, we’re probably just going to store everything in a set of relational database tables (i.e. rows and columns!).

And, a schema is essentially a description of a set of what columns mean what, and what tables you’re going to have. Which is what I’m going to write now, so, without further ado…

What do we actually want?

It’s a good idea to start with a discussion of what data we have, and what we’re trying to get out of that data. We’ve said that we want our new system to support our own funky blend of federation, so that’s going to need to be accounted for. Naturally, a chat service will have a bunch of users, each with their own usernames, passwords, emails, and other information that needs to be stored about them. We’ll probably have some messages as well, given users tend to make a lot of those when you give them the chance to.

As we’ve also discussed before, the primary function of our chat service is to convey messages from their source to their recipient, and do so in a reliable manner. That implies further that, in addition to the messages themselves, we’d also benefit from storing information about message delivery – did the messages get through, in the end, or do we need to try again sometime?

Since we’re supporting group chats, those also need to have some information stored about them. Our federation protocol⁴ requires us to store multiple different ‘versions’ of group chat state (remember, ‘state’ refers to things like the list of members of the chat, who has admin rights, and what the current topic is) – because it’s based on this whole funky consensus protocol stuff, we’ll need to keep track of what servers have proposed changes, and which changes we decided to actually accept.

A model for group chat state

The consensus algorithms we looked into previously allow us to get a bunch of trusted servers (‘sponsoring servers’) to agree on something, where ‘something’ is just some arbitrary value – usually with some kind of monotonically incrementing identifier or version number. It thus follows that we need some model for what that ‘something’ will look like; how will servers communicate information about a room’s state to one another?

Users, administrators, and moderation

Actually, we haven’t even specified what we want this room state to look like yet – there are still some unresolved questions around things as simple as how administrator rights / operator powers should work in group chats. Different platforms do this in different ways, after all:

IRC has a system of ‘channel modes’, where users can be given flags such as +o (operator; can kick, ban, etc.) and +v (voice; can speak in muted channels).
- Some servers then extend this, adding +h (‘half-operator’), +a (‘admin’), +q (‘owner’), and all sorts of other random gunk that confuses people and makes it hard to determine who can do what.
- Of course, server administrators can override all of this and just do what they want, on most implementations.
Matrix has ‘power levels’ - each user has an integer between 0 and 100 assigned to them, which determines what they’re able to do. A set of rules (stored in the room state, alongside the power levels) specify what power levels map to what – for example, only people with power level >= 50 can set the topic, etc.
- You’re not allowed to give people power levels higher than what you already have, and you can’t take away power from people who have more or the same amount of power as you. This is kinda required to make the whole thing work.
- Because Matrix is decentralised, it’s possible to get yourself into a state where everyone’s lost control of a chatroom and you can’t get it back. Of course, though, this is quite easy to avoid, by making the software stop you from shooting yourself in the foot.
WhatsApp has ‘group admins’ and…that’s pretty much it⁵. You either have admin rights, in which case you can do everything, or you don’t and you can’t really do much.
- This is very simple for users to understand.
- However, it makes WhatsApp completely impractical for all sorts of use cases; anyone who’s found themselves kicked out of a WhatsApp chat after someone went crazy and banned everyone after pleading for admin ‘in order to help keep things civil’ probably knows what I’m talking about.
Discord has ‘roles’ - there’s a bit in server settings where you can create sets of permissions, called ‘roles’, that you grant to different users in order to empower them to do specific things⁶.
- This is partially why Discord can play host to massive chatrooms for very popular games like PUBG and Fortnite: the system is very flexible, and suited to large-scale moderation.
- However, it’s also perhaps quite confusing for some people, especially if you’re just using it for smaller-scale stuff like a private group chat.
- Like Matrix, the roles are arranged in a kind of hierarchy; roles at the top of the hierarchy can make changes to roles below them, but not the other way round.

So, hmm, it might seem like there’s a lot to choose from here – but, in fact, it’s a bit more simple than you’d think. It’s immediately apparent that the more flexible Matrix/Discord systems can in fact be used to make more simple systems, like the IRC and WhatsApp ones; if all you want is group admin or not group admin, you can make two Discord roles (one with all permissions, one with none), or have two power levels (100 and 0, say, with appropriate rules for each one), and you’ve essentially got the more simple system using your complex one. (And you can do funky things with your user interface to hide away the added power, for those who don’t really need it.)

Taking some inspiration from this idea, and the age-old concept of an access-control list, here’s a proposed model. We’ll specify a set of capabilities that describe what actions can be taken – for example, speak, change-topic, mute-user, and so on – and then a set of roles, like the Discord roles, that are sets of capabilities. Each user gets a role, which in turn describes what they’re able to do. (If we want to make things work nicely for ex-IRC people, the roles can optionally come with a small letters, like +o and +v.) Unlike Discord and Matrix, there won’t be any hierarchy to the roles. Roles can only be modified by someone holding a capability called change-roles, and that’ll be the end of it. The sponsoring servers in our federation model will do this role check every time they receive a message or a request to change the state in some way, and refuse to apply the change if it’s not permitted.

The list of capabilities will eventually be written in some spec document somewhere, and thereby standardised across different server implementations. Essentially, they’ll work like IRCv3 capabilities, where vendors can make their own capabilities up if they want to (prefixing them with a valid domain name for whatever it is that they built).

To make things easier, the special capability * allows a user to make any change.

In JSON-esque syntax⁷, this would look a bit like:

{
    "roles": {
        "normals": ["speak", "change-topic"],
        "voiced": ["speak", "speak-in-muted-channel", "change-topic", "theta.eu.org/special-extension-power"],
        "admins": ["*"]
    },
    "memberships": {
        "server.org/user1": "normals",
        "server.org/adminuser1": "admins"
    }
}

Of course, there’s also state that doesn’t directly relate to users – the current group chat subject, whether or not guest users are allowed in, etc. Some state may have more complicated structure – for example, the Matrix people have a server ACL state event that lets you ban specific servers from taking part in rooms, which is pretty much its own little object with embedded arrays and booleans – which means we can’t just model this state as a simple stringly-typed key-value store, i.e.

{
    "state": {
        "subject": "example",
        "other_state_key": "other_value",
        // etc
    }
}

The question is, though, how extensible we want things to be: can users (or servers, for that matter) store arbitrary objects in our group chat state, or do they all have to follow some predetermined schema? Matrix takes the approach of letting users chuck in whatever they like (assuming they have the necessary power level) – essentially treating each group chat as a mini database – while pretty much every other platform restricts room state to a set of predetermined things.

Capability negotiation

I’m not entirely sure allowing arbitrary extensibility, in the way that Matrix does, is such a good idea – for two reasons⁸:

Allowing people to just store arbitrary data in your group chat seems a bit too much like an abuse vector. It’s a chat system, not a distributed database; allowing people to just chuck whatever stuff they want in there is a bit much!
- How would you display these arbitrary state events, given you don’t know anything about them?
- You’d need some limit on size, to prevent people just using you as free storage?
- In many jurisdictions, you’re responsible for content that gets put on your server. What if someone uploads stuff you’re not comfortable hosting, and stashes it away in some event your client doesn’t implement (and therefore doesn’t show)?
Usually, things in the group chat state are there for a reason, and it doesn’t make sense for servers to just ignore them.
- For example, consider when the Matrix folks rolled out the server ACL feature: servers not on the latest version of their software would just completely ignore it, which is pretty bad (because then it had no effect, as malicious actors could get into the room via the unpatched servers).
  - They ‘solved’ this by polling all the servers in a room for their current version number, and checking to see which ones still needed updating (which let them badger the server owners until it got updated).

Instead, it’s probably better to actually have some system – like the previously-mentioned IRCv3 capability negotiation⁹ – where servers can say “yes, I support features X, Y, and Z”, thus enabling those features to be used only if all of the sponsoring servers in a group chat actually support them. This solves the issue of extensibility quite nicely: non-user related state is governed by a strict schema, with optional extensions for servers that have managed to negotiate that.

Group Chat state summary

So, to sum up: we’ve managed to get a better idea of what the blob of group chat state, shared across all servers participating in a group chat and agreed upon via the federation consensus protocol, should look like: it’ll contain a capability-based system for regulating what users are allowed to do what, and it’ll contain a set of other pieces of strictly-typed non-user-related state, like the group chat subject! This might all seem a bit abstract for now, but it’ll hopefully become clearer once actual code starts getting written. On that note…

A note about timings

We managed to roughly sketch out group chat state over the course of around ~2,500 words (!), but there’s still all the other stuff to spec out: users, messages, and reliable delivery mechanisms. In addition, there are also a number of less conceptual things to sketch out, like how we’re going to ensure server-to-server transport happens securely (SSL? Encryption?¹⁰) and things like that.

And this all has to be done by March, at the absolute latest. Yay!

On a more hopeful note, we do actually have some code for this whole project – currently, it does the registration flow part of a standard IRC server (basically, the plan is to piggyback off of IRC for the client<->server stuff, to get things going), and has a very untested implementation of the Paxos consensus protocol. We’re using Common Lisp as our main programming language, which is fun¹¹!

Anyway, the rest of the spec stuff will follow in the next blogpost (hopefully quite quickly after this one…!) – if anyone actually has the tenacity to read my ~2,500 words about the annals of random chat protocol stuff, that is! We will also actually show how all of this translates into a database schema, which was sort of the point. Oops.

To be continued…

Err, two months. I was very busy applying to universities, okay! ↩
A large part of my criticism of Matrix centered around the fact that they did something funky – and, in my view, perhaps unnecessary – with theirs. ↩
Using some fancy NoSQL database has undone many a project in the past; SQL databases have had a lot of engineering and research effort put into them over the decades to make them work reliably and quickly, and that shouldn’t be given up lightly! ↩
Read the ‘funky blend of federation’ blog post linked earlier, if you need a refresher! ↩
I think they recently added some sort of limited permissions system for “who can change the topic: admins or all group members?” and things like that, but this is the gist of it. ↩
They also serve a vanity / novelty purpose; look at me, I’ve got a colourful name! ↩
Even though we might not be actually using JSON at the end of the day, it’s a pretty well-understood way to describe things. ↩
This IRCv3 issue is about a very similar problem (allowing arbitrary client-only message tags on IRC), and is also worth a read! ↩
These capabilities are, unfortunately, nothing to do with the user/administrator capabilities discussed earlier… ↩
The Matrix people went and used Perspectives instead of regular SSL PKI for their stuff, and they eventually had to move away from it. It’s worth learning from that mistake! ↩
It was going to be Rust, but then I started doing Lisp stuff and figured this would be more ‘interesting’… ↩

eta.st

Reversing UK mobile rail tickets

Initial explorations

A friendly wolf comes to help

Running the app

Decompiling the app

Making sense of the decompiled output

Interesting uses of RSA

Obtaining the elusive public keys

ttkMobile

Brief aside on freedom of information

Bonus: eTVD logs

Trying this out for yourself

Acknowledgements

Footnotes

Pessimism is a bad defense mechanism

Sending audio to LKV373 HDMI extenders

Testing without the actual devices

Adventures in bitshifting

Partial victory!

Let’s do the Wireshark again

Mysterious packets on port 2067

Heartbeat packets

Incrementing counter

Other gubbins

Putting it all together

But what about audio?

How does digital audio work, anyway?

What format do the extenders use?

Timing is everything

Jitter is unacceptable

Final audio thread code

Conclusions

Why asynchronous Rust doesn't work

A study in async

Hang on a minute…

An aside on naming types

An aside on ‘radioactive’ types

*wibbly wobbly scene transition*

Getting PIV-based SSH working on a YubiKey

Procedure

Setting the tone in a group is very important

Strict COVID-19 restrictions in universities are irresponsible

Writing as a form of relief

The utility of personal content

The title of this post

‘Mental health awareness’

Somewhat contrived schema designing for a new chat system

Unpacking our group chat state object

Let’s do some schema designing!

groupchats: our starting point

groupchat_roles and groupchat_role_capabilities: storing group chat role information

groupchat_memberships: associating users with group chats

groupchat_sponsoring_servers: associating sponsoring servers with group chats

Next steps

Designing group chat state for a new chat system

The importance of a good schema

What do we actually want?

A model for group chat state

Users, administrators, and moderation

Non-user-related state

Capability negotiation

Group Chat state summary

A note about timings

wibbly wobbly scene transition

`groupchats`: our starting point

`groupchat_roles` and `groupchat_role_capabilities`: storing group chat role information

`groupchat_memberships`: associating users with group chats

`groupchat_sponsoring_servers`: associating sponsoring servers with group chats