Drew DeVault's blog

Addressing the harassment

Tue, 21 Apr 2026 00:00:00 +0000

Kiwi Farms is a web forum that facilitates the discussion and harassment of online figures and communities. Their targets are often subject to organized group trolling and stalking, as well as doxing and real-life harassment. Kiwi Farms has been tied to the suicides of three people who were victims of harassment by the website.

– Wikipedia: Kiwi Farms

About three years ago, a thread on Kiwi Farms was opened about me. In the years since, it has grown to about 1,200 posts full of bigots responding to anything and everything I do online with scorn, slurs, and overt bigotry. The thread is full of resources to facilitate harassment, including, among other things, all of my social media profiles, past and present, a history of my residential addresses, my phone numbers, details about my family members, a list of my usernames and password hashes from every leaked database of websites I have accounts on, and so on. Most of my articles or social media posts are archived on Kiwi Farms and then subjected to the most bigoted rebuttals you can imagine. Honestly, it’s mostly just… pathetic. But it’s a problem when it escapes containment, and it’s designed to.

Kiwi Farms is the most organized corner of the harassment which comes my way, but it comes in many forms. On Mastodon, for example, before I deleted my account I would often receive death threats, or graphic images and videos of violence against minorities. I have received a lot of hate and death threats over email, too, several of which I confess that I took some pleasure in forwarding to the sender’s employer.

One of the motivations for this harassment is to “milk” me for “drama”. The idea is to get my hackles up, make me fearful for my safety, and alienate me from my communities, with the hope that it will trigger an entertaining meltdown. Maybe people respond poorly to this kind of harassment – that’s the idea, really – and it often makes the situation worse. Responding to it can legitimize the abuse, elevate it into the discourse, draw more attention to it, and stoke the flames. It can make the victim look bad when they respond emotionally to harassment designed to evoke negative emotions. I have left it unaddressed for a long time in order to subvert this goal, and address it now with a cool head in a relatively quiet period in the harassment campaign.

The harassment waxes and wanes over time, usually picking up whenever I write a progressive blog post that gets some reach. It really took off after a series of incidents in which I called for the Hyprland community and its maintainers to be held to account for the bigotry and harassment on their Discord server (1, 2) and when I spoke out against Richard Stallman’s prolific and problematic public statements regarding the sexual abuse of minors (3).

The abuse crescendoed in October of 2024, when I was involved in editing The Stallman Report. The report is a comprehensive analysis of Richard Stallman’s problematic political discourse regarding sexual harassment, sexual assault, and the sexual abuse of minors, and it depends almost entirely on primary sources – quotes from Stallman’s website which remain online and have not been retracted to this day. The purpose of the report was to make a clear and unassailable case for Stallman’s removal from positions of power, make specific recommendations to address the underlying problems, and to stimulate a period of reflection and reform in the FOSS community. It didn’t achieve much, in the end: the retaliation from Stallman’s defenders was fiercer and more devoted than the support from those who saw the report’s sense.

Myself and the other authors asserted our moral rights to publish anonymously, motivated by our wish to reduce our exposure to the exact sort of harassment I’ve been subjected to over the years. However, I was careless in my opsec during the editing process, and it was possible to plausibly link me to the report as a result, leading to a sharp increase in harassment.

This brings me to a retaliatory, defamatory “report” published about me in the style of the Stallman Report.¹ This report is, essentially, a distillation of the Kiwi Farms thread on me, sanitized of overt bigotry and presented in a readily linkable form in order to stalk me around the internet and enable harassment. It’s used to discredit anything I do online and push for my exclusion from online communities, by dropping the link on Hacker News, Reddit, GitHub or Codeberg issues, etc, anywhere myself or my work is mentioned, or used to discredit the Stallman Report by discrediting one of its unmasked authors.²

The report is pretty obviously written in bad faith and relies on a lot of poor arguments to make the case that I’m a misogynist and a pedophile, charges I deny. It also accuses me of being a hypocrite, which I acknowledge in general terms, because, well, who isn’t. The key thing I want people who encounter this report to keep in mind is that this is the “polite” face of an organized harassment campaign.

Most reasonable readers easily dismiss the report because it is rather transparent in its bad faith. However, someone who reads it in good faith, just trying to do their due diligence, might come away from it with some reasonable concerns. Consider the following quote from my long-deleted Reddit account, /u/sircmpwn:

I’m of the opinion that 14 year old girls should be required to have an IUD installed. Ten years of contraception that requires a visit to the doctor to remove prematurely.

This comment was written 13 years ago, and I don’t stand by what I wrote. I was 19 at the time, and I was a moron. My mother had me when she was 23 years old, and the abuse I suffered at her hands during my childhood was severe, and I generalized this experience to all women. When I wrote this comment, I was one year removed from the abuse, living alone and in poverty, and early in a life-long process of coming to terms with the abuse and figuring out how to be a well-adjusted adult after 18 long years of abuse and isolation.

But an explanation is not an excuse. This comment was reprehensible, as were many of the awful ideas I held at the time. Many years later, I can recognize that this comment is misogynistic, denies the agency of children and women over their own bodies, disparages the many, many mothers who do a wonderful job raising children in difficult circumstances, and is based in argumentation which can reasonably be related to eugenics. This comment was just awful – there’s a reason this was deleted. I apologize to anyone who read it at the time, or comes across it now, and is justifiably insulted.

I don’t feel that it’s necessary to rebuke most of the report. But, there is a grain of truth in the report, the grain of truth that led me to retract my shitty Reddit comments and reflect on myself, and that grain of truth is this: in early adulthood, I was a huge asshole.

I have had more than my fair share of harmful ignorance, bad takes, sexism and misogyny, transphobic and homophobic beliefs, and worse. Moreover, I have verbally abused many people and made many of my own arguments in bad faith to support bad conclusions. Some of the people who read this will recall having found themselves at the wrong end of my verbal abuse and harassment.

It’s important for me to take responsibility for this period of my life, and in dismissing bad faith criticisms of myself to carefully avoid dismissing good faith criticisms in the same fell swoop.

I’m not really sure how to deal with this part of my life appropriately. I have apologized to a few people individually, but it’s not a scalable solution and with many people I have no business re-opening wounds to salve my own conscience. I can offer a general apology, and I will. I’ve never found the right moment to say it, but now will do: I apologise, sincerely, to everyone who I have harmed with verbal abuse and with hateful and problematic rhetoric. If you have had a bad experience or experiences with me, and there’s anything you want from me that can help you heal from that experience – a personal apology, for example – please reach out to me and ask.

That said, apologies alone aren’t enough. I believe in restorative justice, in growing and mending wounds and repairing harm done, and I set myself seriously to this task over many years. I have gone to therapy, spoken with close friends about it, and taken structural action as well: I have founded support groups and worked one-on-one with many of the people whose politics and behavior I object to. I want an amicable end to bigotry and bullying, for bigots and bullies like my former self to look forward to, to provide a path that doesn’t require them to double down. It’s not easy, and not everyone manages, but I have to look at myself and see the path I’ve taken and imagine that it’s possible, because what’s left for the likes of me if not?

This part of my past brings me a great deal of shame, and that shame motivates me to grow as a person. In a certain sense, it is an ironic, cruel privilege to have had so much cause to reflect on myself, to drive me to question myself and my ideas, and become a much better person with much more defensible ideas. It has driven me to study feminism, social justice, racial justice, intersectionality, LGBTQ theory, antifascism, and to find the intersections in my own life and strive to act out of a more legitimate sense of justice.

I’m often still a firebrand, but I’ve chosen much better hills to die on. My passion is invested in making a more just world, building safe and healthy communities, elevating my peers, and calling for justice and a just society. I have taken the lessons I have learned and tried to share them with other people, and to stand up for what I can now say I know is right, both online and in real life. Through a process of learning, reflection, and humility, I acknowledge that I have done a lot harm in my youth. To repair this harm, I have committed myself to doing more than enough good now to make sure that the world is a better place when all is said and done. That’s what justice means to me when I turn my principles inwards and hold myself accountable.

So where do we go from here?

The response to my progressive beliefs and activism is reactionary backlash, doxing, harassment, and death threats targeting me and my family, all of which is likely to escalate in response to this post, and none of which is defensible. On the other hand, I understand that the consequences for my own reactionary past is, in some cases, alienation – and, honestly, fair enough.

But I don’t want you to confuse my honest faults with the defamation and harassment I endure for standing up for my honest strengths. If you feel generous and optimistic about who I am today, and you recognize my growth, and wish for an ally in the fight for what’s right, your good faith and solidarity mean the world to me. I would appreciate it if you would express your support and rebuke harassment when you see it, and help keep me honest as I continue a life-long process of learning and growth.

If I’ve hurt you, and you want to seek reconciliation, I make myself available to you for that purpose. If I’ve hurt you, and you simply don’t care to be hurt again, I’m sorry – I understand where you’re coming from, and have made my peace with it.

Please send words of support and/or death threats to drew@ddevault.org.

Thank you.

Rewrote my blog with Zine

Sun, 19 Apr 2026 00:00:00 +0000

15 years ago, on December 11th, 2010, at the bold age of 17, I wrote my first blog post on the wonders of the Windows Phone 7 on Blogspot. I started blogging as a kid at the behest of a family friend at Microsoft, who promised she’d make sure I would become the youngest Microsoft MVP if I started blogging. That never came to pass, though, because as I entered adulthood and started to grow independent of my Microsoft-friendly family I quickly began down the path to the free and open source software community.

Early blog posts covered intriguing topics such as complaining about my parent’s internet filter, a horrible hack to “replace” the battery of a dead gameboy game, announcing my friend’s Minecraft guild had a new website (in PHP), and so on. After Blogspot, I moved to Jekyll on GitHub pages, publishing You don’t need jQuery in 2013. For a long time this was the oldest post on the site.

I’m pretty proud of my writing skills and have a solid grasp on who I am today, but the further back you go the worse my writing, ideas, values, and politics all get. I was growing up in front of the world on this blog, you know? It’s pretty embarassing to keep all of this old stuff around. But, I decided a long time ago to keep all of it up, so that people can understand where I’ve come from, and that everyone has to start somewhere.¹

At some point – I’m not sure when – I switched from Jekyll to Hugo, and I’ve stuck with it since. But lately I’ve been frustrated with it. I’d like my blog engine to remain relatively stable and simple, but Hugo is quite complex and over the past few years I’ve been bitten by a number of annoying and backwards-incompatible changes. And, as part of my efforts to remove vibe-coded software from my stack, I was disappointed to learn that Hugo is being vibe coded now, and so rewriting my blog went onto the todo list.

Choosing the right static site generator (SSG) was a bit of a frustrating process. Other leading candidates, like Pelican or Zola, are also built from slop now. But a few months ago I found Zine, and after further study I found it to be a pretty promising approach. Over the past few days I have rewritten my templates and ported in nearly 400 (jeesh) blog posts from my archives.

There’s a lot to like about Zine. I’m pretty intrigued by SuperHTML as a templating engine design – the templates are all valid HTML5 and use an interesting approach to conditions, loops, and interpolation. SuperMD has some interesting ideas, but I’m less sold on it. The Scripty language used for interpolation and logic is a bit iffy in terms of design – feels half baked. And the designers had some fun ideas, like devlogs, which I feel are kind of interesting but tend to have an outsized influence on the design, more polished where the polish might have been better spent elsewhere. The development web server tends to hang fairly often and I’ve gotten it to crash with esoteric error messages every now and then.

But what can I say, it’s alpha software – I hope it will improve, and I’m betting that it will by migrating my blog. There’s no official LLM policy (yet) and I hope they will end up migrating to Codeberg, and using Discord for project communication is not something I appreciate, but maybe they’ll change their tune eventually.

In the meantime, I took the opportunity to clean up the code a bit. The canonical links have gone through several rounds of convention and backwards compatibility, and I have replaced them with a consistent theme and set up redirects. I probably broke everyone’s feed readers when rolling these changes out, and I apologise for that. I have gone through the backlog and updated a number of posts as best as I can to account for bitrot, but there are still a lot of broken videos and links when you get far enough back – hopefully I can restore some of that given enough time.

I’ve also gone ahead and imported the really old stuff from Blogspot. The whole lot is garbage, but if you’re curious to see where I started out, these old posts are more accessible now.

🚀 Signal boost: The Future of Everything is Lies, I Guess

Mon, 06 Apr 2026 00:00:00 +0000

🚀 Signal boost: someone else's article that I think you should read

The Future of Everything is Lies, I Guess

tar: a slop-free alternative to rsync

Sat, 28 Mar 2026 00:00:00 +0000

So apparently rsync is slop now. When I heard, I wanted to drop a quick note on my blog to give an alternative: tar. It doesn’t do everything that rsync does, in particular identifying and skipping up-to-date files, but tar + ssh can definitely accomodate the use case of “transmit all of these files over an SSH connection to another host”.

Consider the following:

tar -cz public | ssh example.org tar -C /var/www -xz

This will transfer the contents of ./public/ to example.org:/var/www/public/, preserving file ownership and permissions and so on, with gzip compression. This is roughly the equivalent of:

rsync -a public example.org:/var/www/

Here’s the same thing with a lightweight progress display thanks to pv:

tar -cz public | pv | ssh example.org tar -C /var/www -xz

I know tar is infamously difficult to remember how to use. Honestly, I kind of feel that way about rsync, too. But, here’s a refresher on the most important options for this use-case. To use tar, pick one of the following modes with the command line flags:

-c: create an archive
-x: extract an archive

Use -f <filename> to read from or write to a file. Without this option, tar uses stdin and stdout, which is what the pipelines above rely on. Use -C <path> to change directories before archiving or extracting files. Use -z to compress or decompress the tarball with gzip. That’s basically everything you need to know about tar to use it for this purpose (and for most purposes, really).

With rsync, to control where the files end up you have to memorize some rules about things like whether or not each path has a trailing slash. With tar, the rules are, in my opinion, a bit easier to reason about. The paths which appear on the command line of tar -c are the paths that tar -x will open to create those files. So if you run this:

tar -c public/index.html public/index.css

You get a tarball which has public/index.html and public/index.css in it.

When tar -x opens this tarball, it will call fopen("public/index.html", "w"). So, whatever tar’s working directory is, it will extract this file into ./public/index.html. You can change the working directory before tar does this, on either end, by passing tar -C <path>.

Of course, you could just use scp, but this fits into my brain better.

I hope that’s useful to you!

Update: As a fun little challenge I wrapped up this concept in a small program that makes it easier to use:

https://git.sr.ht/~sircmpwn/xtar

Example:

xtar -R /var/www me@example.org public/*

A eulogy for Vim

Wed, 25 Mar 2026 00:00:00 +0000

Vim is important to me. I’m using it to write the words you’re reading right now. In fact, almost every word I have ever committed to posterity, through this blog, in my code, all of the docs I’ve written, emails I’ve sent, and more, almost all of it has passed through Vim.

My relationship with the software is intimate, almost as if it were an extra limb. I don’t think about what I’m doing when I use it. All of Vim’s modes and keybindings are deeply ingrained in my muscle memory. Using it just feels like my thoughts flowing from my head, into my fingers, into a Vim-shaped extension of my body, and out into the world. The unique and profound nature of my relationship with this software is not lost on me.

A picture of my right hand, with the letters “hjkl” tattooed on the wrist

I didn’t know Bram Moolenaar. We never met, nor exchanged correspondence. But, after I moved to the Netherlands, Bram’s home country, in a strange way I felt a little bit closer to him. He passed away a couple of years after I moved here, and his funeral was held not far from where I lived at the time. When that happened, I experienced an odd kind of mourning. He was still young, and he had affected my own life profoundly. He was a stranger, and I never got to thank him.

The people he entrusted Vim to were not strangers, they knew Bram and worked with him often, and he trusted them. It’s not my place to judge their work as disrespectful to his memory, or out of line with what he would have wanted. Even knowing Bram only through Vim, I know he and I disagreed often. However, the most personal thing I know about Bram, and that many people remember about him, was his altruistic commitment to a single cause: providing education and healthcare to Ugandan children in need. So, at the very least, I know that he cared.

I won’t speculate on how he would have felt about generative AI, but I can say that GenAI is something I care about. It causes a lot of problems for a lot of people. It drives rising energy prices in poor communities, disrupts wildlife and fresh water supplies, increases pollution, and stresses global supply chains. It re-enforces the horrible, dangerous working conditions that miners in many African countries are enduring to supply rare metals like Cobalt for the billions of new chips that this boom demands. And at a moment when the climate demands immediate action to reduce our footprint on this planet, the AI boom is driving data centers to consume a full 1.5% of the world’s total energy production in order to eliminate jobs and replace them with a robot that lies.

Meanwhile, this whole circus is enabling the rising tide of fascism around the world, not only by supercharging propaganda but also by directly financially supporting fascist policies and policymakers. All this to enrich the few, centralize power, reduce competition, and underwrite an enormous bubble that, once it bursts, will ruin the lives of millions of the world’s poor and marginalized classes.

I don’t think it’s cute that someone vibe coded “battleship” in VimScript. I think it’s more important that we stop collectively pretending that we don’t understand how awful all of this is. I don’t want to use software which has slop in it. I do what I can to avoid it, and sadly even Vim now comes under scrutiny in that effort as both Vim and NeoVim are relying on LLMs to develop the software.

So this is how, a few years after Bram’s passing, I find myself in another unusual moment of mourning: mourning Vim itself. What an odd feeling.

To keep my conscience clear, and continue to enjoy the relationship I have with this amazing piece of software, I have forked Vim. You can find my fork here: Vim Classic.

The choice of which version to use as the basis for a fork was a bit difficult. The last version of Vim released during Bram’s lifetime was Vim 9.0. To me, that seems like a good starting point. But, in the end, I chose to base my fork on Vim 8.2.0148 instead. Patch 148 was the patch immediately prior to the introduction of Vim9 Script, Vim 9.0’s flagship feature.

I’m sure Bram worked hard on Vim9 script, and I want to honor that. At the same time, it was still very new when he passed away, and the job of fully realizing its potential was handed down to the current maintainers. Its absence from Vim Classic is an honest assessment that I don’t have the time or energy to try to sort out all of the work on Vim9 which followed in Bram’s footsteps, and decide what stays and what goes. It seems like a useful line to draw in the sand: Vim Classic is compatible with legacy plugins, but not the newfangled stuff.

Since forking from this base, I have backported a handful of patches, most of which address CVEs discovered after this release, but others which address minor bug fixes. I also penned a handful of original patches which bring the codebase from this time up to snuff for building it on newer toolchains. My old vimrc needed very few changes to work on this version of Vim, and all of my plugins work ~~with the exception of fzf.vim, which I would like to fix at some point (or maybe a sympathetic reader is willing to work on backporting the necessary changes)~~. Thanks, dzwdz, for figuring out the issues with fzf.vim!

I plan to use this for a little while, look for sore points and rough edges, collect feedback from other users, and then tag a little release soon. Going forward, maintenance will be slow and quiet. I welcome your patches, particularly to help with maintaining the runtime scripts, stuff like making sure new language features end up in the syntax files. I’ll also gladly accept new bug fixes, and maybe even a few new features if a good case can be made for including them. Backporting small patches from Vim upstream will be considered, with extra scrutiny.

In short, I invite you to use Vim Classic, if you feel the same way as me, and to maintain it with me, contributing the patches you need to support your own use cases.

The cults of TDD and GenAI

Thu, 29 Jan 2026 00:00:00 +0000

I’ve gotten a lot of flack throughout my career over my disdain towards test-driven development (TDD). I have met a lot of people who swear by it! And, I have also met a lot of people who insisted that I adopt it, too, often with the implied threat of appealing to my boss if appealing to me didn’t work.

The basic premise of TDD, for those unaware, is that one first writes a unit test that verifies the expected behavior for some code they want to write, observes the new test fail, and then one writes the implementation, iterating on it until the test passes. The advantage of this approach is, first, to ensure that your codebase is adequately covered by testing, and, second, to provide you a rapid feedback loop to assist in your work.

I have often found elements of TDD to be quite useful. Using a unit test or something similar to provide an efficient rapid feedback loop is a technique which I have employed many times. However, I am and have always been skeptical of the cult which arises around automated software testing and in particular TDD. A lot of people adopt an unquestioning loyalty to TDD, building tools and practices and vibes around the idea. It’s often too much.

The flaw with TDD is that, while it ensures that you have a test for every function you write, it also exerts an influence on the tested codebase, shaping the code to be as “testable” as possible, which only sometimes leads to better code. Moreover, TDD has no means of ensuring that the behavior that your tests verify is the right behavior for your software to have. Software with a thousand passing tests and 100% test coverage could be doing whatever the user or the business or whatever needs it to, but it could just as easily not meet the requirements in spite of those comprehensive tests – and in any case it gives you confidence in your work, which may or may not be misplaced.

The cult of TDD exploits the fact that TDD is very good at making you feel like a good, diligent programmer. That rapid feedback loop not only assists in your work but also enables a powerful dopamine cycle. Add into that a culture of aiming for 100% coverage and you get the bonus hit from watching a number go up. Buy into the whole cult and you get a slew of new README badges to keep green, and lots of cool charts and numbers, hundreds of blinkenlights on your test suite, a bunch of fun Slack messages from Jenkins, and a cute cardboard cut-out of the CTO to keep in the cubicle of whoever last broke the build.¹ All of this pomp and circumstance is fun and it feels good and because it’s all in the name of testing (which is good, right?) it makes you feel like a good programmer even if none of it necessarily contributes to the results your team is supposed to achieve.

All of these flashy traits allow one to adopt the aesthetics of good, diligent software engineering work regardless of how good the work actually is. It’s an intoxicating way to work, especially for someone who struggles with software engineering. It makes you feel like a good programmer and gives you data to “back it up”, stuff you can cite at your performance review. But, software development is really hard, and TDD doesn’t go that far to making it easier. All of the really hard problems are not solved by TDD.

I suspect that coding agents are tapping into the same emotional and psychological reflexes that the cult of TDD gives us an early example of. Software development is still hard, but using an agent allows someone who’s just “so-so” at programming to feel the rush of being great at programming, a rush they might have been chasing for their entire career, and I bet the rush is so much sweeter than watching the lights on your test suite runs tick over to green.

A coding agent permits one to feel as if they have the raw productive power a great programmer can tap into. One may feel like the “10× programmers” they’ve sat next to in the open office for ten years, whose skills they never quite achieved themselves. It scales up the raw output by a factor of ten, and lets one assemble apparently great works in a fraction of the time, solo-coding great cathedrals in the time it used to take them to build, with great difficulty, a homely shack.

But, if it seems too good to be true…

Those cathedrals are not the great works they appear to be. The construction is shoddy and the architecture nonsensical and a great programmer hand-writing code will still outperform any mediocre programmer once the gleam wears off of their respective works and the bugs and problems start showing up. The project has 99.9% coverage on a thousand beautiful green tests, and, inside, the foundations are still rotten.

God, though, I understand why so many people are chasing that dragon, even though it’s going to ruin their careers, and maybe even their lives. I get why people fall for this, in spite of the externalities that they must know of by now. In spite of the colossal waste, the loss of fresh water resources, the fact that AI datacenters are the fastest growing source of carbon emissions, the people suffering sky-rocketing power bill and rolling outages near these new datacenters, the reams and reams of fascist propaganda these machines are producing to tear our society apart, the corruption, the market manipulation, the plain and simple fact that the ultimate purpose of these tools is to put their users out of a job entirely… well, once you finally get a taste of what it feels like to be great… I suppose all of those problems seem so far away.

Redesigning my microkernel from the ground up

Mon, 12 Jan 2026 00:00:00 +0000

As you may recall, circa 2022-2023 I was working on a microkernel written in Hare named Helios. Helios was largely inspired by and modelled after the design of seL4 and was my first major foray into modern OS development that was serious enough to get to a somewhat useful state of functionality, with drives for some real hardware, filesystems, and an environment for running user programs of a reasonable level of sophistication.

Helios development went strong for a while but eventually it slowed and eventually halted in a state of design hell. Since Helios was my first major OS project at this scale and with this much ambition, the design and implementation ended up with a lot of poor assumptions that made it a pretty weak foundation for building a complete OS upon. In late 2023 I more or less gave up on it and moved my OS development work out of the realm of writing code and back into the realm of thinking really hard about how to design operating systems.

What followed was a couple of years of design thinking, developing small scale design experiments, and doing deeper research into prior art – reading papers and studying existing kernels. It was also during this period that I wrote Bunnix, a working Unix clone, motivated in part by a desire to gain some first-hand experience working in the design and implementation of Unix-style operating systems – a fertile environment for learning a lot of the nuts and bolts of OS implementations by working against a complete and proven design.

In August I was finally prepared to have another go. I decided to start over from scratch, importing and adapting and rewriting code from Helios and Bunnix on an as-needed basis to speed things up, and writing from scratch anything where the lessons learned in hindsight outweighed the benefits of adapting existing code.¹

The result is Hermes.

Hermes has not yet reached feature parity with Helios, lacking some IPC features and an aarch64 port, but already it’s significantly more robust and thoughtfully designed than Helios.

The big glitzy feature that most obviously distinguishes Hermes from Helios is that Hermes supports symmetric multiprocessing (SMP), which is to say, running on multiple CPU cores. This time around, I finally listened to the advice I’d been hearing in osdev circles for years and implemented SMP as early as possible to avoid dealing with tons of problems adding multiprocessing to an otherwise mature kernel.

The multicore scheduler at the heart of Hermes is surprisingly simple, actually. It uses relatively ordinary per-CPU run queues. Each new task, once scheduleable, is scheduled, in order of preference, on (1) the CPU matching its affinity, (2) any currently idle CPU, or (3) a random CPU. If a CPU would idle, it first tries to steal a pending task from another CPU. The most important parts of the scheduler are less than 200 lines of code ([1], [2]).

The less obviously impressive improvements from Helios to Hermes are numerous. The syscall and IPC ABIs were rethought from the ground up – one of the major goals of the redesign. I also moved from an seL4-style capability derivation graph – which is quite complex to implement and reason about – to reference counting to manage the lifetimes of kernel resources. Resource management in general is much simpler and should improve the performance of the kernel substantially.

I’ve also taken a much different approach to organizing the code, to allow the kernel and many of the things around it – its bootloaders and the userspace that runs the kernel test suite – share a lot more code than was possible in Helios, making a lot of the non-kernel code a lot easier to write and maintain.

The userspace is also a substantial upgrade in design from Helios, or at least I hope it will be when more of it takes shape. Rather than developing a specialized Hare standard library, independent of the upstream Hare standard library, for writing drivers and low-level services, I have started with a port of the upstream Hare standard library and built low-level driver and service support libraries around it. The userspace is streamlined considerably by doing so, giving these low-level components access to a more comfortable and featureful programming environment and reducing the complexity of the system by making various components more uniform in their design.

Finally, I’ve taken a much more serious approach to testing Hermes and making it as robust and complete as possible in real-world use-cases. I borrowed the EFI bootloader from Bunnix and repurposed it for Hermes, opening up a lot of newer hardware, and I have written a more comprehensive test suite and run and verified it on much more real-world hardware. I have about ten devices which all (consistently!) pass the Hermes test suite. Feel free to try it out on yours as well and let me know how it goes!

That’s all there is to say for now, but I hope to keep you in the loop as I continue working on this for a while. The userspace is starting to take shape and soon(™) I hope to start building out block device drivers, some filesystems, and enough support code to run a shell and a handful of useful programs. In the meantime, feel free to poke around the code and play around with it. There is also some early documentation available for you to read if you wish. I’m hanging out in #ares on Libera Chat if you have any questions.

OpenAI employees… are you okay?

Sat, 08 Nov 2025 00:00:00 +0000

You might have seen an article making the rounds this week, about a young man who ended his life after ChatGPT encouraged him to do so. The chat logs are really upsetting.

Someone two degrees removed from me took their life a few weeks ago. A close friend related the story to me, about how this person had approached their neighbor one evening to catch up, make small talk, and casually discussed their suicidal ideation at some length. At the end of the conversation, they asked to borrow a rope, and their neighbor agreed without giving the request any critical thought. The neighbor found them the next morning.

I didn’t know the deceased, nor their neighbor, but I’m close friends with someone who knew both. I found their story deeply chilling – ice runs through my veins when I imagine how the neighbor must have felt. I had a similar feeling upon reading this article, wondering how the people behind ChatGPT and tools like it are feeling right now.

Two years ago, someone I knew personally took their life as well. I was not friendly with this person – in fact, we were on very poor terms. I remember at the time, I had called a crisis hotline just to ask an expert for advice on how to break this news to other people in my life, many of whom were also on poor terms with a person whose struggles to cope with their mental health issues caused a lot of harm to others.

None of us had to come to terms with any decisions with the same gravity as what that unfortunate neighbor had to face. None of us were ultimately responsible for this person’s troubles or were the impetus for what happened. Nonetheless, the uncomfortable and confronting feelings I experienced in the wake of that event perhaps give me some basis for empathy and understanding towards the neighbor, or for OpenAI employees, and others who find themselves in similar situations.

If you work on LLMs, well… listen, I’ve made my position as an opponent of this technology clear. I feel that these tools are being developed and deployed recklessly, and I believe tragedy is the inevitable result of that recklessness. If you confide in me, I’m not going to validate your career choice. But maybe that’s not necessarily a bad quality to have in a confidant? I still feel empathy towards you and I recognize your humanity and our need to acknowledge each other as people.

If you feel that I can help, I encourage you to reach out. I will keep our conversation in confidence, and you can reach out anonymously if that makes you feel safer. I’m a good listener and I want to know how you’re doing. Email me.

If you’re experiencing a crisis, 24-hour support is available from real people who are experts in getting you the help you need. Please consider reaching out. All you need to do is follow the link.

What's up with FUTO?

Wed, 22 Oct 2025 00:00:00 +0000

Some time ago, I noticed some new organization called FUTO popping up here and there. I’m always interested in seeing new organizations that fund open source popping up, and seeing as they claim several notable projects on their roster, I explored their website with interest and gratitude. I was first confused, and then annoyed by what I found. Confused, because their website is littered with bizzare manifestos,¹ and ultimately annoyed because they were playing fast and loose with the term “open source”, using it to describe commercial source-available software.

FUTO eventually clarified their stance on “open source”, first through satire and then somewhat more soberly, perpetuating the self-serving myth that “open source” software can privilege one party over anyone else and still be called open source. I mentally categorized them as problematic but hoped that their donations or grants for genuinely open source projects would do more good than the harm done by this nonsense.

By now I’ve learned better. tl;dr: FUTO is not being honest about their “grant program”, they don’t have permission to pass off these logos or project names as endorsements, and they collaborate with and promote mask-off, self-proclaimed fascists.

An early sign that something is off with FUTO is in that “sober” explanation of their “disdain for OSI approved licenses”, where they make a point of criticizing the Open Source Initiative for banning Eric S. Raymond (aka ESR) from their mailing lists, citing right-wing reactionary conspiracy theorist Bryan Lunduke’s blog post on the incident. Raymond is, as you may know, one of the founders of OSI and a bigoted asshole. He was banned from the mailing lists, not because he’s a bigoted asshole, but because he was being a toxic jerk on the mailing list in question. Healthy institutions outgrow their founders. That said, FUTO’s citation and perspective on the ESR incident could be generously explained as a simple mistake, and we should probably match generosity with generosity given their prolific portfolio of open source grants.

I visited FUTO again quite recently as part of my research on Cloudflare’s donations to fascists, and was pleased to discover that this portfolio of grants had grown immensely since my last visit, and included a number of respectable projects that I admire and depend on (and some projects I don’t especially admire, hence arriving there during my research on FOSS projects run by fascists). But something felt fishy about this list – surely I would have heard about it if someone was going around giving big grants to projects like ffmpeg, VLC, musl libc, Tor, Managarm, Blender, NeoVim – these projects have a lot of overlap with my social group and I hadn’t heard a peep about it.

So I asked Rich Felker, the maintainer of musl libc, about the FUTO grant, and he didn’t know anything about it. Rich and I spoke about this for a while and eventually Rich uncovered a transaction in his GitHub sponsors account from FUTO: a one-time donation of $1,000. This payment circumvents musl’s established process for donations from institutional sponsors. The donation page that FUTO used includes this explanation: “This offer is for individuals, and may be available to small organizations on request. Commercial entities wishing to be listed as sponsors should inquire by email.” It’s pretty clear that there are special instructions for institutional donors who wish to receive musl’s endorsement as thanks for their contribution.

The extent of the FUTO “grant program”, at least in the case of musl libc, involved ignoring musl’s established process for institutional sponsors, quietly sending a modest one-time donation to one maintainer, and then plastering the logo of a well-respected open source project on a list of “grant recipients” on their home page. Rich eventually posted on Mastodon to clarify that the use of the musl name and logo here was unauthorized.

I also asked someone I know on the ffmpeg project about the grant that they had received from FUTO and she didn’t know anything about it, either. Here’s what she said:

I’m sure we did not get a grant from them, since we tear each other to pieces over everything, and that would be enough to start a flame war. Unless some dev independently got money from them to do something, but I’m sure that we as a project got nothing. The only grant we’ve received is from the STF last year.

Neovim is another project FUTO lists as a grant recipient, and they also have a separate process for institutional sponsors. I didn’t reach out to anyone to confirm, but FUTO does not appear on the sponsor list so presumably the M.O. is the same. This is also the case for Wireshark, Conduit, and KiCad. GrapheneOS is listed prominently as well, but that doesn’t seem to have worked out very well for them. Presumably ffmpeg received a similar quiet donation from FUTO, rather than something more easily recognizable as a grant.

So, it seems like FUTO is doing some shady stuff and putting a bunch of notable FOSS projects on their home page without good reason to justify their endorsement. Who’s behind all of this?

As far as I can tell, the important figures are Eron Wolf² and Louis Rossmann.³ Wolf is the founder of FUTO – a bunch of money fell into his lap from founding Yahoo Games before the bottom fell out of Yahoo, and he made some smart investments to grow his wealth, which he presumably used to fund FUTO. Rossmann is a notable figure in the right to repair movement, with a large following on YouTube, who joined FUTO a year later and ultimately moved to Austin to work more closely with them. His established audience and reputation provides a marketable face for FUTO. I had heard of Rossmann prior to learning about FUTO and held him in generally good regard, despite little specific knowledge of his work, simply because we have a common cause in right to repair.

I hadn’t heard of Wolf before looking into FUTO. However, in the course of my research, several people tipped me off to his association with Curtis Yarvin (aka moldbug), and in particular to the use of FUTO’s platform and the credentials of Wolf and Rossman to platform and promote Yarvin. Curtis Yarvin is a full-blown, mask-off, self-proclaimed fascist. A negligible amount of due diligence is required to verify this, but here’s one source from Politico in January 2025:

I’ve interacted with Vance once since the election. I bumped into him at a party. He said, “Yarvin, you reactionary fascist.” I was like, “Thank you, Mr. Vice President, and I’m glad I didn’t stop you from getting elected.”

– Ian Ward for Politico

Vice President Vance and numerous other figures in the American right have cited Yarvin as a friend and source of inspiration in shaping policy.⁴ Among his many political positions, Yarvin has proclaimed that black people are genetically predisposed to a lower IQ than white people, and moreover suggests that black people are inherently suitable for enslavement.⁵

Yarvin has appeared on FUTO’s social media channels, in particular in an interview published on PeerTube and Odysee, the latter a platform controversial for its role in spreading hate speech and misinformation.⁶ Yarvin also appeared on stage to “debate” Louis Rossmann in June 2022, in which Yarvin is permitted to speak at length with minimal interruptions or rebuttals to argue for an authoritarian techno-monarchy to replace democracy.

Rossmann caught some flack for this “debate” and gave a milquetoast response in a YouTube comment on this video, explaining that he agreed to this on very short notice as a favor to Eron, who had donated “a million” to Rossmann’s non-profit prior to bringing Rossmann into the fold at FUTO. Rossmann does rebuke Yarvin’s thesis, albeit buried in this YouTube comment rather than when he had the opportunity to do so on-stage during the debate. Don’t argue with fascists, Louis – they aren’t arguing with you, they are pitching their ideas⁷ to the audience. Smart fascists are experts at misdirection and bad-faith debate tactics and as a consequence Rossmann just becomes a vehicle for fascist propaganda – consult the YouTube comments to see who this video resonates with the most.

In the end, Rossmann seems to regret agreeing to this debate. I don’t think that Eron Wolf regrets it, though – based on his facilitation of this debate and his own interview with Yarvin on the FUTO channel a month later, I can only assume that Wolf considers Yarvin a close associate. No surprise given that Wolf is precisely the kind of insecure silicon valley techbro Yarvin’s rhetoric is designed to appeal to – moderately wealthy but unknown, and according to Yarvin, fit to be a king. Rossmann probably needs to reflect on why he associates with and lends his reputation to an organization that openly and unapologetically platforms its founder’s fascist friends.

In summary, FUTO is not just the product of some eccentric who founded a grant-making institution that funds open source at the cost of making us read his weird manifestos on free markets and oligopoly. It’s a private, for-profit company that associates with and uses their brand to promote fascists. They push an open-washing narrative and they portray themselves as a grant-making institution when, in truth, they’re passing off a handful of small donations as if they were endorsements from dozens of respectable, high-profile open source projects, in an attempt to legitimize themselves, and, indirectly, legitimize people they platform like Curtis Yarvin.

So, if you read this and discover that your project’s name and logo is being proudly displayed on the front page of a fascist-adjacent, washed-up millionaire’s scummy vanity company, and you don’t like that, maybe you should ask them to knock it off? Eron, Louis – you know that a lot of these logos are trademarked, right?

Updated 2025-10-27:

The FUTO website has been updated to clarify the nature of grants versus donations (before, after) and to reduce the appearance of endorsements from the donation recipients – the site is much better after this change.

I spoke to a representative who spoke for the FUTO leadership, and shared positive feedback regarding the changes to the website. I also asked for a follow-up on the matter of platforming fascist activist Curtis Yarvin on their social media channels, and I was provided this official response:

We prefer to spend our time building great software for people to use, funding interesting projects, and making FUTO better every day. We have no interest in engaging in politics as this is a distraction from our work. We’d like to move past this. We don’t have any further comment that that.

I understand the view that distancing oneself from politics can be a productive approach to your work, and the work FUTO does funding great software like Immich is indeed important work that should be conducted relatively free of distractions. However, FUTO is a fundamentally political organization and it does not distance itself from politics. Consider for example the FUTO statement on Open Source, which takes several political positions:

Disapproval of the Open Source Initiative and their legitimacy as an authority
Disapproval of OSI’s proposed AI standards
Disapproval of the “tech oligopoly”
Advocacy for an “open source” which is inclusive of restrictions on commercial use
Support for Eric S. Raymond’s side in his conflict with OSI
Tacit support for Bryan Lunduke

A truer “apolitical” approach would accept the mainstream definition of open source, would not take positions on conflicts with OSI or Eric Raymond, and would be careful not to cite (or platform) controversial figures such as Lunduke.

It is difficult for FUTO’s work to be apolitical at all. Importantly, there are biases in their grants and donations: their selections have a tendency to privilege projects focused on privacy, decentralization, multimedia, communication, and right to repair, all of which suggest the political priorities of FUTO. The choice to fund “source first” software in addition to open source, or not to fund outright closed source software, or not to vet projects based on their community moderation, are also political factors in their process of selecting funding recipients, or at least are seemingly apolitical decisions which ultimately have political consequences.

This brings us to the political nature of the choice to platform Curtis Yarvin. Yarvin is a self-proclaimed fascist who argues openly for fascist politics. Platforming Yarvin on the FUTO channels legitimizes Yarvin’s ideas and his work, and provides curious listeners a funnel that leads to Yarvin’s more radical ideas and into a far-right rabbit-hole. Platforming Yarvin advances the fascist political program.

It should go without saying that it is political to support fascism or fascists. There is an outrageous moral, intellectual, and political contradiction in claiming that it is apolitical to promote a person whose political program is to dismantle democracy and eject people he disagrees with from the political sphere entirely. FUTO should reflect on their values, acknowledge the political nature of their work, and consider the ways in which their work intersects with politics writ large, then make decisions that align their political actions with their political beliefs.

Cloudflare bankrolls fascists

Wed, 24 Sep 2025 00:00:00 +0000

US politics has been pretty fascist lately. The state is filling up concentration camps, engaging in mass state violence against people on the basis of racialized traits, deporting them to random countries without any respect for habeas corpus, exerting state pressure on the free press to censor speech critical of the current administration, and Trump is openly floating the idea of an unconstitutional third term.

Fascism is clearly on the rise, and they’re winning more and more power. None of this is far removed from us in the FOSS community – there are a number of fascists working in FOSS, same as the rest of society. I don’t call them fascists baselessly – someone who speaks out in support of and expresses solidarity with fascists, or who uses fascists dog-whistles or promotes fascist ideology and talking points, or boosts fascist conspiracy theories – well, they’re a fascist.

If one consistently speaks in support of a certain political position and against the opponents of that position then it is correct to identify them with this political position. Facts, as it were, don’t care about feelings, namely the feelings that get hurt when someone is called a fascist. Fascists naturally do not want to be identified as such and will reject the label, but we shouldn’t take their word for it. People should be much more afraid of being called out as fascist than they are afraid of calling someone a fascist. If someone doesn’t want to be called a fascist, they shouldn’t act like one.

It’s in this disturbing political context that I saw an odd post from the Cloudflare blog pop up in my circles this week: Supporting the future of the open web: Cloudflare is sponsoring Ladybird and Omarchy. Based on Ladybird’s sponsorship terms we can assume that these projects received on the order of $100,000 USD from Cloudflare. I find this odd for a few reasons, in particular because one thing that I know these two projects have in common is that they are both run by fascists.

Even at face value this is an unusual pair of projects to fund. I’m all for FOSS projects getting funded, of course, and I won’t complain about a project’s funding on the solitary basis that it’s an odd choice. I will point out that these are odd choices, though, especially Omarchy.

Ladybird makes some sense, given that it’s aligned in principle with Cloudflare’s stated objective to “support the open web”, though I remain bearish that new web browser engines are even possible to make.¹ Omarchy is a bizarre choice, though – do we really need another pre-customized Arch Linux distribution? And if we do, do we really need a big corporation like Cloudflare to bankroll it? Everyone on /r/unixporn manages to make Arch Linux look pretty for free.

Omarchy is a very weird project to fund, come to think of it. Making an Arch Linux spin technically requires some work, and work is work, I won’t deny it, but most of the work done here is from Arch Linux and Hyprland. Why not fund those, instead? Well, don’t fund Hyprland, since it’s also run by a bunch of fascists, but you get my point.

Anyway, Omarchy and Ladybird are both run by fascists. Omarchy makes this pretty obvious from the outset – on the home page the big YouTube poster image prominently features SuperGrok, which is a pathetically transparent dog-whistle to signal alliance with Elon Musk’s fascist politics. Omarchy is the pet project of David Heinemeier Hansson, aka DHH, who is well known as a rich fascist weirdo.² One need only consult his blog to browse his weird, racist views on immigration, fat-shaming objections to diverse representation, vaguely anti-feminist/homophobic/rapey rants on consent, and, recently, tone-policing antifascists who celebrate the death of notable fascist Charlie Kirk.

Speaking of tributes to Charlie Kirk, that brings us to Andreas Kling, the project lead for Ladybird, who tweeted on the occasion of his assassination:

RIP Charlie Kirk
I hope many more debate nerds carry on his quest to engage young people with words, not fists.

– @awesomekling

Kling has had a few things to say about Kirk on Twitter lately. Here’s another one – give you three guesses as to which “[group]” he objects to punching. You may also recall that Kling achieved some notoriety for his obnoxious response as the maintainer of SerenityOS when someone proposed gender-neutral language for the documentation:

Screenshot of the interaction on GitHub. Kling responds “This project is not an appropriate arena to advertise your personal politics.”

Replacing “he” with “they” in one sentence of the documentation is the kind of “ideologically motivated change” that Serenity’s CONTRIBUTING.md apparently aims to prevent, a classic case of the sexist “identities that are not men are inherently political” nonsense. Ladybird has a similar, weirdly defensive policy on “neutrality”, and a milquetoast code of conduct, which is based on the Ruby Community Conduct Guideline, which has been itself the subject of many controversies due to its inadequacy leading to real-world incidents of harassment and abuse.

Here’s another one – Kling endorsing white replacement theory in June:

White males are actively discriminated against in tech.
It’s an open secret of Silicon Valley.
One of the last meetings I attended before leaving Apple (in 2017) was management asking us to “keep the corporate diversity targets in mind” when interviewing potential new hires.
The phrasing was careful, but the implication was pretty clear.
I knew in my heart this wasn’t wholesome, but I was too scared to rock the boat at the time.

– @awesomekling replying to @danheld³

And in a moment of poetic irony, a few days ago Kling spoke in solidarity with Hansson over his “persecution” for “banal, mainstream positions” on Twitter just a few days ago, in response to Hansson’s tweet signal-boosting another notable reactionary tech fascist, Bryan Lunduke.

So, to sum it up, Kling wears his mask a bit better than Hansson, but as far as I’m concerned it seems clear that both projects are run by fascists. If it walks like a fascist and quacks like a fascist… then why is Cloudflare giving them hundreds of thousands of dollars?

A better future for JavaScript that won't happen

Wed, 17 Sep 2025 00:00:00 +0000

In the wake of the largest supply-chain attack in history, the JavaScript community could have a moment of reckoning and decide: never again. As the panic and shame subsides, after compromised developers finish re-provisioning their workstations and rotating their keys, the ecosystem might re-orient itself towards solving the fundamental flaws that allowed this to happen.

After all, people have been sounding the alarm for years that this approach to dependency management is reckless and dangerous and broken by design. Maybe this is the moment when the JavaScript ecosystem begins to understand the importance and urgency of this problem, and begins its course correction. It could leave behind its sprawling dependency trees full of micro-libraries, establish software distribution based on relationships of trust, and incorporate the decades of research and innovation established by more serious dependency management systems.

Perhaps Google and Mozilla, leaders in JavaScript standards and implementations, will start developing a real standard library for JavaScript, which makes micro-dependencies like left-pad a thing of the past. This could be combined with a consolidation of efforts, merging micro-libraries into larger packages with a more coherent and holistic scope and purpose, which prune their own dependency trees in turn.

This could be the moment where npm comes to terms with its broken design, and with a well-funded effort (recall that, ultimately, npm is GitHub is Microsoft, market cap $3 trillion USD), will develop and roll out the next generation of package management for JavaScript. It could incorporate the practices developed and proven in Linux distributions, which rarely suffer from these sorts of attacks, by de-coupling development from packaging and distribution, establishing package maintainers who assemble and distribute curated collections of software libraries. By introducing universal signatures for packages of executable code, smaller channels and webs of trust, reproducible builds, and the many other straightforward, obvious techniques used by responsible package managers.

Maybe other languages that depend on this broken dependency management model, like Cargo, PyPI, RubyGems, and many more, are watching this incident and know that the very same crisis looms in their future. Maybe they will change course, too, before the inevitable.

Imagine if other large corporations who depend on and profit from this massive pile of recklessly organized software committed their money and resources to it, through putting their engineers to the task of fixing these problems, through coming together to establish and implement new standards, through direct funding of their dependencies and by distributing money through institutions like NLNet, ushering in an era of responsible, sustainable, and secure software development.

This would be a good future, but it’s not the future that lies in wait for us. The future will be more of the same. Expect symbolic gestures – mandatory 2FA will be rolled out in more places, certainly, and the big players will write off meager donations in the name of “OSS security and resilience” in their marketing budgets.

No one will learn their lesson. This has been happening for decades and no one has learned anything from it yet. This is the defining hubris of this generation of software development.

Embedding Wren in Hare

Wed, 20 Aug 2025 00:00:00 +0000

I’ve been on the lookout for a scripting language which can be neatly embedded into Hare programs. Perhaps the obvious candidate is Lua – but I’m not particularly enthusiastic about it. When I was evaluating the landscape of tools which are “like Lua, but not Lua”, I found an interesting contender: Wren.

I found that Wren punches far above its weight for such a simple language. It’s object oriented, which, you know, take it or leave it depending on your use-case, but it’s very straightforwardly interesting for what it is. I found a few things to complain about, of course – its scope rules are silly, the C API has some odd limitations here and there, and in my opinion the “standard library” provided by wren CLI is poorly designed. But, surprisingly, my list of complaints more or less ends there, and I was excited to build a nice interface to it from Hare.

The result is hare-wren. Check it out!

The basic Wren C API is relatively straightforwardly exposed to Hare via the wren module, though I elected to mold it into a more idiomatic Hare interface rather than expose the C API directly to Hare. You can use it something like this:

use wren;

export fn main() void = {
	const vm = wren::new(wren::stdio_config);
	defer wren::destroy(vm);
	wren::interpret(vm, "main", `
		System.print("Hello world!")
	`)!;
};

$ hare run -lc main.ha
Hello world!

Calling Hare from Wren and vice-versa is also possible with hare-wren, of course. Here’s another example:

use fmt;
use wren;

export fn main() void = {
	let config = *wren::stdio_config;
	config.bind_foreign_method = &bind_foreign_method;

	const vm = wren::new(&config);
	defer wren::destroy(vm);

	wren::interpret(vm, "main", `
	class Example {
		foreign static greet(user)
	}

	System.print(Example.greet("Harriet"))
	`)!;
};

fn bind_foreign_method(
	vm: *wren::vm,
	module: str,
	class_name: str,
	is_static: bool,
	signature: str,
) nullable *wren::foreign_method_fn = {
	const is_valid = class_name == "Example" &&
		signature == "greet(_)" && is_static;
	if (!is_valid) {
		return null;
	};
	return &greet_user;
};

fn greet_user(vm: *wren::vm) void = {
	const user = wren::get_string(vm, 1)!;
	const greeting = fmt::asprintf("Hello, {}!", user)!;
	defer free(greeting);
	wren::set_string(vm, 0, greeting);
};

$ hare run -lc main.ha
Hello, Harriet!

In addition to exposing the basic Wren virtual machine to Hare, hare-wren has an optional submodule, wren::api, which implements a simple async runtime based on hare-ev and a modest “standard” library, much like Wren CLI. I felt that the Wren CLI libraries had a lot of room for improvement, so I made the call to implement a standard library which is only somewhat compatible with Wren CLI.

On top of the async runtime, Hare’s wren::api runtime provides some basic features for reading and writing files, querying the process arguments and environment, etc. It’s not much but it is, perhaps, an interesting place to begin building out something a bit more interesting. A simple module loader is also included, which introduces some conventions for installing third-party Wren modules that may be of use for future projects to add new libraries and such.

Much like wren-cli, hare-wren also provides the hwren command, which makes this runtime, standard library, and module loader conveniently available from the command line. It does not, however, support a REPL at the moment.

I hope you find it interesting! I have a few projects down the line which might take advantage of hare-wren, and it would be nice to expand the wren::api library a bit more as well. If you have a Hare project which would benefit from embedding Wren, please let me know – and consider sending some patches to improve it!

What's new with Himitsu 0.9?

Fri, 08 Aug 2025 00:00:00 +0000

Last week, Armin and I worked together on the latest release of Himitsu, a “secret storage manager” for Linux. I haven’t blogged about Himitsu since I announced it three years ago, and I thought it would be nice to give you a closer look at the latest release, both for users eager to see the latest features and for those who haven’t been following along.¹

A brief introduction: Himitsu is like a password manager, but more general: it stores any kind of secret in its database, including passwords but also SSH keys, credit card numbers, your full disk encryption key, answers to those annoying “security questions” your bank obliged you to fill in, and so on. It can also enrich your secrets with arbitrary metadata, so instead of just storing, say, your IMAP password, it can also store the host, port, TLS configuration, and username, storing the complete information necessary to establish an IMAP session.

Another important detail: Himitsu is written in Hare and depends on Hare’s native implementations of cryptographic primitives – neither Himitsu nor the cryptography implementation it depends on have been independently audited.

So, what new and exciting features does Himitsu 0.9 bring to the table? Let me summarize the highlights for you.

A new prompter

The face of Himitsu is the prompter. The core Himitsu daemon has no user interface and only communicates with the outside world through its IPC protocols. One of those protocols is the “prompter”, which Himitsu uses to communicate with the user, to ask you for consent to use your secret keys, to enter the master password, and so on. The prompter is decoupled from the daemon so that it is easy to substitute with different versions which accommodate different use-cases, for example by integrating the prompter more deeply into a desktop environment or to build one that fits better on a touch screen UI like a phone.

But, in practice, given Himitsu’s still-narrow adoption, most people use the GTK+ prompter developed upstream. Until recently, the prompter was written in Python for GTK+ 3, and it was a bit janky and stale. The new hiprompt-gtk changes that, replacing it with a new GTK4 prompter implemented in Hare.

I’m excited to share this one with you – it was personally my main contribution to this release. The prompter is based on Alexey Yerin’s hare-gi, which is a (currently only prototype-quality) code generator which processes GObject Introspection documents into Hare modules that bind to libraries like GTK+. The prompter uses Adwaita for its aesthetic and controls and GTK layer shell for smoother integration on supported Wayland compositors like Sway.

Secret service integration

Armin has been hard at work on a new package, himitsu-secret-service, which provides the long-awaited support for integrating Himitsu with the dbus Secret Service API used by many Linux applications to manage secret keys. This makes it possible for Himitsu to be used as a secure replacement for, say, gnome-keyring.

Editing secret keys

Prior to this release, the only way to edit a secret key was to remove it and re-add it with the desired edits applied manually. This was a tedious and error-prone process, especially when bulk-editing keys. This release includes some work from Armin to improve the process, by adding a “change” request to the IPC protocol and implementing it in the command line hiq client.

For example, if you changed your email address, you could update all of your logins like so:

$ hiq -c email=newemail@example.org email=oldemail@example.org

Don’t worry about typos or mistakes – the new prompter will give you a summary of the changes for your approval before the changes are applied.

You can also do more complex edits with the -e flag – check out the hiq(1) man page for details.

Secret reuse notifications

Since version 0.8, Himitsu has supported “remembering” your choice, for supported clients, to consent to the use of your secrets. This allows you, for example, to remember that you agreed for the SSH agent to use your SSH keys for an hour, or for the duration or your login session, etc. Version 0.9 adds a minor improvement to this feature – you can add a command to himitsu.ini, such as notify-send, which will be executed whenever a client takes advantage of this “remembered” consent, so that you can be notified whenever your secrets are used again, ensuring that any unexpected use of your secrets will get your attention.

himitsu-firefox improvements

There are also some minor improvements landed for himitsu-firefox that I’d like to note. tiosgz sent us a nice patch which makes the identification of login fields in forms more reliable – thanks! And I’ve added a couple of useful programs, himitsu-firefox-import and himitsu-firefox-export, which will help you move logins between Himitsu and Firefox’s native password manager, should that be useful to you.

And the rest

Check out the changelog for the rest of the improvements. Enjoy!

Just speak the truth

Mon, 30 Jun 2025 00:00:00 +0000

Today, we’re looking at two case studies in how to respond when reactionaries appear in your free software community.

Exhibit A

It is a technical decision.
The technical reason is that the security team does not have the bandwidth to provide lifecycle maintenance for multiple X server implementations. Part of the reason for moving X from main to community was to reduce the burden on the security team for long-term maintenance of X. Additionally, nobody so far on the security team has expressed any interest in collaborating with xxxxxx on security concerns.
We have a working relationship with Freedesktop already, while we would have to start from the beginning with xxxxxx.
Why does nobody on the security team have any interest in collaboration with xxxxxx? Well, speaking for myself only here – when I looked at their official chat linked in their README, I was immediately greeted with alt-right propaganda rather than tactically useful information about xxxxxx development. At least for me, I don’t have any interest in filtering through hyperbolic political discussions to find out about CVEs and other relevant data for managing the security lifecycle of X.
Without relevant security data products from xxxxxx, as well as a professionally-behaving security contact, it is unlikely for xxxxxx to gain traction in any serious distribution, because X is literally one of the more complex stacks of software for a security team to manage already.
At the same time, I sympathize with the need to keep X alive and in good shape, and agree that there hasn’t been much movement from freedesktop in maintaining X in the past few years. There are many desktop environments which will never get ported to Wayland and we do need a viable solution to keep those desktop environments working.

I know the person who wrote this, and I know that she’s a smart cookie, and therefore I know that she probably understood at a glance that the community behind this “project” literally wants to lynch her. In response, she takes the high road, avoids confronting the truth directly, and gives the trolls a bunch of talking points to latch on for counter-arguments. Leaves plenty of room for them to bog everyone down in concern trolling and provides ample material to fuel their attention-driven hate machine.

There’s room for improvement here.

Exhibit B

Screenshot of a post by Chimera Linux which reads “any effort to put (redacted) in chimera will be rejected on the technical basis of the maintainers being reactionary dipshits”

Concise, speaks the truth, answers ridiculous proposals with ridicule, does not afford the aforementioned reactionary dipshits an opportunity to propose a counter-argument. A+.

Extra credit for the follow-up:

Screenshot of a follow-up post that reads “just to be clear, given the coverage of the most recent post, we don’t want to be subject to any conspiracy theories arising from that. so i’ll just use this opportunity to declare that we are definitely here to further woke agenda by turning free software gay”

The requirement for a passing grade in this class is a polite but summary dismissal, but additional credit is awarded for anyone who does not indulge far-right agitators as if they were equal partners in maintaining a sense of professional decorum.

If you are a community leader in FOSS, you are not obligated to waste your time coming up with a long-winded technical answer to keep nazis out of your community. They want you to argue with them and give them attention and feed them material for their reactionary blog or whatever. Don’t fall into their trap. Do not answer bad faith with good faith. This is a skill you need to learn in order to be an effective community leader.

If you see nazis 👏👏 you ban nazis 👏👏 — it’s as simple as that.

The name of the project is censored not because it’s particularly hard for you to find, but because all they really want is attention, and you and me are going to do each other a solid by not giving them any of that directly.

To preclude the sorts of reply guys who are going to insist on name-dropping the project and having a thread about the underlying drama in the comments, the short introduction is as follows:

For a few years now, a handful of reactionary trolls have been stoking division in the community by driving a wedge between X11 and Wayland users, pushing a conspiracy theory that paints RedHat as the DEI boogeyman of FOSS and assigning reactionary values to X11 and woke (pejorative) values to Wayland. Recently, reactionary opportunists “forked” Xorg, replaced all of the literature with political manifestos and dog-whistles, then used it as a platform to start shit with downstream Linux distros by petitioning for inclusion and sending concern trolls to waste everyone’s time.

The project itself is of little consequence; they serve our purposes today by providing us with case-studies in dealing with reactionary idiots starting shit in your community.

Unionize or die

Mon, 09 Jun 2025 00:00:00 +0000

Tech workers have long resisted the suggestion that we should be organized into unions. The topic is consistently met with a cold reception by tech workers when it is raised, and no big tech workforce is meaningfully organized. This is a fatal mistake – and I don’t mean “fatal” in the figurative sense. Tech workers, it’s time for you to unionize, and strike, or you and your loved ones are literally going to die.

In this article I will justify this statement and show that it is clearly not hyperbolic. I will explain exactly what you need to do, and how organized labor can and will save your life.

Hey – if you want to get involved in labor organizing in the tech sector you should consider joining the new unitelabor.dev forum. Adding a head’s up here in case you don’t make it to the end of this very long blog post.

The imperative to organize is your economic self-interest

Before I talk about the threats to your life and liberty that you must confront through organized labor, let me re-iterate the economic position for unionizing your workplace. It is important to revisit this now, because the power politics of the tech sector has been rapidly changing over the past few years, and those changes are not in your favor.

The tech industry bourgeoisie has been waging a prolonged war on labor for at least a decade. Far from mounting any kind of resistance, most of tech labor doesn’t even understand that this is happening to them. Your boss is obsessed with making you powerless and replaceable. You may not realize how much leverage you have over your boss, but your boss certainly does – and has been doing everything in their power to undermine you before you wizen up. Don’t let yourself believe you’re a part of their club – if your income depends on your salary, you are part of the working class.

Payroll – that’s you – is the single biggest expense for every tech company. When tech capitalists look at their balance sheet and start thinking of strategies for increasing profits, they see an awful lot of pesky zeroes stacked up next to the line item for payroll and benefits. Long-term, what’s their best play?

It starts with funneling cash and influence into educating a bigger, cheaper generation of compsci graduates to flood the labor market – “everyone can code”. Think about strategic investments in cheap(ish), broadly available courses, online schools and coding “bootcamps” – dangling your high salary as the carrot in front of wannabe coders fleeing dwindling prospects in other industries, certain that the carrot won’t be nearly as big when they all eventually step into a crowded labor market.

The next step is rolling, industry-wide mass layoffs – often obscured under the guise of “stack ranking” or some similar nonsense. Big tech has been callously cutting jobs everywhere, leaving workers out in the cold in batches of thousands or tens of thousands. If you don’t count yourself among them yet, maybe you will soon. What are your prospects for re-hire going to look like if this looming recession materializes in the next few years?

Consider what’s happening now – why do you think tech is driving AI mandates down from the top? Have you been ordered to use an LLM assistant to “help” with your programming? Have you even thought about why the executives would push this crap on you? You’re “training” your replacement. Do you really think that, if LLMs really are going to change the way we code, they aren’t going to change the way we’re paid for it? Do you think your boss doesn’t see AI as a chance to take $100M off of their payroll expenses?

Aren’t you worried you could get laid off and this junior compsci grad or an H1B takes your place for half your salary? You should be – it’s happening everywhere. What are you going to do about it? Resent the younger generation of programmers just entering the tech workforce? Or the immigrant whose family pooled their resources to send them abroad to study and work? Or maybe you weren’t laid off yet, and you fancy yourself better than the poor saps down the hall who were. Don’t be a sucker – your enemy isn’t in the cubicle next to you, or on the other side of the open office. Your enemy has an office with a door on it.

Listen: a tech union isn’t just about negotiating higher wages and benefits, although that’s definitely on the table. It’s about protecting yourself, and your colleagues, from the relentless campaign against labor that the tech leadership is waging against us. And more than that, it’s about seizing some of the awesome, society-bending power of the tech giants. Look around you and see what destructive ends this power is being applied to. You have your hands at the levers of this power if only you rise together with your peers and make demands.

And if you don’t, you are responsible for what’s going to happen next.

The imperative to organize is existential

If global warming is limited to 2°C, here’s what Palo Alto looks like in 2100:¹

Map of Palo Alto showing flooding near the coast

Limiting warming to 2° C requires us to cut global emissions in half by 2030 – in 5 years – but emissions haven’t even peaked yet. Present-day climate policies are only expected to limit warming to 2.5° to 2.9° C by 2100.² Here’s Palo Alto in 75 years if we stay our current course:

Map of Palo Alto showing much more extreme flooding

Here’s the Gulf of Mexico in 75 years:

Gulf of Mexico showing

This is what will happen if things don’t improve. Things aren’t improving – they’re getting worse. The US elected an anti-science president who backed out of the Paris agreement, for a start. Your boss is pouring all of our freshwater into datacenters to train these fucking LLMs and expanding into this exciting new market with millions of tons of emissions as the price of investment. Cryptocurrencies still account for a full 1% of global emissions. Datacenters as a whole account for 2%. That’s on us – tech workers. That is our fucking responsibility.

Climate change is accelerating, and faster than we thought, and the rich and powerful are making it happen faster. Climate catastrophe is not in the far future, it’s not our children or our children’s children, it’s us, it’s already happening. You and I will live to see dozens of global catastrophes playing out in our lifetimes, with horrifying results. Even if we started a revolution tomorrow and overthrew the ruling class and implemented aggressive climate policies right now we will still watch tens or hundreds of millions die.

Let’s say you are comfortably living outside of these blue areas, and you’ll be sitting pretty when Louisiana or Bruges or Fiji are flooded. Well, 13 million Americans are expected to have to migrate out of flooded areas – and 216 million globally³ – within 25 to 30 years. That’s just from the direct causes of climate change – as many as 1 billion could be displaced if we account for the ensuing global conflict and civil unrest.⁴ What do you think will happen to non-coastal cities and states when 4% of the American population is forced to flee their homes? You think you won’t be affected by that? What happens when anywhere from 2.5% to 12% of the Earth’s population becomes refugees?

What are you going to eat? Climate change is going to impact fresh water supplies and reduce the world’s agriculturally productive land. Livestock is expected to be reduced by 7-10% in just 25 years.⁵ Food prices will skyrocket and people will starve. 7% of all species on Earth may already be extinct because of human activities.⁶ You think that’s not going to affect you?

The overwhelming majority of the population supports climate action.⁷ The reason it’s not happening is because, under capitalism, capital is power, and the few have it and the many don’t. We live in a global plutocracy.

The plutocracy has an answer to climate change: fascism. When 12% of the world’s population is knocking at the doors of the global north, their answer will be concentration camps and mass murder. They are already working on it today. When the problem is capitalism, the capitalists will go to any lengths necessary to preserve the institutions that give them power – they always have. They have no moral compass or reason besides profit, wealth, and power. The 1% will burn and pillage and murder the 99% without blinking.

They are already murdering us. 1.2 million Americans are rationing their insulin.⁸ The healthcare industry, organized around the profit motive, murders 68,000 Americans per year.⁹ To the Europeans among my readership, don’t get too comfortable, because I assure you that our leaders are working on destroying our healthcare systems, too.

Someone you love will be laid off, get sick, and die because they can’t afford healthcare. Someone you know, probably many people that you know, will be killed by climate change. It might be someone you love. It might be you.

When you do get laid off mid-recession, your employer replaces you and three of your peers with a fresh bootcamp “graduate” and a GitHub Copilot subscription, and all of the companies you might apply to have done the same… how long can you keep paying rent? What about your friends and family, those who don’t have a cushy tech job or tech worker prospects, what happens when they get laid off or automated away or just priced out of the cost of living? Homelessness is at an all time high and it’s only going to get higher. Being homeless takes 30 years off of your life expectancy.¹⁰ In the United States, there are 28 vacant homes for every homeless person.¹¹

Capitalism is going to murder the people you love. Capitalism is going to murder you.

We need a different answer to the crises that we face. Fortunately, the working class can offer a better solution – one with a long history of success.

Organizing is the only answer and it will work

The rich are literally going to kill you and everyone you know and love just because it will make them richer. Because it is making them richer.

Do you want to do something about any of the real, urgent problems you face? Do you want to make meaningful, rapid progress on climate change, take the catastrophic consequences we are already guaranteed to face in stride, and keep your friends and family safe?

Well, tough shit – you can’t. Don’t tell me you’ll refuse the work, or that it’ll get done anyway without you, or that you can just find another job. They’ll replace you, you won’t find another job, and the world will still burn. You can’t vote your way to a solution, either: elections don’t matter, your vote doesn’t matter, and your voice is worthless to politicians.¹² Martin Gilens and Benjamin Page demonstrated this most clearly in their 2014 study, “Testing Theories of American Politics: Elites, Interest Groups, and Average Citizens”.¹³

Gilens and Page plotted a line chart which shows us the relationship between the odds of a policy proposal being adopted (Y axis) charted against public support for the policy (X axis). If policy adoption was entirely driven by public opinion, we would expect a 45° line (Y=X), where broad public support guarantees adoption and broad public opposition prevents adoption. We could also substitute “public opinion” for the opinions of different subsets of the public to see their relative impact on policy. Here’s what they got:

Two graphs, the first labelled “Average Citizens’ Preferences” and the second “Economic Elites’ Preferences”, showing that the former has little to no correlation with the odds of a policy being adopted, and the latter has a significant impact

For most of us, we get a flat line: Y, policy adoption, is completely unrelated to X, public support. Our opinion has no influence whatsoever on policy adoption. Public condemnation or widespread support has the same effect on a policy proposal, i.e. none. But for the wealthy, it’s a different story entirely. I’ve never seen it stated so plainly and clearly: the only thing that matters is money, wealth, and capital. Money is power, and the rich have it and you don’t.

Nevertheless, you must solve these problems. You must participate in finding and implementing solutions. You will be fucked if you don’t. But it is an unassailable fact that you can’t solve these problems, because you have no power – at least, not alone.

Together, we do have power. In fact, we can fuck with those bastards’ money and they will step in line if, and only if, we organize. It is the only solution, and it will work.

The ultra-rich possess no morals or ideology or passion or reason. They align with fascists because the fascists promise what they want, namely tax cuts, subsidies, favorable regulation, and cracking the skulls of socialists against the pavement. The rich hoard and pillage and murder with abandon for one reason and one reason only: it’s profitable. The rich always do what makes them richer, and only what makes them richer. Consequently, you need to make this a losing strategy. You need to make it more profitable to do what you want. To control the rich, you must threaten the only thing they care about.

Strikes are so costly for companies that they will do anything to prevent them – and if they fail to prevent them, then shareholders will pressure them to capitulate if only to stop the hemorrhaging of profit. This threat is so powerful that it doesn’t have to stop at negotiating your salary and benefits. You could demand your employer participate in boycotting Israel. You could demand that your employer stops anti-social lobbying efforts, or even adopts a pro-social lobbying program. You could demand that your CEO cannot support causes that threaten the lives and dignity of their queer or PoC employees. You could demand that they don’t bend the knee to fascists. If you get them where it hurts – their wallet – they will fall in line. They are more afraid of you than we are afraid of them. They are terrified of us, and it’s time we used that to our advantage.

We know it works because it has always worked. In 2023, United Auto Workers went on strike and most workers won a 25% raise. In February, teachers in Los Angeles went on strike for just 8 days and secured a 19% raise. Nurses in Oregon won a 22% raise, better working schedules, and more this year – and Hawaiian nurses secured an agreement to improve worker/patient ratios in September. Tech workers could take a page out of the Writer’s Guild’s book – in 2023 they secured a prohibition against the use of their work to train AI models and the use of AI to suppress their wages.

Organized labor is powerful and consistently gets concessions from the rich and powerful in a way that no other strategy has ever been able to. It works, and we have a moral obligation to do it. Unions get results.

How to organize step by step

I will give you a step-by-step plan for exactly what you need to do to start moving the needle here. The process is as follows:

Building solidarity and community with your peers
Understanding your rights and how to organize safely
Establishing the consensus to unionize, and doing it
Promoting solidarity across tech workplaces and labor as a whole

Remember that you will not have to do this alone – in fact, that’s the whole point. Step one is building community with your colleagues. Get to know them personally, establish new friendships and grow the friendships you already have. Learn about each other’s wants, needs, passions, and so on, and find ways to support each other. If someone takes a sick day, organize someone to check on them and make them dinner or pick up their kids from school. Organize a board game night at your home with your colleagues, outside of work hours. Make it a regular event!

Talk to your colleagues about work, and your workplace. Tell each other about your salaries and benefits. When you get a raise, don’t be shy, tell your colleagues how much you got and how you negotiated it. Speak positively about each other at performance reviews and save critical feedback for their ears only. Offer each other advice about how to approach their boss to get their needs met, and be each other’s advocate.

Talk about the power you have to work together to accomplish bigger things. Talk about the advantage of collective action. It can start small – perhaps your team collectively refuses to incorporate LLMs into your workflow. Soon enough you and your colleagues will be thinking about unionizing.

Disclaimer: Knowledge about specific processes and legal considerations in this article is US-specific. Your local laws are likely similar, but you should research the differences with your colleagues.

The process of organizing a union in the US is explained step-by-step at workcenter.gov. More detailed resources, including access to union organizers in your neighborhood, are available from the American Federation of Labor and Congress of Industrial Organizations (AFL-CIO). But your biggest resources will be people already organizing in the tech sector: in particular you should consult CODE-CWA, which works with tech workers to provide mentoring and resources on organizing tech workplaces – and has already helped several tech workplaces organize their unions and start making a difference. They’ve got your back.

This is a good time to make sure that you and your colleagues understand your rights. First of all, you would be wise to pool your resources and hire the attention of a lawyer specializing in labor – consult your local bar association to find one (it’s easy, just google it and they’ll have a web thing). Definitely reach out to AFL-CIO and CODE-CWA to meet experienced union organizers who can help you.

You cannot be lawfully fired or punished for discussing unions, workplace conditions, or your compensation and benefits, with your colleagues. You cannot be punished for distributing literature in support of your cause, especially if you do it off-site (even just outside of the front door). Be careful not to make careless remarks about your boss’s appearance, complain about the quality of your company’s products, make disparaging comments about clients or customers, etc – don’t give them an easy excuse. Hold meetings and discussions outside of work if necessary, and perform your duties as you normally would while organizing.

Once you start getting serious about organizing, your boss will start to work against you, but know that they cannot stop you. Nevertheless, you and/or some of your colleagues may run the risk of unlawful retaliation or termination for organizing – this is why you should have a lawyer on retainer. This is also why it’s important to establish systems of mutual aid, so that if one of your colleagues gets into trouble you can lean on each other to keep supporting your families. And, importantly, remember that HR works for the company, not for you. HR are the front lines that are going to execute the unionbusting mandates from above.

Once you have a consensus among your colleagues to organize – which you will know because they will have signed union cards – you can approach your employer to ask them to voluntarily recognize the union. If they agree to opening an organized dialogue amicably, you do so. If not, you will reach out to the National Labor Relations Board (NLRB) to organize a vote to unionize. Only organize a vote that you know you will win. Once your workplace votes to unionize, your employer is obligated to negotiate with you in good faith. Start making collective decisions about what you want from your employer and bring them to the table.

In this process, you will have established a relationship with more experienced union organizers who will continue to help you with conducting your union’s affairs and start getting results. The next step is to make yourself available for this purpose to the next tech workplace that wants to unionize: to share what you’ve learned and support the rest of the industry in solidarity. Talk to your friends across the industry and build solidarity and power in mass.

Prepare for the general strike on May 1st, 2028

The call has gone out: on Labor Day, 2028 – just under three years from now – there will be a general strike in the United States. The United Auto Workers union, one of the largest in the United States, has arranged for their collective bargaining agreements to end on this date, and has called for other unions to do the same across all industries. The American Federation of Teachers and its 1.2 million members are on board, and other unions are sure to follow. Your new union should be among them.

This is how we collectively challenge not just our own employers, but our political institutions as a whole. This is how we turn this nightmare around.

A mass strike is a difficult thing to organize. It is certain to be met with large-scale, coordinated, and well-funded propaganda and retaliation from the business and political spheres. Moreover, a mass strike depends on careful planning and mass mutual aid. We need to be prepared to support each other to get it done, and to plan and organize seriously. When you and your colleagues get organized, discuss this strike amongst yourselves and be prepared to join in solidarity with the rest of the 99% around the country and the world at large.

To commit yourselves to participate or get involved in the planning of the grassroots movement, see generalstrikeus.com.

Join unitelabor.dev

I’ve set up a Discourse instance for discussion, organizing, Q&A, and solidarity among tech workers at unitelabor.dev. Please check it out!

If you have any questions or feedback on this article, please post about it there.

Unionize or die

You must organize, and you must start now, or the worst will come to pass. Fight like your life depends on it, beause it does. It has never been more urgent. The tech industry needs to stop fucking around and get organized.

We are powerful together. We can change things, and we must. Spread the word, in your workplace and with your friends and online. On the latter, be ready to fight just to speak – especially in our online spaces owned and controlled by the rich (ahem – YCombinator, Reddit, Twitter – etc). But fight all the same, and don’t stop fighting until we’re done.

We can do it, together.

Resources

Tech-specific:

General:

Industrial Workers of the World (Global)
International Confederation of Labor (Global)
AFL-CIO (US)
workcenter.gov (US)
generalstrikeus.com (US)
Emergency Workplace Organizing Committee (US)
Labor Notes (US)
Organize Now! (UK)

Send me more resources to add here!

The British Airways position on various border disputes

Mon, 05 May 2025 00:00:00 +0000

My spouse and I are on vacation in Japan, spending half our time seeing the sights and the other half working remotely and enjoying the experience of living in a different place for a while. To get here, we flew on British Airways from London to Tokyo, and I entertained myself on the long flight by browsing the interactive flight map on the back of my neighbor’s seat and trying to figure out how the poor developer who implemented this map solved the thorny problems that displaying a world map implies.

I began my survey by poking through the whole interface of this little in-seat entertainment system¹ to see if I can find out anything about who made it or how it works – I was particularly curious to find a screen listing open source licenses that such such devices often disclose. To my dismay I found nothing at all – no information about who made it or what’s inside. I imagine that there must be some open source software in that thing, but I didn’t find any licenses or copyright statements.

When I turned my attention to the map itself, I did find one copyright statement, the only one I could find in the whole UI. If you zoom in enough, it switches from a satellite view to a street view showing the OpenStreetMap copyright line:

Note that all of the pictures in this article were taken by pointing my smartphone camera at the screen from an awkward angle and fine-tune your expectations accordingly. I don’t have pictures to support every border claim documented in this article, but I did take notes during the flight.

Given that British Airways is the proud flag carrier of the United Kingdom I assume that this is indeed the only off-the-shelf copyrighted material included in this display, and everything else was developed in-house without relying on any open source software that might require a disclosure of license and copyright details. For similar reasons I am going to assume that all of the borders shown in this map are reflective of the official opinion of British Airways on various international disputes.

As I briefly mentioned a moment ago, this map has two views: satellite photography and a very basic street view. Your plane and its route are shown in real-time, and you can touch the screen to pan and zoom the map anywhere you like. You can also rotate the map and change the angle in “3D” if you have enough patience to use complex multitouch gestures on the cheapest touch panel they could find.

The street view is very sparse and only appears when you’re pretty far zoomed in, so it was mostly useless for this investigation. The satellite map, thankfully, includes labels: cities, country names, points of interest, and, importantly, national borders. The latter are very faint, however. Here’s an illustrative example:

We also have our first peek at a border dispute here: look closely between the “Georgia” and “Caucasus Mountains” labels. This ever-so-faint dotted line shows what I believe is the Russian-occupied territory of South Ossetia in Georgia. Disputes implicating Russia are not universally denoted as such – I took a peek at the border with Ukraine and found that Ukraine is shown as whole and undisputed, with its (undotted) border showing Donetsk, Luhansk, and Crimea entirely within Ukraine’s borders.

Of course, I didn’t start at Russian border disputes when I went looking for trouble. I went directly to Palestine. Or rather, I went to Israel, because Palestine doesn’t exist on this map:

I squinted and looked very closely at the screen and I’m fairly certain that both the West Bank and Gaza are outlined in these dotted lines using the borders defined by the 1949 armistice. If you zoom in a bit more to the street view, you can see labels like “West Bank” and the “Area A”, “Area B” labels of the Oslo Accords:

Given that this is British Airways, part of me was surprised not to see the whole area simply labelled Mandatory Palestine, but it is interesting to know that British Airways officially supports the Oslo Accords.

Heading south, let’s take a look at the situation in Sudan:

This one is interesting – three areas within South Sudan’s claimed borders are disputed, and the map only shows two with these dotted lines. The border dispute with Sudan in the northeast is resolved in South Sudan’s favor. Another case where BA takes a stand is Guyana, which has an ongoing dispute with Venezuela – but the map only shows Guyana’s claim, albeit with a dotted line, rather than the usual approach of drawing both claims with dotted lines.

Next, I turned my attention to Taiwan:

The cities of Taipei and Kaohsiung are labelled, but the island as a whole was not labelled “Taiwan”. I zoomed and panned and 3D-zoomed the map all over the place but was unable to get a “Taiwan” label to appear. I also zoomed into the OSM-provided street map and panned that around but couldn’t find “Taiwan” anywhere, either.

The last picture I took is of the Kashmir area:

I find these faint borders difficult to interpret and I admit to not being very familiar with this conflict, but perhaps someone in the know with the patience to look more closely will email me their understanding of the official British Airways position on the Kashmir conflict (here’s the full sized picture).

Here are some other details I noted as I browsed the map:

The Hala’ib Triangle and Bir Tawil are shown with dotted lines
The Gulf of Mexico is labelled as such
Antarctica has no labelled borders or settlements

After this thrilling survey of the official political positions of British Airways, I spent the rest of the flight reading books or trying to sleep.

Resistance from the tech sector

Sun, 20 Apr 2025 00:00:00 +0000

As of late, most of us have been reading the news with a sense of anxious trepidation. At least, those of us who read from a position of relative comfort and privilege. Many more read the news with fear. Some of us are already no longer in a position to read the news at all, having become the unfortunate subjects of the news. Fascism is on the rise worldwide and in the United States the news is particularly alarming. The time has arrived to act.

The enemy wants you to be overwhelmed and depressed, to feel like the situation is out of your control. Propaganda is as effective on me as it is on you, and in my own home the despair and helplessness the enemy aims to engineer in us often prevails in my own life. We mustn’t fall for this gambit.

When it comes to resistance, I don’t have all of the answers, and I cannot present a holistic strategy for effective resistance. Nevertheless, I have put some thought towards how someone in my position, or in my community, can effectively apply ourselves towards resistance.

The fact of the matter is that the tech sector is extraordinarily important in enabling and facilitating the destructive tide of contemporary fascism’s ascent to power. The United States is embracing a technocratic fascism at the hands of Elon Musk and his techno-fetishist “Department of Government Efficiency”. Using memes to mobilize the terminally online neo-right, and “digitizing” and “modernizing” government institutions with the dazzling miracles of modern technology, the strategy puts tech, in its mythologized form – prophesied, even, through the medium of science fiction – at the center of a revolution of authoritarian hate.

And still, this glitz and razzle dazzle act obscures the more profound and dangerous applications of tech hegemony to fascism. Allow me to introduce public enemy number one: Palantir. Under the direction of neo-fascist Peter Thiel and in collaboration with ICE, Palantir is applying the innovations of the last few decades of surveillance capitalism to implementing a database of undesirables the Nazis could have never dreamed of. Where DOGE is hilariously tragic, Palantir is nightmarishly effective.

It’s clear that the regime will be digital. The through line is tech – and the tech sector depends on tech workers. That’s us. This puts us in a position to act, and compels us to act. But then, what should we do?

If there’s one thing I want you to take away from this article, something to write on your mirror and repeat aloud to yourself every day, it’s this: there’s safety in numbers. It is of the utmost importance that we dispense with American individualism and join hands with our allies to resist as one. Find your people in your local community, and especially in your workplace, who you can trust and who believe in what’s right and that you can depend on for support. It’s easier if you’re not going it alone. Talk to your colleagues about your worries and lean on them to ease your fears, and allow them to lean on you in turn.

One of the most important actions you can take is to unionize your workplace. We are long overdue for a tech workers union. If tech workers unionize then we can compel our employers – this regime’s instruments of fascist power – to resist also. If you’re at the bottom looking up at your boss’s boss’s boss cozying up with fascists, know that with a union you can pull the foundations of his power out from beneath him.

More direct means of resistance are also possible, especially for the privileged and highly paid employees of big tech. Maneuver yourself towards the levers of power. At your current job, find your way onto the teams implementing the technology that enables authoritarianism, and fuck it up. Drop the database by “mistake”. Overlook bugs. Be confidently wrong in code reviews and meetings. Apply for a job at Palantir, and be incompetent at it. Make yourself a single point of failure, then fail. Remember too that plausible deniability is key – make them work to figure out that you are the problem.

This sort of action is scary and much riskier than you’re probably immediately comfortable with. Inaction carries risks also. Only you are able to decide what your tolerance for risk is, and what kind of action that calls for. If your appetite for risk doesn’t permit sabotage, you could simply refuse to work on projects that aren’t right. Supporting others is essential resistance, too – be there for your friends, especially those more vulnerable than yourself, and support the people who engage in direct resistance. You didn’t see nuffin, right? If your allies get fired for fucking up an important digital surveillance project – you’ll have a glowing reference for them when they apply for Palantir, right?

Big tech has become the problem, and it’s time for tech workers to be a part of the solution. If this scares you – and it should – I get it. I’m scared, too. It’s okay for it to be scary. It’s okay for you not to do anything about it right now. All you have to do right now is be there for your friends and loved ones, and answer this question: where will you draw the line?

Remember your answer, and if and when it comes to pass… you will know when to act. Don’t let them shift your private goalposts until the frog is well and truly boiled to death.

Hang in there.

A Firefox addon for putting prices into perspective

Fri, 04 Apr 2025 00:00:00 +0000

I had a fun idea for a small project this weekend, and so I quickly put it together over the couple of days. The result is Price Perspective.

Humor me: have you ever bought something, considered the price, and wondered how that price would look to someone else? Someone in the developing world, or a billionaire, or just your friend in Australia? In other words, can we develop an intuition for purchasing power?

The Price Perspective add-on answers these questions. Let’s consider an example: my income is sufficient to buy myself a delivery pizza for dinner without a second thought. How much work does it take for someone in Afghanistan to buy the same pizza? I can fire up Price Perspective to check:

The results are pretty shocking.

How about another example: say I’m looking to buy a house in the Netherlands. I fire up funda.nl and look at a few places in Amsterdam. After a few minutes wondering if I’ll ever be in an economic position to actually afford any of these homes (and speculating on if that day will come before or after I have spent this much money on rent over my lifetime), I wonder what these prices look like from the other side. Let’s see what it’d take for the Zuck to buy this apartment I fancy:

Well… that’s depressing. Let’s experiment with Price Perspective to see what it would take to make a dent in Zuck’s wallet. Let’s add some zeroes.

So, Zuckerberg over-bidding this apartment to the tune of €6.5B would cost him a proportion of his annual income which is comparable to me buying it for €5,000.

How about the reverse? How long would I have to work to buy, say, Jeff Bezos’s new mansion?

Yep. That level of wealth inequality is a sign of a totally normal, healthy, well-functioning society.

Curious to try it out for yourself? Get Price Perspective from addons.mozilla.org, tell it where you live and how much money you make in a year, and develop your own sense of perspective.

Using linkhut to signal-boost my bookmarks

Thu, 27 Mar 2025 00:00:00 +0000

Notice: linkhut has started to use generative AI tools upstream. Consequently, I have withdrawn my endorsement of the project.

It must have been at least a year ago that I first noticed linkhut, and its flagship instance at ln.ht, appear on SourceHut, where it immediately caught my attention for its good taste in inspirations. Once upon a time, I had a Pinboard account, which is a similar concept, but I never used it for anything in the end. When I saw linkhut I had a similar experience: I signed up and played with it for a few minutes before moving on.

I’ve been rethinking my relationship social media lately, as some may have inferred from my unannounced disappearance from Mastodon.¹ While reflecting on this again recently, in a stroke of belated inspiration I suddenly appreciated the appeal of tools like linkhut, especially alongside RSS feeds – signal-boosting stuff I read and found interesting.

The appeal of this reminds me of one of the major appeals of SoundCloud to me, back when I used it circa… 2013? That is: I could listen to the music that artists I liked were listening to, and that was amazing for discovering new music. Similarly, for those of you who enjoy my blog posts, and want to read the stuff I like reading, check out my linkhut feed. You can even subscribe to its RSS feed if you like. There isn’t much there today, but I will be filling it up with interesting articles I see and projects I find online.

I want to read your linkhut feed, too, but it’s pretty quiet there at the moment. If you find the idea interesting, sign up for an account or set up your own instance and start bookmarking stuff – and email me your feed so I can find some good stuff to subscribe to in my own feed reader.

Please stop externalizing your costs directly into my face

Mon, 17 Mar 2025 00:00:00 +0000

This blog post is expressing personal experiences and opinions and doesn’t reflect any official policies of SourceHut.

Over the past few months, instead of working on our priorities at SourceHut, I have spent anywhere from 20-100% of my time in any given week mitigating hyper-aggressive LLM crawlers at scale. This isn’t the first time SourceHut has been at the wrong end of some malicious bullshit or paid someone else’s externalized costs – every couple of years someone invents a new way of ruining my day.

Four years ago, we decided to require payment to use our CI services because it was being abused to mine cryptocurrency. We alternated between periods of designing and deploying tools to curb this abuse and periods of near-complete outage when they adapted to our mitigations and saturated all of our compute with miners seeking a profit. It was bad enough having to beg my friends and family to avoid “investing” in the scam without having the scam break into my business and trash the place every day.

Two years ago, we threatened to blacklist the Go module mirror because for some reason the Go team thinks that running terabytes of git clones all day, every day for every Go project on git.sr.ht is cheaper than maintaining any state or using webhooks or coordinating the work between instances or even just designing a module system that doesn’t require Google to DoS git forges whose entire annual budgets are considerably smaller than a single Google engineer’s salary.

Now it’s LLMs. If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots crawl everything they can find, robots.txt be damned, including expensive endpoints like git blame, every page of every git log, and every commit in every repo, and they do so using random User-Agents that overlap with end-users and come from tens of thousands of IP addresses – mostly residential, in unrelated subnets, each one making no more than one HTTP request over any time period we tried to measure – actively and maliciously adapting and blending in with end-user traffic and avoiding attempts to characterize their behavior or block their traffic.

We are experiencing dozens of brief outages per week, and I have to review our mitigations several times per day to keep that number from getting any higher. When I do have time to work on something else, often I have to drop it when all of our alarms go off because our current set of mitigations stopped working. Several high-priority tasks at SourceHut have been delayed weeks or even months because we keep being interrupted to deal with these bots, and many users have been negatively affected because our mitigations can’t always reliably distinguish users from bots.

All of my sysadmin friends are dealing with the same problems. I was asking one of them for feedback on a draft of this article and our discussion was interrupted to go deal with a new wave of LLM bots on their own server. Every time I sit down for beers or dinner or to socialize with my sysadmin friends it’s not long before we’re complaining about the bots and asking if the other has cracked the code to getting rid of them once and for all. The desperation in these conversations is palpable.

Whether it’s cryptocurrency scammers mining with FOSS compute resources or Google engineers too lazy to design their software properly or Silicon Valley ripping off all the data they can get their hands on at everyone else’s expense… I am sick and tired of having all of these costs externalized directly into my fucking face. Do something productive for society or get the hell away from my servers. Put all of those billions and billions of dollars towards the common good before sysadmins collectively start a revolution to do it for you.

Please stop legitimizing LLMs or AI image generators or GitHub Copilot or any of this garbage. I am begging you to stop using them, stop talking about them, stop making new ones, just stop. If blasting CO₂ into the air and ruining all of our freshwater and traumatizing cheap laborers and making every sysadmin you know miserable and ripping off code and books and art at scale and ruining our fucking democracy isn’t enough for you to leave this shit alone, what is?

If you personally work on developing LLMs et al, know this: I will never work with you again, and I will remember which side you picked when the bubble bursts.

A holistic perspective on intellectual property, part 1

Thu, 13 Feb 2025 00:00:00 +0000

I’d like to write about intellectual property in depth, in this first of a series of blog posts on the subject. I’m not a philosopher, but philosophy is the basis of reasonable politics so buckle up for a healthy Friday afternoon serving of it.

To understand intellectual property, we must first establish at least a shallow understanding of property generally. What is property?¹ An incomplete answer might state that a material object I have power over is my property. An apple I have held in my hand is mine, insofar as nothing prevents me from using it (and, in the process, destroying it), or giving it away, or planting it in the ground. However, you might not agree that this apple is necessarily mine if I took it from a fruit stand without permission. This act is called “theft” — one of many possible transgressions upon property.

It is important to note that the very possibility that one could illicitly assume possession of an object is a strong indication that “property” is a social convention, rather than a law of nature; one cannot defy the law of gravity in the same way as one can defy property. And, given that, we could try to imagine other social conventions to govern the use of things in a society. If we come up with an idea we like, and we’re in a radical mood, we could even challenge the notion of property in society at large and seek to implement a different social convention.

As it stands today, the social convention tells us property is a thing which has an “owner”, or owners, to whom society confers certain rights with respect to the thing in question. That may include, for example, the right to use it, to destroy it, to exclude others from using it, to sell it, or give it away, and so on. Property is this special idea society uses to grant you the authority to use a bunch of verbs with respect to a thing. However, being a social convention, nothing prevents me from using any of these verbs on something society does not recognize as my property, e.g. by selling you this bridge. This is why the social convention must be enforced.

And how is it enforced? We could enforce property rights with shame: stealing can put a stain on one’s reputation, and this shame may pose an impediment to one’s social needs and desires, and as such theft is discouraged. We can also use guilt: if you steal something, but don’t get caught, you could end up remorseful without anyone to shame you for it, particularly with respect to the harm done to the person who suffered a loss of property as a result. Ultimately, in modern society the social convention of property is enforced with, well, force. If you steal something, society has appointed someone with a gun to track you down, restrain you, and eventually lock you up in a miserable room with bars on the windows.

I’d like to take a moment here to acknowledge the hubris of property: we see the bounty of the natural world and impose upon it these imagined rights and privileges, divvy it up and hand it out and hoard it, and resort to cruelty if anyone steps out of line. Indeed this may be justifiable if the system of private property is sufficiently beneficial to society, and the notion of property is so deeply ingrained into our system that it feels normal and unremarkable. It’s worth remembering that it has trade-offs, that we made the whole thing up, and that we can make up something else with different trade-offs. That being said, I’m personally fond of most of my personal property and I’d like to keep enjoying most of my property rights as such, so take from that what you will.²

One way we can justify property rights is by using them as a tool for managing scarcity. If demand for coffee exceeds the supply of coffee beans, a scarcity exists, meaning that not everyone who wants to have coffee gets to have some. But, we still want to enjoy scarce things. Perhaps someone who foregoes coffee will enjoy some other scarce resource, such as tea — then everyone can benefit in some part from some access to scarce resources. I suppose that the social convention of property can derive some natural legitimacy from the fact that some resources are scarce.³ In this sense, private property relates to the problem of distribution.

But a naive solution to distribution has flaws. For example, what of hoarding? Are property rights legitimate when someone takes more than they need or intend to use? This behavior could be motivated by an antagonistic relationship with society at large, such as as a means of driving up prices for private profit; such behavior could be considered anti-social and thus a violation of the social convention as such.

Moreover, property which is destroyed by its use, such as coffee, is one matter, but further questions are raised when we consider durable goods, such as a screwdriver. The screwdriver in my shed spends the vast majority of its time out of use. Is it just for me to assert property rights over my screwdriver when I am not using it? To what extent is the scarcity of screwdrivers necessary? Screwdrivers are not fundamentally scarce, given that the supply of idle screwdrivers far outpaces the demand for screwdriver use, but our modern conception of property has the unintended consequence of creating scarcity where there is none by denying the use of idle screwdrivers where they are needed.

Let’s try to generalize our understanding of property, working our way towards “intellectual property” one step at a time. To begin with, what happens if we expand our understanding of property to include immaterial things? Consider domain names as a kind of property. In theory, domain names are abundant, but some names are more desirable than others. We assert property rights over them, in particular the right to use a name and exclude others from using it, or to derive a profit from exclusive use of a desirable name.

But a domain name doesn’t really exist per-se: it’s just an entry in a ledger. The electric charge on the hard drives in your nearest DNS server’s database exist, but the domain name it represents doesn’t exist in quite the same sense as the electrons do: it’s immaterial. Is applying our conception of property to these immaterial things justifiable?

We can start answering this question by acknowledging that property rights are useful for domain names, in that this gives domain names desirable properties that serve productive ends in society. For example, exclusive control over a domain name allows a sense of authenticity to emerge from its use, so that you understand that pointing your browser to drewdevault.com will return the content that the person, Drew DeVault, wrote for you. We should also acknowledge that there are negative side-effects of asserting property rights over domains, such as domain squatting, extortionate pricing for “premium” domain names, and the advantage one party has over another if they possess a desirable name by mere fact of that possession, irrespective of merit.

On the balance of things, if we concede the legitimacy of personal property⁴ I find it relatively easy to concede the legitimacy of this sort of property, too.

The next step is to consider if we can generalize property rights to govern immaterial, non-finite things, like a story. A book, its paper and bindings and ink, is a material, finite resource, and can be thought of in terms that apply to material property. But what of the words formed by the ink? They can be trivially copied with a pen and paper, or transformed into a new medium by reading it aloud to an audience, and these processes do not infringe on the material property rights associated with the book. This process cannot be thought of as stealing, as the person who possesses a copy of the book is not asserting property rights over the original. In our current intellectual property regime, this person is transgressing via use of the idea, the intellectual property — the thing in the abstract space occupied by the story itself. Is that, too, a just extension of our notion of property?

Imagine with me the relationship one has with one’s property, independent of the social constructs around property. With respect to material property, a relationship of possession exists: I physically possess a thing, and I have the ability to make use of it through my possession of it. If someone else were to deny me access to this thing, they would have to resort to force, and I would have to resort to force should I resist their efforts.

Our relationship with intellectual property is much different. An idea cannot be withheld or seized by force. Instead, our relationship to intellectual property is defined by our history with respect to an idea. In the case of material property, the ground truth is that I keep it locked in my home to deny others access to it, and the social construct formalizes this relationship. With respect to intellectual property, such as the story in a book, the ground truth is that, sometime in the past, I imagined it and wrote it down. The social construct of intellectual property invents an imagined relationship of possession, modelled after our relationship with material property.

Why?

The resource with the greatest and most fundamental scarcity is our time,⁵ and as a consequence the labor which goes into making something is of profound importance. Marx famously argued for a “labor theory of value”, which tells us that the value inherent in a good or service is in the labor which is required to provide it. I think he was on to something!⁶ Intellectual property is not scarce, nor can it be possessed, but it does have value, and that value could ultimately be derived from the labor which produced it.

The social justification for intellectual property as a legal concept is rooted in the value of this labor. We recognize that intellectual labor is valuable, and produces an artifact — e.g. a story — which is valuable, but is not scarce. A capitalist society fundamentally depends on scarcity to function, and so through intellectual property norms we create an artificial scarcity to reward (and incentivize) intellectual labor without questioning our fundamental assumptions about capitalism and value.⁷ But, I digress — let’s revisit the subject in part two.

In part two of this series on intellectual property, I will explain the modern intellectual property regime as I understand it, as well as its history and justification. So equipped with the philosophical and legal background, part three will constitute the bulk of my critique of intellectual property, and my ideals for reform. Part four will examine how these ideas altogether apply in practice to open source, as well as the hairy questions of intellectual property as applied to modern problems in this space, such as the use of LLMs to file the serial numbers off of open source software.

If you want to dive deeper into the philosophy here, a great resource is the Stanford Encyclopedia of Philosophy. Check out their articles on Property and Ownership and Redistribution for a start, which expand on some of the ideas I’ve drawn on here and possess a wealth of citations catalogued with a discipline I can never seem to muster for my blog posts. I am a programmer, not a philosopher, so if you want to learn more about this you should go read from the hundreds of years of philosophers who have worked on this with rigor and written down a bunch of interesting ideas.

Join us to discuss transparency and governance at FOSDEM '25

Thu, 23 Jan 2025 00:00:00 +0000

Good news: it appears that Jack Dorsey’s FOSDEM talk has been cancelled!

This is a follow up to two earlier posts, which you can read here: one and two.

I say it “appears” so, because there has been no official statement from anyone to that effect. There has also been no communication from staff to the protest organizers, including to our email reaching out as requested to discuss fire safety and crowd control concerns with the staff. The situation is a bit unclear, but… we’ll extend FOSDEM the benefit of the doubt, and with it our gratitude. From all of the volunteers who have been organizing this protest action, we extend our heartfelt thanks to the staff for reconsidering the decision to platform Dorsey and Block, Inc. at FOSDEM. All of us – long-time FOSDEM volunteers, speakers, devroom organizers, and attendees – are relieved to know that FOSDEM stands for our community’s interests.

More importantly: what comes next?

The frustration the community felt at learning that Block was sponsoring FOSDEM and one of the keynote slots¹ had been given to Dorsey and his colleagues uncovered some deeper frustrations with the way FOSDEM is run these days. This year is FOSDEM’s 25th anniversary, and it seems sorely overdue for graduating from the “trust us, it’s crazy behind the scenes” governance model to something more aligned with the spirit of open source.

We trust the FOSDEM organizers — we can extend them the benefit of the doubt when they tell us that talk selection is independent of sponsorships. But it strains our presumption of good faith when the talk proposal was rejected by 3 of the 4 independent reviewers and went through anyway. And it’s kind of weird that we have to take them at their word — that the talk selection process isn’t documented anywhere publicly, nor the conflict of interest policy, nor the sponsorship terms, nor almost anything at all about how FOSDEM operates or is governed internally. Who makes decisions? How? We don’t know, and that’s kind of weird for something so important in the open source space.

Esther Payne, a speaker at FOSDEM 2020, summed up these concerns:

Why do we have so little information on the FOSDEM site about the budget and just how incorporated is FOSDEM as an organisation? How do the laws of Belgium affect the legalities of the organisation? How is the bank account administrated? How much money goes into the costs of this year, and how much of the budget goes into startup costs for the next year?

Peter Zaitsev, a long-time devroom organizer and FOSDEM speaker for many years, asked similar questions last year. I’ve spoken to the volunteers who signed up for the protest – we’re relieved that Dorsey’s talk has been cancelled, but we’re still left with big questions about transparency and governance at FOSDEM.

So, what’s next?

Let’s do something useful with that now-empty time slot in Janson. Anyone who planned to attend the protest is encouraged to come anyway on Sunday at 12:00 PM, where we’re going to talk amongst ourselves and anyone else who shows up about what we want from FOSDEM in the future, and what a transparent and participatory model of governance would look like. We would be thrilled if anyone on the FOSDEM staff wants to join the conversation as well, assuming their busy schedule permits. We’ll prepare a summary of our discussion and our findings to submit to the staff and the FOSDEM community for consideration after the event.

Until then – I’ll see you there!

NOTICE: The discussion session has been cancelled. After meeting with many of the protest volunteers and discussing the matter among the organizers, we have agreed that de-platforming Dorsey is mission success and improvising further action isn’t worth the trouble. We’ll be moving for reforms at FOSDEM after the event – I’ll keep you posted.

P.S. It’s a shame we won’t end up handing out our pamphlets. The volunteers working on that came up with this amazing flyer and I think it doesn’t deserve to go unseen:

We will be doing a modest print run for posterity — find one of us at FOSDEM if you want one.

FOSDEM '25 protest

Mon, 20 Jan 2025 00:00:00 +0000

Update: Dorsey’s talk was cancelled! See the update here.

Last week, I wrote to object to Jack Dorsey and his company, Block, Inc., being accepted as main track speakers at FOSDEM, and proposed a protest action in response. FOSDEM issued a statement about our plans on Thursday.

Today, I have some updates for you regarding the planned action.

I would like to emphasize that we are not protesting FOSDEM or its organizers. We are protesting Jack Dorsey and his company, first and foremost, from promoting their business at FOSDEM. We are members of the FOSDEM community. We have variously been speakers, devroom organizers, volunteers, and attendees for years — in other words, we are not activism tourists. We have a deep appreciation for the organizers and all of the work that they have done over the years to make FOSDEM such a success.

That we are taking action demonstrates that we value FOSDEM, that we believe it represents our community, and that we want to defend its — our — ethos. Insofar as we have a message to the FOSDEM organizers, it is one of gratitude, and an appeal to build a more open and participatory process, in the spirit of open source, and especially to improve the transparency of the talk selection process, sponsorship terms, and conflict of interest policies, so protests like ours are not necessary in the future. To be clear, we do not object to the need for sponsors generally at FOSDEM — we understand that FOSDEM is a free, volunteer driven event, many of us having volunteered for years — but we do object specifically to Jack Dorsey and Block, Inc. being selected as sponsors and especially as speakers.

As for the planned action, I have some more information for anyone who wishes to participate. Our purpose is to peacefully disrupt Dorsey’s talk, and only Dorsey’s talk, which is scheduled to take place between 12:00 and 12:30 on Sunday, February 2nd in Janson. If you intend to participate, we will be meeting outside of the upper entrance to Janson at 11:45 AM. We will be occupying the stage for the duration of the scheduled time slot in order to prevent the talk from proceeding as planned.

To maintain the peaceful nature of our protest and minimize the disruption to FOSDEM generally, we ask participants to strictly adhere to the following instructions:

Do not touch anyone else, or anyone else’s property, for any reason.
Do not engage in intimidation.
Remain quiet and peaceful throughout the demonstration.
When the protest ends, disperse peacefully and in a timely manner.
Leave the room the way you found it.

Dorsey’s time slot is scheduled to end at 12:30, but we may end up staying as late as 14:00 to hand the room over to the next scheduled talk.

I’ve been pleased by the response from volunteers (some of whom helped with this update — thanks!), but we still need a few more! I have set up a mailing list for planning the action. If you plan to join, and especially if you’re willing and able to help with additional tasks that need to be organized, please contact me directly to receive an invitation to the mailing list.

Finally, I have some corrections to issue regarding last week’s blog post.

In the days since I wrote my earlier blog post, Dorsey’s talk has been removed from the list of keynotes and moved to the main track, where it will occupy the same time slot in the same room but not necessarily be categorized as a “keynote”.

It has also been pointed out that Dorsey does not bear sole responsibility for Twitter’s sale. However, he is complicit and he profited handsomely from the sale and all of its harmful consequences. The sale left the platform at the disposal of the far right, causing a sharp rise in hate speech and harassment and the layoffs of 3,700 of the Twitter employees that made it worth so much in the first place.

His complicity, along with his present-day activities at Block, Inc. and the priorities of the company that he represents as CEO — its irresponsible climate policy, $120M in fines for enabling consumer fraud, and the layoffs of another 1,000 employees in 2024 despite posting record profits on $5B in revenue — are enough of a threat to our community and its ethos to raise alarm at his participation in FOSDEM. We find this compelling enough to take action to prevent him and his colleagues from using FOSDEM’s platform to present themselves as good actors in our community and sell us their new “AI agentic framework”.

The open source community and FOSDEM itself would not exist without collective action. Our protest to defend its principles is in that spirit. Together we can, and will, de-platform Jack Dorsey.

I’ll see you there!

No billionaires at FOSDEM

Thu, 16 Jan 2025 00:00:00 +0000

Update: Dorsey’s talk was cancelled! See the update here.

Jack Dorsey, former CEO of Twitter, ousted board member of BlueSky, and grifter extraordinaire to the tune of a $5.6B net worth, is giving a keynote at FOSDEM.

The FOSDEM keynote stage is one of the biggest platforms in the free software community. Janson is the biggest venue in the event – its huge auditorium can accommodate over 1,500 of FOSDEM’s 8,000 odd attendees, and it is live streamed to a worldwide audience as the face of one of the free and open source software community’s biggest events of the year. We’ve platformed Red Hat, the NLNet Foundation, NASA, numerous illustrious community leaders, and many smaller projects that embody our values and spirit at this location to talk about their work or important challenges our community faces.

Some of these challenges, as a matter of fact, are Jack Dorsey’s fault. In 2023 this stage hosted Hachyderm’s Kris Nóva to discuss an exodus of Twitter refugees to the fediverse. After Dorsey sold Twitter to Elon Musk, selling the platform out to the far right for a crisp billion-with-a-“B” dollar payout, the FOSS community shouldered the burden – both with our labor and our wallets – of a massive exodus onto our volunteer-operated servers, especially from victims fleeing the hate speech and harassment left in the wake of the sale. Two years later one of the principal architects of, and beneficiaries of, that disaster will step onto the same stage. Even if our community hadn’t been directly harmed by Dorsey’s actions, I don’t think that we owe this honor to someone who took a billion dollars to ruin their project, ostracize their users, and destroy the livelihoods of almost everyone who worked on it.

Dorsey is presumably being platformed in Janson because his blockchain bullshit company is a main sponsor of FOSDEM this year. Dorsey and his colleagues want to get us up to speed on what Block is working on these days. Allow me to give you a preview: in addition to posting $5B in revenue and a 21% increase in YoY profit in 2024, Jack Dorsey laid off 1,000 employees, ordering them not to publicly discuss board member Jay-Z’s contemporary sexual assault allegations on their way out, and announced a new bitcoin mining ASIC in collaboration with Core Scientific, who presumably installed them into their new 100MW Muskogee, OK bitcoin mining installation, proudly served by the Muskogee Generating Station fossil fuel power plant and its 11 million tons of annual CO₂ emissions and an estimated 62 excess deaths in the local area due to pollution associated with the power plant. Nice.

In my view, billionaires are not welcome at FOSDEM. If billionaires want to participate in FOSS, I’m going to ask them to refrain from using our platforms to talk about their AI/blockchain/bitcoin/climate-disaster-as-a-service grifty business ventures, and instead buy our respect by, say, donating 250 million dollars to NLNet or the Sovereign Tech Fund. That figure, as a percentage of Dorsey’s wealth, is proportional to the amount of money I donate to FOSS every year, by the way. That kind of money would keep the FOSS community running for decades.

I do not want to platform Jack Dorsey on this stage. To that end, I am organizing a sit-in, in which I and anyone who will join me are going to sit ourselves down on the Janson stage during his allocated time slot and peacefully prevent the talk from proceeding as scheduled. We will be meeting at 11:45 AM outside of Janson, 15 minutes prior to Dorsey’s scheduled time slot. Once the stage is free from the previous speaker, we will sit on the stage until 12:30 PM. Bring a good book. If you want to help organize this sit-in, or just let me know that you intend to participate, please contact me via email; I’ll set up a mailing list if there’s enough interest in organizing things like printing out pamphlets to this effect, or even preparing an alternative talk to “schedule” in his slot.

Follow-up: FOSDEM ’25 protest

Neurodivergence and accountability in free software

Wed, 25 Sep 2024 00:00:00 +0000

In November of last year, I wrote Richard Stallman’s political discourse on sex, which argues that Richard Stallman, the founder of and present-day voting member of the board of directors of the Free Software Foundation (FSF), endorses and advocates for a harmful political agenda which legitimizes adult attraction to minors, consistently defends adults accused of and convicted of sexual crimes with respect to minors, and more generally erodes norms of consent and manipulates language regarding sexual harassment and sexual assault in his broader political program.

In response to this article, and on many occasions when I have re-iterated my position on Stallman in other contexts, a common response is to assert that my calls to censure Stallman are ableist, on the basis that Stallman is neurodivergent (ND). This line of reasoning suggests that Stallman’s awkward and zealous views on sex are in line with his awkward and zealous positions on other matters (such as his insistence on “GNU/Linux” terminology rather than “Linux”), and that together this illustrates a pattern which suggests neurodivergence is at play. This argumentation is flawed, but I think it presents us with a good opportunity to talk about how neurodivergence and sexism presents in our community.

Neurodivergence (antonymous with “neurotypical”) is an umbrella term that encompasses a wide variety of human experiences, including autism, ADHD, personality disorders, bipolar disorder, and others. The particular claims I’ve heard about Stallman suggest that he is “obviously” autistic, or has Asperger syndrome.¹ The allegation of ableism in my criticisms of Stallman are rooted in this presumption of neurodivergence in Stallman: the argument goes that I am putting his awkwardness on display and mocking him for it, that calling for the expulsion of someone on the basis of being awkward is ableist, and that this has a chilling effect on our community, which is generally thought to have a high incidence of neurodivergence. I will respond to this defense of Stallman today.

A defense of problematic behavior that cites neurodivergence to not only explain, but excuse, said behavior, is ableist and harms neurodivergent people, rather than standing up for them as these arguments portray themselves as doing. To illustrate this, I opened a discussion on the Fediverse asking neurodivergent people to chime in and reached out directly to some ND friends in my social circle.

Aside: Is Stallman neurodivergent?

Stallman’s neurodivergence is an unsolicited armchair diagnosis with no supporting evidence besides “vibes”. This 2008 article summarizes his public statements on the subject:

“During a 2000 profile for the Toronto Star, Stallman described himself to an interviewer as ‘borderline autistic,’ a description that goes a long way toward explaining a lifelong tendency toward social and emotional isolation and the equally lifelong effort to overcome it,” Williams wrote.
When I cited that excerpt from the book during the interview, Stallman said that assessment was “exaggerated.”
“I wonder about it, but that’s as far as it goes,” he said. “Now, it’s clear I do not have [Asperger’s] — I don’t have most of the characteristics of that. For instance, one of those characteristics is having trouble with rhythm. I love the most complicated, fascinating rhythms.” But Stallman did acknowledge that he has “a few of the characteristics” and that he “might have what some people call a ‘shadow’ version of it.”

The theory that Stallman is neurodivergent is usually cited to explain his various off-putting behaviors, but there is no tangible evidence to support the theory. This alone raises some alarms, in that off-putting behavior is sufficient evidence to presume neurodivergence. I agree that some of his behavior, off-putting or otherwise, appears consistent, to my untrained eye, with some of the symptoms of autism. Nevertheless I am not going to forward an armchair diagnosis in either direction. However, because a defense of Stallman on the basis of neurodivergence is contingent on him being neurodivergent, this rest of this article will presume that it is true for the purpose of rebuttal.

tl;dr: we don’t know and the assumption that he is is ableist.

This defense of Stallman is ableist because it infantalizes and denies agency to neurodivergent people. Consider what’s being said here: it only follows that Stallman’s repugnant behavior is excusable because he’s neurodivergent if neurodivergent people cannot help but be repugnant. An autistic person I spoke to, who wishes to remain anonymous, had the following to say:

As an autistic person, I find these statements deeply offensive, because they build on and perpetuate damaging stereotypes.
Research has repeatedly proved that, on average, autistic folks have high empathy and a higher sense of values than the general population. We are not the emotionless robots that the popular imagination believes we are.
But we are not a monolith, and some autistic folks are absolute assholes who should be called out (and held accountable) for the harm that they cause. Autism is context, not an excuse: it can explain why someone might struggle in some situations and need additional support, but it should never be an excuse to harm others. We can all learn and improve.
I have witnessed people pulling the autism card to avoid consequences for CoC violations, then calling out the organization for “not supporting true diversity” when they’re shown the door. This is manipulative and insulting to the other neurodivergent members of the community, and should never be tolerated.

Bram Dingelstad, a neurodivergent person who participated in the discussion, had this to say:

Problematic behaviour is what it is: problematic.
There are a lot of neurodivergent people out there that are able to carry themselves in a way that doesn’t make anyone unsafe or harm victims of sexual assault by dismissing or downplaying their lived experience. In my opinion, using neurodivergence as an excuse for this behaviour only worsens the perception of neurodiversity.
Richard Stallman should be held accountable for his speech and his actions.

Another commenter put it more concisely, if not as eloquently:

It’s fucking ableist to say neurodiversity disposes you towards problematic behaviors. It’s disgusting trying to hide behind it and really quite insulting.

I came away from these discussions with the following understanding: neurodivergence, in particular autism, causes people to struggle to understand unstated social norms and conventions, sometimes with embarrassing or harmful consequences, such as with respect to interpersonal relationships. The people I’ve spoken to call for empathy and understanding in the mistakes which can be made in light of this, but also call for accountability – to be shown what’s right (and, importantly, why it’s so), and then to be expected to behave accordingly, no different from anyone else.

Being neurodivergent doesn’t make someone sexist, but it can make it harder for them to hide sexist views. To associate Stallman’s sexism with his perceived neurodivergence is ableist, and to hold Stallman accountable for his behavior is not. One commenter puts it this way:

I’ve said quite a few times is that sexism is not a symptom of autism. Writing this sort of behaviour off as “caused by” neurodivergence is itself ableist, I’m not a huge fan of the narrative that I have “the neurodevelopmental disorder that makes you a bigot”.
I fundamentally disagree with the idea that the pervasive sexism in tech is because of the high incidence of neurodiversity. It’s because tech has broadly operated as a boys club for decades, and those norms persist.

Using neurodivergence as a cover for sexism and problematic behavior in our communities is a toxic, ableist, and, of course, sexist attitude that serves to provide problematic men with space to be problematic. Note also how intersections between neurodiversity and identity play out: white men tend to be excused on the basis of neurodivergence, whereas for women, transgender people, people of color, etc – the excuse does not apply. Consider the differences in how bipolar disorder is perceived in women – “she’s crazy” – versus how men with autism are accommodated – “he can’t help it”.

So, I reject the notion that it is ableist to criticize problematic behavior that can be explained by neurodivergence. But, even if it were, an anonymous autistic commenter has this to say:

If we accept the hypothesis that it is ableist to condemn behavior which can be explained by neurodivergence (and I don’t), my answer is: be ableist. I don’t like it, but it’s ridiculous to imagine any other option in the physical world, and it’s weird to treat the virtual world so differently.
Here’s an anecdote: when I was at school, a new person, Adam, joined the class. We didn’t want Adam to feel excluded, so we included him in our social events. Adam had narcissistic personality disorder, and likely in part because of this, he was also a serial harasser of women. So what did we do about it?
We stopped inviting Adam. I wish we didn’t have to stop inviting him, but our hands were tied. I’m not going to say it’s something only he could change, because maybe he truly couldn’t change that. Maybe it was ableist to exclude him. But the safety of my friends comes first. The hard part is distinguishing between this situation and a situation where someone is excluded when they are perceived as a threat just because they’re different.

Stallman’s rhetoric and behavior are harmful, and we need to address that harm. The refrain of “criticizing Stallman’s behavior is ableist and alienates neurodiverse individuals in our community” is itself ableist and isn’t doing any favors for our neurodiverse friends.

To conclude this article, I thought I’d take this opportunity to find out what our neurodiverse friends are actually struggling with and how we can better accommodate their needs in our community.

First of all, a recognition of individuals as being autonomous, independent people with agency and independent needs has to come first, with neurodiversity and with everything else. Listen to people when they explain their experiences and their needs as individuals, and don’t rely on romanticized and stereotypical understandings of particular neurodevelopmental conditions such as autism. These stereotypes are often deeply harmful: one person spoke of being accused of incompetence and lying about their neurodivergence in a ploy for sympathy. They experienced severe harassment, at the worst in the form of harassers engineering stressful situations and screenshoting their reactions to humiliate them and damage their reputation.

Standing up for your peers is important, in this as in all things. Not only against harassment, discrimination, and abuse on the basis of neurodivergence, but on any basis, from any person – which I was often reminded is especially important for neurodivergent people who are not cishet white men, as these challenges are amplified in light of these intersectional identities. Talk to people and understand their experiences, their needs, and their worldview. Be patient, but clear and open in your communication. The neurodivergent people I spoke to often found it difficult to learn social mores, moreso than most neurotypical experiences, but nevertheless the vast majority of them felt perfectly capable of it, and the expectation that they weren’t is demeaning and ableist.

I also heard some advice from the neurodivergent community that applies especially to free software community leaders. Clearly stated community norms and expectations, through codes of conducts and visible moderation, is often helpful for neurodivergent people. Many ND people struggle to intuit or “guess” social norms and prefer expectations to be stated unambiguously. Normalizing the use of tone indicators (e.g. “/s”), questions clarifying intent, and conflict de-escalation are also good tools to employ.

Another consideration of merit is accommodations for asynchronous participation in meaningful governance and decision-making processes. Some ND people find it difficult to participate in real-time discussions in chat rooms or in person, and mediums like emails and other long-form slow discussions are easier for them to engage with. Accommodations for sensory sensitivities at in-person events is another good strategy to include more ND folks in your event. Establishing quiet spaces to get away from the busier parts of the event, being considerate of lighting choices, flexible break times, and activities for smaller groups were all highlighted to me by ND people as making their experience more enjoyable.

These are the lessons I took away from speaking to dozens of neurodivergent people in researching this blog post. I encourage you to speak to, and listen to, people in your communities as well, particularly when dealing with an issue which cites their struggles or impacts them directly.

Rust for Linux revisited

Fri, 30 Aug 2024 00:00:00 +0000

Ugh. Drew’s blogging about Rust again.

– You

I promise to be nice.

Two years ago, seeing the Rust-for-Linux project starting to get the ball rolling, I wrote “Does Rust belong in the Linux kernel?”, penning a conclusion consistent with Betteridge’s law of headlines. Two years on we have a lot of experience to draw on to see how Rust-for-Linux is actually playing out, and I’d like to renew my thoughts with some hindsight – and more compassion. If you’re one of the Rust-for-Linux participants burned out or burning out on this project, I want to help. Burnout sucks – I’ve been there.

The people working on Rust-for-Linux are incredibly smart, talented, and passionate developers who have their eyes set on a goal and are tirelessly working towards it – and, as time has shown, with a great deal of patience. Though I’ve developed a mostly-well-earned reputation for being a fierce critic of Rust, I do believe it has its place and I have a lot of respect for the work these folks are doing. These developers are ambitious and motivated to make an impact, and Linux is undoubtedly the highest-impact software in the world, and in theory Linux is enthusiastically ready to accept motivated innovators into its fold to facilitate that impact.

At least in theory. In practice, the Linux community is the wild wild west, and sweeping changes are infamously difficult to achieve consensus on, and this is by far the broadest sweeping change ever proposed for the project. Every subsystem is a private fiefdom, subject to the whims of each one of Linux’s 1,700+ maintainers, almost all of whom have a dog in this race. It’s herding cats: introducing Rust effectively is one part coding work and ninety-nine parts political work – and it’s a lot of coding work. Every subsystem has its own unique culture and its own strongly held beliefs and values.

The consequences of these factors is that Rust-for-Linux has become a burnout machine. My heart goes out to the developers who have been burned in this project. It’s not fair. Free software is about putting in the work, it’s a classical do-ocracy… until it isn’t, and people get hurt. In spite of my critiques of the project, I recognize the talent and humanity of everyone involved, and wouldn’t have wished these outcomes on them. I also have sympathy for many of the established Linux developers who didn’t exactly want this on their plate… but that’s neither here nor there for the purpose of this post, and any of those developers and their fiefdoms who went out of their way to make life difficult for the Rust developers above and beyond what was needed to ensure technical excellence are accountable for these shitty outcomes.¹

So where do we go now?

Well, let me begin by re-iterating something from my last article on the subject: “I wish [Rust-for-Linux] the best of luck and hope to see them succeed”. Their path is theirs to choose, and though I might advise a moment to rest before diving headfirst into this political maelstrom once again, I support you in your endeavours if this is what you choose to do. Not my business. That said, allow me to humbly propose a different path for your consideration.

Here’s the pitch: a motivated group of talented Rust OS developers could build a Linux-compatible kernel, from scratch, very quickly, with no need to engage in LKML politics. You would be astonished by how quickly you can make meaningful gains in this kind of environment; I think if the amount of effort being put into Rust-for-Linux were applied to a new Linux-compatible OS we could have something production ready for some use-cases within a few years.

Novel OS design is hard: projects like Redox are working on this, but it will take a long time to bear fruit and research operating systems often have to go back to the drawing board and make major revisions over and over again before something useful and robust emerges. This is important work – and near to my heart – but it’s not for everyone. However, making an OS which is based on a proven design like Linux is much easier and can be done very quickly. I worked on my own novel OS design for a couple of years and it’s still stuck in design hell and badly in need of being rethought; on the other hand I wrote a passable Unix clone alone in less than 30 days.

Rust is a great fit for a large monolithic kernel design like Linux. Imagine having the opportunity to implement something like the dcache from scratch in Rust, without engaging with the politics – that’s something a small group of people, perhaps as few as one, could make substantial inroads on in a short period of time taking full advantage of what Rust has on offer. Working towards compatibility with an existing design can leverage a much larger talent pool than the very difficult problem of novel OS design, a lot of people can manage with a copy of the ISA manual and a missive to implement a single syscall in a Linux-compatible fashion over the weekend. A small and motivated group of contributors could take on the work of, say, building out io_uring compatibility and start finding wins fast – it’s a lot easier than designing io_uring from scratch. I might even jump in and build out a driver or two for fun myself, that sounds like a good opportunity for me to learn Rust properly with a fun project with a well-defined scope.

Attracting labor shouldn’t be too difficult with this project in mind, either. If there was the Rust OS project, with a well-defined scope and design (i.e. aiming for Linux ABI compatibility), I’m sure there’s a lot of people who’d jump in to stake a claim on some piece of the puzzle and put it together, and the folks working on Rust-for-Linux have the benefit of a great deal of experience with the Linux kernel to apply to oversight on the broader design approach. Having a clear, well-proven goal in mind can also help to attract the same people who want to make an impact in a way that a speculative research project might not. Freeing yourselves of the LKML political battles would probably be a big win for the ambitions of bringing Rust into kernel space. Such an effort would also be a great way to mentor a new generation of kernel hackers who are comfortable with Rust in kernel space and ready to deploy their skillset to the research projects that will build a next-generation OS like Redox. The labor pool of serious OS developers badly needs a project like this to make that happen.

So my suggestion for the Rust-for-Linux project is: you’re burned out and that’s awful, I feel for you. It might be fun and rewarding to spend your recovery busting out a small prototype Unix kernel and start fleshing out bits and pieces of the Linux ABI with your friends. I can tell you from my own experience doing something very much like this that it was a very rewarding burnout recovery project for me. And who knows where it could go?

Once again wishing you the best and hoping for your success, wherever the path ahead leads.

What about drivers?

To pre-empt a response I expect to this article: there’s the annoying question of driver support, of course. This was an annoying line of argumentation back when Linux had poor driver support as well, and it will be a nuisance for a hypothetical Linux-compatible Rust kernel as well. Well, the same frustrated arguments I trotted out then are still ready at hand: you choose your use-cases carefully. General-purpose comes later. Building an OS which supports virtual machines, or a datacenter deployment, or a specific mobile device whose vendor is volunteering labor for drivers, and so on, will come first. You choose the hardware that supports the software, not the other way around, or build the drivers you need.

That said, a decent spread of drivers should be pretty easy to implement with the talent base you have at your disposal, so I wouldn’t worry about it.

So you want to compete with or replace open source

Tue, 16 Jul 2024 00:00:00 +0000

We are living through an interesting moment in source-available software.¹ The open source movement has always had, and continues to have, a solid grounding in grassroots programmers building tools for themselves and forming communities around them. Some looming giants brought on large sums of money – Linux, Mozilla, Apache, and so on – and other giants made do without, like GNU, but for the most part if anyone thought about open source 15 years ago they were mostly thinking about grassroots communities who built software together for fun. With the rise of GitHub and in particular the explosion of web development as an open platform, commercial stakeholders in software caught on to the compelling economics of open source. The open source boom that followed caused open source software to have an enormous impact on everyone working in the software industry, and, in one way or another, on everyone living on planet Earth.

Over the past decade or so, a lot of businesses, particularly startups, saw these economics unfolding in front of them and wanted to get in on this boom. A lot of talented developers started working on open source software with an explicit aim towards capitalizing on it, founding businesses and securing capital investments to build their product – an open source product. A few years following the onset of these startups, the catch started to become apparent. While open source was proven to be incredibly profitable and profoundly useful for the software industry as a whole, the economics of making open source work for one business are much different.

It comes down to the fact that the free and open source software movements are built on collaboration, and all of our success is attributable to this foundation. The economics that drew commercial interest into the movement work specifically because of this collaboration – because the FOSS model allows businesses to share R&D costs and bring together talent across corporate borders into a great melting pot of innovation. And, yes, there is no small amount of exploitation going on as well; businesses are pleased to take advantage of the work of Jane Doe in Ohio’s FOSS project to make themselves money without sharing any of it back. Nevertheless, the revolutionary economics of FOSS are based on collaboration, and are incompatible with competition.

The simple truth of open source is that if you design your business model with an eye towards competition, in which you are the only entity who can exclusively monetize the software product, you must eschew the collaborative aspects of open source – and thus its greatest strength. Collaboration in open source works because the collaborators, all representatives of different institutions, are incentivized to work together for mutual profit. No one is incentivized to work for you, for free, for your own exclusive profit.

More than a few of these open source startups were understandably put out when this reality started to set in. It turns out the market capitalization of a business that has an open source product was often smaller than the investments they had brought in. Under these conditions it’s difficult to give the investors the one and only thing they demand – a return on investment. The unbounded growth demanded by the tech boom is even less likely to be attainable in open source. There are, to be entirely clear, many business models which are compatible with open source. But there are also many which are not. There are many open source projects which can support a thriving business or even a thriving sub-industry, but there are some ideas which, when placed in an open source framing, simply cannot be capitalized on as effectively, or often at all.

Open source ate a lot of lunches. There are some kinds of software which you just can’t make in a classic silicon valley startup fashion anymore. Say you want to write a database server – a sector which has suffered a number of rug-pulls from startups previously committed to open source. If you make it closed source, you can’t easily sell it like you could 10 or 20 years ago, ala MSSQL. This probably won’t work. If you make it open source, no one will pay you for it and you’ll end up moaning about how the major cloud providers are “stealing” your work. The best way to fund the development of something like that is with a coalition of commercial stakeholders co-sponsoring or co-maintaining the project in their respective self-interests, which is how projects like PostgreSQL, Mesa, or the Linux kernel attract substantial paid development resources. But it doesn’t really work as a startup anymore.

Faced with these facts, there have been some challenges to the free and open source model coming up in the past few years, some of which are getting organized and starting to make serious moves. Bruce Perens, one of the founding figures of the Open Source Initiative, is working on the “post-open” project; “Fair Source” is another up-and-coming-effort, and there have been and will be others besides.

What these efforts generally have in common is a desire to change the commercial dynamic of source-available software. In other words, the movers and shakers in these movements want to get paid more, or more charitably, want to start a movement in which programmers that work on source-available software as a broader class get paid more. The other trait they have in common is a view that the open source definition and the four freedoms of free software do not sufficiently provide for this goal.

For my part, I don’t think that this will work. I think that the aim of sole or limited rights to monetization and the desire to foster a collaborative environment are irreconcilable. These movements want to have both, and I simply don’t think that’s possible.

This logic is rooted in a deeper notion of ownership over the software, which is both subtle and very important. This is a kind of auteur theory of software. The notion is that the software they build belongs to them. They possess a sense of ownership over the software, which comes with a set of moral and perhaps legal rights to the software, which, importantly, are withheld from any entity other than themselves. The “developers” enjoy this special relationship with the project – the “developers” being the special class of person entitled to this sense of ownership and the class to whom the up-and-coming source-available movements make an appeal, in the sense of “pay the developers” – and third-party entities who work on the source code are merely “contributors”, though they apply the same skills and labor to the project as the “developers” do. The very distinction between “first-party” and “third-party” developers is contingent on this “auteur” worldview.

This is quite different from how most open source projects have found their wins. If Linux can be said to belong to anyone, it belongs to everyone. It is for this reason that it is in everyone’s interests to collaborate on the project. If it belonged to someone or some entity alone, especially if that sense of ownership is rooted in justifying that entity’s sole right to effectively capitalize on the software, the dynamic breaks down and the incentive for the “third-party” class to participate is gone. It doesn’t work.

That said, clearly the proponents of these new source-available movements feel otherwise. And, to be clear, I wish them well. I respect the right for authors of software to distribute it under whatever terms they wish.² And, for my part, I do believe that source-available is a clear improvement over proprietary software, even though these models fall short of what I perceive as the advantages of open source. However, for these movements to have a shot at success, they need to deeply understand these dynamics and the philosophical and practical underpinnings of the free and open source movements.

However, it is very important to me that we do not muddy the landscape of open source by trying to reform, redefine, or expand our understanding of open source to include movements which contradict this philosophy. My well-wishes are contingent on any movements which aim to compete with open source stopping short of calling themselves open source. This is something I appreciate about the fair source and post-open movements – both movements explicitly disavow the label of open source. If you want to build something new, be clear that it is something new – this is the ground rule.

So you want to compete with open source, or even replace it with something new. Again, I wish you good luck. But this question will be at the heart of your challenge: will you be able to assume the mantle of the auteur and capitalize on this software while still retaining the advantages that made open source successful? Will you be able to appeal to the public in the same way open source does while holding onto these commercial advantages for yourself? Finding a way to answer this question with a “yes” is the task laid before you. It will be difficult; in the end, you will have to give something to the public to get something in return. Simply saying that the software itself is a gift equal to the labor you ask of the public is probably not going to work, especially when this “gift” comes with monetary strings attached.

As for me, I still believe in open source, and even in the commercial potential of open source. It requires creativity and a clever business acumen to identify and exploit market opportunities within this collaborative framework. To win in open source you must embrace this collaboration and embrace the fact that you will share the commercial market for the software with other entities. If you’re up to that challenge, then let’s keep beating the open source drum together. If not, these new movements may be a home for you – but know that a lot of hard work still lies ahead of you in that path.

Writing a Unix clone in about a month

Fri, 24 May 2024 00:00:00 +0000

I needed a bit of a break from “real work” recently, so I started a new programming project that was low-stakes and purely recreational. On April 21st, I set out to see how much of a Unix-like operating system for x86_64 targets that I could put together in about a month. The result is Bunnix. Not including days I didn’t work on Bunnix for one reason or another, I spent 27 days on this project.

You can try it for yourself if you like:

Bunnix 0.0.0 iso

To boot this ISO with qemu:

qemu-system-x86_64 -cdrom bunnix.iso -display sdl -serial stdio

You can also write the iso to a USB stick and boot it on real hardware. It will probably work on most AMD64 machines – I have tested it on a ThinkPad X220 and a Starlabs Starbook Mk IV. Legacy boot and EFI are both supported. There are some limitations to keep in mind, in particular that there is no USB support, so a PS/2 keyboard (or PS/2 emulation via the BIOS) is required. Most laptops rig up the keyboard via PS/2, and YMMV with USB keyboards via PS/2 emulation.

Tip: the DOOM keybindings are weird. WASD to move, right shift to shoot, and space to open doors. Exiting the game doesn’t work so just reboot when you’re done playing. I confess I didn’t spend much time on that port.

What’s there?

The Bunnix kernel is (mostly) written in Hare, plus some C components, namely lwext4 for ext4 filesystem support and libvterm for the kernel video terminal.

The kernel supports the following drivers:

PCI (legacy)
AHCI block devices
GPT and MBR partition tables
PS/2 keyboards
Platform serial ports
CMOS clocks
Framebuffers (configured by the bootloaders)
ext4 and memfs filesystems

There are numerous supported kernel features as well:

A virtual filesystem
A /dev populated with block devices, null, zero, and full psuedo-devices, /dev/kbd and /dev/fb0, serial and video TTYs, and the /dev/tty controlling terminal.
Reasonably complete terminal emulator and somewhat passable termios support
Some 40 syscalls, including for example clock_gettime, poll, openat et al, fork, exec, pipe, dup, dup2, ioctl, etc

Bunnix is a single-user system and does not currently attempt to enforce Unix file modes and ownership, though it could be made multi-user relatively easily with a few more days of work.

Included are two bootloaders, one for legacy boot which is multiboot-compatible and written in Hare, and another for EFI which is written in C. Both of them load the kernel as an ELF file plus an initramfs, if required. The EFI bootloader includes zlib to decompress the initramfs; multiboot-compatible bootloaders handle this decompression for us.

The userspace is largely assembled from third-party sources. The following third-party software is included:

Colossal Cave Adventure (advent)
dash (/bin/sh)
Doom
gzip
less (pager)
lok (/bin/awk)
lolcat
mandoc (man pages)
sbase (core utils)¹
tcc (C compiler)
Vim 5.7

The libc is derived from musl libc and contains numerous modifications to suit Bunnix’s needs. The curses library is based on netbsd-curses.

The system works but it’s pretty buggy and some parts of it are quite slapdash: your milage will vary. Be prepared for it to crash!

How Bunnix came together

I started documenting the process on Mastodon on day 3 – check out the Mastodon thread for the full story. Here’s what it looked like on day 3:

Screenshot of an early Bunnix build, which boots up, sets up available memory, and exercises an early in-memory filesystem

Here’s some thoughts after the fact.

Some of Bunnix’s code stems from an earlier project, Helios. This includes portions of the kernel which are responsible for some relatively generic CPU setup (GDT, IDT, etc), and some drivers like AHCI were adapted for the Bunnix system. I admit that it would probably not have been possible to build Bunnix so quickly without prior experience through Helios.

Two of the more challenging aspects were ext4 support and the virtual terminal, for which I brought in two external dependencies, lwext4 and libvterm. Both proved to be challenging integrations. I had to rewrite my filesystem layer a few times, and it’s still buggy today, but getting a proper Unix filesystem design (including openat and good handling of inodes) requires digging into lwext4 internals a bit more than I’d have liked. I also learned a lot about mixing source languages into a Hare project, since the kernel links together Hare, assembly, and C sources – it works remarkably well but there are some pain points I noticed, particularly with respect to building the ABI integration riggings. It’d be nice to automate conversion of C headers into Hare forward declaration modules. Some of this work already exists in hare-c, but has a ways to go. If I were to start again, I would probably be more careful in my design of the filesystem layer.

Getting the terminal right was difficult as well. I wasn’t sure that I was going to add one at all, but I eventually decided that I wanted to port vim and that was that. libvterm is a great terminal state machine library, but it’s poorly documented and required a lot of fine-tuning to integrate just right. I also ended up spending a lot of time on performance to make sure that the terminal worked smoothly.

Another difficult part to get right was the scheduler. Helios has a simpler scheduler than Bunnix, and while I initially based the Bunnix scheduler on Helios I had to throw out and rewrite quite a lot of it. Both Helios and Bunnix are single-CPU systems, but unlike Helios, Bunnix allows context switching within the kernel – in fact, even preemptive task switching enters and exits via the kernel. This necessitates multiple kernel stacks and a different approach to task switching. However, the advantages are numerous, one of which being that implementing blocking operations like disk reads and pipe(2) are much simpler with wait queues. With a robust enough scheduler, the rest of the kernel and its drivers come together pretty easily.

Another source of frustration was signals, of course. Helios does not attempt to be a Unix and gets away without these, but to build a Unix, I needed to implement signals, big messy hack though they may be. The signal implementation which ended up in Bunnix is pretty bare-bones: I mostly made sure that SIGCHLD worked correctly so that I could port dash.

Porting third-party software was relatively easy thanks to basing my libc on musl libc. I imported large swaths of musl into my own libc and adapted it to run on Bunnix, which gave me a pretty comprehensive and reliable C library pretty fast. With this in place, porting third-party software was a breeze, and most of the software that’s included was built with minimal patching.

What I learned

Bunnix was an interesting project to work on. My other project, Helios, is a microkernel design that’s Not Unix, while Bunnix is a monolithic kernel that is much, much closer to Unix.

One thing I was surprised to learn a lot about is filesystems. Helios, as a microkernel, spreads the filesystem implementation across many drivers running in many separate processes. This works well enough, but one thing I discovered is that it’s quite important to have caching in the filesystem layer, even if only to track living objects. When I revisit Helios, I will have a lot of work to do refactoring (or even rewriting) the filesystem code to this end.

The approach to drivers is also, naturally, much simpler in a monolithic kernel design, though I’m not entirely pleased with all of the stuff I heaped into ring 0. There might be room for an improved Helios scheduler design that incorporates some of the desirable control flow elements from the monolithic design into a microkernel system.

I also finally learned how signals work from top to bottom, and boy is it ugly. I’ve always felt that this was one of the weakest points in the design of Unix and this project did nothing to disabuse me of that notion.

I had also tried to avoid using a bitmap allocator in Helios, and generally memory management in Helios is a bit fussy altogether – one of the biggest pain points with the system right now. However, Bunnix uses a simple bitmap allocator for all conventional pages on the system and I found that it works really, really well and does not have nearly as much overhead as I had feared it would. I will almost certainly take those lessons back to Helios.

Finally, I’m quite sure that putting together Bunnix in just 30 days is a feat which would not have been possible with a microkernel design. At the end of the day, monolithic kernels are just much simpler to implement. The advantages of a microkernel design are compelling, however – perhaps a better answer lies in a hybrid kernel.

What’s next

Bunnix was (note the past tense) a project that I wrote for the purpose of recreational programming, so it’s purpose is to be fun to work on. And I’ve had my fun! At this point I don’t feel the need to invest more time and energy into it, though it would definitely benefit from some. In the future I may spend a few days on it here and there, and I would be happy to integrate improvements from the community – send patches to my public inbox. But for the most part it is an art project which is now more-or-less complete.

My next steps in OS development will be a return to Helios with a lot of lessons learned and some major redesigns in the pipeline. But I still think that Bunnix is a fun and interesting OS in its own right, in no small part due to its demonstration of Hare as a great language for kernel hacking. Some of the priorities for improvements include:

A directory cache for the filesystem and better caching generally
Ironing out ext4 bugs
procfs and top
mmaping files
More signals (e.g. SIGSEGV)
Multi-user support
NVMe block devices
IDE block devices
ATAPI and ISO 9660 support
Intel HD audio support
Network stack
Hare toolchain in the base system
Self hosting

Whether or not it’s me or one of you readers who will work on these first remains to be seen.

In any case, have fun playing with Bunnix!

Copyleft licenses are not “restrictive”

Fri, 19 Apr 2024 00:00:00 +0000

One may observe an axis, or a “spectrum”, along which free and open source software licenses can be organized, where one end is “permissive” and the other end is “copyleft”. It is important to acknowledge, however, that though copyleft can be found at the opposite end of an axis with respect to permissive, it is not synonymous with the linguistic antonym of permissive – that is, copyleft licenses are not “restrictive” by comparison with permissive licenses.

Aside: Free software is not synonymous with copyleft and open source is not synonymous with permissive, though this is a common misconception. Permissive licenses are generally free software and copyleft licenses are generally open source; the distinction between permissive and copyleft is orthogonal to the distinction between free software and open source.

It is a common misunderstanding to construe copyleft licenses as more “restrictive” or “less free” than permissive licenses. This view is predicated on a shallow understanding of freedom, a sort of passive freedom that presents as the absence of obligations. Copyleft is predicated on a deeper understanding of freedom in which freedom is a positive guarantee of rights.^[source]

Let’s consider the matter of freedom, obligation, rights, and restrictions in depth.

Both forms of licenses include obligations, which are not the same thing as restrictions. An example of an obligation can be found in the permissive MIT license:

Permission is hereby granted […] to deal in the Software without restriction […] subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

This obliges the user, when distributing copies of the software, to include the copyright notice. However, it does not restrict the use of the software under any conditions. An example of a restriction comes from the infamous JSON license, which adds the following clause to a stock MIT license:

The Software shall be used for Good, not Evil.

IBM famously petitioned Douglas Crockford for, and received, a license to do evil with JSON.¹ This kind of clause is broadly referred to in the free software jargon as “discrimination against field of endeavour”, and such restrictions contravene both the free software and open source definitions. To quote the Open Source Definition, clause 6:

The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.

No such restrictions are found in free or open source software licenses, be they permissive or copyleft – all FOSS licenses permit the use of the software for any purpose without restriction. You can sell both permissive and copyleft software, use it as part of a commercial cloud service,² use the software as part of a nuclear weapons program,³ or do whatever else you want with it. There are no restrictions on how free software is used, regardless of if it is permissive or copyleft.

Copyleft does not impose restrictions, but it does impose obligations. The obligations exist to guarantee rights to the users of the software – in other words, to ensure freedoms. In this respect copyleft licenses are more free than permissive licenses.

Freedom is a political concept, and in order to understand this, we must consider it in political terms, which is to say as an exercise in power dynamics. Freedom without obligation is a contradiction. Freedom emerges from obligations, specifically obligations imposed on power.

Where does freedom come from?

Consider the United States as an example, a society which sets forth freedom as a core political value.⁴ Freedoms in the US are ultimately grounded in the US constitution and its bill of rights. These tools create freedoms by guaranteeing rights to US citizens through the imposition of obligations on the government. For instance, you have a right to an attorney when accused of a crime in the United States, and as such the government is obliged to provide you with one. It is from obligations such as these that freedom emerges. Freedom of assembly, another example, is guaranteed such that the police are prevented from breaking up peaceful protests – this freedom emerges from a constraint (or restriction, if you must) on power (the government) as a means of guaranteeing the rights and freedom of those with less power by comparison (its citizens).

Who holds the power in the context of software?

Consider non-free software by contrast: software is written by corporations and sold on to users with substantial restrictions on its use. Corporations hold more power than individuals: they have more resources (e.g. money), more influence, and, in a sense more fundamental to the software itself, they retain in private the tools to understand the software, or to modify its behavior, and they dictate the conditions under which it may be used (e.g. only if your license key has not expired, or only for certain purposes). This is true of anyone who retains the source code in private and uses copyright law to enforce their will upon the software – in this way they possess, and exercise, power over the user.

Permissive licenses do not provide any checks on this power; generally they preserve moral rights and little else. Permissive licenses provide for relatively few and narrow freedoms, and are not particularly “free” as such. Copyleft licenses constrain these powers through additional obligations, and from these obligations greater freedoms emerge. Specifically, they oblige reciprocity. They are distinguished from permissive licenses in this manner, but where permissive licenses permit, copyleft does not restrict per-se – better terms might be “reciprocal” and “non-reciprocal”, but perhaps that ship has sailed. “You may use this software if …” is a statement made both by permissive and copyleft licenses, with different ifs. Neither form of license says “you cannot use this software if …”; licenses which do so are non-free.

Permissive licenses and copyleft licenses are both free software, but only the latter provides a guarantee of rights, and while both might be free only the latter provides freedom.

FDO's conduct enforcement actions regarding Vaxry

Tue, 09 Apr 2024 00:00:00 +0000

freedesktop(.org), aka FDO, recently banned Hyprland maintainer Vaxry from the FDO community, and in response Vaxry has taken his case to the court of public opinion, publishing their email exchanges and writing about it on his blog.

It saddens me to bear witness to these events today. I wrote in September of last year about problems with toxicity in the Hyprland community. I initially reached out to Vaxry to discuss these problems in private in February of last year. I failed to get through to him, leading to that blog post in September. I spent some time in the following weeks talking with Vaxry on his behavior and his community’s social norms, again in private, but again, I was unable to get through to him. Unfortunately, we find ourselves again leaving the private sphere and discussing Vaxry’s behavior and the problem posed by the Hyprland community once again.

The fact of the matter is that Hyprland remains a toxic community, enabled and encouraged by its toxic leadership, namely Vaxry. FDO’s decision to ban Vaxry is ultimately a consequence of Vaxry’s behavior, and because he has elected to appeal his case in public, I am compelled to address his behavior in public. I hereby rise firmly in defense of FDO’s decision.

I invite you to start by reading the two email threads, one, and two, which Vaxry has published for your consideration, as well as Vaxry’s follow-ups on his blog, one, and two.

Here’s my read on the situation.

The FDO officer that reached out to Vaxry did it after Vaxry’s problematic behavior was brought to her attention by members of the FDO community, and was acting on her mandate within the FDO conduct enforcement board by investigating complaints submitted to her by this community. It is not a stretch to suggest a close relationship between these communities exists: FDO is the steward of both the Wayland protocol and implementation and the wlroots library, essential dependencies of Hyprland and sources for collaboration between Hyprland and FDO. Vaxry and other members of the Hyprland community had already participated extensively in these projects (mainly in discussions on IRC and GitLab issues) at the time of the email exchange, in spaces where the code of conduct applies.

The FDO officer duly investigated the complaints she had received and found, in collaboration with the other members of the FDO conduct enforcement team, that they were credible, and worrying. There are numerous examples of behavior from Vaxry that contravenes the FDO code of conduct in several different respects, and any number of them would be grounds for an immediate ban. Since these behaviors are concerning, but did not take place in the FDO community, the conduct board decided to issue a warning in private, stating that if this sort of behavior was seen in the FDO community that it would result in enforcement action from the conduct team.

All of the actions from the FDO conduct team are reasonable and show considerable restraint. Vaxry could have taken it in stride with no consequences to himself. Instead, he immediately escalated the situation. He construes the FDO officer’s polite and well-reasoned warning as threats and intimidation. He minimizes examples of his own hate speech by shrugging them off as a joke. He belittles the FDO officer and builds a straw man wherein her email is an official statement on behalf of RedHat, and cites a conspiracy theory about DEI programs at RedHat as justification for calling the FDO officer a hypocrite. He is insulted on my behalf that my name was cited in the FDO officer’s email in lowercase, “drew”, and feels the need to address this.

The FDO officer responds to Vaxry’s unhinged rant with a sarcastic quip clarifying that it was indeed within the FDO conduct team’s remit to ban Vaxry from their GitLab instance – I confess that in my view this was somewhat unprofessional, though I can easily sympathize with the FDO officer given the context. Following this, Vaxry states that Hyprland will cease all communication with FDO’s conduct team and ignore (emphasis his) any future emails from them. Finally, he threatens legal action (on what basis is unclear) and signs the email.

Regardless of how you feel about the conduct team issuing a private warning to Vaxry on the basis of activities outside of FDO community spaces, the email thread that ensues most certainly is within the scope of the FDO code of conduct, and Vaxry’s behavior therein is sufficient justification for a ban from the FDO community as far as I’m concerned. The conduct team cites Vaxry’s stated intention to ignore any future conduct interventions as the ultimate reason for the ban, which I find entirely reasonable on FDO’s part. I have banned people for far less than this, and I stand by it.

Vaxry’s follow-up blog posts only serve to underscore this point. First of all, he immediately opens with a dog-whistle calling for the reader to harass the FDO officer in question: “I don’t condone harassing this person, but here is their full name, employer and contact details”:

I do not condone any hateful messages sent towards any of the parties mentioned.
Recently I have received an email filled with threats to my inbox, from a member of the X.org board, Freedesktop.org, and a Red Hat employee. Their name is [redacted].

Moreover, Vaxry claims to have apologised for his past conduct, which is not true. In lieu of an apology, Vaxry has spent the “1.5 years” since the last incident posting angry rants on his blog calling out minority representation and “social justice warriors” in light of his perceived persecution. Meanwhile the Hyprland community remains a toxic place, welcoming hate, bullying, and harassment, but now prohibiting all “political” speech, which in practice means any discussion of LGBTQ topics, though this is largely unenforced. In the end, the Hyprland community’s fundamental problem is that they’re all “just having fun”, and it seems that they can’t have “fun” unless it’s at someone else’s expense.

The FDO team is right that Hyprland’s community reflects poorly on the Linux desktop community as a whole. Vaxry has created a foothold for hate, transphobia, homophobia, bullying, and harassment in the Linux desktop community. We are right to take action to correct this problem.

Every option other than banning Vaxry has been exhausted over the past year and a half. I personally spent several weeks following my last blog post on the matter discussing Vaxry’s behavior in confidence and helping him understand how to improve, and at my suggestion he joined a private community of positive male role models to discuss these issues in a private and empathetic space. After a few weeks of these private discussions, the last thing he said to me was “I do believe there could be arguments to sway my opinion towards genocide”.¹

There’s nothing left to do but to build a fence around Hyprland and protect the rest of the community from them. I know that there’s a lot of good people who use and contribute to Hyprland, and I’m sorry for those of you who are affected by this problem. But, in the end, actions have consequences. The rest of the community has no choice but to sanction Vaxry.

And, to Vaxry – I know you’re reading this – there are going to continue to be consequences for your actions, but it’s still not too late to change. I know it’s humiliating to be called out like this, and I really would rather not have had to do so. FDO is probably not the last time you’re going to be banned if you don’t change course, and it would reflect better on you if you took it on the chin and didn’t post inflammatory rants on your blog – trust me, you don’t look like the good guy here. You are trapped in an echo chamber of hate, anger, and bigotry. I hope that you find a way out, and that someday you can build a community which is as great as your software is.

And, to the FDO officer in question: I’m so sorry that you’re at the ass end of all of this hate and abuse. You don’t deserve any of it. You did a good job, and I’m proud of you and the rest of the FDO conduct team. If you need any support, someone to talk to, don’t hesitate to reach out and ask, on IRC, Matrix, email, whatever. Don’t read the comments.

And on that note, I condemn in the harshest terms the response from communities like /r/linux on the subject. The vile harassment and hate directed at the FDO officer in question is obscene and completely unjustifiable. I don’t care what window manager or desktop environment you use – this kind of behavior is completely uncalled for. I expect better.

P.S. The Hyprland community has already descended on me before even publishing this post, after I called Vaxry out on Mastodon a few hours ago. My notifications are not full of reasonable objections to my complaints, but instead the response is slurs and death threats. This only serves to prove my characterization of the Hyprland community as deeply toxic.

Why Prusa is floundering, and how you can avoid their fate

Tue, 26 Dec 2023 00:00:00 +0000

Prusa is a 3D printer manufacturer which has a long history of being admired by the 3D printing community for high quality, open source printers. They have been struggling as of late, and came under criticism for making the firmware of their Mk4 printer non-free.¹

Armin Ronacher uses Prusa as a case-study in why open source companies fail, and uses this example to underline his argument that open source needs to adapt for commercial needs, namely by adding commercial exclusivity clauses to its licenses – Armin is one of the principal proponents of the non-free Functional Source License. Armin cites his experience with a Chinese manufactured 3D printer as evidence that intellectual property is at the heart of Prusa’s decline, and goes on to discuss how this dynamic applies to his own work in developing a non-free license for use with Sentry. I find this work pretty interesting – FSL is a novel entry into the non-free license compendium, and it’s certainly a better way to do software than proprietary models, assuming that it’s not characterized as free or open source. But, allow me to use the same case study to draw different conclusions.

It is clear on the face of it that Prusa’s move to a non-free firmware is unrelated to their struggles with the Chinese competition – their firmware was GPL’d, and the cited competitor (Bambu) evidently respects copyleft, and there’s no evidence that Bambu’s printers incorporate derivatives of Prusa’s firmware in a manner which violates the GPL. Making the license non-free is immaterial to the market dynamics between Prusa and Bambu, so the real explanation must lie elsewhere.

If you had asked me 10 years ago what I expected Prusa’s largest risk would be, I would have simply answered “China” and you would have probably said the same. The Chinese economy and industrial base can outcompete Western manufacturing in almost every manufacturing market.² This was always the obvious vulnerability in their business model, and they absolutely needed to be prepared for this situation, or their death was all but certain. Prusa made one of the classic errors in open source business models: they made their product, made it open source, sold it, and assumed that they were done working on their business model.

It was inevitable that someday Chinese manufacturers would undercut Prusa on manufacturing costs. Prusa responded to this certainty by not diversifying their business model whatsoever. There has only ever been one Prusa product: their latest 3D printer model. The Mk4 costs $1,200. You can buy the previous generation (at $1,000), or the MINI (from 2019, $500). You can open your wallet and get their high-end printers, which are neat but fail to address the one thing that most users at this price-point really want, which is more build volume. Or, you can buy an Ender 3 off Amazon right now for $180 and you’ll get better than half of the value of an Mk4 at an 85% discount. You could also buy Creality’s flagship model for a cool $800 and get a product which beats the Mk4 in every respect. China has joined the market, bringing with them all of the competitive advantages their industrial base can bring to bear, and Prusa’s naive strategy is causing their position to fall like a rock.

Someone new to 3D printing will pick up an Ender and will probably be happy with it for 1-2 years. When they upgrade, will they upgrade to a Prusa or an Ender 5? Three to five years a customer spends in someone else’s customer pipeline is an incredibly expensive opportunity cost Prusa is missing out on. This opportunity cost is the kind of arithmetic that would make loss leaders like a cheap, low-end, low-or-negative-margin Prusa printer make financial sense. Hell, Prusa should have made a separate product line of white-labeled Chinese entry-level 3D printers just to get people on the Prusa brand.

Prusa left many stones unturned. Bambu’s cloud slicer is a massive lost opportunity for Prusa. On-demand cloud printing services are another lost opportunity. Prusa could have built a marketplace for models & parts and skimmed a margin off of the top, but they waited until 2022 to launch Printables – waiting until the 11th hour when everyone was fed up with Thingiverse. Imagine a Prusa where it works out of the box, you can fire up a slicer in your browser which auto-connects to your printer and prints models from a Prusa-operated model repository, paying $10 for a premium model, $1 off the top goes to Prusa, with the same saved payment details which ensure that a fresh spool of Prusa filament arrives at your front door when it auto-detects that your printer is almost out. The print you want is too big for your build volume? Click here to have it cloud printed – do you want priority shipping for that? Your hot-end is reaching the end of its life – as one of our valued business customers on our premium support contract we would be happy to send you a temporary replacement printer while yours is shipped in for service.

Prusa’s early foothold in the market was strong, and they were wise to execute the way they did early on. But they absolutely had to diversify their lines of business. Prusa left gaping holes in the market and utterly failed to capitalize on any of them. Prusa could have been synonymous with 3D printing if they had invested in the brand (though they probably needed a better name). I should be able to walk into a Best Buy and pick up an entry-level Prusa for $250-$500, or into a Home Depot and pick up a workshop model for $1000-$2000. I should be able to bring it home, unbox it, scan a QR code to register it with PrusaConnect, and have a Benchy printing in less than 10 minutes.

Chinese manufacturers did all of this and more, and they’re winning. They aren’t just cheaper – they offer an outright better product. These are not cheap knock-offs: if you want the best 3D printer today it’s going to be a Chinese one, regardless of how much you want to spend, but, as it happens, you’re going to spend less.

Note that none of this is material to the license of the product, be it free or non-free. It’s about building a brand, developing a customer relationship, and identifying and exploiting market opportunities. Hackers and enthusiasts who found companies like Prusa tend to imagine that the product is everything, but it’s not. Maybe 10% of the work is developing the 3D printer itself – don’t abandon the other 90% of your business. Especially when you make that 10% open: someone else is going to repurpose it, do the other 90%, and eat your lunch. FOSS is great precisely because it makes that 10% into community property and shares the cost of innovation, but you’d be a fool to act as if that was all there was to it. You need to deal with sales and marketing, chase down promising leads, identify and respond to risks, look for and exploit new market opportunities, and much more to be successful.

This is a classic failure mode of open source businesses, and it’s Prusa’s fault. They had an excellent foothold early in the market, leveraging open source and open hardware to great results and working hand-in-hand with enthusiasts early on to develop the essential technology of 3D printing. Then, they figured they were done developing their business model, and completely dropped the ball as a result. Open source is not an “if you build it, the money will come” situation, and to think otherwise is a grave mistake. Businesses need to identify their risks and then mitigate them, and if they don’t do that due diligence, then it’s their fault when it fails – it’s not a problem with FOSS.

Free and open source software is an incredibly powerful tool, including as a commercial opportunity. FOSS really has changed the world! But building a business is still hard, and in addition to its fantastic advantages, the FOSS model poses important and challenging constraints that you need to understand and work with. You have to be creative, and you must do a risk/reward assessment to understand how it applies to your business and how you can utilize it for commercial success. Do the legwork and you can utilize FOSS for a competitive advantage, but skip this step and you will probably fail within a decade.

Richard Stallman's political discourse on sex

Sat, 25 Nov 2023 00:00:00 +0000

Richard Stallman, the founder of the Free Software Foundation, has been subject to numerous allegations of misconduct. He stepped down in 2019, and following his re-instatement in 2021, a famous open letter was published in which numerous organizations and individuals from throughout the Free Software ecosystem called for his removal from the Free Software Foundation. The letter had no effect; Stallman remains a voting member of the FSF’s board of directors to this day and continues to receive numerous speaking engagements.

Content warning: This article discusses sexual abuse, sexual assault, sexual harassment, and all of the above with respect to minors, as well as the systemic normalization of abuse, and directly quotes statements which participate in the normalization of abuse.

This article presents an analysis of Stallman’s political discourse on sex with the aim of establishing the patterns that cause the sort of discomfort that led to Stallman’s public condemnation. In particular, we will address how Stallman speaks about sexual assault, harassment, consent, and minors in his discourse.

I think that it is important to acknowledge this behavior not as a series of isolated incidents, nor a conflict with Stallman’s “personal style”, but a pattern of behavior from which a political narrative forms, and draws attention to the fact that the meager retractions, excuses, and non-apologies from both Stallman and the Free Software Foundation as a whole fail to account for that pattern in a meaningful way.

The failure of the Free Software community to account for Richard Stallman’s behavior has a chilling effect. The norms set by our leadership influence the norms of our broader community, and many members of the Free Software community look to Stallman as a ideological and political leader. The norms Stallman endorses are harmful and deeply confronting and alienating to many people, in particular women and children. Should these norms be adopted by our movement, we risk creating a community which enables the exploitation of vulnerable people.

Let’s begin to address this by considering Stallman’s retraction of his comments in support of pedophilia. The following comment from Stallman in 2013 drew harsh criticism:

There is little evidence to justify the widespread assumption that willing participation in pedophilia hurts children.

— stallman.org, 04 January 2013 “Pedophilia”

Following much of the criticism directed at Stallman, he had a number of “personal conversations” which reframed his views. Of the many comments Stallman has made which drew ire, this is one of the few for which a correction was made, in September 2019:

Many years ago I posted that I could not see anything wrong about sex between an adult and a child, if the child accepted it.
Through personal conversations in recent years, I’ve learned to understand how sex with a child can harm per psychologically. This changed my mind about the matter: I think adults should not do that. I am grateful for the conversations that enabled me to understand why.

— stallman.org, 14 September 2019 “Sex between an adult and a child is wrong”

This statement from Stallman has been accepted by his defenders as evidence of his capitulation on pedophilia. I argue that this statement is misleading due to the particular way Stallman uses the word “child”. When Stallman uses this word, he does so with a very specific meaning, which he explains on his website:

Children: Humans up to age 12 or 13 are children. After that, they become adolescents or teenagers. Let’s resist the practice of infantilizing teenagers, by not calling them “children”.

— stallman.org, “Anti-glossary”

It seems clear from this definition is that Stallman’s comments are not a capitulation at all. His 2019 retraction, when interpreted using his definition of “children”, does not contradict most of Stallman’s past statements regarding sex and minors, including his widely criticized defenses of many people accused of sexual impropriety with minors.

Stallman’s most recent direct response to his criticism underscores this:

It was right for me to talk about the injustice to Minsky, but it was tone-deaf that I didn’t acknowledge as context the injustice that Epstein did to women or the pain that caused.

— fsf.org, April 12, 2021, “RMS addresses the free software community”

Stallman qualifies his apology by explicitly re-affirming his defense of Marvin Minsky, which is addressed in detail later in this piece. Stallman’s doubling-down here is consistent with the supposition that Stallman maintains the view that minors can have sexual relationships with adults of any age, provided that they aren’t “children” – in other words, provided they’re at least 13 or 14 years old.

Stallman cares deeply about language and its usage. His strange and deliberate usage of the word “children” is also found many times throughout his political notes over the years. For example:

It sounds horrible: “UN peacekeepers accused of child rape in South Sudan.” But the article makes it pretty clear that the “children” involved were not children. They were teenagers.

— stallman.org, 30 April 2018 “UN peacekeepers in South Sudan”

Here Stallman again explicitly distinguishes “teenagers” from children, drawing this distinction especially in the context of sexual relationships between adults and minors. Stallman repeats this pattern many times over the years – we see it again in Stallman’s widely criticized defense of Cody Wilson:

Cody Wilson has been charged with hiring a “child” sex worker. Her age has not been announced, but I think she must surely be a teenager, not a child. Calling teenagers “children” in this context is a way of smearing people with normal sexual proclivities as “perverts”.

— stallman.org, 23 September 2018 “Cody Wilson”

And once more when defending Roy Moore:

Senate candidate Roy Moore tried to start dating/sexual relationships with teenagers some decades ago.
He tried to lead Ms Corfman step by step into sex, but he always respected “no” from her and his other dates. Thus, Moore does not deserve the exaggerated condemnation that he is receiving for this. As an example of exaggeration: one mailing referred to these teenagers as “children”, even the one that was 18 years old. Many teenagers are minors, but none of them are children.
The condemnation is surely sparked by the political motive of wanting to defeat Moore in the coming election, but it draws fuel from ageism and the fashion for overprotectiveness of “children”.

— stallman.org, 27 November 2017 “Roy Moore’s relationships”

Ms. Corfman was 14 at the time Roy Moore is accused of initiating sexual contact with her; Moore was 32 at the time. Here we see an example of him re-iterating his definition of “children”, a distinction he draws especially to suggest that an adult having sex with a minor is socially acceptable.

Note that Stallman refers to Ms. Corfman as Moore’s “date”. Stallman’s use of this word is important: here he normalizes the possibility that a minor and an adult could engage in a healthy dating relationship. In this statement, Stallman cites an article which explains circumstances which do not resemble such a normalized dating experience: Moore isolated Corfman from her mother, drove her directly to his home, and initiated sexual contact there.

Note also that the use of the phrase “step by step” in this quotation is more commonly referred to as “grooming” in the discourse on child sexual exploitation.

Stallman reaches for similar reasoning in other political notes, such as the following:

A British woman is on trial for going to a park and inviting teenage boys to have sex with her there. Her husband acted as a lookout in case someone else passed by. One teenager allegedly visited her at her house repeatedly to have sex with her.
None of these acts would be wrong in any sense, provided they took precautions against spreading infections. The idea that adolescents (of whatever sex) need to be “protected” from sexual experience they wish to have is prudish ignorantism, and making that experience a crime is perverse.

— stallman.org, 26 May 2017, “Prudish ignorantism”

The woman in question, aged 60, had sex with her husband, age 69, in a public space, and invited spectators as young as 11 to participate.

Stallman has also sought to normalize adult attraction to minors, literally describing it as “normal” in September 2018:

Calling teenagers “children” encourages treating teenagers as children, a harmful practice which retards their development into capable adults.
In this case, the effect of that mislabeling is to smear Wilson. It is rare, and considered perverse, for adults to be physically attracted to children. However, it is normal for adults to be physically attracted to adolescents. Since the claims about Wilson is the latter, it is wrong to present it as the former.

— stallman.org, 23 September 2018, “Cody Wilson”

One month prior, Stallman made a statement which similarly normalized adult attraction to minors, and suggests that acting on this attraction should be acceptable to society, likening opposition to this view to homosexual conversion therapy:

This accords with the view that Stendhal reported in France in the 1800s, that a woman’s most beautiful years were from 16 to 20.
Although this attitude on men’s part is normal, the author still wants to present it as wrong or perverted, and implicitly demands men somehow control their attraction to direct it elsewhere. Which is as absurd, and as potentially oppressive, as claiming that homosexuals should control their attraction and direct it towards to the other sex. Will men be pressured to undergo “age conversion therapy” intended to brainwash them to feel attracted mainly to women of their own age?

— stallman.org, 21 August 2018, “Age and attraction”

A trend is thus clearly seen in Stallman’s regular political notes, over several years, wherein Stallman re-iterates his position that “adolescents” or “teenagers” are distinct from “children” for the purpose of having sex with adults, and normalizes and defends adult attraction to minors and adults who perform sexual acts with minors. We see this distinction of the two groups, children and adolescents, outlined again on his “anti-glossary”, which still published on his website today, albeit without the connotations of sex. His regular insistence on a definition of children which excludes adolescents serves such that his redaction of his controversial 2013 comment serves to redact none of the other widely-condemned comments he has made since.

Stallman has often written political notes when people accused of sexual impropriety, particularly with minors, appear in the news, or appear among Stallman’s social circle. Stallman’s comments generally downplay the abuse and manipulate language in a manner which benefits perpetrators of abuse. We see this downplaying in another example from 2019:

Should we accept stretching the terms “sexual abuse” and “molestation” to include looking without touching?
I do not accept it.

— stallman.org, 11 June 2019 “Stretching meaning of terms”

Stallman is writing here in response to a news article outlining accusations of sexual misconduct directed at Ohio State athletics doctor Richard Strauss. Strauss was accused of groping at least 177 students between 1979 and 1997 during routine physical exams, accusations corroborated by at least 50 members of the athletic department staff.

In addition to Stallman’s regular fixation of the use of the word “children” with respect to sex, this political note also draws our attention to the next linguistic fixation of Stallman I want to question: the use of phrases like “sexual abuse” and “sexual assault”. The term “sexual assault” also appears in Stallman’s “Anti-glossary”:

Sexual assault: The term is applied to a broad range of actions, from rape on one end, to the least physical contact on the other, as well as everything in between. It acts as propaganda for treating them all the same. That would be wrong.
The term is further stretched to include sexual harassment, which does not refer to a single act, but rather to a series of acts that amounts to a form of gender bias. Gender bias is rightly prohibited in certain situations for the sake of equal opportunity, but that is a different issue.
I don’t think that rape should be treated the same as a momentary touch. People we accuse have a right to those distinctions, so I am careful not to use the term “sexual assault” to categorize the actions of any person on any specific occasion.

— stallman.org, “Anti-glossary”

Stallman often fixates on the term “sexual assault” throughout his political notes. He feels that the term fails to distinguish between “grave” and “minor” crimes, as he illustrated in 2021:

“Sexual assault” is so vague that it makes no sense as a charge. Because of that term, we can’t whether these journalists were accused of a grave crime or a minor one. However, the charge of espionage shows this is political persecution.

— stallman.org, 21 July 2021, “Imprisonment of journalists”

I would like to find out what kind of crimes Stallman feels the need to distinguish along this axis. His other political notes give us some hints, such as this one regarding Al Franken’s sexual misconduct scandal:

If it is true that he persistently pressured her to kiss him, on stage and off, if he stuck his tongue into her mouth despite her objections, that could well be sexual harassment. He should have accepted no for an answer the first time she said it. However, calling a kiss “sexual assault” is an exaggeration, an attempt to equate it to much graver acts, that are crimes.
The term “sexual assault” encourages that injustice, and I believe it has been popularized specifically with that intention. That is why I reject that term.

— stallman.org, 30 July 2019, “Al Franken”

Stallman also wrote in 2020 to question the use of the phrase again:

In the US, when thugs¹ rape people they say are suspects, it is rare to bring them to justice.
I object to describing any one crime as “sexual assault” because that is vague about the severity of the crime. This article often uses that term to refer to many crimes that differ in severity but raise the same issue. That may be a valid practice.

— stallman.org, 12 August 2020, “When thugs rape people they say are suspects”

In the article Stallman cites in this political note, various unwelcome sexual acts by the police are described, the least severe of which is probably molestation.

More alarmingly, Stallman addresses his views on the term “sexual assault” in this 2017 note, affording for the possibility that a 35-year-old man could have had consensual sex with an 11-year-old girl.

Jelani Maraj (who I had never heard of) could be imprisoned for a long time for “sexual assault”. What does that concretely mean?
Due to the vagueness of the term “sexual assault” together with the dishonest law that labels sex with adolescents as “rape” even if they are willing, we cannot tell from this article what sort of acts Maraj was found to have committed. So we can’t begin to judge whether those acts were wrong.
I see at least three possibilities. Perhaps those acts really constituted rape — it is a possibility. Or perhaps the two had sex willingly, but her parents freaked out and demanded prosecution. Or, intermediate between those two, perhaps he pressured her into having sex, or got her drunk.

— stallman.org, 13 November 2017, “Jelani Maraj”

Another article by Stallman does not explicitly refer to sexual assault, but does engage in a bizarre defense of a journalist who was fired for masturbating during a video conference. In this article Stallman fixates on questions such as whether or not the genitals being in view of the webcam was intentional or not, and suggests that masturbating on a video call would be acceptable should the genitals remain unseen.

The New Yorker’s unpublished note to staff was vague about its grounds for firing Toobin. Indeed, it did not even acknowledge that he had been fired. This is unfair, like convicting someone on unstated charges. Something didn’t meet its “standards of conduct”, but it won’t tell us what — we can only guess. What are the possibilities? Intentionally engaging in video-call sex as a side activity during a work meeting? If he had not made a mistake in keeping that out of view of the coworkers, why would it make a difference what the side activity was?

— stallman.org, November 2020, “On the Firing of Jeffrey Toobin”

Finally, Stallman elaborated on his thoughts on the term most recently in October 2023. This note gives the clearest view of Stallman’s preferred distinction between various sexual crimes:

I warned that the stretchable term “sexual assault”, which extends from grave crimes such as rape through significant crimes such as groping and down to no clear lower bound, could be stretched to criminalize minor things, perhaps even stealing a kiss. Now this has happened.
What next? Will a pat on the arm or a hug be criminalized? There is no clear limit to how far this can go, when a group builds up enough outrage to push it.

— stallman.org, 15 October 2023, “Sexual assault for stealing a kiss”

From Stallman’s statements, we can refine his objection to the term “sexual assault”, and sexual behaviors generally, to further suggest that the following beliefs are held by Stallman on the subject:

Groping and molestation are not sexual assault, but are crimes
Kissing someone without consent is not sexual assault, furthermore it is not wrong
Masturbating during a video conference is not wrong if you are not seen doing so
A 35-year-old man having sex with an 11-year-old girl does not constitute rape, nor sexual assault, but is in fact conscionable

The last of these may be covered under Stallman’s 2019 retraction, even accounting for Stallman’s unconventional use of the word “children”.

Stallman’s fixation on the term “sexual assault” can be understood in his political notes as having the political aims of eroding the meaning of the phrase, questioning the boundaries of consent, downplaying the importance of agency in intimate interactions, appealing for the defense of people accused of sexual assault, and arguing for sexual relationships between minors and adults to be normalized. In one notable case, he has used this political angle to rise to the defense of his friends – in Stallman’s infamous email regarding Marvin Minsky, he writes the following:

The injustice [done to Minsky] is in the word “assaulting”. The term “sexual assault” is so vague and slippery that it facilitates accusation inflation: taking claims that someone did X and leading people to think of it as Y, which is much worse than X.
(…)
The word “assaulting” presumes that he applied force or violence, in some unspecified way, but the article itself says no such thing. Only that they had sex.
We can imagine many scenarios, but the most plausible scenario is that she presented herself to him as entirely willing. Assuming she was being coerced by Epstein, he would have had every reason to tell her to conceal that from most of his associates.
I’ve concluded from various examples of accusation inflation that it is absolutely wrong to use the term “sexual assault” in an accusation.

— Excerpt from Selam G’s recount of Stallman’s email to MIT Computer Science and Artificial Intelligence Laboratory mailing list, September 2019. Selam’s quotation has been corroborated by other sources. Minsky is, in this context, accused of having had a sexual encounter with a minor facilitated by convicted child trafficker Ghislaine Maxwell. The original accusation does not state that this sexual encounter actually occurred; only that the minor in question was instructed to have sex with Minsky. Minsky would have been at least 75 years old at the time of the alleged incident; the minor was 16.

There is an important, but more subtle pattern in Stallman’s statements that I want to draw your attention to here: Stallman appears to have little to no understanding of the role of power dynamics in sexual harassment, assault, and rape. Stallman appears to reject the supposition that these acts could occur without an element of outwardly apparent violent coercion.

This is most obviously evidenced by his statements regarding the sexual abuse of minors; most people understand that minors cannot consent to sex even if they “appear willing”, in particular because an adult in this situation is exploiting a difference in experience and maturity to manipulate the child into sexually satisfying them – in other words, a power differential. Stallman seems to reject this understanding of consent in his various defenses of people accused of sexual impropriety with minors, and in cases where the pretense of consent cannot be easily established, he offers the perpetrator the benefit of the doubt.

We can also find an example of Stallman disregarding power dynamics with respect to adults in the following political note from 2017:

A famous theater director had a habit of pestering women, asking them for sex.
As far as I can tell from this article, he didn’t try to force women into sex.
When women persistently said no, he does not seem to have tried to punish them.
The most he did was ask.
He was a pest, but nothing worse than that.

— stallman.org, 29 October 2017, “Pestering women”

In this case we have an example of “quid pro quo”, a kind of sexual harassment which weaponizes power dynamics for sexual gratification. This kind of sexual harassment is explicitly cited as illegal by Title VII of the US Civil Rights Act. A lack of competence in this respect displayed by Stallman, whose position in the Free Software Foundation board of directors requires that he act in a manner consistent with this law, is alarming.

I have identified this blindness to power dynamics as a recurring theme in Stallman’s comments on sexual abuse, be it with respect to sexual relationships between minors and adults, managers and subordinates, students and teachers, or public figures and their audience. I note for the reader that Stallman has held and currently holds several of these positions of power.

In addition to his position as a voting member of the Free Software Foundation’s Board of Directors, Stallman is still invited to speak at events and conferences. Stallman’s infamous rider prescribes a number of his requirements for attending an event; most of his conditions are relatively reasonable, though amusing. In this document, he states his preference for being accommodated in private, on a “spare couch”, when he travels. At these events, in these private homes, he may be afforded many opportunities to privacy with vulnerable people, including minors that, in his view, can consent to having sex with adults.

In summary, Stallman has a well-documented and oft-professed set of political beliefs which reject the social and legal norms regarding consent. He is not simply quietly misled in these beliefs; rather he advocates for these values using his political platform. He has issued no meaningful retractions of these positions or apologies for harm caused, and has continued to pursue a similar agenda since his return to the FSF board of directors.

This creates a toxic environment not only in the Free Software Foundation and in Stallman’s direct purview, but in the broader Free Software movement. The free software movement is culturally poisoned by our support of Stallman as our ideological leader. The open letter calling for Stallman’s removal received 3,000 signatures; the counter-letter in support of Stallman received 6,876 before it stopped accepting submissions.

Richard Stallman founded the Free Software Foundation in 1985, and has performed innumerable works to the benefit of our community since then. We’ve taken Stallman’s views on software freedom seriously, and they’ve led us to great achievements. It is to Stallman’s credit that the Free Software community is larger than one man. However, one’s political qualifications to speak about free software does not make one qualified to address matters of sex; in this respect Stallman’s persistence presents as dangerous incompetence.

When we consider his speech on sex as a discourse that has been crafted and rehearsed methodically over the years, he asks us to consider him seriously, and so we must. When we analyze the dangerous patterns in this discourse, we have to conclude that he is not fit for purpose in his leadership role, and we must acknowledge the shadow that our legitimization of his discourse casts on our community.

Can I be on your podcast?

Thu, 09 Nov 2023 00:00:00 +0000

I am working on rousing the Hare community to get the word out about our work. I have drafted the Hare evangelism guidelines to this effect, which summarizes how we want to see our community bringing Hare to more people.

We’d like to spread the word in a way which is respectful of the attention of others – we’re explicitly eschewing unsolicited prompts for projects to consider writing/rewriting in Hare, as well as any paid sponsorships or advertising. Blog posts about Hare, videos, participating in (organic) online discussions – much better! And one idea we have is to talk about Hare on podcasts which might be interested in the project.

If that describes your podcast, here’s my bold request: can I make an appearance?

Here are some mini “press kits” to give you a hook and some information that might be useful for preparing an interview.

The Hare programming language

Hare is a systems programming language designed to be simple, stable, and robust. Hare uses a static type system, manual memory management, and a minimal runtime. It is well-suited to writing operating systems, system tools, compilers, networking software, and other low-level, high performance tasks.

Hare has been in development since late 2019 and today has about 100 contributors.

Official website
Source code & development resources
“Introducing the Hare programming language”, video, 2022
GPLv3, MPL 2.0, MIT

A hand-drawn picture of a rabbit

Hare’s official mascot, Harriet. Drawn by Louis Taylor, CC-0

The Ares operating system

Ares is an operating system written in Hare which is under development. It features a micro-kernel oriented design and runs on x86_64 and aarch64. Its design is inspired by the seL4 micro-kernel and Plan 9.

A photo of a laptop running the Ares operating system

A picture of a ThinkPad running Ares and demonstrating some features

Himitsu: a secret storage system

Himitsu is a secure secret storage system for Unix-like systems. It provides an arbitrary key/value store (where values may be secret) and a query language for manipulating the key store.

Himitsu is written in Hare.

Interested?

If any of these topics are relevant for your podcast and you’d like to talk about them, please reach out to me via email: drew@ddevault.org

Thanks!

On "real name" policies

Tue, 31 Oct 2023 00:00:00 +0000

Some free software projects reject anonymous or pseudonymous contributions, requiring you to author patches using your “real name”. Such projects have a so-called “real name” policy; Linux is one well-known example.¹

The root motivations behind such policies vary, but in my experience the most often cited rationale is that it’s important to establish the provenance of the contribution for copyright reasons. In the case of Linux, contributors are asked to “sign-off” their commits to indicate their agreement to the terms of the Developer Certificate of Origin (DCO), which includes clauses like the following:

The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file.

To some extent, the DCO serves as a legal assertion of copyright and an agreement to license a work under given copyright terms (GPLv2 in the case of Linux). This record also means that the author of the code is accountable in case the copyright is challenged; in the case of an anonymous or pseudonymous contributor you’re shit out of luck. At that point, liability over the disagreement would likely fall into the hands of the maintainer that accepted the contribution. It is reasonable for a maintainer to ask a contributor to assert their copyright and accept liability over the provenance of their code in a legally meaningful and accountable form.

The possibility that someone may have something useful to offer to a free software project, but is not comfortable disclosing their name for any number of reasons, is a reasonable supposition. A maintainer whose “real name” policy is challenged on this basis would also be reasonable in saying “I feel for you, but I cannot agree to accept legal liability over the provenance of this code, nor can I communicate that risk to end-users who acquire code under a license that may or may not be valid as such”.

“Real name” policies are controversial in the free software community. I open with this perspective in an attempt to cool down the room. Those who feel marginalized by “real name” policies often skew young, and many treat matters such as copyright and licensing with disdain. Moreover, the problem tends to inflame deeply hurtful sentiments and raise thorny matters of identity and discrimination, and it’s easy to construe the intent of the policymakers as the intent to cause harm. The motivations behind these policies are reasonable.

That said, intent or otherwise, these policies can cause harm. The profile of the contributor who is comfortable using their “real name” is likely to fall more narrowly into over-represented demographics in our community; enforcing a real-name policy will ostracize some people. Those with marginalized identities tend to be less comfortable with disclosing their “real name”. Someone who has been subject to harassment may not be comfortable with this disclosure, since it offers more fuel to harassers keeping tabs on their activities. The use of a “real name” also confers a gender bias; avoiding a “real name” policy neatly eliminates discrimination on this basis. Of course, there are also many falsehoods programmers believe about names which can present in the implementation of such a policy.

There is also one particular problem which has been at the heart of conflict surrounding the use of “real-name” policies in free software: transgender identities. A transgender person is likely to change their name in the process of assuming their new identity. When this happens, their real name changes. However, it may or may not match their legal name – some trans people opt to change it, others don’t; if they do it is a process that takes time. Meanwhile, addressing a trans person by their old name, or “deadname”, is highly uncomfortable. Doing so deliberately, as a matter of policy or otherwise, is a form of discrimination. Many trans people experience deliberate “deadnaming” as a form of harassment in their daily lives, and institutionalizing this behavior is cruel.

The truth is, managing the names of participants is more challenging than anyone would like. On the one hand, names establish accountability and facilitate collaboration, and importantly, credit the authors of a work for services performed. On the other hand, names are highly personal and deeply affecting, and their usage and changes over time are the subject of important consideration at the discretion of their owner. A complicating factor is that handling names properly introduces technical problems which must be overcome.

To embrace the advantages of “real name” policies – establishing provenance, encouraging accountability, fostering a social environment – without causing harm, the approach I have settled on for my projects is to use the DCO to establish provenance and encourage contributors to sign-off and participate under the identity they feel most comfortable with. I encourage people to utilize an identity they use beyond the project’s walls, to foster a social environment and a connection to the broader community, to establish accountability, and to ensure that participants are reachable for further discussion on their work. If a contributor’s identity changes, we make every effort to support this change in contemporary, future, and historical use.

Going off-script

Fri, 13 Oct 2023 00:00:00 +0000

There is a phenomenon in society which I find quite bizarre. Upon our entry to this mortal coil, we are endowed with self-awareness, agency, and free will. Each of the 8 billion members of this human race represents a unique person, a unique worldview, and a unique agency. Yet, many of us have the same fundamental goals and strive to live the same life.

I think of such a life experiences as “following the script”. Society lays down for us a framework for living out our lives. Everyone deviates from the script to some extent, but most people hit the important beats. In Western society, these beats are something like, go to school, go to college, get a degree, build a career, get married, have 1.5 children, retire to Florida, die.

There are a number of reasons that someone may deviate from the script. The most common case is that the deviations are imposed by circumstance. A queer person will face discrimination, for instance, in marriage, or in adopting and raising children. Someone born into the lower class will have reduced access to higher education and their opportunities for career-building are curtailed accordingly; similar experiences follow for people from marginalized groups. Furthermore, more and more people who might otherwise be able to follow the script are finding that they can’t afford a home and don’t have the resources to build a family.

There are nevertheless many people who are afforded the opportunity to follow the script, and when they do so, they often experience something resembling a happy and fulfilling life. Generally this is not the result of a deliberate choice – no one was presented with the script and asked “is this what you want”? Each day simply follows the last and you make the choices that correspond with what you were told a good life looks like, and sometimes a good life follows.

Of course, it is entirely valid to want the “scripted” life. But you were not asked if you wanted it: it was just handed to you on a platter. The average person lacks the philosophical background which underpins their worldview and lifestyle, and consequently cannot explain why it’s “good”, for them or generally. Consider your career. You were told that it was a desirable thing to build for yourself, and you understand how to execute your duties as a member of the working class, but can you explain why those duties are important and why you should spend half of your waking life executing them? Of course, if you are good at following the script, you are rewarded for doing so, generally with money, but not necessarily with self-actualization.

This state of affairs leads to some complex conflicts. This approach to life favors the status quo and preserves existing power structures, which explains in part why it is re-enforced by education and broader social pressures. It also leads to a sense of learned helplessness, a sense that this is the only way things can be, which reduces the initiative to pursue social change – for example, by forming a union.

It can also be uncomfortable to encounter someone who does not follow the script, or even questions the script. You may be playing along, and mostly or entirely exposed to people who play along. Meeting someone who doesn’t – they skipped college, they don’t want kids, they practice polyamory, they identify as a gender other than what you presumed, etc – this creates a moment of dissonance and often resistance. This tends to re-enforce biases and can even present as inadvertent micro-aggressions.

I think it’s important to question the script, even if you decide that you like it. You should be able to explain why you like it. This process of questioning is a radical act. A radical, in its non-pejorative usage, is born when someone questions their life and worldview, decides that they want something else, and seeks out others who came to similar conclusions. They organize, they examine their discomfort and put it to words, and they share these words in the hope that they can explain a similar discomfort that others might feel within themselves. Radical movements, which by definition is any movement which challenges the status quo, are the stories of the birth and spread of radical ideas.

Ask yourself: who are you? Did you choose to be this person? Who do you want to be, and how will you become that person? Should you change your major? Drop out? Quit your job, start a business, found a labor union? Pick up a new hobby? Join or establish a social club? An activist group? Get a less demanding job, move into a smaller apartment, and spend more time writing or making art? However you choose to live, choose it deliberately.

The next step is an exercise in solidarity. How do you feel about others who made their own choices, choices which may be alike or different to your own? Or those whose choices were constrained by their circumstances? What can you do together that you couldn’t do alone?

Who do you want to be? Do you know?

The forbidden topics

Fri, 29 Sep 2023 00:00:00 +0000

There are forbidden topics in the hacker community. One is sternly reprimanded for bringing them up, by their peers, their leaders, and the community at large. In private, one can expect threats and intimidation; in public, outcry and censorship. The forbidden topics are enforced by the moderators of our spaces, taken off of forums, purged from chat rooms, and cleaned up from GitHub issues and mailing lists; the ban-hammers fall swiftly and resolutely. My last article to touch these subjects was removed from Hacker News by the moderators within 30 minutes and landed several death threats in my inbox. The forbidden topics, when raised, are met with a resounding, aggressive dismissal and unconditional condemnation.

Some years ago, the hacker community possessed near-unanimous praise for the ideals of free speech; the hacker position was generally that of what we would now understand as “radical” free speech, which is to say the kind of “shout ‘fire’ in a crowded movie theater” radical, but more specifically the kind that tolerates hate speech. The popular refrain went, “I disapprove of what you say, but I will defend to the death your right to say it”. Many hackers hold this as a virtue to this day. I once held this as a virtue for myself.

However, this was a kind of free speech which was unconsciously contingent on being used for speech with which the listener was comfortable. The hacker community at this time was largely homogeneous, and as such most of the speech we were exposed to was of the comfortable sort. As the world evolved around us, and more people found their voice, this homogeneity began to break down. Critics of radical free speech, victims of hate speech, and marginalized people of all kinds began to appear in hacker communities. The things they had to say were not comfortable.

The free speech absolutists among the old guard, faced with this discomfort, developed a tendency to defend hate speech and demean speech that challenged them. They were not the target of the hate, so it did not make them personally uncomfortable, and defending it would maintain the pretense of defending free speech, of stalwartly holding the line on a treasured part of their personal hacker ethic. Speech which challenged their preconceptions and challenged their power structures was not so easily acceptable. The pretense is dropped and they lash out in anger, calling for the speakers to be excluded from our communities.

Some of the once-forbidden topics are becoming less so. There are carefully chalked-out spaces where we can talk about them, provided they are not too challenging, such as LGBTQ identities or the struggles of women in our spaces. Such discussions are subject to careful management by our leaders and moderators, to the extent necessary to preserve power structures. Those who speak on these topics are permitted to do so relatively free of retaliation provided that they speak from a perspective of humility, a voice that “knows its place”. Any speech which suggests that the listener may find themselves subject to a non-majority-conforming person in a position of power, or even that of a peer, will have crossed the line; one must speak as a victim seeking the pity and grace of your superiors to be permitted space to air your grievances.

Similarly, space is made for opposition to progressive speech, again moderated only insofar as it is necessary to maintain power structures. Some kinds of overt hate speech may rouse a response from our leaders, but those who employ a more subtle approach are permitted their voice. Thus, both progressive speech and hate speech are permitted within a carefully regulated framework of power preservation.

Some topics, however, remain strictly forbidden.

Our community has persistent and pervasive problems of a particular sort which we are not allowed to talk about: sexual harassment and assault. Men who assault, harass, and even rape women in our spaces, are protected. A culture of silence is enforced, and those who call out rape, sexual assault, or harassment, those who criticise they who enable and protect these behaviors, are punished, swiftly and aggressively.

Men are terrified of these kinds of allegations. It seems like a life sentence: social ostracization, limited work opportunities, ruined relationships. We may have events in our past that weigh on our conscience; was she too drunk, did she clearly consent, did she regret it in the morning? Some of us have events in our past that we try not to think about, because if we think too hard, we might realize that we crossed the line. This fills men with guilt and uncertainty, but also fear. We know the consequences if our doubts became known.

So we lash out in this fear. We close ranks. We demand the most stringent standards of evidence to prove anything, evidence that we know is not likely to be there. We refuse to believe that our friends were not the men we thought they were, or to confront that we might not be ourselves. We demand due process under the law, we say they should have gone to the police, that they can’t make accusations of such gravity without hard proof. Think of the alleged perpetrator; we can’t ruin their lives over frivolous accusations.

For victims, the only recourse permitted by society is to suffer in silence. Should they speak, victims are subject to similar persecutions: they are ostracized, struggle to work, and lose their relationships. They have to manage the consequences of a traumatic experience with support resources which are absent or inadequate. Their trauma is disbelieved, their speech is punished, and their assailants walk free among us as equals while they are subject to retaliatory harassment or worse.

Victims have no recourse which will satisfy men. Reporting a crime is traumatic, especially one of this nature. I have heard many stories of disbelief from the authorities, disbelief in the face of overwhelming evidence. They were told it was their fault. They were told they should have been in a different place, or wearing something else, or should have simply been a different person. It’s their fault, not the aggressor’s. It’s about what they, the victim, should have done differently, never mind what the perpetrator should have done differently. It’s estimated that less than 1% of rapes end with the rapist in jail¹ – the remainder go unreported, unprosecuted or fail after years of traumatic legal proceedings for the victims. The legal system does not provide justice: it exacerbates harm. A hacker will demand this process is completed before they will seek justice, or allow justice to be sought. Until then, we will demand silence, and retaliate if our demands are not met.

The strict standards of evidence required by the justice system are there because of the state monopoly on violence: a guilty verdict in a crime will lead to the imprisonment of the accused. We have no such recourse available in private, accordingly there is no need to hold ourselves to such standards. Our job is not to punish the accused, but rather to keep our communities safe. We can establish the need to take action to whatever standard we believe is sufficient, and by setting these standards as strict as the courts we will fail to resolve over 99% of the situations with which we are faced – a standard which is clearly not sufficient to address the problem. I’m behind you if you want to improve the justice system in this regard, but not if you set this as a blocker to seeking any justice at all. What kind of hacker puts their faith in authority?

I find the state of affairs detestable. The hypocrisy of the free speech absolutist who demands censorship of challenging topics. The fact that the famous hacker curiosity can suddenly dry up if satisfying it would question our biases and preconceptions. The complicity of our moderators in censoring progressive voices in the defense of decorum and the status quo. The duplicitous characterization of “polite” hate speech as acceptable in our communities. Our failure to acknowledge our own shortcomings, our fear of seeing the “other” in a position of power, and the socially enforced ignorance of the “other” that naturally leads to failing to curtail discrimination and harassment in our communities. The ridiculously high standard of evidence we require from victims, who simply ask for our belief at a minimum, before we’ll consider doing anything about their grievance, if we could even be convinced in the first place.

Meanwhile, the problems that these forbidden topics seek to discuss are present in our community. That includes the “polite” problems, such as the conspicuous lack of diversity in our positions of power, which may be discussed and commiserated only until someone suggests doing something about it; and also the impolite problems up to and including the protection of the perpetrators of sexual harassment, sexual assault, and, yes, rape.

Most hackers live under the comfortable belief that it “can’t happen here”, but it can and it does. I attended a hacker event this year – HiP Berlin – where I discovered that some of the organizers had cooperated to make it possible for multiple known rapists to participate, working together to find a way to circumvent the event’s code of conduct – a document that they were tasked with enforcing. One of the victims was in attendance, believing the event to be safe. At every hacker event I have attended in recent memory, I have personally witnessed or heard stories of deeply problematic behavior and protection for its perpetrators from the leadership.

Our community has problems, important problems, that every hacker should care about, and we need the bravery and humility to face them, not the cowardice to retaliate against those who speak up. Talk to, listen to, and believe your peers and their stories. Stand up for what’s right, and speak out when you see something that isn’t. Demand that your leaders and moderators do the right thing. Make a platform where people can safely speak about what our community needs to do right by them, and have the courage to listen to them and confront yourself.

You need to be someone who will do something about it.

Edit: Case in point: this post was quietly removed by Hacker News moderators within 40 minutes of its submission.

Hyprland is a toxic community

Sun, 17 Sep 2023 00:00:00 +0000

Hyprland is an open source Wayland compositor based on wlroots, a project I started back in 2017 to make it easier to build good Wayland compositors. It’s a project which is loved by its users for its emphasis on customization and “eye candy” – beautiful graphics and animations, each configuration tailored to the unique look and feel imagined by the user who creates it. It’s a very exciting project!

Unfortunately, the effect is spoilt by an incredibly toxic and hateful community. I cannot recommend Hyprland to anyone who is not prepared to steer well clear of its community spaces. Imagine a high school boys’ locker room come to life on Discord and GitHub and you’ll get an idea of what it’s like.

I became aware of the issues with Hyprland’s community after details of numerous hateful incidents on their Discord came to my attention by way of the grapevine. Most of them stem from the community’s tolerance of hate: community members are allowed to express hateful views with impunity, up to and including astonishing views such as endorsements of eugenics and calls for hate-motivated violence. Such comments are treated as another act in the one big inside joke that is the Hyprland community – the community prefers not to take itself “too seriously”. Hate is moderated only if it is “disruptive” (e.g. presents as spam), but hate presented with a veneer of decorum (or sarcasm) is tolerated, and when challenged, it’s laughed off as a joke.

In one particular incident, the moderators of the Discord server engaged in a harassment campaign against a transgender user, including using their moderator privileges to edit the pronouns in their username from “they/she” to “who/cares”. These roles should be held by trusted community leaders, and it’s from their behavior that the community’s culture and norms stem – they set an example for the community and define what behaviors are acceptable or expected. The problem comes from the top down.

Someone recently pitched a code of conduct – something that this project sorely needs – in a GitHub issue. This thread does not have much overt hate, but it does clearly show how callous and just plain mean the community is, including its leadership (Vaxerski is the original author of Hyprland). Everything is a joke and anyone who wants to be “serious” about anything is mercilessly bullied and made fun of. Quoting this discussion:

I think [a Code of Conduct] is pretty discriminatory towards people that prefer a close, hostile, homogeneous, exclusive, and unhealthy community.

First of all, why would I pledge to uphold any values? Seems like just inconveniencing myself. […] If I’d want to moderate, I’d spend 90% of the time reading kids arguing about bullshit instead of coding.

If you don’t know how to behave without a wall of text explaining how to behave online then you shouldn’t be online.

I am not someone who believes all projects need a code of conduct, if there exists a reasonable standard of conduct in its absence – and that means having a community that does not bully and harass others for expressing differing points of view, let alone for simply having a marginalized identity.

I would have preferred to address these matters in private, so I reached out to Vaxry in February. He responded with a lack of critical awareness over how toxicity presents in his community. However, following my email, he put out a poll for the Discord community to see if the community members experienced harassment in the community – apparently 40% of respondents reported such experiences. Vaxry et al implemented new moderation policies as a result. But these changes did not seem to work: the problems are still present, and the community is still a toxic place that facilitates bullying and hate, including from the community leaders.

Following my email conversation with Vaxry, he appeared on a podcast to discuss toxicity in the Hyprland community. This quote from the interview clearly illustrates the attitude of the leadership:

[A trans person] joined the Discord server and made a big deal out of their pronouns [..] because they put their pronouns in their nickname and made a big deal out of them because people were referring to them as “he” [misgendering them], which, on the Internet, let’s be real, is the default. And so, one of the moderators changed the pronouns in their nickname to “who/cares”. […] Let’s be real, this isn’t like, calling someone the N-word or something.

Later he describes a more moderated community (the /r/unixporn discord server) as having an environment in which everyone is going to “lick your butthole just to be nice”. He compared himself to Terry Davis, the late operating system developer whose struggles with mental illness were broadcast for the world to see, citing a video in which he answers a phone call and refers to the person on the phone by the N-word “ten times” – Vaxry compares this to his approach to answering “stupid questions”.

It really disappoints me to see such an exciting project brought low by a horribly mismanaged community of hate and bullying. Part of what makes open source software great is that it’s great for everyone. It’s unfortunate that someone can discover this cool project, install it and play with it and get excited about it, then join the community to find themselves at the wrong end of this behavior. No one deserves that.

I empathise with Vaxry. I remember being young, smart, productive… and mean. I did some cool stuff, but I deeply regret the way I treated people. It wasn’t really my fault – I was a product of my environment – but it was my responsibility. Today, I’m proud to have built many welcoming communities, where people are rewarded for their involvement, rather than coming away from their experience hurt. What motivates us to build and give away free software if not bringing joy to ourselves and others? Can we be proud of a community which brings more suffering into the world?

My advice to the leadership begins with taking a serious look in the mirror. This project needs a “come to Jesus” moment. Ask yourself what kind of community you can be proud of – can you be proud of a community that people walk away from feeling dejected and hurt? Yours is not a community that brings people joy. What are you going to do about it?

A good start will be to consider the code of conduct proposal seriously, but a change of attitude is also required. My inbox is open to any of the leaders in this project (or any other project facing similar problems) if you want to talk. I’m happy to chat with you in good faith and help you understand what’s needed and why it’s important.

To members of the Hyprland community, I want each of you to personally step up to make the community better. If you see hate and bullying, don’t stay silent. This is a community which proclaims to value radical free speech: test it by using your speech to argue against hate. Participate in the community as you think it should be, not as it necessarily is, and change will follow. If you are sensitive to hate, or a member of a marginalized group, however, I would just advise steering clear of Hyprland until the community improves.

If the leadership fails to account for these problems, it will be up to the community to take their activity elsewhere. You could set up adjacent communities which are less toxic, or fork the software, or simply choose to use something else.

To the victims of harassment, I offer my sincere condolences. I know how hard it is to be the subject of this kind of bullying. You don’t deserve to be treated like this. There are many places in the free software community where you are welcome and celebrated – Hyprland is not the norm. If you need support, I’m always available to listen to your struggles.

To everyone else: please share this post throughout the Hyprland community and adjacent communities. This is a serious problem and it’s not going to change unless its clearly brought to light. The Hyprland maintainers need to be made aware that the broader open source community does not appreciate this kind of behavior.

I sincerely hope that this project improves its community. A serious attitude shift is needed from the top-down, and I hope for the sake of Vaxry, the other leaders, and the community as a whole, that such change comes sooner rather than later. When Vaxry is older and wiser, I want him to look back on the project and community that he’s built with pride and joy, not with regret and shame.

Vaxry has published a response to this post.

I was also privately provided some of the enusing discussion from the Hyprland Discord. Consider that this lacks context and apply your grain of salt accordingly.

Screenshot of a Discord channel with the initial reaction to this post. A user called “slave labor” responds with “no way”, “the computer reddit woke up”

Screenshot of a Discord channel with Vaxry’s initial reaction to this post. “Really, right as I wanted to take a day off because of health reasons I have to reply to this?”. Another user responds “wow this is quite… shallow”, “almost as if it recycles very limited context to get more clicks”

I apologise to Vaxry for interrupting their rest, and wish them a speedy recovery.

Screenshot of a Discord channel. Some notable quotes include “LGBTQ is fucking trash anyways” (someone else responds “fuck off” to this) and “for reclaiming polymc from the leftoids”. The discussion as a whole lacks any sembelance of professionalism.

Here is a plain text log which includes some additional discussion.

AI crap

Tue, 29 Aug 2023 00:00:00 +0000

There is a machine learning bubble, but the technology is here to stay. Once the bubble pops, the world will be changed by machine learning. But it will probably be crappier, not better.

Contrary to the AI doomer’s expectations, the world isn’t going to go down in flames any faster thanks to AI. Contemporary advances in machine learning aren’t really getting us any closer to AGI, and as Randall Monroe pointed out back in 2018:

What will happen to AI is boring old capitalism. Its staying power will come in the form of replacing competent, expensive humans with crappy, cheap robots. LLMs are a pretty good advance over Markov chains, and stable diffusion can generate images which are only somewhat uncanny with sufficient manipulation of the prompt. Mediocre programmers will use GitHub Copilot to write trivial code and boilerplate for them (trivial code is tautologically uninteresting), and ML will probably remain useful for writing cover letters for you. Self-driving cars might show up Any Day Now™, which is going to be great for sci-fi enthusiasts and technocrats, but much worse in every respect than, say, building more trains.

The biggest lasting changes from machine learning will be more like the following:

A reduction in the labor force for skilled creative work
The complete elimination of humans in customer-support roles
More convincing spam and phishing content, more scalable scams
SEO hacking content farms dominating search results
Book farms (both eBooks and paper) flooding the market
AI-generated content overwhelming social media
Widespread propaganda and astroturfing, both in politics and advertising

AI companies will continue to generate waste and CO₂ emissions at a huge scale as they aggressively scrape all internet content they can find, externalizing costs onto the world’s digital infrastructure, and feed their hoard into GPU farms to generate their models. They might keep humans in the loop to help with tagging content, seeking out the cheapest markets with the weakest labor laws to build human sweatshops to feed the AI data monster.

You will never trust another product review. You will never speak to a human being at your ISP again. Vapid, pithy media will fill the digital world around you. Technology built for engagement farms – those AI-edited videos with the grating machine voice you’ve seen on your feeds lately – will be white-labeled and used to push products and ideologies at a massive scale with a minimum cost from social media accounts which are populated with AI content, cultivate an audience, and sold in bulk and in good standing with the Algorithm.

All of these things are already happening and will continue to get worse. The future of media is a soulless, vapid regurgitation of all media that came before the AI epoch, and the fate of all new creative media is to be subsumed into the roiling pile of math.

This will be incredibly profitable for the AI barons, and to secure their investment they are deploying an immense, expensive, world-wide propaganda campaign. To the public, the present-day and potential future capabilities of the technology are played up in breathless promises of ridiculous possibility. In closed-room meetings, much more realistic promises are made of cutting payroll budgets in half.

The propaganda also leans into the mystical sci-fi AI canon: the threat of smart computers with world-ending power, the forbidden allure of a new Manhattan Project and all of its consequences, the long-prophesied singularity. The technology is nowhere near this level, a fact well-known by experts and the barons themselves, but the illusion is maintained in the interests of lobbying lawmakers to help the barons erect a moat around their new industry.

Of course, AI does present a threat of violence, but as Randall points out, it’s not from the AI itself, but rather from the people that employ it. The US military is testing out AI-controlled drones, which aren’t going to be self-aware but will scale up human errors (or human malice) until innocent people are killed. AI tools are already being used to set bail and parole conditions – it can put you in jail or keep you there. Police are using AI for facial recognition and “predictive policing”. Of course, all of these models end up discriminating against minorities, depriving them of liberty and often getting them killed.

AI is defined by aggressive capitalism. The hype bubble has been engineered by investors and capitalists dumping money into it, and the returns they expect on that investment are going to come out of your pocket. The singularity is not coming, but the most realistic promises of AI are going to make the world worse. The AI revolution is here, and I don’t really like it.

Flame bait

I had much more inflammatory article drafted for this topic under the title "ChatGPT is the new techno-atheist's substitute for God". It makes some fairly pointed comparisons between the cryptocurrency cult and the machine learning cult and the religious, unshakeable, and largely ignorant faith in both technologies as the harbingers of progress. It was fun to write, but this is probably the better article. I found this Hacker News comment and quoted it in the original draft: "It's probably worth talking to GPT4 before seeking professional help [to deal with depression]." In case you need to hear it: [do not][suicide] (TW: suicide) seek out OpenAI's services to help with your depression. Finding and setting up an appointment with a therapist can be difficult for a lot of people -- it's okay for it to feel hard. Talk to your friends and ask them to help you find the right care for your needs.

Hello from Ares!

Wed, 09 Aug 2023 00:00:00 +0000

I am pleased to be writing today’s blog post from a laptop running Ares OS. I am writing into an ed(1) session, on a file on an ext4 filesystem on its hard drive. That’s pretty cool! It seems that a lot of interesting stuff has happened since I gave that talk on Helios at FOSDEM in February.

A picture of my ThinkPad while I was editing this blog post

The talk I gave at FOSDEM was no doubt impressive, but it was a bit of a party trick. The system was running on a Raspberry Pi with one process which included both the slide deck as a series of raster images baked into the ELF file, as well as the GPU driver and drawing code necessary to display them, all in one package. This was quite necessary, as it turns out, given that the very idea of “processes” was absent from the system at this stage.

Much has changed since that talk. The system I am writing to you from has support for processes indeed, complete with fork and exec and auxiliary vectors and threads and so on. If I run “ps” I get the following output:

mercury % ps
1 /sbin/usrinit dexec /sbin/drv/ext4 block0 childfs 0 fs 0
2 /etc/driver.d/00-pcibus
3 /etc/pci.d/class/01/06/ahci
4 /etc/driver.d/00-ps2kb
5 /etc/driver.d/99-serial
6 /etc/driver.d/99-vgacons
7 /sbin/drv/ext4 block0
15 ed blog.md
16 ps

Each of these processes is running in userspace, and some of them are drivers. A number of drivers now exist for the system, including among the ones you see here a general-purpose PCI driver, AHCI (SATA), PS/2 keyboard, PC serial, and a VGA console, not to mention the ext4 driver, based on lwext4 (the first driver not written in Hare, actually). Not shown here are additional drivers for the CMOS real-time clock (so Ares knows what time it is, thanks to Stacy Harper), a virtio9pfs driver (thanks also to Tom Leb for the initial work here), and a few more besides.

As of this week, a small number of software ports exist. The ext4 driver is based on lwext4, as I said earlier, which might be considered a port, though it is designed to be portable. The rc shell I have been working on lately has also been ported, albeit with many features disabled, to Mercury. And, of course, I did say I was writing this blog post with ed(1) – I have ported Michael Forney’s ed implementation from sbase, with relatively few features disabled as a matter of fact (the “!” command and signals were removed).

This ed port, and lwext4, are based on our C library, designed with drivers and normal userspace programs in mind, and derived largely from musl libc. This is coming along rather well – a few features (signals again come to mind) are not going to be implemented, but it’s been relatively straightforward to get a large amount of the POSIX/C11 API surface area covered on Ares, and I was pleasantly surprised at how easy it was to port ed(1).

There’s still quite a lot to be done. In the near term, I expect to see the following:

A virtual filesystem
Pipes and more shell features enabled, such as redirects
More filesystem support (mkdir et al)
A framebuffer console
EFI support on x86_64
MBR and GPT partitions

This is more of the basics. As these basics unblock other tasks, a few of the more ambitious projects we might look forward to include:

Networking support (at least ICMP)
Audio support
ACPI support
Basic USB support
A service manager (not systemd…)
An installer, perhaps a package manager
Self-hosting builds
Dare I say Wayland?

I should also probably do something about that whining fan I’m hearing in the background right now. Of course, I will also have to do a fresh DOOM port once the framebuffer situation is improved. There’s also still plenty of kernel work to be done and odds and ends all over the project, but it’s in pretty good shape and I’m having a blast working on it. I think that by now I have answered the original question, “can an operating system be written in Hare”, with a resounding “yes”. Now I’m just having fun with it. Stay tuned!

Now I just have to shut this laptop off. There’s no poweroff command yet, so I suppose I’ll just hold down the power button until it stops making noise.

The rc shell and its excellent handling of whitespace

Mon, 31 Jul 2023 00:00:00 +0000

This blog post is a response to Mark Dominus’ “The shell and its crappy handling of whitespace”.

I’ve been working on a shell for Unix-like systems called rc, which draws heavily from the Plan 9 shell of the same name. When I saw Mark’s post about the perils of whitespace in POSIX shells (or derived shells, like bash), I thought it prudent to see if any of the problems he outlines are present in the shell I’m working on myself. Good news: they aren’t!

Let’s go over each of his examples. First he provides the following example:

for i in *.jpg; do
	cp $i /tmp
done

This breaks if there are spaces in the filenames. Not so with rc:

% cat test.rc
for (i in *.jpg) {
	cp $i subdir
}
% ls
a.jpg   b.jpg  'bite me.jpg'   c.jpg   subdir   test.rc
% rc ./test.rc 
% ls subdir/
a.jpg   b.jpg  'bite me.jpg'   c.jpg

He gives a similar example for a script that renames jpeg to jpg:

for i in *.jpeg; do
  mv $i $(suf $i).jpg
done

This breaks for similar reasons, but works fine in rc:

% cat test.rc  
fn suf(fname) {
	echo $fname | sed -e 's/\..*//'
}

for (i in *.jpeg) {
	mv $i `{suf $i}.jpg
}
% ls 
a.jpeg   b.jpeg  'bite me.jpeg'   c.jpeg   test.rc
% rc ./test.rc  
% ls 
a.jpg   b.jpg  'bite me.jpg'   c.jpg   test.rc

There are other shells, such as fish or zsh, which also have answers to these problems which don’t necessarily call for generous quoting like other shells often do. rc is much simpler than these shells. At the moment it clocks in at just over 3,000 lines of code, compared to fish at ~45,000 and zsh at ~144,000. Admittedly, it’s not done yet, but I would be surprised to see it grow beyond 5,000 lines for version 1.0.¹

The key to rc’s design success in this area is the introduction of a second primitive. The Bourne shell and its derivatives traditionally work with only one primitive: strings. But command lines are made of lists of strings, and so a language which embodies the primitives of the command line ought to also be able to represent those as a first-class feature. In traditional shells a list of strings is denoted inline with the use of spaces within those strings, which raises obvious problems when the members themselves contain spaces; see Mark’s post detailing the errors which ensue. rc adds lists of strings as a formal primitive alongside strings.

% args=(ls --color /) 
% echo $args(1) 
ls
% echo $args(2) 
--color
% echo $#args 
3
% $args 
bin   dev  home  lost+found  mnt  proc  run   srv      swap  tmp  var
boot  etc  lib   media       opt  root  sbin  storage  sys   usr
% args=("foo bar" baz) 
% touch $args 
% ls 
 baz  'foo bar'

Much better, right? One simple change eliminates the need for quoting virtually everywhere. Strings can contain spaces and nothing melts down.

Let me run down the remaining examples from Mark’s post and demonstrate their non-importance in rc. First, regarding $*, it just does what you expect.

% cat yell.rc
#!/bin/rc
shift
echo I am about to run $* now!!!
exec $*
% ls *.jpg
'bite me.jpg'
% ./yell.rc ls *.jpg
I am about to run ls bite me.jpg now!!!
'bite me.jpg'

Note also that there is no need to quote the arguments to “echo” here. Also note the use of shift; $* includes $0 in rc.

Finally, let’s rewrite Mark’s “lastdl” program in rc and show how it works fine in rc’s interactive mode.

#!/bin/rc
cd $HOME/downloads
echo $HOME/downloads/`{ls -t | head -n1}

Its use at the command line works just fine without quotes.

% file `{lastdl} 
/home/sircmpwn/downloads/test image.jpg: JPEG image data, JFIF standard 1.01,
aspect ratio, density 1x1, segment length 16, baseline, precision 8,
5000x2813, components 3

Just for fun, here’s another version of this rc script that renames files with spaces to without, like the last example in Mark’s post:

#!/bin/rc
cd $HOME/downloads
last=`{ls -t | head -n1}
if (~ $last '* *') {
	newname=`{echo $last | tr ' \t' '_'}
	mv $last $HOME/downloads/$newname
	last=$newname
}
echo $HOME/downloads/$last

The only quotes to be found are those which escape the wildcard match testing for a space in the string.² Not bad, right? Like Plan 9’s rc, my shell imagines a new set of primitives for shells, then starts from the ground up and builds a shell which works better in most respects while still being very simple. Most of the problems that have long plagued us with respect to sh, bash, etc, are solved in a simple package with rc, alongside a nice interactive mode reminiscent of the best features of fish.

rc is a somewhat complete shell today, but there is a bit more work to be done before it’s ready for 1.0, most pressingly with respect to signal handling and job control, alongside a small bit of polish and easier features to implement (such as subshells, IFS, etc). Some features which are likely to be omitted, at least for 1.0, include logical comparisons and arithmetic expansion (for which /bin/test and /bin/dc are recommended respectively). Of course, rc is destined to become the primary shell of the Ares operating system project that I’ve been working on, but I have designed it to work on Unix as well.

Check it out!

Alpine Linux does not make the news

Tue, 25 Jul 2023 00:00:00 +0000

My Linux distribution of choice for several years has been Alpine Linux. It’s a small, efficient distribution which ships a number of tools I appreciate for their simplicity, such as musl libc. It has a very nice package manager, apk, which is fast and maintainable. The development community is professional and focuses on diligent maintenance of the distribution and little else. Over the years I have used it, very little of note has happened.

I run Alpine in every context; on my workstation and my laptops but also on production servers, on bare-metal and in virtual machines, on my RISC-V and ARM development boards, at times on my phones, and in many other contexts besides. It has been a boring experience. The system is simply reliable, and the upgrades go over without issue every other quarter,¹ accompanied by high-quality release notes. I’m pleased to maintain several dozen packages in the repositories, and the community is organized such that it is easy for someone like me to jump in and do the work required to maintain it for my use-cases.

Red Hat has been in the news lately for their moves to monetize the distribution, moves that I won’t comment on but which have generally raised no small number of eyebrows, written several headlines, and caused intense flamewars throughout the internet. I don’t run RHEL or CentOS anywhere, in production or otherwise, so I just looked curiously on as all of this took place without calling for any particular action on my part. Generally speaking, Alpine does not make the news.

And so it has been for years, as various controversies come about and die off, be it with Red Hat, Ubuntu, Debian, or anything else, I simply keep running “apk upgrade” every now and then and life goes on uninterrupted. I have high-quality, up-to-date software on a stable system and suffer from no fuss whatsoever.

The Alpine community is a grassroots set of stakeholders who diligently concern themselves with the business of maintaining a good Linux distribution. There is little in the way of centralized governance;² for the most part the distribution is just quietly maintained by the people who use it for the purpose of ensuring its applicability to their use-cases.

So, Alpine does not make the news. There are no commercial entities which are trying to monetize it, at least no more than the loosely organized coalition of commercial entities like SourceHut that depend on Alpine and do their part to keep it in good working order, alongside various users who have no commercial purpose for the system. The community is largely in unanimous agreement about the fundamental purpose of Alpine and the work of the community is focused on maintaining the project such that this purpose is upheld.

This is a good trait for a Linux distribution to have.

Seriously, don't sign a CLA

Tue, 04 Jul 2023 00:00:00 +0000

SourceGraph is making their product closed source, abandoning the Apache 2.0 license it was originally distributed under, so once again we convene in the ritual condemnation we offer to commercial products that piss in the pool of open source. Invoking Bryan Cantrill once more:

Bryan Cantrill on OpenSolaris — YouTube

A contributor license agreement, or CLA, usually (but not always) includes an important clause: a copyright assignment. These agreements are provided by upstream maintainers to contributors to open source software projects, and they demand a signature before the contributor’s work is incorporated into the upstream project. The copyright assignment clause that is usually included serves to offer the upstream maintainers more rights over the contributor’s work than the contributor was offered by upstream, generally in the form of ownership or effective ownership over the contributor’s copyright and the right to license it in any manner they choose in the future, including proprietary distributions.

This is a strategy employed by commercial companies with one purpose only: to place a rug under the project, so that they can pull at the first sign of a bad quarter. This strategy exists to subvert the open source social contract. These companies wish to enjoy the market appeal of open source and the free labor of their community to improve their product, but do not want to secure these contributors any rights over their work.

This is particularly pathetic in cases like that of SourceGraph, which used a permissive Apache 2.0 license. Such licenses already allow their software to be incorporated into non-free commercial works, such is the defining nature of a permissive license, with relatively few obligations: in this case, a simple attribution will suffice. SourceGraph could have been made non-free without a CLA at all if this one obligation was met. The owners of SourceGraph find the simple task of crediting their contributors too onerous. This is disgusting.

SourceGraph once approached SourceHut asking about building an integration between our platforms. They wanted us to do most of the work, which is a bit tacky but reasonable under the reciprocal social contract of open source. We did not prioritize it and I’m glad that we didn’t: our work would have been made non-free.

Make no mistake: a CLA is a promise that a open source software project will one day become non-free. Don’t sign them.

What are my rights as a contributor?

If you sign away your rights by agreeing to a CLA, you retain all of the rights associated with your work.

By default, you own the copyright over your contribution and the contribution is licensed under the same software license the original project uses, thus, your contribution is offered to the upstream project on the same terms that their contribution was offered to you. The copyright for such projects is held collectively by all contributors.

You also always have the right to fork an open source project and distribute your improvements on your own terms, without signing a CLA – the only power upstream holds is authority over the “canonical” distribution. If the rug is pulled from under you, you may also continue to use, and improve, versions of the software from prior to the change in license.

How do I prevent this from happening to my project?

A CLA is a promise that software will one day become non-free; you can also promise the opposite. Leave copyright in the collective hands of all contributors and use a copyleft license.

Without the written consent of all contributors, or performing their labor yourself by re-writing their contributions, you cannot change the license of a project. Skipping the CLA leaves their rights intact.

In the case of a permissive software license, a new license (including proprietary licenses) can be applied to the project and it can be redistributed under those terms. In this way, all future changes can be written with a new license. The analogy is similar to that of a new project with a proprietary license taking a permissively licensed project and incorporating all of the code into itself before making further changes.

You can prevent this as well with a copyleft license: such a license requires the original maintainers to distribute future changes to the work under a free software license. Unless they can get all copyright holders – all of the contributors – to agree to a change in license, they are obligated to distribute their improvements on the same terms.

Thus, the absence of a CLA combined with the use of a copyleft license serves as a strong promise about the future of the project.

Learn more at writefreesoftware.org:

What should I do as a business instead of a CLA?

It is not ethical to demand copyright assignment in addition to the free labor of the open source community. However, there are some less questionable aspects of a contributor license agreement which you may uphold without any ethical qualms, notably to establish provenance.

Many CLAs include clauses which establish the provenance of the contribution and transfer liability to the contributor, such that the contributor agrees that their contribution is either their own work or they are authorized to use the copyright (for example, with permission from their employer). This is a reasonable thing to ask for from contributors, and manages your exposure to legal risks.

The best way to ask for this is to require contributions to be “signed-off” with the Developer Certificate of Origin.

Previously:

Social media and "parasocial media"

Fri, 30 Jun 2023 00:00:00 +0000

A few months ago, as Elon Musk took over Twitter and instituted polices that alienated many people, some of these people fled towards federated, free software platforms like Mastodon. Many people found a new home here, but there is a certain class of refugee who has not found it to their liking.

I got to chatting with one such “refugee” on Mastodon today. NotJustBikes is a creator I enjoy watching on ~~YouTube~~ Invidious, who makes excellent content on urbanism and the design of cities. He’s based in my home town of Amsterdam and his videos do a great job of explaining many of the things I love about this place for general audiences. He’s working on building an audience, expanding his reach, and bringing his message to as many people as possible in the interest of bringing better infrastructure to everyone.

But he’s not satisfied with his move from Twitter to Mastodon, nor are some of his friends among the community of “urbanist” content creators. He yearns for an “algorithm” to efficiently distribute content to his followers, and Mastodon is not providing this for him.

On traditional “social media” platforms, in particular YouTube, the interactions are often not especially social. The platforms facilitate a kind of intellectual consumption moreso than conversation: conversations flow in one direction, from creator to audience, where the creator produces and the audience consumes. I think a better term for these platforms is “parasocial media”: they are optimized for creating parasocial relationships moreso than social relationships.

The fediverse is largely optimized for people having conversations with each other, and not for producing and consuming “content”. Within this framework, a “content creator” is a person only in the same sense that a corporation is, and their conversations are unidirectional, where the other end is also not a person, but an audience. That’s not the model that the fediverse is designed around.

It’s entirely reasonable to want to build an audience and publish content in a parasocial manner, but that’s not what the fediverse is for. And I think that’s a good thing! There are a lot of advantages in having spaces which focus on being genuinely “social”, rather than facilitating more parasocial interactions and helping creators build an audience. This limits the fediverse’s reach, but I think that’s just fine.

Within this model, the fediverse’s model, it’s possible to publish things, and consume things. But you cannot effectively optimize for building the largest possible audience. You will generally be more successful if you focus on the content itself, and not its reach, and on the people you connect with at a smaller scale. Whether or not this is right for you depends on your goals.

I hope you enjoyed this content! Remember to like and subscribe.

Burnout and the quiet failures of the hacker community

Thu, 29 Jun 2023 00:00:00 +0000

This has been a very challenging year for me. You probably read that I suffered from burnout earlier in the year. In some respects, things have improved, and in many other respects, I am still haunted.

You might not care to read this, and so be it, take your leave if you must. But writing is healing for me. Maybe this is a moment for solidarity, sympathy, for reflecting on your own communities. Maybe it’s a vain and needlessly public demonstration of my slow descent into madness. I don’t know, but here we go.

Yesterday was my 30^th birthday. 🎂 It was another difficult day for me. I drafted a long blog post with all of the details of the events leading up to my burnout. You will never read it; I wrote it for myself and it will only be seen by a few confidants, in private, and my therapist. But I do want to give you an small idea of what I’ve been going through, and some of the take-aways that matter for you and the hacker community as a whole.

Here’s a quote from yesterday’s unpublished blog post:

Trigger warnings: child abuse, rape, sexual harassment, suicide, pedophilia, torture.

You won’t read the full story, and trust me, you’re better off for that. Suffice to say that my life has been consumed with trauma and strife all year. I have sought healing, and time for myself, time to process things, and each time a new crisis has landed on my doorstep, most of them worse than the last. A dozen things went wrong this year, horribly wrong, one after another. I have enjoyed no peace in 2023.

Many of the difficulties I have faced this year have been beyond the scope of the hacker community, but several have implicated it in challenging and confronting ways.

The hacker community has been the home I never had, but I’m not really feeling at home here right now. A hacker community that was precious to me failed someone I love and put my friends in danger. Rape and death had come to our community, and was kept silent. But I am a principled person, and I stand for what is right; I spoke the truth and it brought me and my loved ones agonizing stress and trauma and shook our community to the core. Board members resigned. Marriages are on the rocks. When the dust settled, I was initially uncomfortable staying in this community, but things eventually started to get better. Until another member of this community, someone I trusted and thought of as a friend, confessed to me that he had raped multiple women a few years ago. I submitted my resignation from this community last night.

Then I went to GPN, a hacker event in Germany, at the start of June. It was a welcome relief from the stress I’ve faced this year, a chance to celebrate hacker culture and a warm reminder of the beauty of our community. It was wonderful. Then, on the last night, a friend took me aside and confided in me that they are a pedophile, and told me it was okay because they respected the age of consent in Germany – which is 14. What began as a wonderful reminder of what the hacker community can be became a PTSD episode and a reminder that rape culture is fucking everywhere.

I don’t want to be a part of this anymore. Our communities have tolerated casual sexism and misogyny and transphobia and racism and actual fucking rapists, and stamped down on women and queer people and brown people in our spaces with a smile on our face and a fucked-up facsimile of tolerance and inclusion as a cornerstone of the hacker ethic.

This destroys communities. It is destroying our communities. If there’s one thing I came to understand this year, it’s that these problems are pervasive and silent.

Here’s what you need to do: believe the victims. Stand up for what’s right. Have the courage to remove harmful people from your environment, especially if you’re a man and have a voice. Make people feel welcome, and seen. Don’t tolerate casual sexism in the hacker community or anywhere else. Don’t tolerate transphobia or homophobia. Don’t tolerate racists. If you see something, say something. And for fuck’s sake, don’t bitch about that code of conduct that someone wants to add to your community.¹

I’m going to withdraw a bit from the in-person hacker community for the indefinite future. I don’t think I can manage it for a while. I have felt good about working on my software and collaborating with my free software communities online, albeit at a much-reduced capacity. I’m going to keep working, and writing, insofar as I find satisfaction in it. Life goes on.

Be there for the people you love, and love more people, and be there for them, too.

Reforming the free software message

Mon, 19 Jun 2023 00:00:00 +0000

Several weeks ago, I wrote The Free Software Foundation is dying, wherein I enumerated a number of problems with the Free Software Foundation. Some of my criticisms focused on the message: fsf.org and gnu.org together suffer from no small degree of incomprehensibility and inaccessibility which makes it difficult for new participants to learn about the movement and apply it in practice to their own projects.

This is something which is relatively easily fixed! I have a background in writing documentation and a thorough understanding of free software philosophy and practice. Enter writefreesoftware.org: a comprehensive introduction to free software philosophy and implementation.

The goals of this resource are:

Provide an accessible introduction to the most important principles of free software
Offer practical advice on choosing free software licenses from a free software perspective (compare to the OSS perspective at choosealicense.com).
Publish articles covering various aspects of free software in practice, such as how it can be applied to video games

No particular association with any particular free software project or organization
No policy of non-cooperation with the open source movement

Compare writefreesoftware.org with the similar resources provided by GNU (1, 2) and you should get the general idea.

The website is itself free software, CC-BY-SA 4.0. You can check out the source code here and suggest any improvements or articles for the mailing list. Get involved! This resource is not going to solve all of the FSF’s problems, but it is an easy way to start putting the effort in to move the free software movement forward. I hope you like it!

Throwing in the towel on mobile Linux

Fri, 16 Jun 2023 00:00:00 +0000

I have been tinkering with mobile Linux – a phrase I will use here to describe any Linux distribution other than Android running on a mobile device – as my daily driver since about 2019, when I first picked up the PinePhone. For about 3 years I have run mobile Linux as my daily driver on my phone, and as of a few weeks ago, I’ve thrown in the towel and switched to Android.

The distribution I ran for the most time is postmarketOS, which I was mostly quite happy with, running at times sxmo and Phosh. I switched to UBports a couple of months ago. I have tried a variety of hardware platforms to support these efforts, namely:

Pinephone (pmOS)
Pinephone Pro (pmOS)
Xiaomi Poco F1 (pmOS)
Fairphone 4 (UBports)

I have returned to LineageOS as my daily driver and closed the book on mobile Linux for the time being. What put the final nails in the coffin was what I have been calling out as my main concern throughout my experience: reliability, particularly of the telephony components.

Use-case	Importance	postmarketOS	UBports	LineageOS
Basic system reliability	5	2	4	5
Mobile telephony	5	3	3	5
Hotspot	4	5	3	5
2FA	4	4	1	5
Web browsing	4	5	2	4
Mobile banking	4	1	1	5
Bluetooth audio	3	4	2	4
Music player	3	4	1	3
Reading email	3	1	3	4
Navigation aid	3	2	1	5
Camera	3	3	3	5
Password manager	3	5	1	1
sysadmin	3	5	2	3

More on these use-cases and my experiences

Mobile banking: only available through a proprietary vendor-provided Android app. Tried to get it working on Waydroid; did not work on pmOS and almost worked on UBports, but Waydroid is very unreliable. Kind of shit but I don’t have any choice because my bank requires it for 2FA.

Web browsing: I can just run Firefox upstream on postmarketOS. Amazing! UBports cannot do this, and the available web browsers are not nearly as pleasant to use. I run Fennic on Android and it’s fine.

Music player: the music player on UBports is extremely unreliable.

Reading email: This is not entirely pmOS’s fault; I could have used my main client, aerc, which is a testament to pmOS’s general utility, but it is a TUI that is uncomfortable to use on a touchscreen-only device.

Password manager: pmOS gets 5/5 because I could use the password manager I wrote myself, himitsu, out of the box. Non-critical use-case because I could just type passwords in manually on the rare occasion I need to use one.

sysadmin: stuff like being able to SSH into my production boxes from anywhere to troubleshoot stuff.

Among these use-cases, there is one that absolutely cannot be budged on: mobile telephony. My phone is a critical communication device and I need to be able to depend on calls and SMS at all times, therefore the first two rows need to score 4 or 5 before the platform is suitable for my use. I remember struggling with postmarketOS while I was sick with a terrible throat infection – and I could not call my doctor. Not cool.

I really like these projects and I love the work that’s going into them. postmarketOS in particular: being able to run the same environment I run everywhere else, Alpine Linux, on my phone, is fucking amazing. The experience is impressively complete in many respects, all kinds of things, including things I didn’t expect to work well, work great. In the mobile Linux space I think it’s the most compelling option right now.

But pmOS really suffers from reliability issues – both on edge and on stable it seemed like every update broke some things and fixed others, so only a subset of these cool features was working well at any given moment. The breakage would often be minor nuisances, such as the media controls on my bluetooth headphones breaking in one update and being fixed in the next, or major showstoppers such as broken phone calls, SMS, or, in one case, all of my icons disappearing from the UI (with no fallback in most cases, leaving me navigating the UI blind).

So I tried UBports instead, and despite the general lack of good auxiliary features compared to pmOS, the core telephony was more reliable – for a while. But once issues started to appear, particularly around SMS, I could not tolerate it for long in view of the general uselessness of the OS for anything else. I finally gave it up and installed LineageOS.

Mobile Linux is very cool and the community has made tremendous, unprecedented progress towards realizing its potential, and the forward momentum is still strong. I’m excited to see it continue to improve. But I think that before anyone can be expected to use this as a daily driver, the community really needs to batten down the hatches and focus on one thing and one thing only: always, always being usable as a phone. I’ll be back once more reliability is in place.

How to go to war with your employer

Mon, 12 Jun 2023 00:00:00 +0000

There is a power differential between you and your employer, but that doesn’t mean you can’t improve your working conditions. Today I’d like to offer a little bit of advice on how to frame your relationship with your employer in terms which empower you and afford you more agency. I’m going to talk about the typical working conditions of the average white-collar job in a neo-liberal political environment where you are mostly happy to begin with and financially stable enough to take risks, and I’m specifically going to talk about individual action or the actions of small groups rather than large-scale collective action (e.g. unions).

I wish to subvert the expectation here that employees are subordinate to their employers. A healthy employment relationship between an employee and employer is that of two entities who agree to work together on equal terms to strive towards mutual goals, which in the simplest form is that you both make money and in the subtleties also suggests that you should be happy doing it. The sense of “going to war” here should rouse in you an awareness of the resources at your disposal, a willingness to use them to forward your interests, and an acknowledgement of the fact that tactics, strategy, propaganda, and subterfuge are among the tools you can use – and the tools your employer uses to forward their own interests.

You may suppose that you need your employer more than they need you, but with some basic accounting we can get a better view of the veracity of this supposition. Consider at the most fundamental that your employer is a for-profit entity that spends money to make money, and they spend money on you: as a rule of thumb, they expect a return of at least your salary ×1.5 (accounting for overhead, benefits et al) for their investment in you, otherwise it does not make financial sense for them to employ you.

If you have finer-grained insights into your company’s financial situation, you can get a closer view of your worth to them by dividing their annual profit by their headcount, adjusted to your discretion to account for the difference in the profitability of your role compared to your colleagues. It’s also wise to run this math in your head to see how the returns from your employment are affected by conditions in the hiring market, layoffs, etc – having fewer employees increases the company’s return per employee, and a busier hiring market reduces your leverage. In any case, it should be relatively easy for you to justify, in the cold arithmetic of finance that businesses speak, that employees matter to the employer, and the degree to which solidarity between workers is a meaningful force amplifier for your leverage.

In addition to your fundamental value, there are some weak points in the corporate structure that you should be aware of. There are some big levers that you may already be aware of that I have already placed outside of the scope of this blog post, such as the use of collective bargaining, unionization, strikes, and so on, where you need to maximize your collective leverage to really put the screws to your employer. Many neo-liberal workplaces lack the class consciousness necessary to access these levers, and on the day-to-day scale it may be strategically wise to smarten up your colleagues on social economics in preparation for use of these levers. I want to talk about goals on the smaller scale, though. Suppose your goals are, for instance:

You don’t like agile/scrum and want to interact with it from the other end of a six foot pole and/or replace it with another system
Define your own goals and work on the problems you think are important at your own discretion moreso than at the discretion of your manager
Skip meetings you know are wasting your time
Set working hours that suit you or take time off on your terms
Work from home or in-office in an arrangement that meets your own wants/needs
Exercise agency over your tools, such as installing the software you want to use on your work laptop

You might also have more intimidating goals you want to address:

Demand a raise or renegotiating benefits
Negotiate a 4-day workweek
Replace your manager or move teams
Remove a problematic colleague from your working environment

All of these goals are within your power to achieve, and perhaps more easily than you expect.

First of all, you already have more agency than you know. Your job description and assigned tasks tells a narrow story of your role at the business: your real job is ultimately to make money for the business. If you install Linux on your work laptop because it allows you to work more efficiently, then you are doing your job better and making more money for the business; they have no right to object to this and you have a very defensible position for exercising agency in this respect. Likewise if you adapt the workflows around agile (or whatever) to better suit your needs rather than to fall in line with the prescription, if it makes you more productive and happy then it makes the business more money. Remember your real job – to make money – and you can adjust the parameters of your working environment relatively freely provided that you are still aligned with this goal.

Often you can simply exercise agency in cases like these, but in other cases you may have to reach for your tools. Say you don’t just want to have maintain a personal professional distance from agile, but you want to replace it entirely: now you need to talk to your colleagues. You can go straight to management and start making your case, but another option – probably the more effective one – is to start with your immediate colleagues. Your team also possesses a collective agency, and if you agree together, without anyone’s permission, to work according to your own terms, then so long as you’re all doing your jobs – making money – then no one is going to protest. This is more effective than following the chain of command and asking them to take risks they don’t understand. Be aware of the importance of optics here: you need not only to make money, but to be seen making money. How you are seen to be doing this may depend on how far up the chain you need to justify yourself to; if your boss doesn’t like it then make sure your boss’s boss does.

Ranked in descending order of leverage within the business: your team, your boss, you.

More individual-oriented goals such as negotiating a different working schedule or skipping meetings calls for different tools. Simple cases, such as coming in at ten and leaving at four every day, are a case of simple exercise of agency; so long as you’re making the company money no one is going to raise a fuss. If you want, for instance, a four day work-week, or to work from home more often, you may have to justify yourself to someone. In such cases you may be less likely to have your team’s solidarity at your disposal, but if you’re seen to be doing your job – making money – then a simple argument that it makes you better at that job will often suffice.

You can also be clever. “Hey, I’ll be working from home on Friday” works better than “can I work from home on Friday?” If you want to work from home every Friday, however, then you can think strategically: keeping mum about your final goal of taking all Fridays from home may be wise if you can start by taking some Fridays at home to establish that you’re still productive and fulfilling the prime directive¹ under those terms and allow yourself to “accidentally” slip into a new normal of working home every Friday without asking until it’s apparent that the answer will be yes. Don’t be above a little bit of subversion and deception; your employer is using those tools against you too.

Then there are the big guns: human resources. HR is the enemy; their job is to protect the company from you. They can, however, be useful if you understand the risks they’re trying to manage and press the right buttons with them. If your manager is a dick, HR may be the tool to use to fix this, but you need to approach it the right way. HR does not give two fucks that you don’t like your manager, if your manager is making money then they are doing their job. What HR does give a fuck about is managing the company’s exposure to lawsuits.

They can also make your life miserable. If HR does not like you then you are going to suffer, so when you talk to them it is important to know your enemy and to make strategic use of them without making them realize you know the game. They present themselves as your ally, let them think you believe it’s so. At the same time, there is a coded language you can use that will get them to act in your interest. HR will perk up as soon as they smell “unsafe working conditions”, “sexual harassment”, “collective action”, and so on – the risks they were hired to manage – over the horizon. The best way to interact with HR is for them to conclude that you are on a path which ends in these problems landing on their desk without making them think you are a subversive element within the organization. And if you are prepared to make your knowledge of and willingness to use these tools explicit, all communication which suggests as much should be delivered to HR with your lawyer’s signature and only when you have a new job offer lined up as a fallback. HR should either view you as mostly harmless or look upon you with fear, but nothing in between.

These are your first steps towards class consciousness as a white-collar employee. Know your worth, know the leverage you have, and be prepared to use the tools at your disposal to bring about the outcomes you desire, and know your employer will be doing the same. Good luck out there, and don’t forget to actually write some code or whatever when you’re not busy planning a corporate coup.

Burnout

Mon, 01 May 2023 00:00:00 +0000

It kind of crept up on me. One day, sitting at my workstation, I stopped typing, stared blankly at the screen for a few seconds, and a switch flipped in my head.

On the night of New Year’s Eve, my backpack was stolen from me on the train from Berlin to Amsterdam, and with it about $2000 worth of equipment, clothes, and so on. A portent for the year that was to come. I generally keep my private and public lives carefully separated, but perhaps I will offer you a peek behind the curtain today.

It seems like every week or two this year, another crisis presented itself, each manageable in isolation. Some were independent events, others snowballed as the same problems escalated. Gossip at the hackerspace, my personal life put on display and mocked. A difficult break-up in February, followed by a close friend facing their own relationship’s hurtful end. Another close friend – old, grave problems, once forgotten, remembered, and found to still be causing harm. Yet another friend, struggling to deal with depression and emotional abuse at the hands of their partner. Another friendship still: lost, perhaps someday to be found again.

Dependable Drew, an ear to listen, a shoulder to cry on, always knowing the right words to say, ready to help and proud to be there for his friends. Friends who, amidst these crises, are struggling to be there for him.

These events, set over the background of a world on fire.

One of the more difficult crises in my purview reached its crescendo one week ago, culminating in death. A selfish end for a selfish person, a person who had hurt people I love; a final, cruel cut to the wounds we were trying to heal.

I took time for myself throughout these endless weeks, looked after myself as best I could, and allowed my productivity to wane as necessary, unburdened by guilt in so doing. I marched on when I had the energy to, and made many achievements I’m proud of.

Something changed this week. I have often remarked that when you’re staring down a hard problem, one which might take years or even decades to finish, that you have two choices: give up or get to work. The years are going to pass either way. I am used to finding myself at the base of a mountain, picking up my shovel, and getting started. Equipped with this mindset, I have patiently ground down more than one mountain in my time. But this week, for the first time in my life, as I gazed upon that mountain, I felt intimidated.

I’m not sure what the purpose of this blog post is. Perhaps I’m sharing an experience that others might be able to relate to. Perhaps it’s healing in some way. Maybe it’s just indulgent.

I’m going to take the time I need to rest. I enjoy the company of wonderful colleagues at SourceHut, who have been happy to pick up some of the slack. I have established a formal group of maintainers for Hare and given them my blessing to work without seeking my approval. My projects will remain healthy as I take a leave. See you soon.

Who should lead us?

Mon, 24 Apr 2023 00:00:00 +0000

Consider these two people, each captured in the midst of delivering a technical talk.

Based on appearances alone, what do you think of them?

The person on the left is a woman. She’s also pretty young, one might infer something about her level of experience accordingly. I imagine that she has led a much different life than I have, and may have a much different perspective, worldview, identity, and politics than I. Does she complain about sexism and discrimination in her work? Is she a feminist? Does she lean left or right on the political spectrum?

The person on the right looks like most of the hackers I’ve met. You’ve met someone who looks like this a thousand times. He is a man, white and middle-aged – that suggests a fair bit of experience. He probably doesn’t experience or concern himself with race or gender discrimination in the course of his work. He just focuses on the software. His life experiences probably map relatively well onto my own, and we may share a similar worldview and identity.

Making these assumptions is a part of human nature – it’s a useful shortcut in many situations. But they are assumptions based only on appearances. What are the facts?

The person on the right is Scott Guthrie, Vice President of Cloud and AI at Microsoft, giving a talk about Azure’s cloud services. He lives in an $11M house in Hunts Point, Washington. On the left is Alyssa Rosenzweig, main developer for the free software Panfrost GPU drivers and a trans woman, talking about how she reverse engineers proprietary graphics hardware.

You and I have a lot more in common with Alyssa than with Scott. The phone I have in my pocket right now would not work without her drivers. Alyssa humbles me with her exceptional talent and dedication, and the free software community is indebted to her. If you use ARM devices with free software, you owe something to Alyssa. As recently as February, her Wikipedia page was vandalized by someone who edited “she” and “her” to “he” and “him”.

Appearances should not especially matter when considering the merit of someone considered for a leadership role in our community, be it as a maintainer, thought leader, member of our foundations’ boards, etc. I am myself a white man, and I think I perform well in my leadership roles throughout the free software ecosystem. But it’s not my appearance that causes any controversy: someone with the approximate demographic shape of myself or Guthrie would cause no susurration when taking the stage.

It’s those like Alyssa, who aside from anything else is eminently qualified and well-deserving of her leadership role, who are often the target of ire and discrimination in the community. This is an experience shared by many people whose gender expression, skin color, or other traits differ from the “norm”. They’ve been telling us so for years.

Is it any wonder that our community is predominantly made up of white cisgendered men when anyone else is ostracized? It’s not because we’re predisposed to be better at this kind of work. It’s patently absurd to suppose that hackers whose identities and life experience differ from yours or mine cannot be good participants in and leaders of our movement. In actual fact, diverse teams produce better results. While the labor pool is disproportionately filled with white men, we can find many talented hackers who cannot be described as such. If we choose to be inspired by them, and led by them, we will discover new perspectives on our software, and on our movement and its broader place in the world. They can help us create a safe and inviting space for other talented hackers who identify with them. We will be more effective at our mission of bringing free software to everyone with their help.

Moreover, there are a lot of damned good hackers who don’t look like me, and I would be happy to follow their lead regardless of any other considerations.

The free software ecosystem (and the world at large) is not under threat from some woke agenda – a conspiracy theory which has been fabricated out of whole cloth. The people you fear are just people, much like you and I, and they only want to be treated as such. Asking them to shut up and get in line, to suppress their identity, experiences, and politics, to avoid confronting you with uncomfortable questions about your biases and privileges by way of their existence alone – it’s not right.

Forget the politics and focus on the software? It’s simply not possible. Free software is politics. Treating other people with respect, maturity, and professionalism, and valuing their contributions at any level, including leadership, regardless of their appearance or identity – that’s just part of being a good person. That is apolitical.

Alyssa gave her blessing regarding the use of her image and her example in this post. Thanks!

rc: a new shell for Unix

Tue, 18 Apr 2023 00:00:00 +0000

rc is a Unix shell I’ve been working on over the past couple of weeks, though it’s been in the design stages for a while longer than that. It’s not done or ready for general use yet, but it is interesting, so let’s talk about it.

As the name (which is subject to change) implies, rc is inspired by the Plan 9 rc shell. It’s not an implementation of Plan 9 rc, however: it departs in many notable ways. I’ll assume most readers are more familiar with POSIX shell or Bash and skip many of the direct comparisons to Plan 9. Also, though most of the features work as described, the shell is a work-in-progress and some of the design I’m going over today has not been implemented yet.

Let’s start with the basics. Simple usage works much as you’d expect:

name=ddevault
echo Hello $name

But there’s already something important that might catch your eye here: the lack of quotes around $name. One substantial improvement rc makes over POSIX shells and Bash right off the bat is fixing our global shell quoting nightmare. There’s no need to quote variables!

# POSIX shell
x="hello world"
printf '%s\n' $x
# hello
# world

# rc
x="hello world"
printf '%s\n' $x
# hello world

Of course, the POSIX behavior is actually useful sometimes. rc provides for this by acknowledging that shells have not just one fundamental type (strings), but two: strings and lists of strings, i.e. argument vectors.

x=(one two three)
echo $x(1)  # prints first item ("one")
echo $x     # expands to arguments (echo "one" "two" "three")
echo $#x    # length operator: prints 3

x="echo hello world"
$x
# echo hello world: command not found

x=(echo hello world)
$x
# hello world

# expands to a string, list values separated with space:
$"x
# echo hello world: command not found

You can also slice up lists and get a subset of items:

x=(one two three four five)
echo $x(-4) # one two three four
echo $x(2-) # two three four five
echo $x(2-4) # two three four

A departure from Plan 9 rc is that the list operators can be used with strings for string operations as well:

x="hello world"
echo $#x     # 11
echo $x(2)   # e
echo $x(1-5) # hello

rc also supports loops. The simple case is iterating over the command line arguments:

% cat test.rc 
for (arg) {
	echo $arg
}
% rc test.rc one two three 
one
two
three

{ } is a command like any other; this can be simplified to for (arg) echo $arg. You can also enumerate any list with in:

list=(one two three)
for (item in $list) {
	echo $item
}

We also have while loops and if:

while (true) {
	if (test $x -eq 10) {
		echo ten
	} else {
		echo $x
	}
}

Functions are defined like so:

fn greet {
	echo Hello $1
}

greet ddevault

Again, any command can be used, so this can be simplified to fn greet echo $1. You can also add named parameters:

fn greet(user time) {
	echo Hello $user
	echo It is $time
}

greet ddevault `{date}

Note the use of `{script…} instead of $() for command expansion. Additional arguments are still placed in $*, allowing for the user to combine variadic-style functions with named arguments.

Here’s a more complex script that I run to perform sanity checks before applying patches:

#!/bin/rc
fn check_branch(branch) {
	if (test `{git rev-parse --abbrev-ref HEAD} != $branch) {
		echo "Error: not on master branch"
		exit 1
	}
}

fn check_uncommitted {
	if (test `{git status -suno | wc -l} -ne 0) {
		echo "Error: you have uncommitted changes"
		exit 1
	}
}

fn check_behind {
	if (test `{git rev-list "@{u}.." | wc -l} -ne 0) {
		echo "Error: your branch is behind upstream"
		exit 1
	}
}

check_branch master
check_uncommitted
check_behind
exec git pull

That’s a brief introduction to rc! Presently it clocks in at about 2500 lines of Hare. It’s not done yet, so don’t get too excited, but much of what’s described here is already working. Some other stuff which works but I didn’t mention include:

Boolean compound commands (x && y, x || y)
Pipelines, which can pipe arbitrary file descriptors (“x |[2] y”)
Redirects, also including arbitrary fds (“x >[2=1] file”)

It also has a formal context-free grammar, which is a work-in-progress but speaks to our desire to have a robust description of the shell available for users and other implementations. We use Ember Sawady’s excellent madeline for our interactive mode, which supports command line editing, history, ^r, and fish-style forward completion OOTB.

Future plans include:

Simple arithmetic expansion
Named pipe expansions
Sub-shells
switch statements
Port to ares
Find a new name, perhaps

It needs a small amount of polish, cleanup, and bugs fixed as well.

I hope you find it interesting! I will let you know when it’s done. Feel free to play with it in the meanwhile, and maybe send some patches?

The Free Software Foundation is dying

Tue, 11 Apr 2023 00:00:00 +0000

The Free Software Foundation is one of the longest-running missions in the free software movement, effectively defining it. It provides a legal foundation for the movement and organizes activism around software freedom. The GNU project, closely related, has its own long story in our movement as the coding arm of the Free Software Foundation, taking these principles and philosophy into practice by developing free software; notably the GNU operating system that famously rests atop GNU/Linux.

Today, almost 40 years on, the FSF is dying.

Their achievements are unmistakable: we must offer them our gratitude and admiration for decades of accomplishments in establishing and advancing our cause. The principles of software freedom are more important than ever, and the products of these institutions remain necessary and useful – the GPL license family, GCC, GNU coreutils, and so on. Nevertheless, the organizations behind this work are floundering.

The Free Software Foundation must concern itself with the following ahead of all else:

Disseminating free software philosophy
Developing, publishing, and promoting copyleft licenses
Overseeing the health of the free software movement

It is failing in each of these regards, and as its core mission fails, the foundation is investing its resources into distractions.

In its role as the thought-leaders of free software philosophy, the message of the FSF has a narrow reach. The organization’s messaging is tone-deaf, ineffective, and myopic. Hammering on about “GNU/Linux” nomenclature, antagonism towards our allies in the open source movement, maligning the audience as “useds” rather than “users”; none of this aids the cause. The pages and pages of dense philosophical essays and poorly organized FAQs do not provide a useful entry point or reference for the community. The message cannot spread like this.

As for copyleft, well, it’s no coincidence that many people struggle with the FSF’s approach. Do you, dear reader, know the difference between free software and copyleft? Many people assume that the MIT license is not free software because it’s not viral. The GPL family of licenses are essential for our movement, but few people understand its dense and esoteric language, despite the 16,000-word FAQ which supplements it. And hip new software isn’t using copyleft: over 1 million npm packages use a permissive license while fewer than 20,000 use the GPL; cargo sports a half-million permissive packages and another 20,000 or so GPL’d.

And is the free software movement healthy? This one gets an emphatic “yes!” – thanks to the open source movement and the near-equivalence between free software and open source software. There’s more free software than ever and virtually all new software contains free software components, and most people call it open source.

The FOSS community is now dominated by people who are beyond the reach of the FSF’s message. The broader community is enjoying a growth in the diversity of backgrounds and values represented, and the message does not reach these people. The FSF fails to understand its place in the world as a whole, or its relationship to the progressive movements taking place in the ecosystem and beyond. The foundation does not reach out to new leaders in the community, leaving them to form insular, weak institutions among themselves with no central leadership, and leaving us vulnerable to exploitation from growing movements like open core and commercial attacks on the free and open source software brand.

Reforms are sorely needed for the FSF to fulfill it basic mission. In particular, I call for the following changes:

Reform the leadership. It’s time for Richard Stallman to go. His polemeic rhetoric rivals even my own, and the demographics he represents – to the exclusion of all others – is becoming a minority within the free software movement. We need more leaders of color, women, LGBTQ representation, and others besides. The present leadership, particularly from RMS, creates an exclusionary environment in a place where inclusion and representation are important for the success of the movement.
Reform the institution. The FSF needs to correct its myopic view of the ecosystem, reach out to emerging leaders throughout the FOSS world, and ask them to take charge of the FSF’s mission. It’s these leaders who hold the reins of the free software movement today – not the FSF. If the FSF still wants to be involved in the movement, they need to recognize and empower the leaders who are pushing the cause forward.
Reform the message. People depend on the FSF to establish a strong background in free software philosophy and practices within the community, and the FSF is not providing this. The message needs to be made much more accessible and level in tone, and the relationship between free software and open source needs to be reformed so that the FSF and OSI stand together as the pillars at the foundations of our ecosystem.
Decouple the FSF from the GNU project. FSF and GNU have worked hand-in-hand over decades to build the movement from scratch, but their privileged relationship has become obsolete. The GNU project represents a minute fraction of the free software ecosystem today, and it’s necessary for the Free Software Foundation to stand independently of any particular project and focus on the health of the ecosystem as a whole.
Develop new copyleft licenses. The GPL family of licenses has served us well, but we need to do better. The best copyleft license today is the MPL, whose terse form and accessible language outperforms the GPL in many respects. However, it does not provide a comprehensive answer to the needs of copyleft, and new licenses are required to fill other niches in the market – the FSF should write these licenses. Furthermore, the FSF should present the community with a free software perspective on licenses as a resource that project leaders can depend on to understand the importance of their licensing choice such that they understand the appeal of copyleft licenses without feeling pushed away from permissive approaches.

The free software movement needs a strong force uniting it: we face challenges from many sides, and today’s Free Software Foundation is not equal to the task. The FOSS ecosystem is flourishing, and it’s time for the FSF to step up to the wheel and direct its coming successes in the name of software freedom.

Writing Helios drivers in the Mercury driver environment

Sat, 08 Apr 2023 00:00:00 +0000

Helios is a microkernel written in the Hare programming language and is part of the larger Ares operating system. You can watch my FOSDEM 2023 talk introducing Helios on PeerTube.

Let’s take a look at the new Mercury driver development environment for Helios.

As you may remember from my FOSDEM talk, the Ares operating system is built out of several layers which provide progressively higher-level environments for an operating system. At the bottom is the Helios microkernel, and today we’re going to talk about the second layer: the Mercury environment, which is used for writing and running device drivers in userspace. Let’s take a look at a serial driver written against Mercury and introduce some of the primitives used by driver authors in the Mercury environment.

Drivers for Mercury are written as normal ELF executables with an extra section called .manifest, which includes a file similar to the following (the provided example is for the serial driver we’ll be examining today):

[driver]
name=pcserial
desc=Serial driver for x86_64 PCs

[capabilities]
0:ioport = min=3F8, max=400
1:ioport = min=2E8, max=2F0
2:note = 
3:irq = irq=3, note=2
4:irq = irq=4, note=2
_:cspace = self
_:vspace = self
_:memory = pages=32

[services]
devregistry=

Helios uses a capability-based design, in which access to system resources (such as I/O ports, IRQs, or memory) is governed by capability objects. Each process has a capability space, which is a table of capabilities assigned to that process, and when performing operations (such as writing to an I/O port) the user provides the index of the desired capability in a register when invoking the appropriate syscall.

The manifest first specifies a list of capabilities required to operate the serial port. It requests, assigned static capability addresses, capabilities for the required I/O ports and IRQs, as well as a notification object which the IRQs will be delivered to. Some capability types, such as I/O ports, have configuration parameters, in this case the minimum and maximum port numbers which are relevant. The IRQ capabilities require a reference to a notification as well.

Limiting access to these capabilities provides very strong isolation between device drivers. On a monolithic kernel like Linux, a bug in the serial driver could compromise the entire system, but a vulnerability in our driver could, at worst, write garbage to your serial port. This model also provides better security than something like OpenBSD’s pledge by declaratively specifying what we need and nothing else.

Following the statically allocated capabilities, we request our own capability space and virtual address space, the former so we can copy and destroy our capabilities, and the latter so that we can map shared memory to perform reads and writes for clients. We also request 32 pages of memory, which we use to allocate page tables to perform those mappings; this will be changed later. These capabilities do not require any specific address for the driver to work, so we use “_” to indicate that any slot will suit our needs.

Mercury uses some vendor extensions over the System-V ABI to communicate information about these capabilities to the runtime. Notes about each of the _’d capabilities are provided by the auxiliary vector, and picked up by the Mercury runtime – for instance, the presence of a memory capability is detected on startup and is used to set up the allocator; the presence of a vspace capability is automatically wired up to the mmap implementation.

Each of these capabilities is implemented by the kernel, but additional services are available in userspace via endpoint capabilities. Each of these endpoints implements a particular API, as defined by a protocol definition file. This driver requires access to the device registry, so that it can create devices for its serial ports and expose them to clients.

These protocol definitions are written in a domain-specific language and parsed by ipcgen to generate client and server implementations of each. Here’s a simple protocol to start us off:

namespace io;

# The location with respect to which a seek operation is performed.
enum whence {
	# From the start of the file
	SET,
	# From the current offset
	CUR,
	# From the end of the file
	END,
};

# An object with file-like semantics.
interface file {
	# Reads up to amt bytes of data from a file.
	call read{pages: page...}(buf: uintptr, amt: size) size;

	# Writes up to amt bytes of data to a file.
	call write{pages: page...}(buf: uintptr, amt: size) size;

	# Seeks a file to a given offset, returning the new offset.
	call seek(offs: i64, w: whence) size;
};

Each interface includes a list of methods, each of which can take a number of capabilities and parameters, and return a value. The “read” call here, when implemented by a file-like object, accepts a list of memory pages to perform the read or write with (shared memory), as well as a pointer to the buffer address and size. Error handling is still a to-do.

ipcgen consumes these files and writes client or server code as appropriate. These are generated as part of the Mercury build process and end up in *_gen.ha files. The generated client code is filed away into the relevant modules (this protocol ends up at io/file_gen.ha), alongside various hand-written files which provide additional functionality and often wrap the IPC calls in a higher-level interface. The server implementations end up in the “serv” module, e.g. serv/io/file_gen.ha.

Let’s look at some of the generated client code for io::file objects:

// This file was generated by ipcgen; do not modify by hand
use helios;
use rt;

// ID for the file IPC interface.
export def FILE_ID: u32 = 0x9A533BB3;

// Labels for operations against file objects.
export type file_label = enum u64 {
	READ = FILE_ID << 16u64 | 1,
	WRITE = FILE_ID << 16u64 | 2,
	SEEK = FILE_ID << 16u64 | 3,
};

export fn file_read(
	ep: helios::cap,
	pages: []helios::cap,
	buf: uintptr,
	amt: size,
) size = {
	// ...
};

Each interface has a unique ID (generated from the FNV-1a hash of its fully qualified name), which is bitwise-OR’d with a list of operations to form call labels. The interface ID is used elsewhere; we’ll refer to it again later. Then each method generates an implementation which arranges the IPC details as necessary and invokes the “call” syscall against the endpoint capability.

The generated server code is a bit more involved. Some of the details are similar – FILE_ID is generated again, for instance – but there are some additional details as well. First is the generation of a vtable defining the functions implementing each operation:

// Implementation of a [[file]] object.
export type file_iface = struct {
	read: *fn_file_read,
	write: *fn_file_write,
	seek: *fn_file_seek,
};

We also define a file object which is subtyped by the implementation to store implementation details, and which provides to the generated code the required bits of state.

// Instance of an file object. Users may subtype this object to add
// instance-specific state.
export type file = struct {
	_iface: *file_iface,
	_endpoint: helios::cap,
};

Here’s an example of a subtype of file used by the initramfs to store additional state:

// An open file in the bootstrap filesystem
type bfs_file = struct {
	serv::io::file,
	fs: *bfs,
	ent: tar::entry,
	cur: io::off,
	padding: size,
};

The embedded serv::io::file structure here is populated with an implementation of file_iface, here simplified for illustrative purposes:

const bfs_file_impl = serv_io::file_iface {
	read = &bfs_file_read,
	write = &bfs_file_write,
	seek = &bfs_file_seek,
};

fn bfs_file_read(
	obj: *serv_io::file,
	pages: []helios::cap,
	buf: uintptr,
	amt: size,
) size = {
	let file = obj: *bfs_file;
	const fs = file.fs;
	const offs = (buf & rt::PAGEMASK): size;
	defer helios::destroy(pages...)!;

	assert(offs + amt <= len(pages) * rt::PAGESIZE);
	const buf = helios::map(rt::vspace, 0, map_flags::W, pages...)!: *[*]u8;

	let buf = buf[offs..offs+amt];
	// Not shown: reading the file data into this buffer
};

The implementation can prepare a file object and call dispatch on it to process client requests: this function blocks until a request arrives, decodes it, and invokes the appropriate function. Often this is incorporated into an event loop with poll to service many objects at once.

// Prepare a file object
const ep = helios::newendpoint()!;
append(fs.files, bfs_file {
	_iface = &bfs_file_impl,
	_endpoint = ep,
	fs = fs,
	ent = ent,
	cur = io::tell(fs.buf)!,
	padding = fs.rd.padding,
});

// ...

// Process requests associated with this file
serv::io::file_dispatch(file);

Okay, enough background: back to the serial driver. It needs to implement the following protocol:

namespace dev;
use io;

# TODO: Add busy error and narrow semantics

# Note: TWO is interpreted as 1.5 for some char lengths (5)
enum stop_bits {
	ONE,
	TWO,
};

enum parity {
	NONE,
	ODD,
	EVEN,
	MARK,
	SPACE,
};

# A serial device, which implements the file interface for reading from and
# writing to a serial port. Typical implementations may only support one read
# in-flight at a time, returning errors::busy otherwise.
interface serial :: io::file {
	# Returns the baud rate in Hz.
	call get_baud() uint;

	# Returns the configured number of bits per character.
	call get_charlen() uint;

	# Returns the configured number of stop bits.
	call get_stopbits() stop_bits;

	# Returns the configured parity setting.
	call get_parity() parity;

	# Sets the baud rate in Hz.
	call set_baud(hz: uint) void;

	# Sets the number of bits per character. Must be 5, 6, 7, or 8.
	call set_charlen(bits: uint) void;

	# Configures the number of stop bits to use.
	call set_stopbits(bits: stop_bits) void;

	# Configures the desired parity.
	call set_parity(parity: parity) void;
};

This protocol inherits the io::file interface, so the serial port is usable like any other file for reads and writes. It additionally defines serial-specific methods, such as configuring the baud rate or parity. The generated interface we’ll have to implement looks something like this, embedding the io::file_iface struct:

export type serial_iface = struct {
	io::file_iface,
	get_baud: *fn_serial_get_baud,
	get_charlen: *fn_serial_get_charlen,
	get_stopbits: *fn_serial_get_stopbits,
	get_parity: *fn_serial_get_parity,
	set_baud: *fn_serial_set_baud,
	set_charlen: *fn_serial_set_charlen,
	set_stopbits: *fn_serial_set_stopbits,
	set_parity: *fn_serial_set_parity,
}

Time to dive into the implementation. Recall the driver manifest, which provides the serial driver with a suitable environment:

[driver]
name=pcserial
desc=Serial driver for x86_64 PCs

[capabilities]
0:ioport = min=3F8, max=400
1:ioport = min=2E8, max=2F0
2:note = 
3:irq = irq=3, note=2
4:irq = irq=4, note=2
_:cspace = self
_:vspace = self
_:memory = pages=32

[services]
devregistry=

I/O ports for reading and writing to the serial devices, IRQs for receiving serial-related interrupts, a device registry to add our serial devices to the system, and a few extra things for implementation needs. Some of these are statically allocated, some of them are provided via the auxiliary vector. Our serial driver opens by defining constants for the statically allocated capabilities:

def IOPORT_A: helios::cap = 0;
def IOPORT_B: helios::cap = 1;
def IRQ: helios::cap = 2;
def IRQ3: helios::cap = 3;
def IRQ4: helios::cap = 4;

The first thing we do on startup is create a serial device.

export fn main() void = {
	let serial0: helios::cap = 0;
	const registry = helios::service(sys::DEVREGISTRY_ID);
	sys::devregistry_new(registry, dev::SERIAL_ID, &serial0);
	helios::destroy(registry)!;
	// ...

The device registry is provided via the aux vector, and we can use helios::service to look it up by its interface ID. Then we use the devregistry::new operation to create a serial device:

# Device driver registry.
interface devregistry {
	# Creates a new device implementing the given interface ID using the
	# provided endpoint capability and returns its assigned serial number.
	call new{; out}(iface: u64) uint;
};

After this we can destroy the registry – we won’t need it again and it’s best to get rid of it so that we can work with the minimum possible privileges at runtime. After this we initialize the serial port, acknowledge any interrupts that might have been pending before we got started, an enter the main loop.

com_init(&ports[0], serial0);

helios::irq_ack(IRQ3)!;
helios::irq_ack(IRQ4)!;

let poll: [_]pollcap = [
	pollcap { cap = IRQ, events = pollflags::RECV, ... },
	pollcap { cap = serial0, events = pollflags::RECV, ... },
];
for (true) {
	helios::poll(poll)!;
	if (poll[0].revents & pollflags::RECV != 0) {
		dispatch_irq();
	};
	if (poll[1].revents & pollflags::RECV != 0) {
		dispatch_serial(&ports[0]);
	};
};

The dispatch_serial function is of interest, as this provides the implementation of the serial object we just created with the device registry.

type comport = struct {
	dev::serial,
	port: u16,
	rbuf: [4096]u8,
	wbuf: [4096]u8,
	rpending: []u8,
	wpending: []u8,
};

fn dispatch_serial(dev: *comport) void = {
	dev::serial_dispatch(dev);
};

const serial_impl = dev::serial_iface {
	read = &serial_read,
	write = &serial_write,
	seek = &serial_seek,
        get_baud = &serial_get_baud,
        get_charlen = &serial_get_charlen,
        get_stopbits = &serial_get_stopbits,
        get_parity = &serial_get_parity,
        set_baud = &serial_set_baud,
        set_charlen = &serial_set_charlen,
        set_stopbits = &serial_set_stopbits,
        set_parity = &serial_set_parity,
};

fn serial_read(
	obj: *io::file,
	pages: []helios::cap,
	buf: uintptr,
	amt: size,
) size = {
	const port = obj: *comport;
	const offs = (buf & rt::PAGEMASK): size;
	const buf = helios::map(rt::vspace, 0, map_flags::W, pages...)!: *[*]u8;
	const buf = buf[offs..offs+amt];

	if (len(port.rpending) != 0) {
		defer helios::destroy(pages...)!;
		return rconsume(port, buf);
	};

	pages_static[..len(pages)] = pages[..];
	pending_read = read {
		reply = helios::store_reply(helios::CADDR_UNDEF)!,
		pages = pages_static[..len(pages)],
		buf = buf,
	};
	return 0;
};

// (other functions omitted)

We’ll skip much of the implementation details for this specific driver, but I’ll show you how read works at least. It’s relatively straightforward: first we mmap the buffer provided by the caller. If there’s already readable data pending from the serial port (stored in that rpending slice in the comport struct, which is a slice of the statically-allocated rbuf field), we copy it into the buffer and return the number of bytes we had ready. Otherwise, we stash details about the caller, storing the special reply capability in our cspace (this is one of the reasons we need cspace = self in our manifest) so we can reply to this call once data is available. Then we return to the main loop.

The main loop also wakes up on an interrupt, and we have an interrupt unmasked on the serial device to wake us whenever there’s data ready to be read. Eventually this gets us here, which finishes the call we saved earlier:

// Reads data from the serial port's RX FIFO.
fn com_read(com: *comport) size = {
	let n: size = 0;
	for (comin(com.port, LSR) & RBF == RBF; n += 1) {
		const ch = comin(com.port, RBR);
		if (len(com.rpending) < len(com.rbuf)) {
			// If the buffer is full we just drop chars
			static append(com.rpending, ch);
		};
	};

	if (pending_read.reply != 0) {
		const n = rconsume(com, pending_read.buf);
		helios::send(pending_read.reply, 0, n)!;
		pending_read.reply = 0;
		helios::destroy(pending_read.pages...)!;
	};

	return n;
};

I hope that gives you a general idea of how drivers work in this environment! I encourage you to read the full implementation if you’re curious to know more about the serial driver in particular – it’s just 370 lines of code.

The last thing I want to show you is how the driver gets executed in the first place. When Helios boots up, it starts /sbin/sysinit, which is provided by Mercury and offers various low-level userspace runtime services, such as the device registry and bootstrap filesystem we saw earlier. After setting up its services, sysinit executes /sbin/usrinit, which is provided by the next layer up (Gaia, eventually) and sets up the rest of the system according to user policy, mounting filesystems and starting up drivers and such. At the moment, usrinit is fairly simple, and just runs a little demo. Here it is in full:

use dev;
use fs;
use helios;
use io;
use log;
use rt;
use sys;

export fn main() void = {
	const fs = helios::service(fs::FS_ID);
	const procmgr = helios::service(sys::PROCMGR_ID);
	const devmgr = helios::service(sys::DEVMGR_ID);
	const devload = helios::service(sys::DEVLOADER_ID);

	log::printfln("[usrinit] Running /sbin/drv/serial");
	let proc: helios::cap = 0;
	const image = fs::open(fs, "/sbin/drv/serial")!;
	sys::procmgr_new(procmgr, &proc);
	sys::devloader_load(devload, proc, image);
	sys::process_start(proc);

	let serial: helios::cap = 0;
	log::printfln("[usrinit] open device serial0");
	sys::devmgr_open(devmgr, dev::SERIAL_ID, 0, &serial);

	let buf: [rt::PAGESIZE]u8 = [0...];
	for (true) {
		const n = match (io::read(serial, buf)!) {
		case let n: size =>
			yield n;
		case io::EOF =>
			break;
		};

		// CR => LF
		for (let i = 0z; i < n; i += 1) {
			if (buf[i] == '\r') {
				buf[i] = '\n';
			};
		};

		// echo
		io::write(serial, buf[..n])!;
	};
};

Each of the services shown at the start are automatically provided in usrinit’s aux vector by sysinit, and includes all of the services required to bootstrap the system. This includes a filesystem (the initramfs), a process manager (to start up new processes), the device manager, and the driver loader service.

usrinit starts by opening up /sbin/drv/serial (the serial driver, of course) from the provided initramfs using fs::open, which is a convenience wrapper around the filesystem protocol. Then we create a new process with the process manager, which by default has an empty address space – we could load a normal process into it with sys::process_load, but we want to load a driver, so we use the devloader interface instead. Then we start the process and boom: the serial driver is online.

The serial driver registers itself with the device registry, which means that we can use the device manager to open the 0th device which implements the serial interface. Since this is compatible with the io::file interface, it can simply be used normally with io::read and io::write to utilize the serial port. The main loop simply echos data read from the serial port back out. Simple!

That’s a quick introduction to the driver environment provided by Mercury. I intend to write a few more drivers soon myself – PC keyboard, framebuffer, etc – and set up a simple shell. We have seen a few sample drivers written pre-Mercury which would be nice to bring into this environment, such as virtio networking and block devices. It will be nice to see them re-introduced in an environment where they can provide useful services to the rest of userspace.

If you’re interested in learning more about Helios or Mercury, consult ares-os.org for documentation – though beware of the many stub pages. If you have any questions or want to get involved in writing some drivers yourself, jump into our IRC channel: #helios on Libera Chat.

When to comment that code

Thu, 09 Mar 2023 00:00:00 +0000

My software tends to have a surprisingly low number of comments. One of my projects, scdoc, has 25 comments among its 1,133 lines of C code, or 2%, compared to the average of 19%.¹ Naturally, I insist that my code is well-written in spite of this divergence from the norm. Allow me to explain.

The philosophy and implementation of code comments varies widely in the industry, and some view comment density as a proxy for code quality.² I’ll state my views here, but will note that yours may differ and I find that acceptable; I am not here to suggest that your strategy is wrong and I will happily adopt it when I write a patch for your codebase.

Let’s begin with an illustrative example from one of my projects:

// Reads the next entry from an EFI [[FILE_PROTOCOL]] handle of an open
// directory. The return value is statically allocated and will be overwritten
// on the next call.
export fn readdir(dir: *FILE_PROTOCOL) (*FILE_INFO | void | error) = {
	// size(FILE_INFO) plus reserve up to 512 bytes for file name (FAT32
	// maximum, times two for wstr encoding)
	static let buf: [FILE_INFO_SIZE + 512]u8 = [0...];
	const n = read(dir, buf)?;
	if (n == 0) {
		return;
	};
	return &buf[0]: *FILE_INFO;
};

This code illustrates two of my various approaches to writing comments. The first comment is a documentation comment: the intended audience is the consumer of this API. The call-site has access to the following information:

This comment
The name of the function, and the module in which it resides (efi::readdir)
The parameter names and types
The return type

The goal is for the user of this function to gather enough information from these details to correctly utilize this API.

The module in which it resides suggests that this function interacts with the EFI (Extensible Firmware Interface) standard, and the user would be wise to pair a reading of this code (or API) with skimming the relevant standard. Indeed, the strategic naming of the FILE_PROTOCOL and FILE_INFO types (notably written in defiance of the Hare style guide), provide hints to the relevant parts of the EFI specification to read for a complete understanding of this code.

The name of the function is also carefully chosen to carry some weight: it is a reference to the Unix readdir function, which brings with it an intuition about its purpose and usage for programmers familiar with a Unix environment.

The return type also provides hints about the function’s use: it may return either a FILE_INFO pointer, void (nothing), or an error. Without reading the documentation string, and taking the name and return type into account, we might (correctly) surmise that we need to call this function repeatedly to read file details out of a directory until it returns void, indicating that all entries have been processed, handling any errors which might occur along the way.

We have established a lot of information about this function without actually reading the comment; in my philosophy of programming I view this information as a critical means for the author to communicate to the user, and we can lean on it to reduce the need for explicit documentation. Nevertheless, the documentation comment adds something here. The first sentence is a relatively information-sparse summary of the function’s purpose, and mainly exists to tick a box in the Hare style guide.³ The second sentence is the only real reason this comment exists: to clarify an important detail for the user which is not apparent from the function signature, namely the storage semantics associated with the return value.

Let’s now study the second comment’s purpose:

// size(FILE_INFO) plus reserve up to 512 bytes for file name (FAT32
// maximum, times two for wstr encoding)
static let buf: [FILE_INFO_SIZE + 512]u8 = [0...];

This comment exists to explain the use of the magic constant of 512. The audience of this comment is someone reading the implementation of this function. This audience has access to a different context than the user of the function, for instance they are expected to have a more comprehensive knowledge of EFI and are definitely expected to be reading the specification to a much greater degree of detail. We can and should lean on that context to make our comments more concise and useful.

An alternative writing which does not rely on this context, and which in my view is strictly worse, may look like the following:

// The FILE_INFO structure includes the file details plus a variable length
// array for the filename. The underlying filesystem is always FAT32 per the
// EFI specification, which has a maximum filename length of 256 characters. The
// filename is encoded as a wide-string (UCS-2), which encodes two bytes per
// character, and is not NUL-terminated, so we need to reserve up to 512 bytes
// for the filename.
static let buf: [FILE_INFO_SIZE + 512]u8 = [0...];

The target audience of this comment should have a reasonable understanding of EFI. We simply need to clarify that this constant is the FAT32 max filename length, times two to account for the wstr encoding, and our magic constant is sufficiently explained.

Let’s move on to another kind of comment I occasionally write: medium-length prose. These often appear at the start of a function or the start of a file and serve to add context to the implementation, to justify the code’s existence or explain why it works. Another sample:

fn init_pagetables() void = {
	// 0xFFFF0000xxxxxxxx - 0xFFFF0200xxxxxxxx: identity map
	// 0xFFFF0200xxxxxxxx - 0xFFFF0400xxxxxxxx: identity map (dev)
	// 0xFFFF8000xxxxxxxx - 0xFFFF8000xxxxxxxx: kernel image
	//
	// L0[0x000]    => L1_ident
	// L0[0x004]    => L1_devident
	// L1_ident[*]  => 1 GiB identity mappings
	// L0[0x100]    => L1_kernel
	// L1_kernel[0] => L2_kernel
	// L2_kernel[0] => L3_kernel
	// L3_kernel[0] => 4 KiB kernel pages
	L0[0x000] = PT_TABLE | &L1_ident: uintptr | PT_AF;
	L0[0x004] = PT_TABLE | &L1_devident: uintptr | PT_AF;
	L0[0x100] = PT_TABLE | &L1_kernel: uintptr | PT_AF;
	L1_kernel[0] = PT_TABLE | &L2_kernel: uintptr | PT_AF;
	L2_kernel[0] = PT_TABLE | &L3_kernel: uintptr | PT_AF;
	for (let i = 0u64; i < len(L1_ident): u64; i += 1) {
		L1_ident[i] = PT_BLOCK | (i * 0x40000000): uintptr |
			PT_NORMAL | PT_AF | PT_ISM | PT_RW;
	};
	for (let i = 0u64; i < len(L1_devident): u64; i += 1) {
		L1_devident[i] = PT_BLOCK | (i * 0x40000000): uintptr |
			PT_DEVICE | PT_AF | PT_ISM | PT_RW;
	};
};

This comment shares a trait with the previous example: its purpose, in part, is to justify magic constants. It explains the indices of the arrays by way of the desired address space, and a perceptive reader will notice that 1 GiB = 1073741824 bytes = 0x40000000 bytes.

To fully understand this, we must again consider the intended audience. This is an implementation comment, so the reader is an implementer. They will need to possess some familiarity with the behavior of page tables to be productive in this code, and they likely have the ARM manual up on their second monitor. This comment simply fills in the blanks for an informed reader.

There are two additional kinds of comments I often write: TODO and XXX.

A TODO comment indicates some important implementation deficiency; it must be addressed at some point in the future and generally indicates that the function does not meet its stated interface and is often accompanied by an assertion, or a link to a ticket on the bug tracker, or both.

assert(ep.send == null); // TODO: support multiple senders

This function should support multiple senders, but does not; an assertion here prevents the code from running under conditions it does not yet support and the TODO comment indicates that this should be addressed in the future. The target audience for this comment is someone who brings about these conditions and runs into the assertion failure.

fn memory_empty(mem: *memory) bool = {
	// XXX: This O(n) linked list traversal is bad
	let next = mem.next;
	let pages = 0u;
	for (next != FREELIST_END; pages += 1) {
		const addr = mem.phys + (next * mem::PAGESIZE): uintptr;
		const ptr = mem::phys_tokernel(addr): *uint;
		next = *ptr;
	};
	return pages == mem.pages;
};

Here we find an example of an XXX comment. This code is correct: it implements the function’s interface perfectly. However, given its expected usage, a performance of O(n) is not great: this function is expected to be used in hot paths. This comment documents the deficiency, and provides a hint to a reader that might be profiling this code in regards to a possible improvement.

One final example:

// Invalidates the TLB for a virtual address.
export fn invalidate(virt: uintptr) void = {
	// TODO: Notify other cores (XXX SMP)
	invlpg(virt);
};

This is an atypical usage of XXX, but one which I still occasionally reach for. Here we have a TODO comment which indicates a case which this code does not consider, but which must be addressed in the future: it will have to raise an IPI to get other cores to invalidate the affected virtual address. However, this is one of many changes which fall under a broader milestone of SMP support, and the “XXX SMP” comment is here to make it easy to grep through the codebase for any places which are known to require attention while implementing SMP support. An XXX comment is often written for the purpose of being easily found with grep.

That sums up most of the common reasons I will write a comment in my software. Each comment is written considering a target audience and the context provided by the code in which it resides, and aims to avoid stating redundant information within these conditions. It’s for this reason that my code is sparse on comments: I find the information outside of the comments equally important and aim to be concise such that a comment is not redundant with information found elsewhere.

Hopefully this post inspired some thought in you, to consider your comments deliberately and to be more aware of your ability to communicate information in other ways. Even if you chose to write your comments more densely than I do, I hope you will take care to communicate well through other mediums in your code as well.

Porting Helios to aarch64 for my FOSDEM talk, part one

Mon, 20 Feb 2023 00:00:00 +0000

Helios is a microkernel written in the Hare programming language, and the subject of a talk I did at FOSDEM earlier this month. You can watch the talk here if you like:

A while ago I promised someone that I would not do any talks on Helios until I could present them from Helios itself, and at FOSDEM I made good on that promise: my talk was presented from a Raspberry Pi 4 running Helios. The kernel was originally designed for x86_64 (though we were careful to avoid painting ourselves into any corners so that we could port it to more architectures later on), and I initially planned to write an Intel HD Graphics driver so that I could drive the projector from my laptop. But, after a few days spent trying to comprehend the IHD manuals, I decided it would be much easier to port the entire system to aarch64 and write a driver for the much-simpler RPi GPU instead. 42 days later the port was complete, and a week or so after that I successfully presented the talk at FOSDEM. In a series of blog posts, I will take a look at those 42 days of work and explain how the aarch64 port works. Today’s post focuses on the bootloader.

The Helios boot-up process is:

Bootloader starts up and loads the kernel, then jumps to it
The kernel configures the system and loads the init process
Kernel provides runtime services to init (and any subsequent processes)

In theory, the port to aarch64 would address these steps in order, but in practice step (2) relies heavily on the runtime services provided by step (3), so much of the work was ordered 1, 3, 2. This blog post focuses on part 1, I’ll cover parts 2 and 3 and all of the fun problems they caused in later posts.

In any case, the bootloader was the first step. Some basic changes to the build system established boot/+aarch64 as the aarch64 bootloader, and a simple qemu-specific ARM kernel was prepared which just gave a little “hello world” to demonstrate the multi-arch build system was working as intended. More build system refinements would come later, but it’s off to the races from here. Targeting qemu’s aarch64 virt platform was useful for most of the initial debugging and bring-up (and is generally useful at all times, as a much easier platform to debug than real hardware); the first tests on real hardware came much later.

Booting up is a sore point on most systems. It involves a lot of arch-specific procedures, but also generally calls for custom binary formats and annoying things like disk drivers — which don’t belong in a microkernel. So the Helios bootloaders are separated from the kernel proper, which is a simple ELF executable. The bootloader loads this ELF file into memory, configures a few simple things, then passes some information along to the kernel entry point. The bootloader’s memory and other resources are hereafter abandoned and are later reclaimed for general use.

On aarch64 the boot story is pretty abysmal, and I wanted to avoid adding the SoC-specific complexity which is endemic to the platform. Thus, two solutions are called for: EFI and device trees. At the bootloader level, EFI is the more important concern. For qemu-virt and Raspberry Pi, edk2 is the free-software implementation of choice when it comes to EFI. The first order of business is producing an executable which can be loaded by EFI, which is, rather unfortunately, based on the Windows COFF/PE32+ format. I took inspiration from Linux and made an disgusting EFI stub solution, which involves hand-writing a PE32+ header in assembly and doing some truly horrifying things with binutils to massage everything into order. Much of the header is lifted from Linux:

.section .text.head
.global base
base:
.L_head:
	/* DOS header */
	.ascii "MZ"
	.skip 58
	.short .Lpe_header - .L_head
	.align 4
.Lpe_header:
	.ascii "PE\0\0"
	.short 0xAA64                              /* Machine = AARCH64 */
	.short 2                                   /* NumberOfSections */
	.long 0                                    /* TimeDateStamp */
	.long 0                                    /* PointerToSymbolTable */
	.long 0                                    /* NumberOfSymbols */
	.short .Lsection_table - .Loptional_header /* SizeOfOptionalHeader */
	/* Characteristics:
	 * IMAGE_FILE_EXECUTABLE_IMAGE |
	 * IMAGE_FILE_LINE_NUMS_STRIPPED |
	 * IMAGE_FILE_DEBUG_STRIPPED */
	.short 0x206
.Loptional_header:
	.short 0x20b                     /* Magic = PE32+ (64-bit) */
	.byte 0x02                       /* MajorLinkerVersion */
	.byte 0x14                       /* MinorLinkerVersion */
	.long _data - .Lefi_header_end   /* SizeOfCode */
	.long __pecoff_data_size         /* SizeOfInitializedData */
	.long 0                          /* SizeOfUninitializedData */
	.long _start - .L_head           /* AddressOfEntryPoint */
	.long .Lefi_header_end - .L_head /* BaseOfCode */
.Lextra_header:
	.quad 0                          /* ImageBase */
	.long 4096                       /* SectionAlignment */
	.long 512                        /* FileAlignment */
	.short 0                         /* MajorOperatingSystemVersion */
	.short 0                         /* MinorOperatingSystemVersion */
	.short 0                         /* MajorImageVersion */
	.short 0                         /* MinorImageVersion */
	.short 0                         /* MajorSubsystemVersion */
	.short 0                         /* MinorSubsystemVersion */
	.long 0                          /* Reserved */

	.long _end - .L_head             /* SizeOfImage */

	.long .Lefi_header_end - .L_head /* SizeOfHeaders */
	.long 0                          /* CheckSum */
	.short 10                        /* Subsystem = EFI application */
	.short 0                         /* DLLCharacteristics */
	.quad 0                          /* SizeOfStackReserve */
	.quad 0                          /* SizeOfStackCommit */
	.quad 0                          /* SizeOfHeapReserve */
	.quad 0                          /* SizeOfHeapCommit */
	.long 0                          /* LoaderFlags */
	.long 6                          /* NumberOfRvaAndSizes */

	.quad 0 /* Export table */
	.quad 0 /* Import table */
	.quad 0 /* Resource table */
	.quad 0 /* Exception table */
	.quad 0 /* Certificate table */
	.quad 0 /* Base relocation table */

.Lsection_table:
	.ascii ".text\0\0\0"              /* Name */
	.long _etext - .Lefi_header_end   /* VirtualSize */
	.long .Lefi_header_end - .L_head  /* VirtualAddress */
	.long _etext - .Lefi_header_end   /* SizeOfRawData */
	.long .Lefi_header_end - .L_head  /* PointerToRawData */
	.long 0                           /* PointerToRelocations */
	.long 0                           /* PointerToLinenumbers */
	.short 0                          /* NumberOfRelocations */
	.short 0                          /* NumberOfLinenumbers */
	/* IMAGE_SCN_CNT_CODE | IMAGE_SCN_MEM_READ | IMAGE_SCN_MEM_EXECUTE */
	.long 0x60000020

	.ascii ".data\0\0\0"        /* Name */
	.long __pecoff_data_size    /* VirtualSize */
	.long _data - .L_head       /* VirtualAddress */
	.long __pecoff_data_rawsize /* SizeOfRawData */
	.long _data - .L_head       /* PointerToRawData */
	.long 0                     /* PointerToRelocations */
	.long 0                     /* PointerToLinenumbers */
	.short 0                    /* NumberOfRelocations */
	.short 0                    /* NumberOfLinenumbers */
	/* IMAGE_SCN_CNT_INITIALIZED_DATA | IMAGE_SCN_MEM_READ | IMAGE_SCN_MEM_WRITE */
	.long 0xc0000040

.balign 0x10000
.Lefi_header_end:

.global _start
_start:
	stp x0, x1, [sp, -16]!

	adrp x0, base
	add x0, x0, #:lo12:base
	adrp x1, _DYNAMIC
	add x1, x1, #:lo12:_DYNAMIC
	bl relocate
	cmp w0, #0
	bne 0f

	ldp x0, x1, [sp], 16

	b bmain

0:
	/* relocation failed */
	add sp, sp, -16
	ret

The specific details about how any of this works are complex and unpleasant, I’ll refer you to the spec if you’re curious, and offer a general suggestion that cargo-culting my work here would be a lot easier than understanding it should you need to build something similar.¹

Note the entry point for later; we store two arguments from EFI (x0 and x1) on the stack and eventually branch to bmain.

This file is assisted by the linker script:

ENTRY(_start)
OUTPUT_FORMAT(elf64-littleaarch64)

SECTIONS {
	/DISCARD/ : {
		*(.rel.reloc)
		*(.eh_frame)
		*(.note.GNU-stack)
		*(.interp)
		*(.dynsym .dynstr .hash .gnu.hash)
	}

	. = 0xffff800000000000;

	.text.head : {
		_head = .;
		KEEP(*(.text.head))
	}

	.text : ALIGN(64K) {
		_text = .;
		KEEP(*(.text))
		*(.text.*)
		. = ALIGN(16);
		*(.got)
	}

	. = ALIGN(64K);
	_etext = .;

	.dynamic : {
		*(.dynamic)
	}

	.data : ALIGN(64K) {
		_data = .;
		KEEP(*(.data))
		*(.data.*)

		/* Reserve page tables */
		. = ALIGN(4K);
		L0 = .;
		. += 512 * 8;
		L1_ident = .;
		. += 512 * 8;
		L1_devident = .;
		. += 512 * 8;
		L1_kernel = .;
		. += 512 * 8;
		L2_kernel = .;
		. += 512 * 8;
		L3_kernel = .;
		. += 512 * 8;
	}

	.rela.text : {
		*(.rela.text)
		*(.rela.text*)
	}
	.rela.dyn : {
		*(.rela.dyn)
	}
	.rela.plt : {
		*(.rela.plt)
	}
	.rela.got : {
		*(.rela.got)
	}
	.rela.data : {
		*(.rela.data)
		*(.rela.data*)
	}

	.pecoff_edata_padding : {
		BYTE(0);
		. = ALIGN(512);
	}
	__pecoff_data_rawsize = ABSOLUTE(. - _data);
	_edata = .;

	.bss : ALIGN(4K) {
		KEEP(*(.bss))
		*(.bss.*)
		*(.dynbss)
	}

	. = ALIGN(64K);
	__pecoff_data_size = ABSOLUTE(. - _data);
	_end = .;
}

Items of note here are the careful treatment of relocation sections (cargo-culted from earlier work on RISC-V with Hare; not actually necessary as qbe generates PIC for aarch64)² and the extra symbols used to gather information for the PE32+ header. Padding is also added in the required places, and static aarch64 page tables are defined for later use.

This is built as a shared object, and the Makefile ~~mutilates~~ reformats the resulting ELF file to produce a PE32+ executable:

$(BOOT)/bootaa64.so: $(BOOT_OBJS) $(BOOT)/link.ld
	$(LD) -Bsymbolic -shared --no-undefined \
		-T $(BOOT)/link.ld \
		$(BOOT_OBJS) \
		-o $@

$(BOOT)/bootaa64.efi: $(BOOT)/bootaa64.so
	$(OBJCOPY) -Obinary \
		-j .text.head -j .text -j .dynamic -j .data \
		-j .pecoff_edata_padding \
		-j .dynstr -j .dynsym \
		-j .rel -j .rel.* -j .rel* \
		-j .rela -j .rela.* -j .rela* \
		$< $@

With all of this mess sorted, and the PE32+ entry point branching to bmain, we can finally enter some Hare code:

export fn bmain(
	image_handle: efi::HANDLE,
	systab: *efi::SYSTEM_TABLE,
) efi::STATUS = {
    // ...
};

Getting just this far took 3 full days of work.

Initially, the Hare code incorporated a lot of proof-of-concept work from Alexey Yerin’s “carrot” kernel prototype for RISC-V, which also booted via EFI. Following the early bringing-up of the bootloader environment, this was refactored into a more robust and general-purpose EFI support layer for Helios, which will be applicable to future ports. The purpose of this module is to provide an idiomatic Hare-oriented interface to the EFI boot services, which the bootloader makes use of mainly to read files from the boot media and examine the system’s memory map.

Let’s take a look at the first few lines of bmain:

efi::init(image_handle, systab)!;

const eficons = eficons_init(systab);
log::setcons(&eficons);
log::printfln("Booting Helios aarch64 via EFI");

if (readel() == el::EL3) {
	log::printfln("Booting from EL3 is not supported");
	return efi::STATUS::LOAD_ERROR;
};

let mem = allocator { ... };
init_mmap(&mem);
init_pagetables();

Significant build system overhauls were required such that Hare modules from the kernel like log (and, later, other modules like elf) could be incorporated into the bootloader, simplifying the process of implementing more complex bootloaders. The first call of note here is init_mmap, which scans the EFI memory map and prepares a simple high-watermark allocator to be used by the bootloader to allocate memory for the kernel image and other items of interest. It’s quite simple, it just finds the largest area of general-purpose memory and sets up an allocator with it:

// Loads the memory map from EFI and initializes a page allocator using the
// largest area of physical memory.
fn init_mmap(mem: *allocator) void = {
	const iter = efi::iter_mmap()!;
	let maxphys: uintptr = 0, maxpages = 0u64;
	for (true) {
		const desc = match (efi::mmap_next(&iter)) {
		case let desc: *efi::MEMORY_DESCRIPTOR =>
			yield desc;
		case void =>
			break;
		};
		if (desc.DescriptorType != efi::MEMORY_TYPE::CONVENTIONAL) {
			continue;
		};
		if (desc.NumberOfPages > maxpages) {
			maxphys = desc.PhysicalStart;
			maxpages = desc.NumberOfPages;
		};
	};
	assert(maxphys != 0, "No suitable memory area found for kernel loader");
	assert(maxpages <= types::UINT_MAX);
	pagealloc_init(mem, maxphys, maxpages: uint);
};

init_pagetables is next. This populates the page tables reserved by the linker with the desired higher-half memory map, illustrated in the comments shown here:

fn init_pagetables() void = {
	// 0xFFFF0000xxxxxxxx - 0xFFFF0200xxxxxxxx: identity map
	// 0xFFFF0200xxxxxxxx - 0xFFFF0400xxxxxxxx: identity map (dev)
	// 0xFFFF8000xxxxxxxx - 0xFFFF8000xxxxxxxx: kernel image
	//
	// L0[0x000]    => L1_ident
	// L0[0x004]    => L1_devident
	// L1_ident[*]  => 1 GiB identity mappings
	// L0[0x100]    => L1_kernel
	// L1_kernel[0] => L2_kernel
	// L2_kernel[0] => L3_kernel
	// L3_kernel[0] => 4 KiB kernel pages
	L0[0x000] = PT_TABLE | &L1_ident: uintptr | PT_AF;
	L0[0x004] = PT_TABLE | &L1_devident: uintptr | PT_AF;
	L0[0x100] = PT_TABLE | &L1_kernel: uintptr | PT_AF;
	L1_kernel[0] = PT_TABLE | &L2_kernel: uintptr | PT_AF;
	L2_kernel[0] = PT_TABLE | &L3_kernel: uintptr | PT_AF;
	for (let i = 0u64; i < len(L1_ident): u64; i += 1) {
		L1_ident[i] = PT_BLOCK | (i * 0x40000000): uintptr |
			PT_NORMAL | PT_AF | PT_ISM | PT_RW;
	};
	for (let i = 0u64; i < len(L1_devident): u64; i += 1) {
		L1_devident[i] = PT_BLOCK | (i * 0x40000000): uintptr |
			PT_DEVICE | PT_AF | PT_ISM | PT_RW;
	};
};

In short, we want three larger memory regions to be available: an identity map, where physical memory addresses correlate 1:1 with virtual memory, an identity map configured for device MMIO (e.g. with caching disabled), and an area to load the kernel image. The first two are straightforward, they use uniform 1 GiB mappings to populate their respective page tables. The latter is slightly more complex, ultimately the kernel is loaded in 4 KiB pages so we need to set up intermediate page tables for that purpose.

We cannot actually enable these page tables until we’re finished making use of the EFI boot services — the EFI specification requires us to preserve the online memory map at this stage of affairs. However, this does lay the groundwork for the kernel loader: we have an allocator to provide pages of memory, and page tables to set up virtual memory mappings that can be activated once we’re done with EFI. bmain thus proceeds with loading the kernel:

const kernel = match (efi::open("\\helios", efi::FILE_MODE::READ)) {
case let file: *efi::FILE_PROTOCOL =>
	yield file;
case let err: efi::error =>
	log::printfln("Error: no kernel found at /helios");
	return err: efi::STATUS;
};

log::printfln("Load kernel /helios");
const kentry = match (load(&mem, kernel)) {
case let err: efi::error =>
	return err: efi::STATUS;
case let entry: uintptr =>
	yield entry: *kentry;
};
efi::close(kernel)!;

The loader itself (the “load” function here) is a relatively straightforward ELF loader; if you’ve seen one you’ve seen them all. Nevertheless, you may browse it online if you so wish. The only item of note here is the function used for mapping kernel pages:

// Maps a physical page into the kernel's virtual address space.
fn kmmap(virt: uintptr, phys: uintptr, flags: uintptr) void = {
	assert(virt & ~0x1ff000 == 0xffff800000000000: uintptr);
	const offs = (virt >> 12) & 0x1ff;
	L3_kernel[offs] = PT_PAGE | PT_NORMAL | PT_AF | PT_ISM | phys | flags;
};

The assertion enforces a constraint which is implemented by our kernel linker script, namely that all loadable kernel program headers are located within the kernel’s reserved address space. With this constraint in place, the implementation is simpler than many mmap implementations; we can assume that L3_kernel is the correct page table and just load it up with the desired physical address and mapping flags.

Following the kernel loader, the bootloader addresses other items of interest, such as loading the device tree and boot modules — which includes, for instance, the init process image and an initramfs. It also allocates & populates data structures with information which will be of later use to the kernel, including the memory map. This code is relatively straightforward and not particularly interesting; most of these processes takes advantage of the same straightforward Hare function:

// Loads a file into continuous pages of memory and returns its physical
// address.
fn load_file(
	mem: *allocator,
	file: *efi::FILE_PROTOCOL,
) (uintptr | efi::error) = {
	const info = efi::file_info(file)?;
	const fsize = info.FileSize: size;
	let npage = fsize / PAGESIZE;
	if (fsize % PAGESIZE != 0) {
		npage += 1;
	};

	let base: uintptr = 0;
	for (let i = 0z; i < npage; i += 1) {
		const phys = pagealloc(mem);
		if (base == 0) {
			base = phys;
		};

		const nbyte = if ((i + 1) * PAGESIZE > fsize) {
			yield fsize % PAGESIZE;
		} else {
			yield PAGESIZE;
		};
		let dest = (phys: *[*]u8)[..nbyte];
		const n = efi::read(file, dest)?;
		assert(n == nbyte);
	};

	return base;
};

It is not necessary to map these into virtual memory anywhere, the kernel later uses the identity-mapped physical memory region in the higher half to read them. Tasks of interest resume at the end of bmain:

efi::exit_boot_services();
init_mmu();
enter_kernel(kentry, ctx);

Once we exit boot services, we are free to configure the MMU according to our desired specifications and make good use of all of the work done earlier to prepare a kernel memory map. Thus, init_mmu:

// Initializes the ARM MMU to our desired specifications. This should take place
// *after* EFI boot services have exited because we're going to mess up the MMU
// configuration that it depends on.
fn init_mmu() void = {
	// Disable MMU
	const sctlr_el1 = rdsctlr_el1();
	wrsctlr_el1(sctlr_el1 & ~SCTLR_EL1_M);

	// Configure MAIR
	const mair: u64 =
		(0xFF << 0) | // Attr0: Normal memory; IWBWA, OWBWA, NTR
		(0x00 << 8);  // Attr1: Device memory; nGnRnE, OSH
	wrmair_el1(mair);

	const tsz: u64 = 64 - 48;
	const ips = rdtcr_el1() & TCR_EL1_IPS_MASK;
	const tcr_el1: u64 =
		TCR_EL1_IPS_42B_4T |	// 4 TiB IPS
		TCR_EL1_TG1_4K |	// Higher half: 4K granule size
		TCR_EL1_SH1_IS |	// Higher half: inner shareable
		TCR_EL1_ORGN1_WB |	// Higher half: outer write-back
		TCR_EL1_IRGN1_WB |	// Higher half: inner write-back
		(tsz << TCR_EL1_T1SZ) |	// Higher half: 48 bits
		TCR_EL1_TG0_4K |	// Lower half: 4K granule size
		TCR_EL1_SH0_IS |	// Lower half: inner sharable
		TCR_EL1_ORGN0_WB |	// Lower half: outer write-back
		TCR_EL1_IRGN0_WB |	// Lower half: inner write-back
		(tsz << TCR_EL1_T0SZ);	// Lower half: 48 bits
	wrtcr_el1(tcr_el1);

	// Load page tables
	wrttbr0_el1(&L0[0]: uintptr);
	wrttbr1_el1(&L0[0]: uintptr);
	invlall();

	// Enable MMU
	const sctlr_el1: u64 =
		SCTLR_EL1_M |		// Enable MMU
		SCTLR_EL1_C |		// Enable cache
		SCTLR_EL1_I |		// Enable instruction cache
		SCTLR_EL1_SPAN |	// SPAN?
		SCTLR_EL1_NTLSMD |	// NTLSMD?
		SCTLR_EL1_LSMAOE |	// LSMAOE?
		SCTLR_EL1_TSCXT |	// TSCXT?
		SCTLR_EL1_ITD;		// ITD?
	wrsctlr_el1(sctlr_el1);
};

There are a lot of bits here! Figuring out which ones to enable or disable was a project in and of itself. One of the major challenges, funnily enough, was finding the correct ARM manual to reference to understand all of these registers. I’ll save you some time and link to it directly, should you ever find yourself writing similar code. Some question marks in comments towards the end point out some flags that I’m still not sure about. The ARM CPU is very configurable and identifying the configuration that produces the desired behavior for a general-purpose kernel requires some effort.

After this function completes, the MMU is initialized and we are up and running with the kernel memory map we prepared earlier; the kernel is loaded in the higher half and the MMU is prepared to service it. So, we can jump to the kernel via enter_kernel:

@noreturn fn enter_kernel(entry: *kentry, ctx: *bootctx) void = {
	const el = readel();
	switch (el) {
	case el::EL0 =>
		abort("Bootloader running in EL0, breaks EFI invariant");
	case el::EL1 =>
		// Can boot immediately
		entry(ctx);
	case el::EL2 =>
		// Boot from EL2 => EL1
		//
		// This is the bare minimum necessary to get to EL1. Future
		// improvements might be called for here if anyone wants to
		// implement hardware virtualization on aarch64. Good luck to
		// this future hacker.

		// Enable EL1 access to the physical counter register
		const cnt = rdcnthctl_el2();
		wrcnthctl_el2(cnt | 0b11);

		// Enable aarch64 in EL1 & SWIO, disable most other EL2 things
		// Note: I bet someday I'll return to this line because of
		// Problems
		const hcr: u64 = (1 << 1) | (1 << 31);
		wrhcr_el2(hcr);

		// Set up SPSR for EL1
		// XXX: Magic constant I have not bothered to understand
		wrspsr_el2(0x3c4);

		enter_el1(ctx, entry);
	case el::EL3 =>
		// Not supported, tested earlier on
		abort("Unsupported boot configuration");
	};
};

Here we see the detritus from one of many battles I fought to port this kernel: the EL2 => EL1 transition. aarch64 has several “exception levels”, which are semantically similar to the x86_64 concept of protection rings. EL0 is used for userspace code, which is not applicable under these circumstances; an assertion sanity-checks this invariant. EL1 is the simplest case, this is used for normal kernel code and in this situation we can jump directly to the kernel. The EL2 case is used for hypervisor code, and this presented me with a challenge. When I tested my bootloader in qemu-virt, it worked initially, but on real hardware it failed. After much wailing and gnashing of teeth, the cause was found to be that our bootloader was started in EL2 on real hardware, and EL1 on qemu-virt. qemu can be configured to boot in EL2, which was crucial in debugging this problem, via -M virt,virtualization=on. From this environment I was able to identify a few important steps to drop to EL1 and into the kernel, though from the comments you can probably ascertain that this process was not well-understood. I do have a better understanding of it now than I did when this code was written, but the code is still serviceable and I see no reason to change it at this stage.

At this point, 14 days into the port, I successfully reached kmain on qemu-virt. Some initial kernel porting work was done after this, but when I was prepared to test it on real hardware I ran into this EL2 problem — the first kmain on real hardware ran at T+18.

That sums it up for the aarch64 EFI bootloader work. 24 days later the kernel and userspace ports would be complete, and a couple of weeks after that it was running on stage at FOSDEM. The next post will cover the kernel port (maybe more than one post will be required, we’ll see), and the final post will address the userspace port and the inner workings of the slidedeck demo that was shown on stage. Look forward to it, and thanks for reading!

Should private platforms engage in censorship?

Mon, 30 Jan 2023 00:00:00 +0000

Private service providers are entitled to do business with whom they please, or not to. Occasionally, a platform will take advantage of this to deny service to a particular entity on any number of grounds, often igniting a flood of debate online regarding whether or not censorship in this form is just. Recently, CloudFlare pulled the plug on a certain forum devoted to the coordinated harassment of its victims. Earlier examples include the same service blocking a far-right imageboard, or Namecheap cancelling service for a neo-Nazi news site.

In each of these cases, a private company elected to terminate service for a customer voluntarily, without a court order. Absent from these events was any democratic or judicial oversight. A private company which provides some kind of infrastructure for the Internet simply elected to unilaterally terminate service for a customer or class of customers.

When private companies choose with whom they do or do not do business with, this is an exercise of an important freedom: freedom of association. Some companies have this right limited by regulation — for instance, utility companies are often required to provide power to everyone who wants it within their service area. Public entities are required to provide their services to everyone — for instance, the US postal service cannot unilaterally choose not to deliver your mail. However, by default, private companies are generally allowed to deny their services to whomever they please.¹

Are they right to?

An argument is often made that, when a platform reaches a given size (e.g. Facebook), or takes on certain ambitions (e.g. CloudFlare), it may become large and entrenched enough in our society that it should self-impose a role more analogous to a public utility than a private company. Under such constraints, such a platform would choose to host any content which is not explicitly illegal, and defer questions over what content is appropriate to the democratic process. There are a number of angles from which we can examine this argument.

For a start, how might we implement the scenario called for by this argument? Consider one option: regulation. Power companies are subject to regulations regarding how and with whom they do business; they must provide service to everyone and they are not generally allowed to shut off your heat in the cold depths of winter. Similarly, we could regulate digital platforms to require them to provide a soapbox for all legally expressible viewpoints, then utilize the democratic process to narrow this soapbox per society’s mutually-agreed-upon views regarding matters such as neo-Nazi propaganda.²

It’s important when making this argument to note that regulation of this sort imposes obligations on private businesses which erode their own right to free association; radical free speech for individuals requires radical curtailing of free association for businesses. Private businesses are owned and staffed by individuals, and requiring them to allow all legal forms of content on their platform is itself a limitation on their freedom. The staff of a newspaper may not appreciate being required by law to provide space in the editorials for KKK members to espouse their racist philosophy, but would nevertheless be required to typeset such articles under such an arrangement.

Another approach to addressing this argument is not to question the rights of a private business, but instead to question whether or not they should be allowed to grow to a size such that their discretion in censorship constitutes a disruption to society due to their scale and entrenched market position. Under this lens, we can suggest another government intervention that does not take the form of regulation, but of an application of antitrust law. With more platforms to choose from, we can explore more approaches to moderation and censorship, and depend on the market’s invisible hand to lead us true.

The free speech absolutist who makes similar arguments may find themselves in a contradiction: expanding free speech for some people (platform users) requires, in this scenario, curtailing freedoms for others (platform owners and staff). Someone in this position may concede that, while they support the rights of individuals, they might not offer the same rights to businesses who resemble utilities. The tools for implementing this worldview, however, introduce further contradictions when combined with the broader political profile of a typical free speech absolutist: calling for regulation isn’t very consistent with any “small government” philosophy; and those who describe themselves as Libertarian and make either of these arguments provide me with no small amount of amusement.

There is another flaw in this line of thinking which I want to highlight: the presumption that the democratic process can address these problems in the first place. Much of the legitimacy of this argument rests on the assumption that the ability for maligned users to litigate their grievances is not only more just, but also equal to the threat posed by hate speech and other concerns which are often the target of censorship on private platforms. I don’t think that this is true.

The democratic and judicial processes are often corrupt and inefficient. It is still the case that the tone of your skin has an outsized effect on the outcome of your court case; why shouldn’t similar patterns emerge when de-platformed racists are given their day before a judge? Furthermore, the pace of government interventions are generally insufficient. Could Facebook appeal a court for the right to remove the Proud Boys from their platform faster than they could organize an attack on the US Capitol building? And can lawmakers keep up with innovation at a pace sufficient to address new forms and mediums for communicating harmful content before they’re a problem?

We should also question if the democratic process will lead to moral outcomes. Minorities are, by definition, in the minority, and a purely democratic process will only favor their needs subject to the will of the majority. Should the rights of trans people to live free of harassment be subject to the pleasure of the cisgendered majority?

These systems, when implemented, will perform as they always have: they will provide disproportionately unfavorable outcomes for disadvantaged members of society. I am a leftist: if asked to imagine a political system which addresses these problems, I will first imagine sweeping reforms to our existing system, point out that the free market isn’t, lean in favor of regulation and nationalization of important industries, and seek to empower the powerless against the powerful. It will require a lot of difficult, ongoing work to get there, and I imagine most of this work will be done in spite of the protests of the typical free speech absolutist.

I am in favor of these reforms, but they are decades away from completion, and many will disagree on the goals and their implementation. But I am also a pragmatic person, and when faced with the system in which we find ourselves today, I seek a pragmatic solution to this problem; ideally one which is not predicated on revolution. When faced with the question, “should private platforms engage in censorship?”, what is the pragmatic answer?

To provide such an answer, we must de-emphasize idealism in favor of an honest examination of the practical context within which our decision-making is done. Consider again the status quo: private companies are generally permitted to exercise their right to free association by kicking people off of their platforms. A pragmatic framework for making these decisions examines the context in which they are made. In the current political climate, this context should consider the threats faced by many different groups of marginalized people today: racism is still alive and strong, what few LGBT rights exist are being dismantled, and many other civil liberties are under attack.

When someone (or some entity such as business) enjoys a particular freedom, the way they exercise it is meaningful. Inaction is a form of complicity; allowing hate to remain on your platform is an acknowledgement of your favor towards the lofty principles outlined in the arguments above in spite of the problems enumerated here and the realities faced by marginalized people today. A purely moral consideration thus suggests that exercising your right to free association in your role as a decision-maker at a business is a just response to this status quo.

I expect the people around me (given a definition of “around me” that extends to the staff at businesses I patronize) to possess a moral compass which is compatible with my own, and to act in accordance with it; in the absence of this I will express my discontent by voting with my feet. However, businesses in the current liberal economic regime often disregard morals in favor of profit-oriented decision making. Therefore, in order for the typical business behave morally, their decision-making must exist within a context where the moral outcomes align with the profitable outcomes.

We are seeing increasing applications of private censorship because this alignment is present. Businesses depend on two economic factors which are related to this issue: access to a pool of profitable users, and access to a labor pool with which to develop and maintain their profits. Businesses which platform bigots are increasingly finding public opinion turning against them; marginalized people and moderates tend to flee to less toxic spaces and staff members are looking to greener pastures. The free market currently rewards private censorship, therefore in a system wherein the free market reigns supreme we observe private censorship.

I reject the idea that it is appropriate for businesses to sideline morality in favor of profit, and I don’t have much faith in the free market to produce moral outcomes. For example, the market is responding poorly to the threat of climate change. However, in the case of private censorship, the incentives are aligned such that the outcomes we’re observing match the outcomes I would expect.

This is a complex topic which we have examined from many angles. In my view, freedom of association is just as important as freedom of speech, and its application to private censorship is not clearly wrong. If you view private censorship as an infringement of the principle of free speech, but agree that freedom of association is nevertheless important, we must resolve this contradiction. The democratic or judicial processes are an enticing and idealistic answer, but these are flawed processes that may not produce just outcomes. If I were to consider these tools to address this question, I’m going to present solutions from a socialist perspective which may or may not jibe with your sensibilities.

Nevertheless, the system as it exists today produces outcomes which approximate both rationality and justice, and I do not stand in opposition to the increased application of private censorship under the current system, flawed though it may be.

My plans at FOSDEM: SourceHut, Hare, and Helios

Tue, 24 Jan 2023 00:00:00 +0000

FOSDEM is right around the corner, and finally in person after long years of dealing with COVID. I’ll be there again this year, and I’m looking forward to it! I have four slots on the schedule (wow! Thanks for arranging these, FOSDEM team) and I’ll be talking about several projects. There is a quick lightning talk on Saturday to introduce Helios and tease a full-length talk on Sunday, a meetup for the Hare community, and a meetup for the SourceHut community. I hope to see you there!

Lightning talk: Introducing Helios

Saturday 12:00 at H.2215 (Ferrer)

Helios is a simple microkernel written in part to demonstrate the applicability of the Hare programming language to kernels. This talk briefly explains why Helios is interesting and is a teaser for a more in-depth talk in the microkernel room tomorrow.
Hare is a systems programming language designed to be simple, stable, and robust. Hare uses a static type system, manual memory management, and a minimal runtime. It is well-suited to writing operating systems, system tools, compilers, networking software, and other low-level, high performance tasks. Helios uses Hare to implement a microkernel, largely inspired by seL4.

BoF: The Hare programming language

Saturday 15:00 at UB2.147

Hare is a systems programming language designed to be simple, stable, and robust. Hare uses a static type system, manual memory management, and a minimal runtime. It is well-suited to writing operating systems, system tools, compilers, networking software, and other low-level, high performance tasks.
At this meeting we’ll sum up the state of affairs with Hare, our plans for the future, and encourage discussions with the community. We’ll also demonstrate a few interesting Hare projects, including Helios, a micro-kernel written in Hare, and encourage each other to work on interesting projects in the Hare community.

BoF: SourceHut meetup

Saturday 16:00 at UB2.147

SourceHut is a free software forge for developing software projects, providing git and mercurial hosting, continuous integration, mailing lists, and more. We’ll be meeting here again in 2023 to discuss the platform and its community, the completion of the GraphQL rollout and the migration to the EU, and any other topics on the minds of the attendees.

Introducing Helios

Sunday 13:00 at H.1308 (Rolin)

Helios is a simple microkernel written in part to demonstrate the applicability of the Hare programming language to kernels. This talk will introduce the design and rationale for Helios, address some details of its implementation, compare it with seL4, and elaborate on the broader plans for the system.

Setting a new focus for my blog

Sun, 22 Jan 2023 00:00:00 +0000

Just shy of two months ago, I published I shall toil at a reduced volume, which addressed the fact that I’m not getting what I want from my blog anymore, and I would be taking an indefinite break. Well, I am ready to resume my writing, albeit with a different tone and focus than before.

Well, that was fast.

– Everyone

Since writing this, I have been considering what exactly the essential subject of my dissatisfaction with my writing has been. I may have found the answer: I lost sight of my goals. I got so used to writing that I would often think to myself, “I want to write a blog post!”, then dig a topic out of my backlog (which is 264 items long) and write something about it. This is not the way; much of the effort expended on writing in this manner is not spent on the subjects I care about most, or those which most urgently demand an expenditure of words.

The consequences of this misalignment of perspective are that my writing has often felt dull and rote. It encourages shallower takes and lends itself to the rants or unthoughtful criticisms that my writings are, unfortunately, (in)famous for. When I take an idea off of the shelf, or am struck by an idea that, in the moment, seemingly demands to be spake of, I often end up with a disappointing result when the fruit of this inspiration is published a few hours later.

Over the long term, these issues manifest as demerits to my reputation, and deservedly so. What’s more, when a critical tone is well-justified, the posts which utilize it are often overlooked by readers due to the normalization of this tone throughout less important posts. Take for instance my recent post on Rust in Linux. Though this article could have been written with greater nuance, I still find its points about the value of conservatism in software decision-making accurate and salient. However, the message is weakened riding on the coat-tails of my long history of less poignant critiques of Rust. As I resume my writing, I will have to take a more critical examination of myself and the broader context of my writing before reaching for a negative tone as a writing tool.

With these lessons in mind, I am seeking out stronger goals to align my writing with, in the hope that the writing is both more fulfilling for me, and more compelling for the reader. Among these goals I have identified two particularly important ones, whose themes resonate through my strongest articles throughout the years:

The applicability of software to the just advancement of society, its contextualization within the needs of the people who use it, a deep respect for these people and the software’s broader impact on the world, and the use of free software to acknowledge and fulfill these needs.
The principles of good software engineering, such that software built to meet these goals is reliable, secure, and comprehensible. It is in the service of this goal that I beat the drum of simplicity with a regular rhythm.

Naturally many people have important beliefs on these subjects. I simply aim to share my own perspective, and I find it rewarding when I am able to write compelling arguments which underline these goals.

There is another kind of blog post that I enjoy writing and plan to resume: in-depth technical analysis of my free software projects. I’m working on lots of interesting and exciting projects, and I want to talk about them more, and I think people enjoy reading about them. I just spent six weeks porting Helios to aarch64, for instance, and have an essay on the subject half-written in the back of my head. I would love to type it in and publish it.

So, I will resume writing, and indeed at a “reduced volume”, with a renewed focus on the message and its context, and an emphasis on serving the goals I care about the most. Hopefully I find it more rewarding to write in this manner, and you find the results more compelling to read! Stay tuned.

$ rm ~/sources/drewdevault.com/todo.txt

I shall toil at a reduced volume

Thu, 01 Dec 2022 00:00:00 +0000

Over the last nine years I have written 300,000 words for this blog on the topics which are important to me. I am not certain that I have much left to say.

I can keep revisiting these topics for years, each time adding a couple more years of wisdom and improvements to my writing skills to present my arguments more effectively. However, I am starting to feel diminishing returns from my writing. It does not seem like my words are connecting with readers anymore. And, though the returns on my work seem to be diminishing, the costs are not. Each new article spurs less discussion than the last, but provides an unwavering supply of spiteful responses.

Software is still the same mess it was when I started writing and working, or perhaps even worse. You can’t overcome perverse incentives. As Cantrill once famously noted, the lawnmower can’t have empathy. The truth he did not speak is that we all have some Oracle in our hearts, and the lawnmower is the size of the entire industry.

I have grown tired of it. I will continue my work quietly, building the things I believe in, and remaining true to my principles. I do not yet know if this is a cessation or a siesta, but I do know that I will not write again for some time. Thank you for reading, and good luck in your endeavours. I hope you found something of value in these pages.

Here are some of the blog posts I am most proud of, should you want to revisit them today or the next time you happen upon my website:

Codegen in Hare v2

Sat, 26 Nov 2022 00:00:00 +0000

I spoke about code generation in Hare back in May when I wrote a tool for generating ioctl numbers. I wrote another code generator over the past few weeks, and it seems like a good time to revisit the topic on my blog to showcase another approach, and the improvements we’ve made for this use-case.

In this case, I wanted to generate code to implement IPC (inter-process communication) interfaces for my operating system. I have designed a DSL for describing these interfaces — you can read the grammar here. This calls for a parser, which is another interesting topic for Hare, but I’ll set that aside for now and focus on the code gen. Assume that, given a file like the following, we can parse it and produce an AST:

namespace hello;

interface hello {
	call say_hello() void;
	call add(a: uint, b: uint) uint;
};

The key that makes the code gen approach we’re looking at today is the introduction of strings::template to the Hare standard library. This module is inspired by a similar feature from Python, string.Template. An example of its usage is provided in Hare’s standard library documentation:

const src = "Hello, $user! Your balance is $$$balance.\n";
const template = template::compile(src)!;
defer template::finish(&template);
template::execute(&template, os::stdout,
	("user", "ddevault"),
	("balance", 1000),
)!; // "Hello, ddevault! Your balance is $1000.

Makes sense? Cool. Let’s see how this can be applied to code generation. The interface shown above compiles to the following generated code:

// This file was generated by ipcgen; do not modify by hand
use errors;
use helios;
use rt;

def HELLO_ID: u32 = 0xC01CAAC5;

export type fn_hello_say_hello = fn(object: *hello) void;
export type fn_hello_add = fn(object: *hello, a: uint, b: uint) uint;

export type hello_iface = struct {
	say_hello: *fn_hello_say_hello,
	add: *fn_hello_add,
};

export type hello_label = enum u64 {
	SAY_HELLO = HELLO_ID << 16u64 | 1,
	ADD = HELLO_ID << 16u64 | 2,
};

export type hello = struct {
	_iface: *hello_iface,
	_endpoint: helios::cap,
};

export fn hello_dispatch(
	object: *hello,
) void = {
	const (tag, a1) = helios::recvraw(object._endpoint);
	switch (rt::label(tag): hello_label) {
	case hello_label::SAY_HELLO =>
		object._iface.say_hello(
			object,
		);
		match (helios::reply(0)) {
		case void =>
			yield;
		case errors::invalid_cslot =>
			yield; // callee stored the reply
		case errors::error =>
			abort(); // TODO
		};
	case hello_label::ADD =>
		const rval = object._iface.add(
			object,
			a1: uint,
			rt::ipcbuf.params[1]: uint,
		);
		match (helios::reply(0, rval)) {
		case void =>
			yield;
		case errors::invalid_cslot =>
			yield; // callee stored the reply
		case errors::error =>
			abort(); // TODO
		};
	case =>
		abort(); // TODO
	};
};

Generating this code starts with the following entry-point:

// Generates code for a server to implement the given interface.
export fn server(out: io::handle, doc: *ast::document) (void | io::error) = {
	fmt::fprintln(out, "// This file was generated by ipcgen; do not modify by hand")!;
	fmt::fprintln(out, "use errors;")!;
	fmt::fprintln(out, "use helios;")!;
	fmt::fprintln(out, "use rt;")!;
	fmt::fprintln(out)!;

	for (let i = 0z; i < len(doc.interfaces); i += 1) {
		const iface = &doc.interfaces[i];
		s_iface(out, doc, iface)?;
	};
};

Here we start with some simple use of basic string formatting via fmt::fprintln. We see some of the same approach repeated in the meatier functions like s_iface:

fn s_iface(
	out: io::handle,
	doc: *ast::document,
	iface: *ast::interface,
) (void | io::error) = {
	const id: ast::ident = [iface.name];
	const name = gen_name_upper(&id);
	defer free(name);

	let id: ast::ident = alloc(doc.namespace...);
	append(id, iface.name);
	defer free(id);
	const hash = genhash(&id);

	fmt::fprintfln(out, "def {}_ID: u32 = 0x{:X};\n", name, hash)!;

Our first use of strings::template appears when we want to generate type aliases for interface functions, via s_method_fntype. This is where some of the trade-offs of this approach begin to present themselves.

const s_method_fntype_src: str =
	`export type fn_$iface_$method = fn(object: *$object$params) $result;`;
let st_method_fntype: tmpl::template = [];

@init fn s_method_fntype() void = {
	st_method_fntype= tmpl::compile(s_method_fntype_src)!;
};

fn s_method_fntype(
	out: io::handle,
	iface: *ast::interface,
	meth: *ast::method,
) (void | io::error) = {
	assert(len(meth.caps_in) == 0); // TODO
	assert(len(meth.caps_out) == 0); // TODO

	let params = strio::dynamic();
	defer io::close(&params)!;
	if (len(meth.params) != 0) {
		fmt::fprint(&params, ", ")?;
	};
	for (let i = 0z; i < len(meth.params); i += 1) {
		const param = &meth.params[i];
		fmt::fprintf(&params, "{}: ", param.name)!;
		ipc_type(&params, &param.param_type)!;

		if (i + 1 < len(meth.params)) {
			fmt::fprint(&params, ", ")!;
		};
	};

	let result = strio::dynamic();
	defer io::close(&result)!;
	ipc_type(&result, &meth.result)!;

	tmpl::execute(&st_method_fntype, out,
		("method", meth.name),
		("iface", iface.name),
		("object", iface.name),
		("params", strio::string(&params)),
		("result", strio::string(&result)),
	)?;
	fmt::fprintln(out)?;
};

The simple string substitution approach of strings::template prevents it from being as generally useful as a full-blown templating engine ala jinja2. To work around this, we have to write Hare code which does things like slurping up the method parameters into a strio::dynamic buffer where we might instead reach for something like {% for param in method.params %} in jinja2. Once we have prepared all of our data in a format suitable for a linear string substitution, we can pass it to tmpl::execute. The actual template is stored in a global which is compiled during @init, which runs at program startup. Anything which requires a loop to compile, such as the parameter list, is fetched out of the strio buffer and passed to the template.

We can explore a slightly different approach when we generate this part of the code, back up in the s_iface function:

export type hello_iface = struct {
	say_hello: *fn_hello_say_hello,
	add: *fn_hello_add,
};

To output this code, we render several templates one after another, rather than slurping up the generated code into heap-allocated string buffers to be passed into a single template.

const s_iface_header_src: str =
	`export type $iface_iface = struct {`;
let st_iface_header: tmpl::template = [];

const s_iface_method_src: str =
	`	$method: *fn_$iface_$method,`;
let st_iface_method: tmpl::template = [];

@init fn s_iface() void = {
	st_iface_header = tmpl::compile(s_iface_header_src)!;
	st_iface_method = tmpl::compile(s_iface_method_src)!;
};

// ...

tmpl::execute(&st_iface_header, out,
	("iface", iface.name),
)?;
fmt::fprintln(out)?;

for (let i = 0z; i < len(iface.methods); i += 1) {
	const meth = &iface.methods[i];
	tmpl::execute(&st_iface_method, out,
		("iface", iface.name),
		("method", meth.name),
	)?;
	fmt::fprintln(out)?;
};

fmt::fprintln(out, "};\n")?;

The remainder of the code is fairly similar.

strings::template is less powerful than a more sophisticated templating system might be, such as Golang’s text/template. A more sophisticated templating engine could be implemented for Hare, but it would be more challenging — no reflection or generics in Hare — and would not be a great candidate for the standard library. This approach hits the sweet spot of simplicity and utility that we’re aiming for in the Hare stdlib. strings::template is implemented in a single ~180 line file.

I plan to continue polishing this tool so I can use it to describe interfaces for communications between userspace drivers and other low-level userspace services in my operating system. If you have any questions, feel free to post them on my public inbox, or shoot them over to my new fediverse account. Until next time!

In praise of Plan 9

Sat, 12 Nov 2022 00:00:00 +0000

Plan 9 is an operating system designed by Bell Labs. It’s the OS they wrote after Unix, with the benefit of hindsight. It is the most interesting operating system that you’ve never heard of, and, in my opinion, the best operating system design to date. Even if you haven’t heard of Plan 9, the designers of whatever OS you do use have heard of it, and have incorporated some of its ideas into your OS.

Plan 9 is a research operating system, and exists to answer questions about ideas in OS design. As such, the Plan 9 experience is in essence an exploration of the interesting ideas it puts forth. Most of the ideas are small. Many of them found a foothold in the broader ecosystem — UTF-8, goroutines, /proc, containers, union filesystems, these all have their roots in Plan 9 — but many of its ideas, even the good ones, remain unexplored outside of Plan 9. As a consequence, Plan 9 exists at the center of a fervor of research achievements which forms a unique and profoundly interesting operating system.

One example I often raise to illustrate the design ideals of Plan 9 is to compare its approach to network programming with that of the Unix standard, Berkeley sockets. BSD sockets fly in the face of Unix sensibilities and are quite alien on the system, though by now everyone has developed stockholm syndrome with respect to them so they don’t notice. When everything is supposed to be a file on Unix, why is it that the networking API is entirely implemented with special-purpose syscalls and ioctls? On Unix, creating a TCP connection involves calling the “socket” syscall to create a magic file descriptor, then the “connect” syscall to establish a connection. Plan 9 is much more Unix in its approach: you open /net/tcp/clone to reserve a connection, and read the connection ID from it. Then you open /net/tcp/n/ctl and write “connect 127.0.0.1!80” to it, where “n” is that connection ID. Now you can open /net/tcp/n/data and that file is a full-duplex stream. No magic syscalls, and you can trivially implement it in a shell script.

This composes elegantly with another idea from Plan 9: the 9P protocol. All file I/O on the entire system uses the 9P protocol, which defines operations like read and write. This protocol is network transparent, and you can mount remote servers into your filesystem namespace and access their files over 9P. You can do something similar on Unix, but on Plan 9 you get much more mileage from the idea because everything is actually a file, and there are no magic syscalls or ioctls. For instance, your Ethernet interface is at /net/ether0, and everything in there is just a file. Say you want to establish a VPN: you simply mount a remote server’s /net/ether0 at /net/ether1, and now you have a VPN. That’s it.

The mountpoints are interesting as well, because they exist within a per-process filesystem namespace. Mounting filesystems does not require special permissions like on Unix, because these mounts only exist within the process tree that creates them, rather than modifying global state. The filesystems can also be implemented in userspace rather trivially via the 9P protocol, similar to FUSE but much more straightforward. Many programs provide a programmable/scriptable interface via a special filesystem such as this.

Userspace programs can also provide filesystems compatible with those normally implemented by kernel drivers, like /net/ether0, and provide these to processes in their namespace. For example, /dev/draw is analogous to a framebuffer device: you open it to write pixels to the screen. The window manager, Rio, implements a /dev/draw-like interface in userspace, then mounts it in the filesystem namespace of its children. All GUI programs can thus be run both on a framebuffer or in a window, without any awareness of which it’s using. The same is also true over the network: to implement VNC-like functionality, just mount your local /dev/draw and /dev/kbd on a remote server. Add /dev/audio if you like.

These ideas can also be built upon to form something resembling a container runtime, pre-dating even early concepts like BSD jails by several years, and implementing them much more effectively. Recall that everything really is just a file on Plan 9, unlike Unix. Access to the hardware is provided through normal files, and per-process namespaces do not require special permissions to modify mountpoints. Making a container is thus trivial: just unmount all of the hardware you don’t want the sandboxed program to have access to. Done. You don’t even have to be root. Want to forward a TCP port? Write an implementation of /net/tcp which is limited to whatever ports you need — perhaps with just a hundred lines of shell scripting — and mount it into the namespace.

The shell, rc, is also wonderful. The debugger is terribly interesting, and its ideas didn’t seem to catch on with the likes of gdb. The editors, acme and sam, are also interesting and present a unique user interface that you can’t find anywhere else. The plumber is cool, it’s like “what if xdg-open was good actually”. The kernel is concise and a pleasure to read. The entire operating system, kernel and userspace, can be built from source code on my 12 year old laptop in about 5 minutes. The network database, ndb, is brilliant. The entire OS is stuffed to the brim with interesting ideas, all of them implemented with elegance, conciseness, and simplicity.

Plan 9 failed, in a sense, because Unix was simply too big and too entrenched by the time Plan 9 came around. It was doomed by its predecessor. Nevertheless, its design ideas and implementation resonate deeply with me, and have provided an endless supply of inspiration for my own work. I think that everyone owes it to themselves to spend a few weeks messing around with and learning about Plan 9. The dream is kept alive by 9front, which is the most actively maintained fork of Plan 9 available today. Install it on your ThinkPad and mess around.

I will offer a caveat, however: leave your expectations at the door. Plan 9 is not Unix, it is not Unix-compatible, and it is certainly not yet another Linux distribution. Everything you’re comfortable and familiar with in your normal Unix setup will not translate to Plan 9. Come to Plan 9 empty handed, and let it fill those hands with its ideas. You will come away from the experience as a better programmer.

Notes from kernel hacking in Hare, part 3: serial driver

Thu, 27 Oct 2022 00:00:00 +0000

Today I would like to show you the implementation of the first userspace driver for Helios: a simple serial driver. All of the code we’re going to look at today runs in userspace, not in the kernel, so strictly speaking this should be “notes from OS hacking in Hare”, but I won’t snitch if you don’t.

Note: In the previous entry to this series, I promised to cover the userspace threading API in this post. I felt like covering this instead. Sorry!

A serial port provides a simple protocol for transferring data between two systems. It generalizes a bit, but for our purposes we can just think of this as a terminal which you can use over a simple cable and a simple protocol. It’s a standard x86_64 feature (though one which has been out of style for a couple of decades now), and its simple design (and high utility) makes it a good choice for the first driver to write for Helios. We’re going to look at the following details today:

The system’s initramfs
The driver loader
The serial driver itself

The initramfs used by Helios, for the time being, is just a tarball. I imported format::tar from the standard library, a module which I designed for this express purpose, and made a few minor tweaks to make it suitable for Helios’ needs. I also implemented seeking within a tar entry to make it easier to write an ELF loader from it. The bootloader loads this tarball into memory, the kernel provides page capabilities to init for it, and then we can map it into memory and study it, something like this:

let base = rt::map_range(rt::vspace, 0, 0, &desc.pages)!;
let slice = (base: *[*]u8)[..desc.length];

const buf = bufio::fixed(slice, io::mode::READ);
const rd = tar::read(&buf);

Pulling a specific driver out of it looks like this:

// Loads a driver from the bootstrap tarball.
fn earlyload(fs: *bootstrapfs, path: str) *process = {
	tar::reset(&fs.rd)!;
	path = strings::ltrim(path, '/');

	for (true) {
		const ent = match (tar::next(&fs.rd)) {
		case io::EOF =>
			break;
		case let ent: tar::entry =>
			yield ent;
		case let err: tar::error =>
			abort("Invalid bootstrap.tar file");
		};
		defer tar::skip(&ent)!;

		if (ent.name == path) {
			// TODO: Better error handling here
			const proc = match (load_driver(&ent)) {
			case let err: io::error =>
				abort("Failed to load driver from boostrap");
			case let err: errors::error =>
				abort("Failed to load driver from boostrap");
			case let proc: *process =>
				yield proc;
			};
			helios::task_resume(proc.task)!;
			return proc;
		};
	};

	abort("Missing bootstrap driver");
};

This code finds a file in the tarball with the given path (e.g. drivers/serial), creates a process with the driver loader, then resumes the thread and the driver is running. Let’s take a look at that driver loader next. The load_driver entry point takes an I/O handle to an ELF file and loads it:

fn load_driver(image: io::handle) (*process | io::error | errors::error) = {
	const loader = newloader(image);
	let earlyconf = driver_earlyconfig {
		cspace_radix = 12,
	};
	load_earlyconfig(&earlyconf, &loader)?;

	let proc = newprocess(earlyconf.cspace_radix)?;
	load(&loader, proc)?;
	load_config(proc, &loader)?;

	let regs = helios::context {
		rip = loader.header.e_entry,
		rsp = INIT_STACK_ADDR,
		...
	};
	helios::task_writeregisters(proc.task, &regs)?;
	return proc;
};

This is essentially a standard ELF loader, which it calls via the more general “newprocess” and “load” functions, but drivers have an extra concern: the driver manifest. The “load_earlyconfig” processes manifest keys which are necessary to configure prior to loading the ELF image, and the “load_config” function takes care of the rest of the driver configuration. The remainder of the code configures the initial thread.

The actual driver manifest is an INI file which is embedded in a special ELF section in driver binaries. The manifest for the serial driver looks like this:

[driver]
name=pcserial
desc=Serial driver for x86_64 PCs

[cspace]
radix=12

[capabilities]
0:serial =
1:note = 
2:cspace = self
3:ioport = min=3F8, max=400
4:ioport = min=2E8, max=2F0
5:irq = irq=3, note=1
6:irq = irq=4, note=1

Helios is a capability-oriented system, and in order to do anything useful, each process needs to have capabilities to work with. Each driver declares exactly what capabilities it needs and receives only these capabilities, and nothing else. This provides stronger isolation than Unix systems can offer (even with something like OpenBSD’s pledge(2)) — this driver cannot even allocate memory.

A standard x86_64 ISA serial port uses two I/O port ranges, 0x3F8-0x400 and 0x2E8-0x2F0, as well as two IRQs, IRQ 3 and 4, together providing support for up to four serial ports. The driver first requests a “serial” capability, which is a temporary design for an IPC endpoint that the driver will use to actually process read or write requests. This will be replaced with a more sophisticated device manager system in the future. It also creates a notification capability, which is later used to deliver the IRQs, and requests a capability for its own cspace so that it can manage capability slots. This will be necessary later on. Following this it requests capabilities for the system resources it needs, namely the necessary I/O ports and IRQs, the latter configured to be delivered to the notification in capability slot 1.

With the driver isolated in its own address space, running in user mode, and only able to invoke this set of capabilities, it’s very limited in what kind of exploits it’s vulnerable to. If there’s a vulnerability here, the worst that could happen is that a malicious actor on the other end of the serial port could crash the driver, which would then be rebooted by the service manager. On Linux, a bug in the serial driver can be used to compromise the entire system.

So, the driver loader parses this file and allocates the requested capabilities for the driver. I’ll skip most of the code, it’s just a boring INI file parser, but the important bit is the table for capability allocations:

type capconfigfn = fn(
	proc: *process,
	addr: uint,
	config: const str,
) (void | errors::error);

// Note: keep these tables alphabetized
const capconfigtab: [_](const str, *capconfigfn) = [
	("cspace", &cap_cspace),
	("endpoint", &cap_endpoint),
	("ioport", &cap_ioport),
	("irq", &cap_irq),
	("note", &cap_note),
	("serial", &cap_serial),
	// TODO: More
];

This table defines functions which, when a given INI key in the [capabilities] section is found, provisions the requested capabilities. This list is not complete; in the future all kernel objects will be added as well as userspace-defined interfaces (similar to serial) which implement various driver interfaces, such as ‘fs’ or ‘gpu’. Let’s start with the notification capability:

fn cap_note(
	proc: *process,
	addr: uint,
	config: const str,
) (void | errors::error) = {
	if (config != "") {
		return errors::invalid;
	};
	const note = helios::newnote()?;
	defer helios::destroy(note)!;
	helios::copyto(proc.cspace, addr, note)?;
};

This capability takes no configuration arguments, so we first simply check that the value is empty. Then we create a notification, copy it into the driver’s capability space at the requested capability address, then destroy our copy. Simple!

The I/O port capability is a bit more involved: it does accept configuration parameters, namely what I/O port range the driver needs.

fn cap_ioport(
	proc: *process,
	addr: uint,
	config: const str,
) (void | errors::error) = {
	let min = 0u16, max = 0u16;
	let have_min = false, have_max = false;

	const tok = strings::tokenize(config, ",");
	for (true) {
		let tok = match (strings::next_token(&tok)) {
		case void =>
			break;
		case let tok: str =>
			yield tok;
		};
		tok = strings::trim(tok);

		const (key, val) = strings::cut(tok, "=");
		let field = switch (key) {
		case "min" =>
			have_min = true;
			yield &min;
		case "max" =>
			have_max = true;
			yield &max;
		case =>
			return errors::invalid;
		};

		match (strconv::stou16b(val, base::HEX)) {
		case let u: u16 =>
			*field = u;
		case =>
			return errors::invalid;
		};
	};

	if (!have_min || !have_max) {
		return errors::invalid;
	};

	const ioport = helios::ioctl_issue(rt::INIT_CAP_IOCONTROL, min, max)?;
	defer helios::destroy(ioport)!;
	helios::copyto(proc.cspace, addr, ioport)?;
};

Here we split the configuration string on commas and parse each as a key/value pair delimited by an equal sign (”=”), looking for a key called “min” and another called “max”. At the moment the config parsing is just implemented in this function directly, but in the future it might make sense to write a small abstraction for capability configurations like this. Once we know the I/O port range the user wants, then we issue an I/O port capability for that range and copy it into the driver’s cspace.

IRQs are a bit more involved still. An IRQ capability must be configured to deliver IRQs to a notification object.

fn cap_irq(
	proc: *process,
	addr: uint,
	config: const str,
) (void | errors::error) = {
	let irq = 0u8, note: helios::cap = 0;
	let have_irq = false, have_note = false;

	// ...config string parsing omitted...

	const _note = helios::copyfrom(proc.cspace, note, helios::CADDR_UNDEF)?;
	defer helios::destroy(_note)!;
	const (ct, _) = rt::identify(_note)!;
	if (ct != ctype::NOTIFICATION) {
		// TODO: More semantically meaningful errors would be nice
		return errors::invalid;
	};

	const irq = helios::irqctl_issue(rt::INIT_CAP_IRQCONTROL, _note, irq)?;
	defer helios::destroy(irq)!;
	helios::copyto(proc.cspace, addr, irq)?;
};

In order to do this, the driver loader copies the notification capability from the driver’s cspace and into the loader’s cspace, then creates an IRQ with that notification. It copies the new IRQ capability into the driver, then destroys its own copy of the IRQ and notification.

In this manner, the driver can declaratively state which capabilities it needs, and the loader can prepare an environment for it with these capabilities prepared. Once these capabilities are present in the driver’s cspace, the driver can invoke them by addressing the numbered capability slots in a send or receive syscall.

To summarize, the loader takes an I/O object (which we know is sourced from the bootstrap tarball) from which an ELF file can be read, finds a driver manifest, then creates a process and fills the cspace with the requested capabilities, loads the program into its address space, and starts the process.

Next, let’s look at the serial driver that we just finished loading.

Let me first note that this serial driver is a proof-of-concept at this time. A future serial driver will take a capability for a device manager object, then probe each serial port and provision serial devices for each working serial port. It will define an API which supports additional serial-specific features, such as configuring the baud rate. For now, it’s pretty basic.

This driver implements a simple event loop:

Configure the serial port
Wait for an interrupt or a read/write request from the user
On interrupt, process the interrupt, writing buffered data or buffering readable data
On a user request, buffer writes or unbuffer reads
GOTO 2

The driver starts by defining some constants for the capability slots we set up in the manifest:

def EP: helios::cap = 0;
def IRQ: helios::cap = 1;
def CSPACE: helios::cap = 2;
def IRQ3: helios::cap = 5;
def IRQ4: helios::cap = 6;

It also defines some utility code for reading and writing to the COM registers, and constants for each of the registers defined by the interface.

// COM1 port
def COM1: u16 = 0x3F8;

// COM2 port
def COM2: u16 = 0x2E8;

// Receive buffer register
def RBR: u16 = 0;

// Transmit holding regiser
def THR: u16 = 0;

// ...other registers omitted...

const ioports: [_](u16, helios::cap) = [
	(COM1, 3), // 3 is the I/O port capability address
	(COM2, 4),
];

fn comin(port: u16, register: u16) u8 = {
	for (let i = 0z; i < len(ioports); i += 1) {
		const (base, cap) = ioports[i];
		if (base != port) {
			continue;
		};
		return helios::ioport_in8(cap, port + register)!;
	};
	abort("invalid port");
};

fn comout(port: u16, register: u16, val: u8) void = {
	for (let i = 0z; i < len(ioports); i += 1) {
		const (base, cap) = ioports[i];
		if (base != port) {
			continue;
		};
		helios::ioport_out8(cap, port + register, val)!;
		return;
	};
	abort("invalid port");
};

We also define some statically-allocated data structures to store state for each COM port, and a function to initialize the port:

type comport = struct {
	port: u16,
	rbuf: [4096]u8,
	wbuf: [4096]u8,
	rpending: []u8,
	wpending: []u8,
};

let ports: [_]comport = [
	comport { port = COM1, ... },
	comport { port = COM2, ... },
];

fn com_init(com: *comport) void = {
	com.rpending = com.rbuf[..0];
	com.wpending = com.wbuf[..0];
	comout(com.port, IER, 0x00);	// Disable interrupts
	comout(com.port, LCR, 0x80);	// Enable divisor mode
	comout(com.port, DL_LSB, 0x01);	// Div Low:  01: 115200 bps
	comout(com.port, DL_MSB, 0x00);	// Div High: 00
	comout(com.port, LCR, 0x03);	// Disable divisor mode, set parity
	comout(com.port, FCR, 0xC7);	// Enable FIFO and clear
	comout(com.port, IER, ERBFI);	// Enable read interrupt
};

The basics are in place. Let’s turn our attention to the event loop.

export fn main() void = {
	com_init(&ports[0]);
	com_init(&ports[1]);

	helios::irq_ack(IRQ3)!;
	helios::irq_ack(IRQ4)!;

	let poll: [_]pollcap = [
		pollcap { cap = IRQ, events = pollflags::RECV },
		pollcap { cap = EP, events = pollflags::RECV },
	];
	for (true) {
		helios::poll(poll)!;
		if (poll[0].events & pollflags::RECV != 0) {
			poll_irq();
		};
		if (poll[1].events & pollflags::RECV != 0) {
			poll_endpoint();
		};
	};
};

We initialize two COM ports first, using the function we were just reading. Then we ACK any IRQs that might have already been pending when the driver starts up, and we enter the event loop proper. Here we are polling on two capabilities, the notification to which IRQs are delivered, and the endpoint which provides the serial driver’s external API.

The state for each serial port includes a read buffer and a write buffer, defined in the comport struct shown earlier. We configure the COM port to interrupt when there’s data available to read, then pull it into the read buffer. If we have pending data to write, we configure it to interrupt when it’s ready to write more data, otherwise we leave this interrupt turned off. The “poll_irq” function handles these interrupts:

fn poll_irq() void = {
	helios::wait(IRQ)!;
	defer helios::irq_ack(IRQ3)!;
	defer helios::irq_ack(IRQ4)!;

	for (let i = 0z; i < len(ports); i += 1) {
		const iir = comin(ports[i].port, IIR);
		if (iir & 1 == 0) {
			port_irq(&ports[i], iir);
		};
	};
};

fn port_irq(com: *comport, iir: u8) void = {
	if (iir & (1 << 2) != 0) {
		com_read(com);
	};
	if (iir & (1 << 1) != 0) {
		com_write(com);
	};
};

The IIR register is the “interrupt identification register”, which tells us why the interrupt occurred. If it was because the port is readable, we call “com_read”. If the interrupt occurred because the port is writable, we call “com_write”. Let’s start with com_read. This interrupt is always enabled so that we can immediately start buffering data as the user types it into the serial port.

// Reads data from the serial port's RX FIFO.
fn com_read(com: *comport) size = {
	let n: size = 0;
	for (comin(com.port, LSR) & RBF == RBF; n += 1) {
		const ch = comin(com.port, RBR);
		if (len(com.rpending) < len(com.rbuf)) {
			// If the buffer is full we just drop chars
			static append(com.rpending, ch);
		};
	};

	// This part will be explained later:
	if (pending_read.reply != 0) {
		const n = rconsume(com, pending_read.buf);
		helios::send(pending_read.reply, 0, n)!;
		pending_read.reply = 0;
	};

	return n;
};

This code is pretty simple. For as long as the COM port is readable, read a character from it. If there’s room in the read buffer, append this character to it.

How about writing? Well, we need some way to fill the write buffer first. This part is pretty straightforward:

// Append data to a COM port read buffer, returning the number of bytes buffered
// successfully.
fn com_wbuffer(com: *comport, data: []u8) size = {
	let z = len(data);
	if (z + len(com.wpending) > len(com.wbuf)) {
		z = len(com.wbuf) - len(com.wpending);
	};
	static append(com.wpending, data[..z]...);
	com_write(com);
	return z;
};

This code just adds data to the write buffer, making sure not to exceed the buffer length (note that in Hare this would cause an assertion, not a buffer overflow). Then we call “com_write”, which does the actual writing to the COM port.

// Writes data to the serial port's TX FIFO.
fn com_write(com: *comport) size = {
	if (comin(com.port, LSR) & THRE != THRE) {
		const ier = comin(com.port, IER);
		comout(com.port, IER, ier | ETBEI);
		return 0;
	};

	let i = 0z;
	for (i < 16 && len(com.wpending) != 0; i += 1) {
		comout(com.port, THR, com.wpending[0]);
		static delete(com.wpending[0]);
	};

	const ier = comin(com.port, IER);
	if (len(com.wpending) == 0) {
		comout(com.port, IER, ier & ~ETBEI);
	} else {
		comout(com.port, IER, ier | ETBEI);
	};

	return i;
};

If the COM port is not ready to write data, we enable an interrupt which will tell us when it is and return. Otherwise, we write up to 16 bytes — the size of the COM port’s FIFO — and remove them from the write buffer. If there’s more data to write, we enable the write interrupt, or we disable it if there’s nothing left. When enabled, this will cause an interrupt to fire when (1) we have data to write and (2) the serial port is ready to write it, and our event loop will call this function again.

That covers all of the code for driving the actual serial port. What about the interface for someone to actually use this driver?

The “serial” capability defined in the manifest earlier is a temporary construct to provision some means of communicating with the driver. It provisions an endpoint capability (which is an IPC primitive on Helios) and stashes it away somewhere in the init process so that I can write some temporary test code to actually read or write to the serial port. Either request is done by “call”ing the endpoint with the desired parameters, which will cause the poll in the event loop to wake as the endpoint becomes receivable, calling “poll_endpoint”.

fn poll_endpoint() void = {
	let addr = 0u64, amt = 0u64;
	const tag = helios::recv(EP, &addr, &amt);
	const label = rt::label(tag);
	switch (label) {
	case 0 =>
		const addr = addr: uintptr: *[*]u8;
		const buf = addr[..amt];
		const z = com_wbuffer(&ports[0], buf);
		helios::reply(0, z)!;
	case 1 =>
		const addr = addr: uintptr: *[*]u8;
		const buf = addr[..amt];
		if (len(ports[0].rpending) == 0) {
			const reply = helios::store_reply(helios::CADDR_UNDEF)!;
			pending_read = read {
				reply = reply,
				buf = buf,
			};
		} else {
			const n = rconsume(&ports[0], buf);
			helios::reply(0, n)!;
		};
	case =>
		abort(); // TODO: error
	};
};

“Calls” in Helios work similarly to seL4. Essentially, when you “call” an endpoint, the calling thread blocks to receive the reply and places a reply capability in the receiver’s thread state. The receiver then processes their message and “replies” to the reply capability to wake up the calling thread and deliver the reply.

The message label is used to define the requested operation. For now, 0 is read and 1 is write. For writes, we append the provided data to the write buffer and reply with the number of bytes we buffered, easy breezy.

Reads are a bit more involved. If we don’t immediately have any data in the read buffer, we have to wait until we do to reply. We copy the reply from its special slot in our thread state into our capability space, so we can use it later. This operation is why our manifest requires cspace = self. Then we store the reply capability and buffer in a variable and move on, waiting for a read interrupt. On the other hand, if there is data buffered, we consume it and reply immediately.

fn rconsume(com: *comport, buf: []u8) size = {
	let amt = len(buf);
	if (amt > len(ports[0].rpending)) {
		amt = len(ports[0].rpending);
	};
	buf[..amt] = ports[0].rpending[..amt];
	static delete(ports[0].rpending[..amt]);
	return amt;
};

Makes sense?

That basically covers the entire serial driver. Let’s take a quick peek at the other side: the process which wants to read from or write to the serial port. For the time being this is all temporary code to test the driver with, and not the long-term solution for passing out devices to programs. The init process keeps a list of serial devices configured on the system:

type serial = struct {
	proc: *process,
	ep: helios::cap,
};

let serials: []serial = [];

fn register_serial(proc: *process, ep: helios::cap) void = {
	append(serials, serial {
		proc = proc,
		ep = ep,
	});
};

This function is called by the driver manifest parser like so:

fn cap_serial(
	proc: *process,
	addr: uint,
	config: const str,
) (void | errors::error) = {
	if (config != "") {
		return errors::invalid;
	};
	const ep = helios::newendpoint()?;
	helios::copyto(proc.cspace, addr, ep)?;
	register_serial(proc, ep);
};

We make use of the serial port in the init process’s main function with a little test loop to echo reads back to writes:

export fn main(bi: *rt::bootinfo) void = {
	log::println("[init] Hello from Mercury!");

	const bootstrap = bootstrapfs_init(&bi.modules[0]);
	defer bootstrapfs_finish(&bootstrap);
	earlyload(&bootstrap, "/drivers/serial");

	log::println("[init] begin echo serial port");
	for (true) {
		let buf: [1024]u8 = [0...];
		const n = serial_read(buf);
		serial_write(buf[..n]);
	};
};

The “serial_read” and “serial_write” functions are:

fn serial_write(data: []u8) size = {
	assert(len(data) <= rt::PAGESIZE);
	const page = helios::newpage()!;
	defer helios::destroy(page)!;
	let buf = helios::map(rt::vspace, 0, map_flags::W, page)!: *[*]u8;
	buf[..len(data)] = data[..];
	helios::page_unmap(page)!;

	// TODO: Multiple serial ports
	const port = &serials[0];
	const addr: uintptr = 0x7fff70000000; // XXX arbitrary address
	helios::map(port.proc.vspace, addr, 0, page)!;

	const reply = helios::call(port.ep, 0, addr, len(data));
	return rt::ipcbuf.params[0]: size;
};

fn serial_read(buf: []u8) size = {
	assert(len(buf) <= rt::PAGESIZE);
	const page = helios::newpage()!;
	defer helios::destroy(page)!;

	// TODO: Multiple serial ports
	const port = &serials[0];
	const addr: uintptr = 0x7fff70000000; // XXX arbitrary address
	helios::map(port.proc.vspace, addr, map_flags::W, page)!;

	const (label, n) = helios::call(port.ep, 1, addr, len(buf));

	helios::page_unmap(page)!;

	let out = helios::map(rt::vspace, 0, 0, page)!: *[*]u8;
	buf[..n] = out[..n];
	return n;
};

There is something interesting going on here. Part of this code is fairly obvious — we just invoke the IPC endpoint using helios::call, corresponding nicely to the other end’s use of helios::reply, with the buffer address and size. However, the buffer address presents a problem: this buffer is in the init process’s address space, so the serial port cannot read or write to it!

In the long term, a more sophisticated approach to shared memory management will be developed, but for testing purposes I came up with this solution. For writes, we allocate a new page, map it into our address space, and copy the data we want to write to it. Then we unmap it, map it into the serial driver’s address space instead, and perform the call. For reads, we allocate a page, map it into the serial driver, call the IPC endpoint, then unmap it from the serial driver, map it into our address space, and copy the data back out of it. In both cases, we destroy the page upon leaving this function, which frees the memory and automatically unmaps the page from any address space. Inefficient, but it works for demonstration purposes.

And that’s really all there is to it! Helios officially has its first driver. The next step is to develop a more robust solution for describing capability interfaces and device APIs, then build a PS/2 keyboard driver and a BIOS VGA mode 3 driver for driving the BIOS console, and combine these plus the serial driver into a tty on which we can run a simple shell.

TOTP for 2FA is incredibly easy to implement. So what's your excuse?

Tue, 18 Oct 2022 00:00:00 +0000

Time-based one-time passwords are one of the more secure approaches to 2FA — certainly much better than SMS. And it’s much easier to implement than SMS as well. The algorithm is as follows:

Divide the current Unix timestamp by 30
Encode it as a 64-bit big endian integer
Write the encoded bytes to a SHA-1 HMAC initialized with the TOTP shared key
Let offs = hmac[-1] & 0xF
Let hash = decode hmac[offs .. offs + 4] as a 32-bit big-endian integer
Let code = (hash & 0x7FFFFFFF) % 1000000
Compare this code with the user’s code

You’ll need a little dependency to generate QR codes with the otpauth:// URL scheme, a little UI to present the QR code and store the shared secret in your database, and a quick update to your login flow, and then you’re good to go.

Here’s the implementation SourceHut uses in Python. I hereby release this code into the public domain, or creative commons zero, at your choice:

import base64
import hashlib
import hmac
import struct
import time

def totp(secret, token):
    tm = int(time.time() / 30)
    key = base64.b32decode(secret)

    for ix in range(-2, 3):
        b = struct.pack(">q", tm + ix)
        hm = hmac.HMAC(key, b, hashlib.sha1).digest()
        offset = hm[-1] & 0x0F
        truncatedHash = hm[offset:offset + 4]
        code = struct.unpack(">L", truncatedHash)[0]
        code &= 0x7FFFFFFF
        code %= 1000000
        if token == code:
            return True

    return False

This implementation has a bit of a tolerance added to make clock skew less of an issue, but that also means that the codes are longer-lived. Feel free to edit these tolerances if you so desire.

Here’s another one written in Hare, also public domain/CC-0.

use crypto::hmac;
use crypto::mac;
use crypto::sha1;
use encoding::base32;
use endian;
use time;

// Computes a TOTP code for a given time and key.
export fn totp(when: time::instant, key: []u8) uint = {
	const now = time::unix(when) / 30;
	const hmac = hmac::sha1(key);
	defer mac::finish(&hmac);

	let buf: [8]u8 = [0...];
	endian::beputu64(buf, now: u64);
	mac::write(&hmac, buf);

	let mac: [sha1::SIZE]u8 = [0...];
	mac::sum(&hmac, mac);

	const offs = mac[len(mac) - 1] & 0xF;
	const hash = mac[offs..offs+4];
	return ((endian::begetu32(hash)& 0x7FFFFFFF) % 1000000): uint;
};

@test fn totp() void = {
	const secret = "3N2OTFHXKLR2E3WNZSYQ====";
	const key = base32::decodestr(&base32::std_encoding, secret)!;
	defer free(key);
	const now = time::from_unix(1650183739);
	assert(totp(now, key) == 29283);
};

In any language, TOTP is just a couple of dozen lines of code even if there isn’t already a library — and there is probably already a library. You don’t have to store temporary SMS codes in the database, you don’t have to worry about phishing, you don’t have to worry about SIM swapping, and you don’t have to sign up for some paid SMS API like Twilio. It’s more secure and it’s trivial to implement — so implement it already! Please!

Update 2022-10-19 @ 07:45 UTC: A reader pointed out that it’s important to have rate limiting on your TOTP attempts, or else a brute force attack can be effective. Fair point!

Status update, October 2022

Sat, 15 Oct 2022 00:00:00 +0000

After a few busy and stressful months, I decided to set aside October to rest. Of course, for me, rest does not mean a cessation of programming, but rather a shift in priorities towards more fun and experimental projects. Consequently, it has been a great month for Helios!

Hare upstream has enjoyed some minor improvements, such as from Pierre Curto’s patch to support parsing IPv6 addresses with a port (e.g. “[::1]:80”) and Kirill Primak’s improvements to the UTF-8 decoder. On the whole, improvements have been conservative. However, queued up for integration once qbe upstream support is merged is support for @threadlocal variables, which are useful for Helios and for ABI compatibility with C. I also drafted up a proof-of-concept for @inline functions, but it still needs work.

Now for the main event: Helios. The large-scale redesign and refactoring I mentioned in the previous status update is essentially complete, and the kernel reached (and exceeded) feature parity with the previous status quo. Since Helios has been my primary focus for the past couple of weeks, I have a lot of news to share about it.

First, I got back into userspace a few days after the last status update, and shortly thereafter implemented a new scheduler. I then began to rework the userspace API (uapi) in the kernel, which differs substantially from its prior incarnation. The kernel object implementations present themselves as a library for kernel use, and the new uapi module handles all interactions with this module from userspace, providing a nice separation of concerns. The uapi module handles more than syscalls now — it also implements send/recv for kernel objects, for instance. As of a few days ago, uapi also supports delivering faults to userspace supervisor processes:

A screenshot of a thread on Helios causing a page fault, then its parent thread receives details of the fault and maps a page onto the address of the attempted write. The child thread is resumed and is surprised to find that the write succeeded (because a page was mapped underneath the write).

@test fn task::pagefault() void = {
	const fault = helios::newendpoint()!;
	defer helios::destroy(fault)!;

	const thread = threads::new(&_task_pagefault)!;
	threads::set_fault(thread, fault)!;
	threads::start(thread)!;

	const fault = helios::recv_fault(fault);
	assert(fault.addr == 0x100);

	const page = helios::newpage()!;
	defer helios::destroy(page)!;
	helios::map(rt::vspace, 0, map_flags::W | map_flags::FIXED, page)!;

	threads::resume(thread)!;
	threads::join(thread)!;
};

fn _task_pagefault() void = {
	let ptr: *int = 0x100: uintptr: *int;
	*ptr = 1337;
	assert(*ptr == 1337);
};

The new userspace threading API is much improved over the hack job in the earlier design. It supports TLS and many typical threading operations, such as join and detach. This API exists mainly for testing the kernel via Vulcan, and is not anticipated to see much use beyond this (though I will implement pthreads for the POSIX C environment at some point). For more details, see this blog post. Alongside this and other userspace libraries, Vulcan has been fleshed out into a kernel test suite once again, which I have been frequently testing on real hardware:

A picture of a laptop showing 15 passing kernel tests

Here’s an ISO you can boot on your own x86_64 hardware to see if it works for you, too. If you have problems, take a picture of the issue, boot Linux and email me said picture, the output of lscpu, and any other details you deem relevant.

The kernel now supports automatic capability address allocation, which is a marked improvement over seL4. The new physical page allocator is also much improved, as it supports allocation and freeing and can either allocate pages sparsely or continuously depending on the need. Mapping these pages in userspace was also much improved, with a better design of the userspace virtual memory map and a better heap, complete with a (partial) implementation of mmap.

I have also broken ground on the next component of the OS, Mercury, which provides a more complete userspace environment for writing drivers. It has a simple tar-based initramfs based on Hare’s format::tar implementation, which I wrote in June for this purpose. It can load ELF files from this tarball into new processes, and implements some extensions that are useful for driver loading. Consequently, the first Mercury driver is up and running:

Demo of a working serial driver

This driver includes a simple driver manifest, which is embedded into its ELF file and processed by the driver loader to declaratively specify the capabilities it needs:

[driver]
name=pcserial
desc=Serial driver for x86_64 PCs

[cspace]
radix=12

[capabilities]
0:endpoint =
1:ioport = min=3F8, max=400
2:ioport = min=2E8, max=2F0
3:note = 
4:irq = irq=3, note=3
5:irq = irq=4, note=3

The driver loader prepares capabilities for the COM1 and COM2 I/O ports, as well as IRQ handlers for IRQ 3 and 4, based on this manifest, then loads them into the capability table for the driver process. The driver is sandboxed very effectively by this: it can only use these capabilities. It cannot allocate memory, modify its address space, or even destroy any of these capabilities. If a bad actor was on the other end of the serial port and exploited a bug, the worst thing it could do is crash the serial driver, which would then be rebooted by the supervisor. On Linux and other monolithic kernels like it, exploiting the serial driver compromises the entire operating system.

The resulting serial driver implementation is pretty small and straightforward, if you’d like to have a look.

This manifest format will be expanded in the future for additional kinds of drivers, such as with details specific to each bus (i.e. PCI vendor information or USB details), and will also have details for device trees when RISC-V and ARM support (the former is already underway) are brought upstream.

Next steps are to implement an I/O abstraction on top of IPC endpoints, which first requires call & reply support — the latter was implemented last night and requires additional testing. Following this, I plan on writing a getty-equivalent which utilizes this serial driver, and a future VGA terminal driver, to provide an environment in which a shell can be run. Then I’ll implement a ramfs to host commands for the shell to run, and we’ll really be cookin’ at that point. Disk drivers and filesystem drivers will be next.

That’s all for now. Quite a lot of progress! I’ll see you next time.

In praise of ffmpeg

Wed, 12 Oct 2022 00:00:00 +0000

My last “In praise of” article covered qemu, a project founded by Fabrice Bellard, and today I want to take a look at another work by Bellard: ffmpeg. Bellard has a knack for building high-quality software which solves a problem so well that every other solution becomes obsolete shortly thereafter, and ffmpeg is no exception.

ffmpeg has been described as the Swiss army knife of multimedia. It incorporates hundreds of video, audio, and image decoders and encoders, muxers and demuxers, filters and devices. It provides a CLI and a set of libraries for working with its tools, and is the core component of many video and audio players as a result (including my preferred multimedia player, mpv). If you want to do almost anything with multimedia files — re-encode them, re-mux them, live stream it, whatever — ffmpeg can handle it with ease.

Let me share an example.

I was recently hanging out at my local hackerspace and wanted to play some PS2 games on my laptop. My laptop is not powerful enough to drive PCSX2, but my workstation on the other side of town certainly was. So I forwarded my game controller to my workstation via USB/IP and pulled up the ffmpeg manual to figure out how to live-stream the game to my laptop. ffmpeg can capture video from KMS buffers directly, use the GPU to efficiently downscale them, grab audio from pulse, encode them with settings tuned for low-latency, and mux it into a UDP socket. On the other end I set up mpv to receive the stream and play it back.

ffmpeg \
  -f pulse \
  -i alsa_output.platform-snd_aloop.0.analog-surround-51.monitor \
  -f kmsgrab \
  -thread_queue_size 64 \   # reduce input latency
  -i - \
  # Capture and downscale frames on the GPU:
  -vf 'hwmap=derive_device=vaapi,scale_vaapi=1280:720,hwdownload,format=bgr0' \
  -c:v libx264 \
  -preset:v superfast \     # encode video as fast as possible
  -tune zerolatency \       # tune encoder for low latency
  -intra-refresh 1 \        # reduces latency and mitigates dropped packets
  -f mpegts \               # mux into mpegts stream, well-suited to this use-case
  -b:v 3M \                 # configure target video bandwidth
  udp://$hackerspace:41841

With an hour of tinkering and reading man pages, I was able to come up with a single command which produced a working remote video game streaming setup from scratch thanks to ffmpeg. ffmpeg is amazing.

I have relied on ffmpeg for many tasks and for many years. It has always been there to handle any little multimedia-related task I might put it to for personal use — re-encoding audio files so they fit on my phone, taking clips from videos to share, muxing fonts into mkv files, capturing video from my webcam, live streaming hacking sessions on my own platform, or anything else I can imagine. It formed the foundation of MediaCrush back in the day, where we used it to optimize multimedia files for efficient viewing on the web, back when that was more difficult than “just transcode it to a webm”.

ffmpeg is notable for being one of the first large-scale FOSS projects to completely eradicate proprietary software in its niche. Virtually all multimedia-related companies rely on ffmpeg to do their heavy lifting. It took a complex problem and solved it, with free software. The book is now closed on multimedia: ffmpeg is the solution to almost all of your problems. And if it’s not, you’re more likely to patch ffmpeg than to develop something new. The code is accessible and the community are experts in your problem domain.

ffmpeg is one of the foremost pillars of achievement in free software. It has touched the lives of every reader, whether they know it or not. If you’ve ever watched TV, or gone to a movie, or watched videos online, or listened to a podcast, odds are that ffmpeg was involved in making it possible. It is one of the most well-executed and important software projects of all time.

Does Rust belong in the Linux kernel?

Mon, 03 Oct 2022 00:00:00 +0000

I am known to be a bit of a polemic when it comes to Rust. I will be forthright with the fact that I don’t particularly care for Rust, and that my public criticisms of it might set up many readers with a reluctance to endure yet another Rust Hot Take from my blog. My answer to the question posed in the title is, of course, “no”. However, let me assuage some of your fears by answering a different question first: does Hare belong in the Linux kernel?

If I should owe my allegiance to any programming language, it would be Hare. Not only is it a systems programming language that I designed myself, but I am using it to write a kernel. Like Rust, Hare is demonstrably useful for writing kernels with. One might even go so far as to suggest that I consider it superior to C for this purpose, given that I chose to to write Helios in Hare rather than C, despite my extensive background in C. But the question remains: does Hare belong in the Linux kernel?

In my opinion, Hare does not belong in the Linux kernel, and neither does Rust. Some of the reasoning behind this answer is common to both, and some is unique to each, but I will be focusing on Rust today because Rust is the language which is actually making its way towards mainline Linux. I have no illusions about this blog post changing that, either: I simply find it an interesting case-study in software engineering decision-making in a major project, and that’s worth talking about.

Each change in software requires sufficient supporting rationale. What are the reasons to bring Rust into Linux? A kernel hacker thinks about these questions differently than a typical developer in userspace. One could espouse the advantages of Cargo, generics, whatever, but these concerns matter relatively little to kernel hackers. Kernels operate in a heavily constrained design space and a language has to fit into that design space. This is the first and foremost concern, and if it’s awkward to mold a language to fit into these constraints then it will be a poor fit.

Some common problems that a programming language designed for userspace will run into when being considered for kernelspace are:

Strict constraints on memory allocation
Strict constraints on stack usage
Strict constraints on recursion
No use of floating point arithmetic
Necessary evils, such as unsafe memory use patterns or integer overflow
The absence of a standard library, runtime, third-party libraries, or other conveniences typically afforded to userspace

Most languages can overcome these constraints with some work, but their suitability for kernel use is mainly defined by how well they adapt to them — there’s a reason that kernels written in Go, C#, Java, Python, etc, are limited to being research curiosities and are left out of production systems.

As Linus recently put it, “kernel needs trump any Rust needs”. The kernel is simply not an environment which will bend to accommodate a language; it must go the other way around. These constraints have posed, and will continue to pose, a major challenge for Rust in Linux, but on the whole, I think that it will be able to rise to meet them, though perhaps not with as much grace as I would like.

If Rust is able to work within these constraints, then it satisfies the ground rules for playing in ring 0. The question then becomes: what advantages can Rust bring to the kernel? Based on what I’ve seen, these essentially break down to two points:¹

Memory safety
Trendiness

I would prefer not to re-open the memory safety flamewar, so we will simply move forward with the (dubious) assumptions that memory safety is (1) unconditionally desirable, (2) compatible with the kernel’s requirements, and (3) sufficiently provided for by Rust. I will offer this quote from an unnamed kernel hacker, though:

There are possibly some well-designed and written parts which have not suffered a memory safety issue in many years. It’s insulting to present this as an improvement over what was achieved by those doing all this hard work.

Regarding “trendiness”, I admit that this is a somewhat unforgiving turn of phrase. In this respect I refer to the goal of expanding the kernel’s developer base from a bunch of aging curmudgeons writing C² towards a more inclusive developer pool from a younger up-and-coming language community like Rust. C is boring³ — it hasn’t really excited anyone in decades. Rust is exciting, and its community enjoys a huge pool of developers building their brave new world with it. Introducing Rust to the kernel will certainly appeal to a broader audience of potential contributors.

But there is an underlying assumption to this argument which is worth questioning: is the supply of Linux developers dwindling, and, if so, is it to such and extent that it demands radical change?

Well, no. Linux has consistently enjoyed a tremendous amount of attention from the software development community. This week’s release of Linux 6.0, one of the largest Linux releases ever, boasted more than 78,000 commits by almost 5,000 different authors since 5.15. Linux has a broad developer base reaching from many different industry stakeholders and independent contributors working on the careful development and maintenance of its hundreds of subsystems. The scale of Linux development is on a level unmatched by any other software project — free software or otherwise.

Getting Rust working in Linux is certainly an exciting project, and I’m all for developers having fun. However, it’s not likely to infuse Linux with a much-needed boost in its contributor base, because Linux has no such need. What’s more, Linux’s portability requirements prevent Rust from being used in most of the kernel in the first place. Most work on Rust in Linux is simply working on getting the systems to cooperate with each other or writing drivers which are redundant with existing C drivers, but cannot replace them due to Rust’s limited selection of targets.⁴ Few to none of the efforts from the Rust-in-Linux team are likely to support the kernel’s broader goals for some time.

We are thus left with memory safety as the main benefit offered by Rust to Linux, and for the purpose of this article we’re going to take it at face value. So, with the ground rules set and the advantages enumerated, what are some of the problems that Rust might face in Linux?

There are a few problems which could be argued over, such as substantial complexity of Rust compared to C, the inevitable doubling of Linux’s build time, the significant shift in design sensibilities required to support an idiomatic Rust design, the fragile interface which will develop on the boundaries between Rust and C code, or the challenges the kernel’s established base of C developers will endure when learning and adapting to a new language. To avoid letting this post become too subjective or lengthy, I’ll refrain from expanding on these. Instead, allow me to simply illuminate these issues as risk factors.

Linux is, on the whole, a conservative project. It is deployed worldwide in billions of devices and its reliability is depended on by a majority of Earth’s population. Risks are carefully evaluated in Linux as such. Every change presents risks and offers advantages, which must be weighed against each other to justify the change. Rust is one of the riskiest bets Linux has ever considered, and, in my opinion, the advantages may not weigh up. I think that the main reason we’re going to see Rust in the kernel is not due to a careful balancing of risk and reward, but because the Rust community wants Rust in Linux, and they’re large and loud enough to not be worth the cost of arguing with.

I don’t think that changes on this scale are appropriate for most projects. I prefer to encourage people to write new software to replace established software, rather than rewriting the established software. Some projects, such as Redox, are doing just that with Rust. However, operating systems are in a difficult spot in this respect. Writing an operating system is difficult work with a huge scope — few projects can hope to challenge Linux on driver support, for example. The major players have been entrenched for decades, and any project seeking to displace them will have decades of hard work ahead of them and will require a considerable amount of luck to succeed. Though I think that new innovations in kernels are badly overdue, I must acknowledge that there is some truth to the argument that we’re stuck with Linux. In this framing, if you want Rust to succeed in a kernel, getting it into Linux is the best strategy.

But, on the whole, my opinion is that the benefits of Rust in Linux are negligible and the costs are not. That said, it’s going to happen, and the impact to me is likely to be, at worst, a nuisance. Though I would have chosen differently, I wish them the best of luck and hope to see them succeed.

Notes from kernel hacking in Hare, part 2: multi-threading

Sun, 02 Oct 2022 00:00:00 +0000

I have long promised that Hare would not have multi-threading, and it seems that I have broken that promise. However, I have remained true to the not-invented-here approach which is typical of my style by introducing it only after designing an entire kernel to implement it on top of.¹

For some background, Helios is a micro-kernel written in Hare. In addition to the project, the Vulcan system is a small userspace designed to test the kernel.

A picture of a laptop running Helios and showing the results of the Vulcan test suite

While I don’t anticipate multi-threaded processes playing a huge role in the complete Ares system in the future, they do have a place. In the long term, I would like to be able to provide an implementation of pthreads for porting existing software to the system. A more immediate concern is how to test the various kernel primitives provided by Helios, such as those which facilitate inter-process communication (IPC). It’s much easier to test these with threads than with processes, since spawning threads does not require spinning up a new address space.

@test fn notification::wait() void = {
	const note = helios::newnote()!;
	defer helios::destroy(note)!;

	const thread = threads::new(&notification_wait, note)!;
	threads::start(thread)!;
	defer threads::join(thread)!;

	helios::signal(note)!;
};

fn notification_wait(note: u64) void = {
	const note = note: helios::cap;
	helios::wait(note)!;
};

So how does it work? Let’s split this up into two domains: kernelspace and userspace.

Threads in the kernel

The basic primitive for threads and processes in Helios is a “task”, which is simply an object which receives some CPU time. A task has a capability space (so it can invoke operations against kernel objects), an virtual address space (so it has somewhere to map the process image and memory), and some state, such as the values of its CPU registers. The task-related structures are:

// A task capability.
export type task = struct {
	caps::capability,
	state: uintptr,
	@offset(caps::LINK_OFFS) link: caps::link,
};

// Scheduling status of a task.
export type task_status = enum {
	ACTIVE,
	BLOCKED, // XXX: Can a task be both blocked and suspended?
	SUSPENDED,
};

// State for a task.
export type taskstate = struct {
	regs: arch::state,
	cspace: caps::cslot,
	vspace: caps::cslot,
	ipc_buffer: uintptr,
	status: task_status,
	// XXX: This is a virtual address, should be physical
	next: nullable *taskstate,
	prev: nullable *taskstate,
};

Here’s a footnote to explain some off-topic curiosities about this code: ²

The most interesting part of this structure is arch::state, which stores the task’s CPU registers. On x86_64,³ this structure is defined as follows:

export type state = struct {
	fs: u64,
	fsbase: u64,

	r15: u64,
	r14: u64,
	r13: u64,
	r12: u64,
	r11: u64,
	r10: u64,
	r9: u64,
	r8: u64,
	rbp: u64,
	rdi: u64,
	rsi: u64,
	rdx: u64,
	rcx: u64,
	rbx: u64,
	rax: u64,

	intno: u64,
	errcode: u64,

	rip: u64,
	cs: u64,
	rflags: u64,
	rsp: u64,
	ss: u64,
};

This structure is organized in part according to hardware constraints and in part at the discretion of the kernel implementer. The last five fields, from %rip to %ss, are constrained by the hardware. When an interrupt occurs, the CPU pushes each of these registers to the stack, in this order, then transfers control to the system interrupt handler. The next two registers serve a special purpose within our interrupt implementation, and the remainder are ordered arbitrarily.

In order to switch between two tasks, we need to save all of this state somewhere, then load the same state for another task when returning from the kernel to userspace. The save/restore process is handled in the interrupt handler, in assembly:

.global isr_common
isr_common:
	_swapgs
	push %rax
	push %rbx
	push %rcx
	push %rdx
	push %rsi
	push %rdi
	push %rbp
	push %r8
	push %r9
	push %r10
	push %r11
	push %r12
	push %r13
	push %r14
	push %r15

	// Note: fsbase is handled elsewhere
	push $0
	push %fs

	cld

	mov %rsp, %rdi
	mov $_kernel_stack_top, %rsp
	call arch.isr_handler
_isr_exit:
	mov %rax, %rsp

	// Note: fsbase is handled elsewhere
	pop %fs
	pop %r15

	pop %r15
	pop %r14
	pop %r13
	pop %r12
	pop %r11
	pop %r10
	pop %r9
	pop %r8
	pop %rbp
	pop %rdi
	pop %rsi
	pop %rdx
	pop %rcx
	pop %rbx
	pop %rax

	_swapgs

	// Clean up error code and interrupt #
	add $16, %rsp

	iretq

I’m not going to go into too much detail on interrupts for this post (maybe in a later post), but what’s important here is the chain of push/pop instructions. This automatically saves the CPU state for each task when entering the kernel. The syscall handler has something similar.

This suggests a question: where’s the stack?

Helios has a single kernel stack,⁴ which is moved to %rsp from $_kernel_stack_top in this code. This is different from systems like Linux, which have one kernel stack per thread; the rationale behind this design choice is out of scope for this post.⁵ However, the “stack” being pushed to here is not, in fact, a traditional stack.

x86_64 has an interesting feature wherein an interrupt can be configured to use a special “interrupt stack”. The task state segment is a bit of a historical artifact which is of little interest to Helios, but in long mode (64-bit mode) it serves a new purpose: to provide a list of addresses where up to seven interrupt stacks are stored. The interrupt descriptor table includes a 3-bit “IST” field which, when nonzero, instructs the CPU to set the stack pointer to the corresponding address in the TSS when that interrupt fires. Helios sets all of these to one, then does something interesting:

// Stores a pointer to the current state context.
export let context: **state = null: **state;

fn init_tss(i: size) void = {
	cpus[i].tstate = taskstate { ... };
	context = &cpus[i].tstate.ist[0]: **state;
};

// ...

export fn save() void = {
	// On x86_64, most registers are saved and restored by the ISR or
	// syscall service routines.
	let active = *context: *[*]state;
	let regs = &active[-1];

	regs.fsbase = rdmsr(0xC0000100);
};

export fn restore(regs: *state) void = {
	wrmsr(0xC0000100, regs.fsbase);

	const regs = regs: *[*]state;
	*context = &regs[1];
};

We store a pointer to the active task’s state struct in the TSS when we enter userspace, and when an interrupt occurs, the CPU automatically places that state into %rsp so we can trivially push all of the task’s registers into it.

There is some weirdness to note here: the stack grows downwards. Each time you push, the stack pointer is decremented, then the pushed value is written there. So, we have to fill in this structure from the bottom up. Accordingly, we have to do something a bit unusual here: we don’t store a pointer to the context object, but a pointer to the end of the context object. This is what &active[-1] does here.

Hare has some memory safety features by default, such as bounds testing array accesses. Here we have to take advantage of some of Hare’s escape hatches to accomplish the goal. First, we cast the pointer to an unbounded array of states — that’s what the *[*] is for. Then we can take the address of element -1 without the compiler snitching on us.

There is also a separate step here to save the fsbase register. This will be important later.

This provides us with enough pieces to enter userspace:

// Immediately enters this task in userspace. Only used during system
// initialization.
export @noreturn fn enteruser(task: *caps::capability) void = {
	const state = objects::task_getstate(task);
	assert(objects::task_schedulable(state));
	active = state;
	objects::vspace_activate(&state.vspace)!;
	arch::restore(&state.regs);
	arch::enteruser();
};

What we need next is a scheduler, and a periodic interrupt to invoke it, so that we can switch tasks every so often.

Scheduler design is a complex subject which can have design, performance, and complexity implications ranging from subtle to substantial. For Helios’s present needs we use a simple round-robin scheduler: each task gets the same time slice and we just switch to them one after another.

The easy part is simply getting periodic interrupts. Again, this blog post isn’t about interrupts, so I’ll just give you the reader’s digest:

arch::install_irq(arch::PIT_IRQ, &pit_irq);
arch::pit_setphase(100);

// ...

fn pit_irq(state: *arch::state, irq: u8) void = {
	sched::switchtask();
	arch::pic_eoi(arch::PIT_IRQ);
};

The PIT, or programmable interrupt timer, is a feature on x86_64 which provides exactly what we need: periodic interrupts. This code configures it to tick at 100 Hz and sets up a little IRQ handler which calls sched::switchtask to perform the actual context switch.

Recall that, by the time sched::switchtask is invoked, the CPU and interrupt handler have already stashed all of the current task’s registers into its state struct. All we have to do now is pick out the next task and restore its state.

// see idle.s
let idle: arch::state;

// Switches to the next task.
export fn switchtask() void = {
	// Save state
	arch::save();

	match (next()) {
	case let task: *objects::taskstate =>
		active = task;
		objects::vspace_activate(&task.vspace)!;
		arch::restore(&task.regs);
	case null =>
		arch::restore(&idle);
	};
};

fn next() nullable *objects::taskstate = {
	let next = active.next;
	for (next != active) {
		if (next == null) {
			next = tasks;
			continue;
		};
		const cand = next as *objects::taskstate;
		if (objects::task_schedulable(cand)) {
			return cand;
		};
		next = cand.next;
	};
	const next = next as *objects::taskstate;
	if (objects::task_schedulable(next)) {
		return next;
	};
	return null;
};

Pretty straightforward. The scheduler maintains a linked list of tasks, picks the next one which is schedulable,⁶ then runs it. If there are no schedulable tasks, it runs the idle task.

Err, wait, what’s the idle task? Simple: it’s another state object (i.e. a set of CPU registers) which essentially works as a statically allocated do-nothing thread.

const idle_frame: [2]uintptr = [0, &pause: uintptr];

// Initializes the state for the idle thread.
export fn init_idle(idle: *state) void = {
	*idle = state {
		cs = seg::KCODE << 3,
		ss = seg::KDATA << 3,
		rflags = (1 << 21) | (1 << 9),
		rip = &pause: uintptr: u64,
		rbp = &idle_frame: uintptr: u64,
		...
	};
};

“pause” is a simple loop:

.global arch.pause
arch.pause:
	hlt
	jmp arch.pause

In the situation where every task is blocking on I/O, there’s nothing for the CPU to do until the operation finishes. So, we simply halt and wait for the next interrupt to wake us back up, hopefully unblocking some tasks so we can schedule them again. A more sophisticated kernel might take this opportunity to go into a lower power state, perhaps, but for now this is quite sufficient.

With this last piece in place, we now have a multi-threaded operating system. But there is one more piece to consider: when a task yields its time slice.

Just because a task receives CPU time does not mean that it needs to use it. A task which has nothing useful to do can yield its time slice back to the kernel through the “yieldtask” syscall. On the face of it, this is quite simple:

// Yields the current time slice and switches to the next task.
export @noreturn fn yieldtask() void = {
	arch::sysret_set(&active.regs, 0, 0);
	switchtask();
	arch::enteruser();
};

The “sysret_set” updates the registers in the task state which correspond with system call return values to (0, 0), indicating a successful return from the yield syscall. But we don’t actually return at all: we switch to the next task and then return to that.

In addition to being called from userspace, this is also useful whenever the kernel blocks a thread on some I/O or IPC operation. For example, tasks can wait on “notification” objects, which another task can signal to wake them up — a simple synchronization primitive. The implementation makes good use of sched::yieldtask:

// Blocks the active task until this notification is signalled. Does not return
// if the operation is blocking.
export fn wait(note: *caps::capability) uint = {
	match (nbwait(note)) {
	case let word: uint =>
		return word;
	case errors::would_block =>
		let note = note_getstate(note);
		assert(note.recv == null); // TODO: support multiple receivers
		note.recv = sched::active;
		sched::active.status = task_status::BLOCKED;
		sched::yieldtask();
	};
};

Finally, that’s the last piece.

Threads in userspace

Phew! That was a lot of kernel pieces to unpack. And now for userspace… in the next post! This one is getting pretty long. Here’s what you have to look forward to:

Preparing the task and all of the objects it needs (such as a stack)
High-level operations: join, detach, exit, suspend, etc
Thread-local storage…
- in the Hare compiler
- in the ELF loader
- at runtime
Putting it all together to test the kernel

We’ll see you next time!

The phrase "open source" (still) matters

Fri, 16 Sep 2022 00:00:00 +0000

In 1988, “Resin Identification Codes” were introduced by the plastic industry. These look exactly like the recycling symbol ♺, which is not trademarked or regulated, except that a number is enclosed within the triangle. These symbols simply identify what kind of plastic was used. The vast majority of plastic is non-recyclable, but has one of these symbols on it to suggest otherwise. This is a deceptive business practice which exploits the consumer’s understanding of the recycling symbol to trick them into buying more plastic products.

The meaning of the term “open source” is broadly understood to be defined by the Open Source Initiative’s Open Source Definition, the “OSD”. Under this model, open source has enjoyed a tremendous amount of success, such that virtually all software written today incorporates open source components.

The main advantage of open source, to which much of this success can be attributed, is that it is a product of many hands. In addition to the work of its original authors, open source projects generally accept code contributions from anyone who would offer them. They also enjoy numerous indirect benefits, through the large community of Linux distros which package and ship the software, or people who write docs or books or blog posts about it, or the many open source dependencies it is likely built on top of.

Under this model, the success of an open source project is not entirely attributable to its publisher, but to both the publisher and the community which exists around the software. The software does not belong to its publisher, but to its community. I mean this not only in a moral sense, but also in a legal sense: every contributor to an open source project retains their copyright and the project’s ownership is held collectively between its community of contributors.¹

The OSD takes this into account when laying out the conditions for commercialization of the software. An argument for exclusive commercialization of software by its publishers can be made when the software is the result of investments from that publisher alone, but this is not so for open source. Because it is the product of its community as a whole, the community enjoys the right to commercialize it, without limitation. This is a fundamental, non-negotiable part of the open source definition.

However, we often see the odd company or organization trying to forward an unorthodox definition of the “open source”. Generally, their argument goes something like this: “open” is just an adjective, and “source” comes from “source code”, so “open source” just means source code you can read, right?

This argument is wrong,² but it usually conceals the speaker’s real motivations: they want a commercial monopoly over their project.³ Their real reason is “I should be able to make money from open source, but you shouldn’t”. An argument for an unorthodox definition of “open source” from this perspective is a form of motivated reasoning.

Those making this argument have good reason to believe that they will enjoy more business success if they get away with it. The open source brand is incredibly strong — one of the most successful brands in the entire software industry. Leveraging that brand will drive interest to their project, especially if, on the surface, it looks like it fits the bill (generally by being source available).

When you get down to it, this behavior is dishonest and anti-social. It leverages the brand of open source, whose success has been dependent on the OSD and whose brand value is associated with the user’s understanding of open source, but does not provide the same rights. The deception is motivated by selfish reasons: to withhold those rights from the user for their own exclusive use. This is wrong.

You can publish software under any terms that you wish, with or without commercial rights, with or without source code, whatever — it’s your right. However, if it’s not open source, it’s wrong to call it open source. There are better terms — “source available”, “fair code”, etc. If you describe your project appropriately, whatever the license may be, then I wish you nothing but success.

Status update, September 2022

Thu, 15 Sep 2022 00:00:00 +0000

I have COVID-19 and I am halfway through my stockpile of tissues, so I’m gonna keep this status update brief.

In Hare news, I finally put the last pieces into place to make cross compiling as easy as possible. Nothing else particularly world-shattering going on here. I have a bunch of new stuff in my patch queue to review once I’m feeling better, however, including bigint stuff — a big step towards TLS support. Unrelatedly, TLS support seems to be progressing upstream in qbe. (See what I did there?)

powerctl is a small new project I wrote to configure power management states on Linux. I’m pretty pleased with how it turned out. It makes for a good case study on Hare for systems programming.

In Helios, I have been refactoring the hell out of everything, rewriting or redesigning large parts of it from scratch. Presently this means that a lot of the functionality which was previously present was removed, and is being slowly re-introduced with substantial changes. The key is reworking these features to take better consideration of the full object lifecycle — creating, copying, and destroying capabilities. An improvement which ended up being useful in the course of this work is adding address space IDs (PCIDs on x86_64), which is going to offer a substantial performance boost down the line.

Alright, time for a nap. Bye!

Notes from kernel hacking in Hare, part 1

Wed, 07 Sep 2022 00:00:00 +0000

One of the goals for the Hare programming language is to be able to write kernels, such as my Helios project. Kernels are complex beasts which exist in a somewhat unique problem space and have constraints that many userspace programs are not accustomed to. To illustrate this, I’m going to highlight a scenario where Hare’s low-level types and manual memory management approach shines to enable a difficult use-case.

Helios is a micro-kernel. During system initialization, its job is to load the initial task into memory, prepare the initial set of kernel objects for its use, provide it with information about the system, then jump to userspace and fuck off until someone needs it again. I’m going to focus on the “providing information” step here.

The information the kernel needs to provide includes details about the capabilities that init has access to (such as working with I/O ports), information about system memory, the address of the framebuffer, and so on. This information is provided to init in the bootinfo structure, which is mapped into its address space, and passed to init via a register which points to this structure.¹

// The bootinfo structure.
export type bootinfo = struct {
	argv: str,

	// Capability ranges
	memory: cap_range,
	devmem: cap_range,
	userimage: cap_range,
	stack: cap_range,
	bootinfo: cap_range,
	unused: cap_range,

	// Other details
	arch: *arch_bootinfo,
	ipcbuf: uintptr,
	modules: []module_desc,
	memory_info: []memory_desc,
	devmem_info: []memory_desc,
	tls_base: uintptr,
	tls_size: size,
};

Parts of this structure are static (such as the capability number ranges for each capability assigned to init), and others are dynamic - such as structures describing the memory layout (N items where N is the number of memory regions), or the kernel command line. But, we’re in a kernel – dynamically allocating data is not so straightforward, especially for units smaller than a page!² Moreover, the data structures allocated here need to be visible to userspace, and kernel memory is typically not available to userspace. A further complication is the three different address spaces we’re working with here: a bootinfo object has a physical memory address, a kernel address, and a userspace address — three addresses to refer to a single object in different contexts.

Here’s an example of what the code shown in this article is going to produce:

A 64 by 64 grid of cells representing a page of physical memory. The first set of cells are colored blue; the next set green; then purple; the remainder are brown.

This is a single page of physical memory which has been allocated for the bootinfo data, where each cell is a byte. The bootinfo structure itself comes first, in blue. Following this is an arch-specific bootinfo structure, in green:

// x86_64-specific boot information
export type arch_bootinfo = struct {
	// Page table capabilities
	pdpt: cap_range,
	pd: cap_range,
	pt: cap_range,

	// vbe_mode_info physical address from multiboot (or zero)
	vbe_mode_info: uintptr,
};

After this, in purple, is the kernel command line. These three structures are always consistently allocated for any boot configuration, so the code which sets up the bootinfo page (the code we’re going to read now) always provisions them. Following these three items is a large area of free space (indicated in brown) which will be used to populate further dynamically allocated bootinfo structures, such as descriptions of physical memory regions.

The code to set this up is bootinfo_init, which is responsible for allocating a suitable page, filling in the bootinfo structure, and preparing a vector to dynamically allocate additional data on this page. It also sets up the arch bootinfo and argv, so the page looks like this diagram when the function returns. And here it is, in its full glory:

// Initializes the bootinfo context.
export fn bootinfo_init(heap: *heap, argv: str) bootinfo_ctx = {
	let cslot = caps::cslot { ... };
	let page = objects::init(ctype::PAGE, &cslot, &heap.memory)!;
	let phys = objects::page_phys(page);
	let info = mem::phys_tokernel(phys): *bootinfo;

	const bisz = size(bootinfo);
	let bootvec = (info: *[*]u8)[bisz..arch::PAGESIZE][..0];

	let ctx = bootinfo_ctx {
		page = cslot,
		info = info,
		arch = null: *arch_bootinfo, // Fixed up below
		bootvec = bootvec,
	};

	let (vec, user) = mkbootvec(&ctx, size(arch_bootinfo), size(uintptr));
	ctx.arch = vec: *[*]u8: *arch_bootinfo;
	info.arch = user: *arch_bootinfo;

	let (vec, user) = mkbootvec(&ctx, len(argv), 1);
	vec[..] = strings::toutf8(argv)[..];
	info.argv = *(&types::string {
		data = user: *[*]u8,
		length = len(argv),
		capacity = len(argv),
	}: *str);

	return ctx;
};

The first three lines are fairly straightforward. Helios uses capability-based security, similar in design to seL4. All kernel objects — such as pages of physical memory — are utilized through the capability system. The first two lines set aside a slot to store the page capability in, then allocate a page using that slot. The next two lines grab the page’s physical address and use mem::phys_tokernel to convert it to an address in the kernel’s virtual address space, so that the kernel can write data to this page.

The next two lines are where it starts to get a little bit interesting:

const bisz = size(bootinfo);
let bootvec = (info: *[*]u8)[bisz..arch::PAGESIZE][..0];

This casts the “info” variable (of type *bootinfo) to a pointer to an unbounded array of bytes (*[*]u8). This is a little bit dangerous! Hare’s arrays are bounds tested by default and using an unbounded type disables this safety feature. We want to get a bounded slice again soon, which is what the first slicing operator here does: [bisz..arch::PAGESIZE]. This obtains a bounded slice of bytes which starts from the end of the bootinfo structure and continues to the end of the page.

The last expression, another slicing expression, is a little bit unusual. A slice type in Hare has the following internal representation:

type slice = struct {
	data: nullable *void,
	length: size,
	capacity: size,
};

When you slice an unbounded array, you get a slice whose length and capacity fields are equal to the length of the slicing operation, in this case arch::PAGESIZE - bisz. But when you slice a bounded slice, the length field takes on the length of the slicing expression but the capacity field is calculated from the original slice. So by slicing our new bounded slice to the 0th index ([..0]), we obtain the following slice:

slice {
	data = &(info: *[*]bootinfo)[1]: *[*]u8,
	length = 0,
	capacity = arch::PAGESIZE - bisz,
};

In plain English, this is a slice whose base address is the address following the bootinfo structure and whose capacity is the remainder of the free space on its page, with a length of zero. This is something we can use static append with!³

// Allocates a buffer in the bootinfo vector, returning the kernel vector and a
// pointer to the structure in the init vspace.
fn mkbootvec(info: *bootinfo_ctx, sz: size, al: size) ([]u8, uintptr) = {
	const prevlen = len(info.bootvec);
	let padding = 0z;
	if (prevlen % al != 0) {
		padding = al - prevlen % al;
	};
	static append(info.bootvec, [0...], sz + padding);
	const vec = info.bootvec[prevlen + padding..];
	return (vec, INIT_BOOTINFO_ADDR + size(bootinfo): uintptr prevlen: uintptr);
};

In Hare, slices can be dynamically grown and shrunk using the append, insert, and delete keywords. This is pretty useful, but not applicable for our kernel — remember, no dynamic memory allocation. Attempting to use append in Helios would cause a linking error because the necessary runtime code is absent from the kernel’s Hare runtime. However, you can also statically append to a slice, as shown here. So long as the slice has a sufficient capacity to store the appended data, a static append or insert will succeed. If not, an assertion is thrown at runtime, much like a normal bounds test.

This function makes good use of it to dynamically allocate memory from the bootinfo page. Given a desired size and alignment, it statically appends a suitable number of zeroes to the page, takes a slice of the new data, and returns both that slice (in the kernel’s address space) and the address that data will have in the user address space. If we return to the earlier function, we can see how this is used to allocate space for the arch_bootinfo structure:

let (vec, user) = mkbootvec(&ctx, size(arch_bootinfo), size(uintptr));
ctx.arch = vec: *[*]u8: *arch_bootinfo;
info.arch = user: *arch_bootinfo;

The “ctx” variable is used by the kernel to keep track of its state while setting up the init task, and we stash the kernel’s pointer to this data structure in there, and the user’s pointer in the bootinfo structure itself.

This is also used to place argv into the bootinfo page:

let (vec, user) = mkbootvec(&ctx, len(argv), 1);
vec[..] = strings::toutf8(argv)[..];
info.argv = *(&types::string {
	data = user: *[*]u8,
	length = len(argv),
	capacity = len(argv),
}: *str);

Here we allocate a vector whose length is the length of the argument string, with an alignment of one, and then copy argv into it as a UTF-8 slice. Slice copy expressions like this one are a type-safe and memory-safe way to memcpy in Hare. Then we do something a bit more interesting.

Like slices, strings have an internal representation in Hare which includes a data pointer, length, and capacity. The types module provides a struct with this representation so that you can do low-level string manipulation in Hare should the task call for it. Hare’s syntax allows us to take the address of a literal value, such as a types::string struct, using the & operator. Then we cast it to a pointer to a string and dereference it. Ta-da! We set the bootinfo argv field to a str value which uses the user address of the argument vector.

Some use-cases call for this level of fine control over the precise behavior of your program. Hare’s goal is to accommodate this need with little fanfare. Here we’ve drawn well outside of the lines of Hare’s safety features, but sometimes it’s useful and necessary to do so. And Hare provides us with the tools to get the safety harness back on quickly, such as we saw with the construction of the bootvec slice. This code is pretty weird but to an experienced Hare programmer (which, I must admit, the world has very few of) it should make sense.

I hope you found this interesting! I’m going back to kernel hacking. Next up is loading the userspace ELF image into its address space. I had this working before but decided to rewrite it. Wish me good luck!

In praise of qemu

Fri, 02 Sep 2022 00:00:00 +0000

qemu is another in a long line of great software started by Fabrice Bellard. It provides virtual machines for a wide variety of software architectures. Combined with KVM, it forms the foundation of nearly all cloud services, and it runs SourceHut in our self-hosted datacenters. Much like Bellard’s ffmpeg revolutionized the multimedia software industry, qemu revolutionized virtualisation.

qemu comes with a large variety of studiously implemented virtual devices, from standard real-world hardware like e1000 network interfaces to accelerated virtual hardware like virtio drives. One can, with the right combination of command line arguments, produce a virtual machine of essentially any configuration, either for testing novel configurations or for running production-ready virtual machines. Network adapters, mouse & keyboard, IDE or SCSI or SATA drives, sound cards, graphics cards, serial ports — the works. Lower level, often arch-specific features, such as AHCI devices, SMP, NUMA, and so on, are also available and invaluable for testing any conceivable system configurations. And these configurations work, and work reliably.

I have relied on this testing quite a bit when working on kernels, particularly on my own Helios kernel. With a little bit of command line magic, I can run a fully virtualised system with a serial driver connected to the parent terminal, with a hardware configuration appropriate to whatever I happen to be testing, in a manner such that running and testing my kernel is no different from running any other program. With -gdb I can set up gdb remote debugging and even debug my kernel as if it were a typical program. Anyone who remembers osdev in the Bochs days — or even earlier — understands the unprecedented luxury of such a development environment. Should I ever find myself working on a hardware configuration which is unsupported by qemu, my very first step will be patching qemu to support it. In my reckoning, qemu support is nearly as important for bringing up a new system as a C compiler is.

And qemu’s implementation in C is simple, robust, and comprehensive. On the several occasions when I’ve had to read the code, it has been a pleasure. Furthermore, the comprehensive approach allows you to build out a virtualisation environment tuned precisely to your needs, whatever they may be, and it is accommodating of many needs. Sure, it’s low level — running a qemu command line is certainly more intimidating than, say, VirtualBox — but the trade-off in power afforded to the user opens up innumerable use-cases that are simply not available on any other virtualisation platform.

One of my favorite, lesser-known features of qemu is qemu-user, which allows you to register a binfmt handler to run executables compiled for an arbitrary architecture on Linux. Combined with a little chroot, this has made cross-arch development easier than it has ever been before, something I frequently rely on when working on Hare. If you do cross-architecture work and you haven’t set up qemu-user yet, you’re missing out.

$ uname -a
Linux taiga 5.15.63-0-lts #1-Alpine SMP Fri, 26 Aug 2022 07:02:59 +0000 x86_64 GNU/Linux
$ doas chroot roots/alpine-riscv64/ /bin/sh
# uname -a
Linux taiga 5.15.63-0-lts #1-Alpine SMP Fri, 26 Aug 2022 07:02:59 +0000 riscv64 Linux

This is amazing.

qemu also holds a special place in my heart as one of the first projects I contributed to over email 🙂 And they still use email today, and even recommend SourceHut to make the process easier for novices.

So, to Fabrice, and the thousands of other contributors to qemu, I offer my thanks. qemu is one of my favorite pieces of software.

powerctl: A small case study in Hare for systems programming

Sun, 28 Aug 2022 00:00:00 +0000

powerctl is a little weekend project I put together to provide a simple tool for managing power states on Linux. I had previously put my laptop into suspend with a basic “echo mem | doas tee /sys/power/state”, but this leaves a lot to be desired. I have to use doas to become root, and it’s annoying to enter my password — not to mention difficult to use in a script or to attach to a key binding. powerctl is the solution: a small 500-line Hare program which provides comprehensive support for managing power states on Linux for non-privileged users.

This little project ended up being a useful case-study in writing a tight systems program in Hare. It has to do a few basic tasks which Hare shines in:

setuid binaries
Group lookup from /etc/group
Simple string manipulation
Simple I/O within sysfs constraints

Linux documents these features here, so it’s a simple matter of rigging it up to a nice interface. Let’s take a look at how it works.

First, one of the base requirements for this tool is to run as a non-privileged user. However, since writing to sysfs requires root, this program will have to be setuid, so that it runs as root regardless of who executes it. To prevent any user from suspending the system, I added a “power” group and only users who are in this group are allowed to use the program. Enabling this functionality in Hare is quite simple:

use fmt;
use unix;
use unix::passwd;

def POWER_GROUP: str = "power";

// Determines if the current user is a member of the power group.
fn checkgroup() bool = {
	const uid = unix::getuid();
	const euid = unix::geteuid();
	if (uid == 0) {
		return true;
	} else if (euid != 0) {
		fmt::fatal("Error: this program must be installed with setuid (chmod u+s)");
	};

	const group = match (passwd::getgroup(POWER_GROUP)) {
	case let grent: passwd::grent =>
		yield grent;
	case void =>
		fmt::fatal("Error: {} group missing from /etc/group", POWER_GROUP);
	};
	defer passwd::grent_finish(&group);

	const gids = unix::getgroups();
	for (let i = 0z; i < len(gids); i += 1) {
		if (gids[i] == group.gid) {
			return true;
		};
	};

	return false;
};

The POWER_GROUP variable allows distributions that package powerctl to configure exactly which group is allowed to use this tool. Following this, we compare the uid and effective uid. If the uid is zero, we’re already running this tool as root, so we move on. Otherwise, if the euid is nonzero, we lack the permissions to continue, so we bail out and tell the user to fix their installation.

Then we fetch the details for the power group from /etc/group. Hare’s standard library includes a module for working with this file. Once we have the group ID from the string, we check the current user’s supplementary group IDs to see if they’re a member of the appropriate group. Nice and simple. This is also the only place in powerctl where dynamic memory allocation is required, to store the group details, which are freed with “defer passwd::grent_finish”.

The tool also requires some simple string munging to identify the supported set of states. If we look at /sys/power/disk, we can see the kind of data we’re working with:

$ cat /sys/power/disk 
[platform] shutdown reboot suspend test_resume

These files are a space-separated list of supported states, with the currently enabled state enclosed in square brackets. Parsing these files is a simple matter for Hare. We start with a simple utility function which reads the file and prepares a string tokenizer which splits the string on spaces:

fn read_states(path: str) (strings::tokenizer | fs::error | io::error) = {
	static let buf: [512]u8 = [0...];

	const file = os::open(path)?;
	defer io::close(file)!;

	const z = match (io::read(file, buf)?) {
	case let z: size =>
		yield z;
	case =>
		abort("Unexpected EOF from sysfs");
	};
	const string = strings::rtrim(strings::fromutf8(buf[..z]), '\n');
	return strings::tokenize(string, " ");
};

The error handling here warrants a brief note. This function can fail if the file does not exist or if there is an I/O error when reading it. I don’t think that I/O errors are possible in this specific case (they can occur when writing to these files, though), but we bubble it up regardless using “io::read()?”. The file might not exist if these features are not supported by the current kernel configuration, in which case it’s bubbled up as “errors::noentry” via “os::open()?”. These cases are handled further up the call stack. The other potential error site is “io::close”, which can fail but only in certain circumstances (such as closing the same file twice), and we use the error assertion operator (”!”) to indicate that the programmer believes this case cannot occur. The compiler will check our work and abort at runtime should this assumption be proven wrong in practice.

In the happy path, we read the file, trim off the newline, and return a tokenizer which splits on spaces. The storage for this string is borrowed from “buf”, which is statically allocated.

The usage of this function to query supported disk suspend behaviors is here:

fn read_disk_states() ((disk_state, disk_state) | fs::error | io::error) = {
	const tok = read_states("/sys/power/disk")?;

	let states: disk_state = 0, active: disk_state = 0;
	for (true) {
		let tok = match (strings::next_token(&tok)) {
		case let s: str =>
			yield s;
		case void =>
			break;
		};
		const trimmed = strings::trim(tok, '[', ']');

		const state = switch (trimmed) {
		case "platform" =>
			yield disk_state::PLATFORM;
		case "shutdown" =>
			yield disk_state::SHUTDOWN;
		case "reboot" =>
			yield disk_state::REBOOT;
		case "suspend" =>
			yield disk_state::SUSPEND;
		case "test_resume" =>
			yield disk_state::TEST_RESUME;
		case =>
			continue;
		};
		states |= state;
		if (trimmed != tok) {
			active = state;
		};
	};

	return (states, active);
};

This function returns a tuple which includes all of the supported disk states OR’d together, and a value which indicates which state is currently enabled. The loop iterates through each of the tokens from the tokenizer returned by read_states, trims off the square brackets, and adds the appropriate state bits. We also check the trimmed token against the original token to detect which state is currently active.

There’s two edge cases to be taken into account here: what happens if Linux adds more states in the future, and what happens if none of the states are active? In the former case, we have the continue branch of the switch statement mid-loop. Hare requires all switch statements to be exhaustive, so the compiler forces us to consider this edge case. For the latter case, the return value will be zero, simply indicating that none of these states are active. This is not actually possible given the invariants for this kernel interface, but we could end up in this situation if the kernel adds a new disk mode and that disk mode is active when this code runs.

When the time comes to modify these states, either to put the system to sleep or to configure its behavior when put to sleep, we use the following function:

fn write_state(path: str, state: str) (void | fs::error | io::error) = {
	const file = os::open(path, fs::flags::WRONLY | fs::flags::TRUNC)?;
	defer io::close(file)!;
	let buf: [128]u8 = [0...];
	const file = &bufio::buffered(file, [], buf);
	fmt::fprintln(file, state)?;
};

This code is working within a specific constraint of sysfs: it must complete the write operation in a single syscall. One of Hare’s design goals is giving you sufficient control over the program’s behavior to plan for such concerns. The means of opening the file — WRONLY | TRUNC — was also chosen deliberately. The “single syscall” is achieved by using a buffered file, which soaks up writes until the buffer is full and then flushes them out all at once. The buffered stream flushes automatically on newlines by default, so the “ln” of “fprintln” causes the write to complete in a single call.

With this helper in place, we can write power states. The ones which configure the kernel, but don’t immediately sleep, are straightforward:

// Sets the current mem state.
fn set_mem_state(state: mem_state) (void | fs::error | io::error) = {
	write_state("/sys/power/mem_sleep", mem_state_unparse(state))?;
};

The star of the show, however, has some extra concerns:

// Sets the current sleep state, putting the system to sleep.
fn set_sleep_state(state: sleep_state) (void | fs::error | io::error) = {
	// Sleep briefly so that the keyboard driver can process the key up if
	// the user runs this program from the terminal.
	time::sleep(250 * time::MILLISECOND);
	write_state("/sys/power/state", sleep_state_unparse(state))?;
};

If you enter sleep with a key held down, key repeat will kick in for the duration of the sleep, so when running this from the terminal you’ll resume to find a bunch of new lines. The time::sleep call is a simple way to avoid this, by giving the system time to process your key release event before sleeping. A more sophisticated solution could open the uinput devices and wait for all keys to be released, but that doesn’t seem entirely necessary.

Following this, we jump into the dark abyss of a low-power coma.

And that’s all there is to it! A few hours of work and 500 lines of code later and we have a nice little systems program to make suspending my laptop easier. I was pleasantly surprised to find out how well this little program plays to Hare’s strengths. I hope you found it interesting! And if you happen to need a simple tool for suspending your Linux machines, powerctl might be the program for you.

A review of postmarketOS on the Xiaomi Poco F1

Thu, 25 Aug 2022 00:00:00 +0000

I have recently had cause to start looking into mainline Linux phones which fall outside of the common range of grassroots phones like the PinePhone (which was my daily driver for the past year). The postmarketOS wiki is a great place to research candidate phones for this purpose, and the phone I landed on is the Xiaomi Poco F1, which I picked up on Amazon.nl (for ease of return in case it didn’t work out) for 270 Euro. Phones of this nature have a wide range of support from Linux distros like postmarketOS, from “not working at all” to “mostly working”. The essential features I require in a daily driver phone are (1) a working modem and telephony support, (2) mobile data, and (3) reasonably good performance and battery life; plus of course some sane baseline expectations like a working display and touchscreen driver.

The use of mainline Linux on a smartphone requires a certain degree of bullshit tolerance, and the main question is whether or not the bullshit exceeds your personal threshold. The Poco F1 indeed comes with some bullshit, but I’m pleased to report that it falls short of my threshold and represents a significant quality-of-life improvement over the PinePhone setup I have been using up to now.

The bullshit I have endured for the Poco F1 setup can be categorized into two parts: initial setup and ongoing problems. Of the two, the initial setup is by far the worst. These phones are designed to run Android first, rather than the mainline Linux first approach seen in devices like the PinePhone and Librem 5. This means that it’s back to dealing with things like Android recovery, fastboot, and so on, during the initial setup. The most severe pain point for Xiaomi phones is unlocking the bootloader.

The only officially supported means of doing this is via a Windows-only application published by Xiaomi. A reverse engineered Java application supposedly provides support for completing this process on Linux. However, this approach comes with the typical bullshit of setting up a working Java environment, and, crucially, Xiaomi appears to have sabotaged this effort via a deliberate attempt to close the hole by returning error messages from this reverse engineered API which direct the user to the official tool instead. On top of this, Xiaomi requires you to associate the phone to be unlocked with a user account on their services, paired to a phone number, and has a 30-day waiting period between unlocks. I ultimately had to resort to a Windows 10 VM with USB passthrough to get the damn thing unlocked. This is very frustrating and far from the spirit of free software; Xiaomi earns few points for openness in my books.

Once unlocked, the “initial setup bullshit” did not cease. The main issue is that the postmarketOS flashing tool (which is just a wrapper around fastboot) seemed to have problems writing a consistent filesystem. I was required to apply a level of Linux expertise which exceeds that of even most enthusiasts to obtain a shell in the initramfs, connect to it over postmarketOS’s telnet debugging feature, and run fsck.ext4 to fix the filesystem. Following this, I had to again apply a level of Alpine Linux expertise which exceeds that of many enthusiasts to repair installed packages and get everything up to a baseline of workitude. Overall, it took me the better part of a day to get to a baseline of “running a working installation of postmarketOS”.

However: following the “initial setup bullshit”, I found a very manageable scale of “ongoing problems”. The device’s base performance is excellent, far better than the PinePhone — it just performs much like I would expect from a normal phone. PostmarketOS is, as always, brilliant, and all of the usual mainline Alpine Linux trimmings I would expect are present — I can SSH in, I easily connected it to my personal VPN, and I’m able to run most of the software I’m already used to from desktop Linux systems (though, of course, GUI applications range widely in their ability to accomodate touch screens and a portrait mobile form-factor). I transferred my personal data over from my PinePhone using a method which is 100% certifiably absent of bullshit, namely just rsyncing over my home directory. Excellent!

Telephony support also works pretty well. Audio profiles are a bit buggy, and I can often find my phone using my headphone output while I don’t have them plugged in instead of the speakers, having to resort to manually switching between them from time to time. However, I have never had an issue with the audio profiles being wrong during a phone call (the modem works, by the way); earpiece and speakerphone both work as expected. That said, I have heard complaints from recipients of my phone calls about hearing an echo of their own voice. Additionally, DTMF tones do not work, but the fix has already been merged and is expected in the next release of ModemManager. SMS and mobile data work fine, and mobile data works with a lesser degree of bullshit than I was prepared to expect after reading the pmOS wiki page for this device.

Another problem is that the phone’s onboard cameras do not work at all, and it seems unlikely that this will be solved in the near future. This is not really an issue for me. Another papercut is that Phosh handles the display notch poorly, and though pmOS provides a “tweak” tool which can move the clock over from behind the notch, it leaves something to be desired. The relevant issue is being discused on the Phosh issue tracker and a fix is presumably coming soon — it doesn’t seem particularly difficult to solve. I have also noted that, though GPS works fine, Mepo renders incorrectly and Gnome Maps has (less severe) display issues as well.

The battery life is not as good as the PinePhone, which itself is not as good as most Android phones. However, it meets my needs. It seems to last anywhere from 8 to 10 hours depending on usage, following a full night’s charge. As such, I can leave it off of the juice when I go out without too much fear. That said, I do keep a battery bank in my backpack just in case, but that’s also just a generally useful thing to have around. I think I’ve lent it to others more than I’ve used it myself.

There are many other apps which work without issues. I found that Foliate works great for reading e-books and Evince works nicely for PDFs (two use-cases which one might perceive as related, but which I personally have different UI expectations for). Firefox has far better performance on this device than on the PinePhone and allows for very comfortable web browsing. I also discovered Gnome Feeds which, while imperfect, accommodates my needs regarding an RSS feed reader. All of the “standard” mobile Linux apps that worked fine on the PinePhone also work fine here, such as Lollypop for music and the Porfolio file manager.

I was pleasantly surprised that, after enduring some more bullshit, I was able to get Waydroid to work, allowing me to run Android applications on this phone. My expectations for this were essentially non-existent, so any degree of workitude was a welcome surprise, and any degree of non-workitude was the expected result. On the whole, I’m rather impressed, but don’t expect anything near perfection. The most egregious issue is that I found that internal storage simply doesn’t work, so apps cannot store or read common files (though they seem to be able to persist their own private app data just fine). The camera does not work, so the use-case I was hoping to accommodate here — running my bank’s Android app — is not possible. However, I was able to install F-Droid and a small handful of Android apps that work with a level of performance which is indistinguishable from native Android performance. It’s not quite there yet, but Waydroid has a promising future and will do a lot to bridge the gap between Android and mainline Linux on mobile.

On the whole, I would rate the Poco F1’s bullshit level as follows:

Initial setup: miserable
Ongoing problems: minor

I have a much higher tolerance for “initial setup” bullshit than for ongoing problems bullshit, so this is a promising result for my needs. I have found that this device is ahead of the PinePhone that I had been using previously in almost all respects, and I have switched to it as my daily driver. In fact, this phone, once the initial bullshit is addressed, is complete enough that it may be the first mainline Linux mobile experience that I might recommend to others as a daily driver. I’m glad that I made the switch.

PINE64 has let its community down

Thu, 18 Aug 2022 00:00:00 +0000

Context for this post:

I know that apologising and taking responsibility for your mistakes is difficult. It seems especially difficult for commercial endeavours, which have fostered a culture of cold disassociation from responsibility for their actions, where admitting to wrongdoing is absolutely off the table. I disagree with this culture, but I understand where it comes from, and I can empathise with those who find themselves in the position of having to reconsider their actions in the light of the harm they have done. It’s not easy.

But, the reckoning must come. I have been a long-time supporter of PINE64. On this blog I have written positively about the PinePhone and PineBook Pro.¹ I believed that PINE64 was doing the right thing and was offering something truly revolutionary on the path towards getting real FOSS systems into phones. I use a PinePhone as my daily driver,² and I also own a PineBook Pro, two RockPro64s, a PinePhone Pro, and a PineNote as well. All of these devices have issues, some of them crippling, but PINE64’s community model convinced me to buy these with confidence in the knowledge that they would be able to work with the community to address these flaws given time.

However, PINE64’s treatment of its community has been in a steady decline for the past year or two, culminating in postmarketOS developer Martijn Braam’s blog post outlining a stressful and frustrating community to participate in, a lack of respect from PINE64 towards this community, and a model moving from a diverse culture that builds working software together to a Manjaro mono-culture that doesn’t. PINE64 offered a disappointing response. In their blog post, they dismiss the problems Martijn brings up, paint his post as misguided at best and disingenuous at worst, and fail to take responsibility for their role in any of these problems.

The future of PINE64’s Manjaro mono-culture is dim. Manjaro is a very poorly run Linux distribution with a history of financial mismanagement, ethical violations, security incidents, shipping broken software, and disregarding the input of its peers in the distribution community. Just this morning they allowed their SSL certificates to expire — for the fourth time. An open letter, signed jointly by 16 members of the Linux mobile community, called out bad behaviors which are largely attributable to Manjaro. I do not respect their privileged position in the PINE64 community, which I do not expect to be constructive or in my best interests. I have never been interested in running Manjaro on a PINE64 device and once they turn their back on the lush ecosystem they promised, I no longer have any interest in the platform.

It’s time for PINE64 to take responsibility for these mistakes, and make clear plans to correct them. To be specific:

Apologise for mistreatment of community members.
Make a tangible commitment to honoring and respecting the community.
Rescind their singular commitment to Manjaro.
Re-instate community editions and expand the program.
Deal with this stupid SPI problem. The community is right, listen to them.

I understand that it’s difficult to acknowledge our mistakes. But it is also necessary, and important for the future of PINE64 and the future of mobile Linux in general. I call on TL Lim, Marek Kraus, and Lukasz Erecinski to personally answer for these problems.

There are three possible outcomes to this controversy, depending on PINE64’s response. If PINE64 refuses to change course, the community will continue to decay and fail — the community PINE64 depends on to make its devices functional and useful. Even the most mature PINE64 products still need a lot of work, and none of the new products are even remotely usable. This course of events will be the end of PINE64 and deal a terrible blow to the mobile FOSS movement.

The other option for PINE64 to change its behavior. They do this with grace, or without. If they crumble under public pressure and, for example, spitefully agree to re-instate community editions without accepting responsibility for their wrongdoings, it does not bode well for addressing the toxic environment which is festering in the PINE64 community. This may be better than the worst case, but may not be enough. New community members may hesitate to join, maligned members may not offer their forgiveness, and PINE64’s reputation will suffer for a long time.

The last option is for PINE64 to act with grace and humility. Acknowledge your mistakes and apologise to those who have been hurt. Re-commit to honoring your community and treating your peers with respect. Remember, the community are volunteers. They have no obligation to make peace, so it’s on you to mend these wounds. It will still be difficult to move forward, but doing it with humility, hand in hand with the community, will set PINE64 up with the best chance of success. We’re counting on you to do the right thing.

Status update, August 2022

Tue, 16 Aug 2022 00:00:00 +0000

It is a blessedly cool morning here in Amsterdam. I was busy moving house earlier this month, so this update is a bit quieter than most.

For a fun off-beat project this month, I started working on a GameBoy emulator written in Hare. No promises on when it will be functional or how much I plan on working on it – just doing it for fun. In more serious Hare news, I have implemented Thread-Local Storage (TLS) for qbe, our compiler backend. Hare’s standard library does not support multi-threading, but I needed this for Helios, whose driver library does support threads. It will also presumably be of use for cproc once it lands upstream.

Speaking of Helios, it received the runtime components for TLS support on x86_64, namely the handling of %fs and its base register MSR in the context switch, and updates to the ELF loader for handling .tdata/.tbss sections. I have also implemented support for moving and copying capabilities, which will be useful for creating new processes in userspace. Significant progress towards capability destructors was also made, with some capabilities — pages and page tables in particular — being reclaimable now. Next goal is to finish up all of this capability work so that you can freely create, copy, move, and destroy capabilities, then use all of these features to implement a simple shell. There is also some refactoring due at some point soon, so we’ll see about that.

Other Hare progress has been slow this month, as I’m currently looking at a patch queue 123 emails backed up. When I’m able to sit down and get through these, we can expect a bunch of updates in short order.

SourceHut news will be covered in the “what’s cooking” post later today. That’s all for now! Thanks for tuning in.

How I wish I could organize my thoughts

Wed, 10 Aug 2022 00:00:00 +0000

I keep a pen & notebook on my desk, which I make liberal use of to jot down my thoughts. It works pretty well: ad-hoc todo lists, notes on problems I’m working on, tables, flowcharts, etc. It has some limitations, though. Sharing anything out of my notebook online is an awful pain in the ass. I can’t draw a straight line to save my life, so tables and flowcharts are a challenge. No edits, either, so lots of crossed-out words and redrawn or rewritten pages. And of course, my handwriting sucks and I can type much more efficiently than I can write. I wish this was a digital medium, but there are not any applications available which can support the note-taking paradigm that I wish I could have. What would that look like?

Well, like this (click for full size):

A mock-up of an application. A4 pages are arranged ad-hoc on a grid. Handwritten notes and drawings appear in red across the grid and over the pages. A flowchart is shown outside of a page.

I don’t have the bandwidth to take on a new project of this scope, so I’ll describe what I think this should look like in the hopes that it will inspire another team to work on something like this. Who knows!

The essential interface would be an infinite grid on which various kinds of objects can be placed by the user. The most important of these objects would be pages, at a page size configurable by the user (A4 by default). You can zoom in on a page (double click it or something) to make it your main focus, zooming in automatically to an appropriate level for editing, then type away. A simple WYSIWYG paradigm would be supported here, perhaps supporting only headings, bold/italic text, and ordered and unordered lists — enough to express your thoughts but not a full blown document editor/typesetter.¹ When you run out of page, another is generated next to the current page, either to the right or below — configurable.

Other objects would include flowcharts, tables, images, hand-written text and drawings, and so on. These objects can be placed free form on the grid, or embedded in a page, or moved between each mode.

The user input paradigm should embrace as many modes of input as the user wants to provide. Mouse and keyboard: middle click to pan, scroll to zoom in or out, left click and drag to move objects around, shift+click to select objects, etc. A multi-point trackpad should support pinch to zoom, two finger pan, etc. Touch support is fairly obvious. Drawing tablet support is also important: the user should be able to use one to draw and write free-form. I’d love to be able to make flowcharts by drawing boxes and arrows and having the software recognize them and align them to the grid as first-class vector objects. Some drawing tablets support trackpad and touch-screen-like features as well — so all of those interaction options should just werk.

Performance is important here. I should be able to zoom in and out and pan around while all of the objects rasterize themselves in real-time, never making the user suffer through stuttery interactions. There should also be various ways to export this content. A PDF exporter should let me arrange the pages in the desired linear order. SVG exporters should be able to export objects like flowcharts and diagrams. Other potential features includes real-time collaboration or separate templates for presentations.

Naturally this application should be free software and should run on Linux. However, I would be willing to pay a premium price for this tool — a one-time fee of as much as $1000, or subscriptions on the order of $100/month if real-time collaboration or cloud synchronization are included. If you’d like some ideas for how to monetize free software projects like this, feel free to swing by my talk on the subject in Italy early this September to talk about it.

Well, that’s enough dreaming for now. I hope this inspired you, and in the meantime it’s back to pen and paper for me.

Conciseness

Tue, 26 Jul 2022 00:00:00 +0000

Conciseness is often considered a virtue among hackers and software engineers. FOSS maintainers in particular generally prefer to keep bug reports, questions on mailing lists, discussions in IRC channels, and so on, close to the point and with minimal faff. It’s not considered impolite to skip the formalities — quite the opposite. So: keep your faffery to a minimum. A quick “thanks!” at the end of a discussion will generally suffice. And, when someone is being direct with you, don’t interpret it as a slight: simply indulge in the blissful freedom of a discussion absent of faffery.

The past and future of open hardware

Mon, 25 Jul 2022 00:00:00 +0000

They say a sucker is born every day, and at least on the day of my birth, that certainly may have been true. I have a bad habit of spending money on open hardware projects that ultimately become vaporware or seriously under-deliver on their expectations. In my ledger are EOMA68, DragonBox Pyra, the Jolla Tablet — which always had significant non-free components — and the Mudita Pure, though I did successfully receive a refund for the latter two.¹

There are some success stories, though. My Pine64 devices work great — though they have non-free components — and I have a HiFive Unmatched that I’m reasonably pleased with. Raspberry Pi is going well, if you can find one — also with non-free components — and Arduino and products like it are serving their niche pretty well. I hear the MNT Reform went well, though by then I had learned to be a bit more hesitant to open my wallet for open hardware, so I don’t have one myself. Pebble worked, until it didn’t. Caveats abound in all of these projects.

What does open hardware need to succeed, and why have many projects failed? And why do the successful products often have non-free components and poor stock? We can’t blame it all on the chip shortage and/or COVID: it’s been an issue for a long time.

I don’t know the answers, but I hope we start seeing improvements. I hope that the successful projects will step into a mentorship role to provide up-and-comers with tips on how they made their projects work, and that we see a stronger focus on liberating non-free components. Perhaps Crowd Supply can do some work in helping to secure investment² for open hardware projects, and continue the good work they’re already doing on guiding them through the development and production processes.

Part of this responsibility comes down to the consumer: spend your money on free projects, and don’t spend your money on non-free projects. But, we also need to look closely at the viability of each project, and open hardware projects need to be transparent about their plans, lest we get burned again. Steering the open hardware movement out of infancy will be a challenge for all involved.

Are you working on a cool open hardware project? Let me know. Explain how you plan on making it succeed and, if I’m convinced that your idea has promise, I’ll add a link here.

Code review at the speed of email

Mon, 25 Jul 2022 00:00:00 +0000

I’m a big proponent of the email workflow for patch submission and code review. I have previously published some content (How to use git.sr.ht’s send-email feature, Forks & pull requests vs email, git-send-email.io) which demonstrates the contributor side of this workflow, but it’s nice to illustrate the advantages of the maintainer workflow as well. For this purpose, I’ve recorded a short video demonstrating how I manage code review as an email-oriented maintainer.

Disclaimer: I am the founder of SourceHut, a platform built on this workflow which competes with platforms like GitHub and GitLab. This article’s perspective is biased.

This blog post provides additional material to supplement this video, and also includes all of the information from the video itself. For those who prefer reading over watching, you can just stick to reading this blog post. Or, you can watch the video and skim the post. Or you can just do something else! When was the last time you called your grandmother?

With hundreds of hours of review experience on GitHub, GitLab, and SourceHut, I can say with confidence the email workflow allows me to work much faster than any of the others. I can review small patches in seconds, work quickly with multiple git repositories, easily test changes and make tweaks as necessary, rebase often, and quickly chop up and provide feedback for larger patches. Working my way through a 50-email patch queue usually takes me about 20 minutes, compared to an hour or more for the same number of merge requests.

This workflow also works entirely offline. I can read and apply changes locally, and even reply with feedback or to thank contributors for their patch. My mail setup automatically downloads mail from IMAP using isync and outgoing mails are queued with postfix until the network is ready. I have often worked through my patch queue on an airplane or a train with spotty or non-functional internet access without skipping a beat. Working from low-end devices like a Pinebook or a phone are also no problem — aerc is very lightweight in the terminal and the SourceHut web interface is much lighter & faster than any other web forge.

The centerpiece of my setup is an email client I wrote specifically for software development using this workflow: aerc.¹ The stock configuration of aerc is pretty good, but I make a couple of useful additions specifically for development on SourceHut. Specifically, I add a few keybindings to ~/.config/aerc/binds.conf:

[messages]
ga = :flag<Enter>:pipe -mb git am -3<Enter>
gp = :term git push<Enter>
gl = :term git log<Enter>

rt = :reply -a -Tthanks<Enter>
Rt = :reply -qa -Tquoted_thanks<Enter>

[compose::review]
V = :header -f X-Sourcehut-Patchset-Update NEEDS_REVISION<Enter>
A = :header -f X-Sourcehut-Patchset-Update APPLIED<Enter>
R = :header -f X-Sourcehut-Patchset-Update REJECTED<Enter>

The first three commands, ga, gp, and gl, are for invoking git commands. “ga” applies the current email as a patch, using git am, and “gp” simply runs git push. “gl” is useful for quickly reviewing the git log. ga also flags the email so that it shows up in the UI as having been applied, which is useful as I’m jumping all over a patch queue. I also make liberal use of \ (:filter) to filter my messages to patches applicable to specific projects or goals.

rt and Rt use aerc templates installed at ~/.config/aerc/templates/ to reply to emails after I’ve finished reviewing them. The “thanks” template is:

X-Sourcehut-Patchset-Update: APPLIED

Thanks!

{{exec "{ git remote get-url --push origin; git reflog -2 origin/master --pretty=format:%h | xargs printf '%s\n' | tac; } | xargs printf 'To %s\n   %s..%s  master -> master'" ""}}

And quoted_thanks is:

X-Sourcehut-Patchset-Update: APPLIED

Thanks!

{{exec "{ git remote get-url --push origin; git reflog -2 origin/master --pretty=format:%h | xargs printf '%s\n' | tac; } | xargs printf 'To %s\n   %s..%s  master -> master'" ""}}

On {{dateFormat (.OriginalDate | toLocal) "Mon Jan 2, 2006 at 3:04 PM MST"}}, {{(index .OriginalFrom 0).Name}} wrote:
{{wrapText .OriginalText 72 | quote}}

Both of these add a magic “X-Sourcehut-Patchset-Update” header, which updates the status of the patch on the mailing list. They also include a shell pipeline which adds some information about the last push from this repository, to help the recipient understand what happened to their patch. I often make some small edits to request the user follow-up with a ticket for some future work, or add other timely comments. The second template, quoted_reply, is also particularly useful for this: it quotes the original message so I can reply to specific parts of it, in the commit message, timely commentary, or the code itself, often pointing out parts of the code that I made some small tweaks to before applying.

And that’s basically it! You can browse all of my dotfiles here to see more details about my system configuration. With this setup I am able to work my way through a patch queue easier and faster than ever before. That’s why I like the email workflow so much: for power users, no alternative is even close in terms of efficiency.

Of course, this is the power user workflow, and it can be intimidating to learn all of these things. This is why we offer more novice-friendly tools, which lose some of the advantages but are often more intuitive. For instance, we are working on user interface on the web for patch review, mirroring our existing web interface for patch submission. But, in my opinion, it doesn’t get better than this for serious FOSS maintainers.

Feel free to reach out on IRC in #sr.ht.watercooler on Libera Chat, or via email, if you have any questions about this workflow and how you can apply it to your own projects. Happy hacking!

Status update, July 2022

Mon, 18 Jul 2022 00:00:00 +0000

Hello there! It’s been a hot July week in Amsterdam, and I expect hotter days are still to come. I wish air conditioning was more popular in Europe, but alas. This month of FOSS development enjoyed a lot of small improvements in a lot of different projects.

For Hare, I have introduced a number of improvements. I wrote a new standard library module for string templates, strings::template, and a new third-party library for working with pixel buffers, pixbuf. The templating is pretty simple — as is typical for the standard library — but allows a fairly wide range of formatting options. We’ll be extending this a little bit more in the future, but it will not be a complete solution like you see in things like Jinja2. Nevertheless, it makes some use-cases, like code generation, a lot cleaner, without introducing a weighty or complex dependency.

pixbuf is pretty neat, and is the first in a line of work I have planned for graphics on Hare. It’s similar to pixman, but with a much smaller scope — it only deals with pixel buffers, handling pixel format conversions and doing small operations like fill and copy. In the future I will add simple buffer compositing as well, and extending modules like hare-png to support loading data into these buffers. Later, I plan on writing a simple vector graphics library, capable at least of rendering TinyVG images and perhaps later TinySVG as well. I’m also working on hare-wayland again, to provide a place to display these buffers.

I also introduced format::tar, which will serve as the basis of initramfs-alike functionality for Helios. On the subject of Helios, much work has been completed. I have implemented a PCI driver and a small proof-of-concept AHCI driver (for reading from SATA disks). Alexey Yerin has also been hard at work on the RISC-V port, and has successfully implemented an e1000 ethernet driver which can send and receive ICMP (ping) packets. I also completed IRQ control for userspace, so that userspace device drivers can process interrupts, and used it to write a keyboard driver for a functional DOOM port. The full DOOM port required a fair bit of work — check out that blog post for the complete details. The idle thread was also added, so that all processes can be blocked waiting on interrupts, signals, endpoints, etc. Non-blocking send, receive, and wait syscalls were also added this month.

I’m working on splitting memory capabilities into separate device- and general-purpose capabilities, then adding support for destroying capabilities when they’re no longer required. I also implemented pre-emptive multi-tasking early this month, and the vulcan test suite now has several multi-threaded tests to verify IPC functionality. However, a couple of pieces are missing — the ability to create and work with new cspaces and vspaces — in order to spawn new processes. I’ll be focusing on these tasks in the coming weeks. With these pieces in place, we can start working on Mercury and Vulcan: the driver system.

I’ll save the SourceHut news for the “what’s cooking” post later today, so that’s all for now. Until next time!

The Fediverse can be pretty toxic

Sat, 09 Jul 2022 00:00:00 +0000

Mastodon, inspired by GNU social, together with Pleroma, form the most popular components of what we know as the “Fediverse” today. All of them are, in essence, federated, free software Twitter clones, interoperable with each other via the ActivityPub protocol.

In many respects, the Fediverse is a liberating force for good. Its federated design distributes governance and costs across many independent entities, something I view as a very strong design choice. Its moderation tools also do a pretty good job of keeping neo-nazis out of your feeds and providing a comfortable space to express yourself in, especially if your form of expression is maligned by society. Large groups of Fediverse members have found in it a home for self-expression which is denied to them elsewhere on the basis of their sexuality, gender expression, politics, or other characteristics. It’s also essentially entirely free from commercial propaganda.

But it’s still just a Twitter clone, and many of the social and psychological ills which come with that are present in the Fediverse. It’s a feed of other people’s random thoughts, often unfiltered, presented to you without value judgement — even when a value judgement may be wise. Features like boosting and liking posts, chasing after follower counts and mini-influencers, these rig up dopamine reinforcement like any other social network does. The increased character limit does not really help; most posts are pretty short and no one wants to read an essay aggressively word-wrapped in a narrow column.

The Fediverse is an environment optimized for flame wars. Arguments in this medium are held under these constraints, in public, with the peanut gallery of followers from either side stepping in and out to reinforce their position and flame the opponents. Progress is measured in gains of ideological territory and in the rising and falling swells of participants dotting their comments throughout huge threads. You are not just arguing your position, but performing it to your audience, and to your opponent’s audience.

Social networks are not good for you. The Fediverse brought out the worst in me, and it can bring out the worst in you, too. The behaviors it encourages are plainly defined as harassment, a behavior which is not unique to any ideological condition. People get hurt on the Fediverse. Keep that in mind. Consider taking a look in the mirror and asking yourself if your relationship with the platform is healthy for you and for the people around you.

Porting Doom to Helios

Fri, 01 Jul 2022 00:00:00 +0000

Doom was an incredibly popular video game by Id software which, six years following its release, was made open source under the GPLv2 license. Thanks to this release, combined with the solid software design and lasting legacy of backwards compatibility in C, Doom has been ported to countless platforms by countless programmers. And I recently added myself to this number :)

I’m working on a new kernel called Helios, and I thought that porting Doom would present a good opportunity for proving the kernel design — you never know if you have a good design until you try to use it for real. Doom is a good target because it does not require much to get working, but it is a useful (and fun) program to port. It calls for the following features:

A working C programming environment
Dynamic memory allocation
A place to draw the screen (a framebuffer)
Keyboard input

As I was working, I gradually came to understand that Helios was pretty close to supporting all of these features, and thought that the time to give Doom a shot was coming soon. In my last status update, I shared a picture of a Helios userspace program utilizing the framebuffer provided by multiboot, ticking one box. We’ve had dynamic memory allocation in userspace working since June 8th. The last pieces were a keyboard driver and a C library.

I started with the keyboard driver, since that would let me continue to work on Hare for a little bit longer, providing a more direct benefit to the long-term goals (rather than the short-term goal of “get Doom to work”). Since Helios is a micro-kernel, the keyboard driver is implemented in userspace. A PS/2 keyboard driver requires two features which are reserved to ring 0: I/O ports and IRQ handling. To simplify the interface to the essentials for this use-case, pressing or releasing a key causes IRQ 1 to be fired on the PIC, then reading from port 0x60 provides a scancode. We already had support for working with I/O ports in userspace, so the blocker here was IRQ handling.

Helios implements IRQs similarly to seL4, by using a “notification” object (an IPC primitive) which is signalled by the kernel when an IRQ occurs. I was pleased to have this particular blocker, as developing out our IPC implementation further was a welcome task. The essential usage of a notification involves two operations: wait and signal. The former blocks until the notification is signalled, and the later signals the notification and unblocks any tasks which are waiting on it. Unlike sending messages to endpoints, signal never blocks.

After putting these pieces together, I was able to write a simple PS/2 keyboard driver which echos pressed keys to the kernel console:

const irq1_notify = helios::newnotification()?;
const irq1 = helios::irqcontrol_issue(rt::INIT_CAP_IRQCONTROL, irq1_notify, 1)?;
const ps2 = helios::iocontrol_issue(rt::INIT_CAP_IOCONTROL, 0x60, 0x64)?;

for (true) {
	helios::wait(irq1_notify)?;
	const scancode = helios::ioport_in8(ps2, 0x60)?;
	helios::irq_ack(irq1)!;
};

This creates a notification capability to wait on IRQs, then creates a capability for IRQ 1 registered for that notification. It also issues an I/O port capability for the PS/2 ports, 0x60-0x64 (inclusive). Then it loops, waiting until an interrupt occurs, reading the scancode from the port, and printing it. Simple!

I now turned my attention to a C library for Doom. The first step for writing userspace programs in C for a new operating system is to produce a suitable C cross-compiler toolchain. I adapted the instructions from this OSdev wiki tutorial for my needs and produced the working patches for binutils and gcc. I started on a simple C library that included some assembly glue for syscalls, an entry point, and a couple of syscall wrappers. With great anticipation, I wrote the following C program and loaded it into Helios:

#include <helios/syscall.h>
#include <string.h>

int main() {
	const char *message = "Hello from userspace in C!\n";
	sys_writecons(message, strlen(message));
	return 0;
}

$ qemu-system-x86_64 -m 1G -no-reboot -no-shutdown \
	-drive file=boot.iso,format=raw \
	-display none \
	-chardev stdio,id=char0 \
	-serial chardev:char0
Booting Helios kernel
Hello from userspace in C!

Woohoo! After a little bit more work setting up the basics, I started rigging doomgeneric (a Doom fork designed to be easy to port) up to my cross environment and seeing what would break.

As it turned out, a lot of stuff would break. doomgeneric is designed to be portable, but it actually depends on a lot of stuff to be available from the C environment: stdio, libmath, string.h stuff, etc. Not too much, but more than I cared to write from scratch. So, I started pulling in large swaths of musl libc, trimming out as much as I could, and wriggling it into a buildable state. I also wrote a lot of shims to fake out having a real Unix system to run it in, like this code for defining stdout & stderr to just write to the kernel console:

static size_t writecons(FILE *f, const unsigned char *buf, size_t size) {
	sys_writecons(f->wbase, f->wpos - f->wbase);
	sys_writecons(buf, size);
	f->wend = f->buf + f->buf_size;
	f->wpos = f->wbase = f->buf;
	return size;
}

#undef stdout
static unsigned char stdoutbuf[BUFSIZ+UNGET];
hidden FILE __stdout_FILE = {
	.buf = stdoutbuf+UNGET,
	.buf_size = sizeof stdoutbuf-UNGET,
	.fd = 1,
	.flags = F_PERM | F_NORD,
	.lbf = '\n',
	.write = &writecons,
	.seek = NULL,
	.close = NULL,
	.lock = -1,
};
FILE *const stdout = &__stdout_FILE;
FILE *volatile __stdout_used = &__stdout_FILE;

#undef stderr
static unsigned char stderrbuf[UNGET];
hidden FILE __stderr_FILE = {
	.buf = stderrbuf+UNGET,
	.buf_size = 0,
	.fd = 2,
	.flags = F_PERM | F_NORD,
	.lbf = -1,
	.write = &writecons,
	.seek = NULL,
	.close = NULL,
	.lock = -1,
};
FILE *const stderr = &__stderr_FILE;
FILE *volatile __stderr_used = &__stderr_FILE;

The result of all of this hacking and slashing is quite a mess, and none of this is likely to be useful in the long term. I did this work over the course of a couple of afternoons just to get everything “working” enough to support Doom, but an actual useful C programming environment for Helios is likely some ways off. Much of the near-term work will be in Mercury, which will be a Hare environment for writing drivers, and we won’t see a serious look at better C support until we get to Luna, the POSIX compatibility layer a few milestones away.

Anyway, in addition to pulling in lots of musl libc, I had to write some original code to create C implementations of the userspace end for working with Helios kernel services. Some of this is pretty straightforward, such as the equivalent of the helios::ioport_issue code from the keyboard driver you saw earlier:

cap_t
iocontrol_issue(cap_t ctrl, uint16_t min, uint16_t max)
{
	uint64_t tag = mktag(IO_ISSUE, 1, 1);
	cap_t cap = capalloc();
	ipc_buffer->caddr = cap;
	struct sysret ret = sys_send(ctrl, tag, min, max, 0);
	assert(ret.status == 0);
	return cap;
}

A more complex example is the code which maps a page of physical memory into the current process’s virtual address space. In Helios, similar to L4, userspace must allocate its own page tables. However, these page tables are semantically owned by userspace, but they’re not actually reachable by userspace — the page tables themselves are not mapped into their address space (for obvious reasons, I hope). A consequence of this is that the user cannot examine the page tables to determine which, if any, intermediate page tables have to be allocated in order to perform a desired memory mapping. The solution is to try the mapping anyway, and if the page tables are missing, the kernel will reply telling you which table it needs to complete the mapping request. You allocate the appropriate table and try again.

Some of this workload falls on userspace. I had already done this part in Hare, but I had to revisit it in C:

struct sysret
page_map(cap_t page, cap_t vspace, uintptr_t vaddr)
{
	uint64_t tag = mktag(PAGE_MAP, 1, 1);
	ipc_buffer->caps[0] = vspace;
	return sys_send(page, tag, (uint64_t)vaddr, 0, 0);
}

static void
map_table(uintptr_t vaddr, enum pt_type kind)
{
	int r;
	cap_t table;
	switch (kind) {
	case PT_PDPT:
		r = retype(&table, CT_PDPT);
		break;
	case PT_PD:
		r = retype(&table, CT_PD);
		break;
	case PT_PT:
		r = retype(&table, CT_PT);
		break;
	default:
		assert(0);
	}
	assert(r == 0);

	struct sysret ret = page_map(table, INIT_CAP_VSPACE, vaddr);
	if (ret.status == -MISSING_TABLES) {
		map_table(vaddr, ret.value);
		map_table(vaddr, kind);
	}
}

void *
map(cap_t page, uintptr_t vaddr)
{
	while (1) {
		struct sysret ret = page_map(page, INIT_CAP_VSPACE, vaddr);
		if (ret.status == -MISSING_TABLES) {
			map_table(vaddr, ret.value);
		} else {
			assert(ret.status == 0);
			break;
		}
	}
	return (void *)vaddr;
}

Based on this work, I was able to implement a very stupid malloc, which rounds all allocations up to 4096 and never frees them. Hey! It works, okay?

uintptr_t base = 0x8000000000;

static cap_t
page_alloc()
{
	cap_t page;
	int r = retype(&page, CT_PAGE);
	assert(r == 0);
	return page;
}

void *
malloc(size_t n)
{
	if (n % 4096 != 0) {
		n += 4096 - (n % 4096);
	}
	uintptr_t ret = base;
	while (n != 0) {
		cap_t page = page_alloc();
		map(page, base);
		base += 4096;
		n -= 4096;
	}
	return (void *)ret;
}

There is also devmap, which you can read in your own time, which is used for mapping device memory into your address space. This is neccessary to map the framebuffer. It’s more complex because it has to allocate a specific physical page address into userspace, rather than whatever page happens to be free.

So, to revisit our progress, we have:

✓ A working C programming environment

✓ Dynamic memory allocation

✓ A place to draw the screen (a framebuffer)

✓ Keyboard input

It’s time for Doom, baby. Doomgeneric expects the porter to implement the following functions:

DG_Init
DG_DrawFrame
DG_GetKey
DG_SetWindowTitle
DG_SleepMs
DG_GetTicksMs

Easy peasy. Uh, except for that last one. I forgot that our requirements list should have included a means of sleeping for a specific period of time. Hopefully that won’t be a problem later.

I started with DG_Init, allocating the pieces that we’ll need and stashing the important bits in some globals.

int fb_width, fb_height, fb_pitch;
uint8_t *fb;
cap_t irq1_notify;
cap_t irq1;
cap_t ps2;

void DG_Init()
{
	uintptr_t vbeaddr = bootinfo->arch->vbe_mode_info;
	uintptr_t vbepage = vbeaddr / 4096 * 4096;
	struct vbe_mode_info *vbe = devmap(vbepage, 1) + (vbeaddr % 4096);
	fb_width = vbe->width;
	fb_height = vbe->height;
	fb_pitch = vbe->pitch;
	assert(vbe->bpp == 32);
	unsigned int npage = (vbe->pitch * vbe->height) / 4096;
	fb = devmap((uintptr_t)vbe->framebuffer, npage);

	irq1_notify = mknotification();
	irq1 = irqcontrol_issue(INIT_CAP_IRQCONTROL, irq1_notify, 1);
	ps2 = iocontrol_issue(INIT_CAP_IOCONTROL, 0x60, 0x64);
}

If the multiboot loader is configured to set up a framebuffer, it gets handed off to the kernel, and Helios provides it to userspace as mappable device memory, so that saves us from doing all of the annoying VBE crap (or heaven forbid, write an actual video driver). This lets us map the framebuffer into our process. Second, we do the same notification+IRQ+IOControl thing we did from the keyboard driver you saw earlier, except in C, so that we can process scancodes later.

Next is DG_DrawFrame, which is pretty straightforward. We just copy scanlines from the internal buffer to the framebuffer whenever it asks us to.

void DG_DrawFrame()
{
  for (int i = 0; i < DOOMGENERIC_RESY; ++i) {
    memcpy(fb + i * fb_pitch, DG_ScreenBuffer + i * DOOMGENERIC_RESX, DOOMGENERIC_RESX * 4);
  }
}

Then we have DG_GetKey, similar to our earlier keyboard driver, plus actually interpeting the scancodes we get, plus making use of a new non-blocking wait syscall I added to Helios:

int DG_GetKey(int *pressed, unsigned char *doomKey)
{
	struct sysret ret = sys_nbwait(irq1_notify);
	if (ret.status != 0) {
		return 0;
	}

	uint8_t scancode = ioport_in8(ps2, 0x60);
	irq_ack(irq1);

	uint8_t mask = (1 << 7);
	*pressed = (scancode & mask) == 0;
	scancode = scancode & ~mask;
	switch (scancode) {
	case K_AD05:
		*doomKey = KEY_ENTER;
		break;
	case K_AE08:
		*doomKey = KEY_UPARROW;
		break;
	case K_AD07:
		*doomKey = KEY_LEFTARROW;
		break;
	case K_AD08:
		*doomKey = KEY_DOWNARROW;
		break;
	case K_AD09:
		*doomKey = KEY_RIGHTARROW;
		break;
	case K_AB03:
		*doomKey = KEY_FIRE;
		break;
	case K_AB06:
		*doomKey = KEY_USE;
		break;
	case 1:
		*doomKey = KEY_ESCAPE;
		break;
	}

	return *doomKey;
}

Then, uh, we have a problem. Here’s what I ended up doing for DG_SleepMs:

uint32_t ticks = 0;

void DG_SleepMs(uint32_t ms)
{
	// TODO: sleep properly
	int64_t _ms = ms;
	while (_ms > 0) {
		sys_yield();
		ticks += 5;
		_ms -= 5;
	}
}

uint32_t DG_GetTicksMs()
{
	return ticks;
}

Some fellow on IRC said he’d implement a sleep syscall for Helios, but didn’t have time before I was ready to carry on with this port. So instead of trampling on his feet, I just yielded the thread (which immediately returns to the caller, since there are no other threads at this point) and pretend it took 5ms to do so, hoping for the best. It does not work! This port plays at wildly different speeds depending on the performance of the hardware you run it on.

I’m not too torn up about it, though. My goal was not to make a particularly nice or fully featured port of Doom. The speed is problematic, I hardcoded the shareware doom1.wad as the only supported level, you can’t save the game, and it crashes when you try to pick up the shotgun. But it does its job: it demonstrates the maturity of the kernel’s features thus far and provides good feedback on the API design and real-world utility.

If you’d like to try it, you can download a bootable ISO.

You can run it on qemu like so:

$ qemu-system-x86_64 -m 1G -no-reboot -no-shutdown \
		-drive file=doom.iso,format=raw \
		-display sdl \
		-chardev stdio,id=char0 \
		-serial chardev:char0

Enter to start, WASD to move, right shift to fire, space to open doors. It might work on real hardware, but the framebuffer stuff is pretty hacky and not guaranteed to work on most stuff, and the PS/2 keyboard driver will only work with a USB keyboard if you have legacy USB emulation configured in your BIOS, and even then it might not work well. YMMV. It works on my ThinkPad X230. Have fun!

GitHub Copilot and open source laundering

Thu, 23 Jun 2022 00:00:00 +0000

Disclaimer: I am the founder of a company which competes with GitHub. I am also a long-time advocate for and developer of free and open source software, with a broad understanding of free and open source software licensing and philosophy. I will not name my company in this post to reduce the scope of my conflict of interest.

We have seen an explosion in machine learning in the past decade, alongside an explosion in the popularity of free software. At the same time as FOSS has come to dominate software and found its place in almost all new software products, machine learning has increased dramatically in sophistication, facilitating more natural interactions between humans and computers. However, despite their parallel rise in computing, these two domains remain philosophically distant.

Though some audaciously-named companies might suggest otherwise, the machine learning space has enjoyed almost none of the freedoms forwarded by the free and open source software movement. Much of the actual code related to machine learning is publicly available, and there are many public access research papers available for anyone to read. However, the key to machine learning is access to a high-quality dataset and heaps of computing power to process that data, and these two resources are still kept under lock and key by almost all participants in the space.¹

The essential barrier to entry for machine learning projects is overcoming these two problems, which are often very costly to secure. A high-quality, well tagged data set generally requires thousands of hours of labor to produce,² a task which can potentially cost millions of dollars. Any approach which lowers this figure is thus very desirable, even if the cost is making ethical compromises. With Amazon, it takes the form of gig economy exploitation. With GitHub, it takes the form of disregarding the terms of free software licenses. In the process, they built a tool which facilitates the large-scale laundering of free software into non-free software by their customers, who GitHub offers plausible deniability through an inscrutable algorithm.

Free software is not an unqualified gift. There are terms for its use and re-use. Even so-called “liberal” software licenses impose requirements on re-use, such as attribution. To quote the MIT license:

Permission is hereby granted […] subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

Or the equally “liberal” BSD license:

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

On the other end of the spectrum, copyleft licenses such as GNU General Public License or Mozilla Public License go further, demanding not only attribution for derivative works, but that such derived works are also released with the same license. Quoting GPL:

You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions:
[…]
You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy.

And MPL:

All distribution of Covered Software in Source Code Form, including any Modifications that You create or to which You contribute, must be under the terms of this License. You must inform recipients that the Source Code Form of the Covered Software is governed by the terms of this License, and how they can obtain a copy of this License. You may not attempt to alter or restrict the recipients’ rights in the Source Code Form.

Free software licenses impose obligations on the user through terms governing attribution, sublicensing, distribution, patents, trademarks, and relationships with laws like the Digital Millennium Copyright Act. The free software community is no stranger to the difficulties in enforcing compliance with these obligations, which some groups view as too onerous. But as onerous as one may view these obligations to be, one is nevertheless required to comply with them. If you believe that the force of copyright should protect your proprietary software, then you must agree that it equally protects open source works, despite the inconvenience or cost associated with this truth.

GitHub’s Copilot is trained on software governed by these terms, and it fails to uphold them, and enables customers to accidentally fail to uphold these terms themselves. Some argue about the risks of a “copyleft surprise”, wherein someone incorporates a GPL licensed work into their product and is surprised to find that they are obligated to release their product under the terms of the GPL as well. Copilot institutionalizes this risk and any user who wishes to use it to develop non-free software would be well-advised not to do so, else they may find themselves legally liable to uphold these terms, perhaps ultimately being required to release their works under the terms of a license which is undesirable for their goals.

Essentially, the argument comes down to whether or not the model constitutes a derivative work of its inputs. Microsoft argues that it does not. However, these licenses are not specific regarding the means of derivation; the classic approach of copying and pasting from one project to another need not be the only means for these terms to apply. The model exists as the result of applying an algorithm to these inputs, and thus the model itself is a derivative work of its inputs. The model, then used to create new programs, forwards its obligations to those works.

All of this assumes the best interpretation of Microsoft’s argument, with a heavy reliance on the fact that the model becomes a general purpose programmer, having meaningfully learned from its inputs and applying this knowledge to produce original work. Should a human programmer take the same approach, studying free software and applying those lessons, but not the code itself, to original projects, I would agree that their applied knowledge is not creating derivative works. However, that is not how machine learning works. Machine learning is essentially a glorified pattern recognition and reproduction engine, and does not represent a genuine generalization of the learning process. It is perhaps capable of a limited amount of originality, but is also capable of degrading to the simple case of copy and paste. Here is an example of Copilot reproducing, verbatim, a function which is governed by the GPL, and would thus be governed by its terms:

Source: Armin Ronacher via Twitter

The license reproduced by Copilot is not correct, neither in form nor function. This code was not written by V. Petkov and the GPL imposes much stronger obligations than those suggested by the comment. This small example was deliberately provoked with a suggestive prompt (this famous function is known as the “fast inverse square root”) and the “float Q_”, but it’s not a stretch to assume someone can accidentally do something similar with any particularly unlucky English-language description of their goal.

Of course, the use of a suggestive prompt to convince Copilot to print GPL licensed code suggests another use: deliberately laundering FOSS source code. If Microsoft’s argument holds, then indeed the only thing which is necessary to legally circumvent a free software license is to teach a machine learning algorithm to regurgitate a function you want to use.

This is a problem. I have two suggestions to offer to two audiences: one for GitHub, and another for free software developers who are worried about Copilot.

To GitHub: this is your Oracle v Google moment. You’ve invested in building a platform on top of which the open source revolution was built, and leveraging this platform for this move is a deep betrayal of the community’s trust. The law applies to you, and banking on the fact that the decentralized open source community will not be able to mount an effective legal challenge to your $7.5B Microsoft war chest does not change this. The open source community is astonished, and the astonishment is slowly but surely boiling over into rage as our concerns fall on deaf ears and you push forward with the Copilot release. I expect that if the situation does not change, you will find a group motivated enough to challenge this. The legitimacy of the free software ecosystem may rest on this problem, and there are many companies who are financially incentivized to see to it that this legitimacy stands. I am certainly prepared to join a class action lawsuit as a maintainer, or alongside other companies with interests in free software making use of our financial resources to facilitate a lawsuit.

The tool can be improved, probably still in time to avoid the most harmful effects (harmful to your business, that is) of Copilot. I offer the following specific suggestions:

Allow GitHub users and repositories to opt-out of being incorporated into the model. Better, allow them to opt-in. Do not tie this flag into unrelated projects like Software Heritage and the Internet Archive.
Track the software licenses which are incorporated into the model and inform users of their obligations with respect to those licenses.
Remove copyleft code from the model entirely, unless you want to make the model and its support code free software as well.
Consider compensating the copyright owners of free software projects incorporated into the model with a margin from the Copilot usage fees, in exchange for a license permitting this use.

Your current model probably needs to be thrown out. The GPL code incorporated into it entitles anyone who uses it to receive a GPL’d copy of the model for their own use. It entitles these people to commercial use, to build a competing product with it. But, it presumably also includes works under incompatible licenses, such as the CDDL, which is… problematic. The whole thing is a legal mess.

I cannot speak for the rest of the community that have been hurt by this project, but for my part, I would be okay with not pursuing the answers to any of these questions with you in court if you agreed to resolve these problems now.

And, my advice to free software maintainers who are pissed that their licenses are being ignored. First, don’t use GitHub and your code will not make it into the model (for now). I’ve written before about why it’s generally important for free software projects to use free software infrastructure, and this only re-enforces that fact. Furthermore, the old “vote with your wallet” approach is a good way to show your disfavor. That said, if it occurs to you that you don’t actually pay for GitHub, then you may want to take a moment to consider if the incentives created by that relationship explain this development and may lead to more unfavorable outcomes for you in the future.

You may also be tempted to solve this problem by changing your software licenses to prohibit this behavior. I’ll say upfront that according to Microsoft’s interpretation of the situation (invoking fair use), it doesn’t matter to them which license you use: they’ll use your code regardless. In fact, some proprietary code was found to have been incorporated into the model. However, I still support your efforts to address this in your software licenses, as it provides an even stronger legal foundation upon which we can reject Copilot.

I will caution you that the way you approach that clause of your license is important. Whenever writing or changing a free and open source software license, you should consider whether or not it will still qualify as free or open source after your changes. To be specific, a clause which outright forbids the use of your code for training a machine learning model will make your software non-free, and I do not recommend this approach. Instead, I would update your licenses to clarify that incorporating the code into a machine learning model is considered a form of derived work, and that your license terms apply to the model and any works produced with that model.

To summarize, I think that GitHub Copilot is a bad idea as designed. It represents a flagrant disregard of FOSS licensing in of itself, and it enables similar disregard — deliberate or otherwise — among its users. I hope they will heed my suggestions, and I hope that my words to the free software community offer some concrete ways to move forward with this problem.

Introducing the Himitsu keyring & password manager for Unix

Mon, 20 Jun 2022 00:00:00 +0000

Himitsu is a new approach to storing secret information on Unix systems, such as passwords or private keys, and I released version 0.1 this morning. It’s available on Alpine Linux community and the Arch User Repository, with more distributions hopefully on the way soon.

So, what is Himitsu and what makes it special? The following video introduces the essential concepts and gives you an idea of what’s possible:

If you prefer reading to watching, this blog post includes everything that’s in the video.

What is Himitsu?

Himitsu draws inspiration from Plan 9’s factotum, but polished up and redesigned for Unix. At its core, Himitsu is a key/value store and a simple protocol for interacting with it. For example, a web login could be stored like so:

proto=web host=example.org user=jdoe password!=hunter2

Himitsu has no built-in knowledge of web logins, it just stores arbitrary keys and values. The bang (!) indicates that the password is a “secret” value, and the “proto” key defines additional conventions for each kind of secret. For proto=web, each key/value pair represents a form field on a HTML login form.

We can query the key store using the “hiq” command. For instance, we can obtain the example key above by querying for any key with “proto=web”, any “host”, “user”, and “password” value, and an optional “comment” value:

$ hiq proto=web host user password! comment?
proto=web host=example.org user=jdoe password!

You’ll notice that the password is hidden here. In order to obtain it, we must ask for the user’s consent.

$ hiq -d proto=web host user password! comment?

A screenshot of a GTK+ dialog confirming the operation

proto=web host=example.org user=jdoe password!=hunter2

You can also use hiq to add or delete keys, or incorporate it into a shell pipeline:

$ hiq -dFpassword host=example.org
hunter2

A simple, extensible protocol

The protocol is a simple line-oriented text protocol, which is documented in the himitsu-ipc(5) manual page. We can also use it via netcat:

$ nc -U $XDG_RUNTIME_DIR/himitsu
query host=example.org
key proto=web host=example.org user=jdoe password!
end
query -d host=example.org
key proto=web host=example.org user=jdoe password!=hunter2
end

The consent prompter also uses a standardized protocol, documented by himitsu-prompter(5). Based on this, you can implement new prompters for Qt, or the TTY, or any other technology appropriate to your system, or implement a more novel approach, such as sending a push notification to your phone to facilitate consent.

Additional frontends

Based on these protocols, a number of additional integrations are possible. Martijn Braam has written a nice GTK+ frontend called keyring:

A screenshot of the GTK+ frontend

There’s also a Firefox add-on which auto-fills forms for keys with proto=web:

Screenshot of himitsu-firefox

We also have a package called himitsu-ssh which provides an SSH agent:

$ hissh-import < ~/.ssh/id_ed25519
Enter SSH key passphrase: 
key proto=ssh type=ssh-ed25519 pkey=pF7SljE25sVLdWvInO4gfqpJbbjxI6j+tIUcNWzVTHU= skey! comment=sircmpwn@homura
$ ssh-add -l
256 SHA256:kPr5ZKTNE54TRHGSaanhcQYiJ56zSgcpKeLZw4/myEI sircmpwn@homura (ED25519)
$ ssh git@git.sr.ht
Hi sircmpwn! You've successfully authenticated, but I do not provide an interactive shell. Bye!
Connection to git.sr.ht closed.

I hope to see an ecosystem of tools built around Himitsu to grow. New frontends like keyring would be great, and new integrations like GPG agents would also be nice to see.

Zero configuration

Himitsu-aware software can discover your credentials and connection details without any additional configuration. For example, a mail client might look for proto=imap and proto=smtp and discover something like this:

proto=imap host=imap.migadu.com user=drew@ddevault.org password! port=993 enc=tls
proto=smtp host=imap.migadu.com user=drew@ddevault.org password! port=465 enc=tls

After a quick consent prompt, the software can load your IMAP and SMTP configuration and get connected without any manual steps. With an agent like himitsu-ssh, it could even connect without actually handling your credentials directly — a use-case we want to support with improvements to the prompter UI (to distinguish between a case where an application will view versus use your credentials).

The cryptography

Your key store is located at $XDG_DATA_HOME/himitsu/. The key is derived by mixing your password with argon2, and the resulting key is used for AEAD with XChaCha20+Poly1305. The “index” file contains a list of base64-encoded encrypted blobs, one per line, enumerating the keys in the key store.¹ Secret keys are encrypted and stored separately in files in this directory. If you like the pass approach to storing your keys in git, you can easily commit this directory to a git repository, or haul it along to each of your devices with whatever other means is convenient to you.

Himitsu is written in Hare and uses cryptography primitives available from its standard library. Note that these have not been audited.

Future plans

I’d like to expand on Himitsu in the future. One idea is to store your full disk encryption password in Himitsu and stick a subset of your key store into the initramfs, which you unlock during early boot, pull FDE keys out of, and then pre-authorize the keyring for your desktop session - which you’re logged in to automatically on the basis that you were pre-authorized during boot.

We also want to add key sharing and synchronization tools. The protocol could easily be moved to TCP and authorized with your existing key store key (we could make an ed25519 key out of it, or generate and store one separately), so setting up key synchronization might be as simple as:

$ hiq -a proto=sync host=himitsu.sr.ht

You could also use Himitsu for service discovery — imagine a key ring running on your datacenter LAN with entries for your Postgres database, SMTP credentials, and so on.

There are some other ideas that we could use your help with:

himitsu-firefox improvements (web devs welcome!)
Chromium support (web devs welcome!)
Himitsu apps for phones (mobile devs welcome!)
More key management frontends (maybe a TUI?)
More security options — smart cards? U2F?
hare-ssh improvements (e.g. RSA keys)
PGP support
Anything else you can think of

Please join us! We hang out on IRC in #himitsu on Libera Chat. Give Himitsu a shot and let us know what you think.

Alright, back to kernel hacking. I got multi-tasking working yesterday!

Status update, June 2022

Wed, 15 Jun 2022 00:00:00 +0000

Hello again! I would like to open this post by acknowledging the response to my earlier post, “bleh”. Since it was published, I have received several hundred emails expressing support and kindness. I initially tried to provide these with thoughtful replies, then shorter replies, then I had to stop replying at all, but I did read every one. Thank you, everyone, for sending these. I appreciate it very much, and it means a lot to me.

I have actually had a lot more fun programming this month than usual, since I decided to spend more time on experimental and interesting projects and less time on routine maintenance or long-term developments. So, the feature you’ve been waiting for in SourceHut might be delayed, but in return, there’s cool progress on the projects that you didn’t even know you were waiting for. Of course, the SourceHut workload never dips below a dull roar, as I have to attend to business matters and customer support promptly, and keep a handle on the patch queue, and the other SourceHut staff and contributors are always hard at work — so there’ll be plenty to discuss in the “what’s cooking” later.

The bulk of my focus has been on the Helios kernel this month, a project I introduced a couple of days ago. I spent a lot of time furiously refactoring, reworking the existing kernel code for evalutaing features like page allocation and virtual address space management into capability-oriented kernel services that can be provided to userspace, then overhauling our startup code to provision a useful set of capabilities for the init process to take advantage of. I also implemented x86_64 I/O port services, which allowed for the first few drivers to be written in userspace — serial ports and simple VBE graphics. We also got interrupts working properly and brought up the PIT, which is another major step towards multi-tasking. I also implemented a new syscall ABI with error handling, and refactored a lot of the arch-specific code to make new ports easier. The kernel is in a much better state now than it was a month ago (and to think it’s only three months old!).

A picture of Helios drawing to a framebuffer

There was also a lot of progress on Himitsu, which I plan on presenting in a video and blog post in a few days time. The Firefox add-on actually works now (though some features remain to be done), and Alexey Yerin fixed several important bugs and contributed several new features. The user is now prompted to consent before deleting keys, and we have a new GTK+ prompter written in Python, which is much more reliable and feature-full thanks to Martijn Braam’s help (rewriting it in C again is a long-term TODO item for any interested contributor). I also made some progress towards what will ultimately become full-disk encryption support.

himitsu-keyring, a new GTK+ keyring manager from Martijn Braam

Hare also enjoyed many improvements this month. We have some new improvements to date/time support, including fixes for Martian time ;) I also mostly implemented cross-compiling, which you can try out with hare build -t riscv64 or something similar. The major outstanding pain point here is that the Hare cache is not arch-aware, so you need to rm -rf ~/.cache/hare each time you switch architectures for now. We now have complex number support, as well as improvements to encoding::json and net::uri.

A screenshot of a fractal rendered with the aid of Hare’s new complex number support

That’s all for today. Until next time!

The Helios microkernel

Mon, 13 Jun 2022 00:00:00 +0000

I’ve been working on a cool project lately that I’d like to introduce you to: the Helios microkernel. Helios is written in Hare and currently targets x86_64, and riscv64 and aarch64 are on the way. It’s very much a work-in-progress: don’t expect to pick this up and start building anything with it today.

A picture of a ThinkPad running Helios, demonstrating userspace memory allocation

Drawing some inspiration from seL4, Helios uses a capability-based design for isolation and security. The kernel offers primitives for allocating physical pages, mapping them into address spaces, and managing tasks, plus features like platform-specific I/O (e.g. reading and writing x86 ports). The entire system is written in Hare, plus some necessary assembly for the platform bits (e.g. configuring the GDT or IDT).

Things are still quite early, but I’m pretty excited about this project. I haven’t had this much fun hacking in some time :) We have several kernel services working, including memory management and virtual address spaces, and I’ve written a couple of simple drivers in userspace (serial and BIOS VGA consoles). Next up is preemptive multi-tasking — we already have interrupts working reliably, including the PIT, so all that’s left for multi-tasking is to actually implement the context switch. I’d like to aim for an seL4-style single-stack system, though some finageling will be required to make that work.

Again, much of the design comes from seL4, but unlike seL4, we intend to build upon this kernel and develop a userspace as well. Each of the planned components is named after celestial bodies, getting further from the sun as they get higher-level:

Helios: the kernel
Mercury: low-level userspace services & service bus
Venus: real-world driver collection
Gaia: high-level programming environment
Ares: a complete operating system; package management, GUI, etc

A few other components are planned — “Vulcan” is the userspace kernel testing framework, named for the (now disproved) hypothetical planet between Mercury and the Sun, and “Luna” is the planned POSIX compatibility layer. One of the goals is to be practical for use on real-world hardware. I’ve been testing it continuously on my ThinkPads to ensure real-world hardware support, and I plan on writing drivers for its devices — Intel HD graphics, HD Audio, and Intel Gigabit Ethernet at the least. A basic AMD graphics driver is also likely to appear, and perhaps drivers for some SoC’s, like Raspberry Pi’s VideoCore. I have some neat ideas for the higher-level components as well, but I’ll save those for later.

Why build a new operating system? Well, for a start, it’s really fun. But I also take most of my projects pretty seriously and aim for real-world usability, though it remains to be seen if this will be achieved. This is a hugely ambitious project, or, in other words, my favorite kind of project. Even if it’s not ultimately useful, it will drive the development of a lot of useful stuff. We’re planning to design a debugger that will be ported to Linux as well, and we’ll be developing DWARF support for Hare to facilitate this. The GUI toolkit we want to build for Ares will also be generally applicable. And Helios and Mercury together have a reasonably small scope and makes for an interesting and useful platform in their own right, even if the rest of the stack never completely materializes. If nothing else, it will probably be able to run DOOM fairly soon.

The kernel is a microkernel, so it is fairly narrow in scope and will probably be more-or-less complete in the foreseeable future. The next to-do items are context switching, so we can set up multi-tasking, IPC, fault handling, and userspace support for interrupts. We’ll also need to parse the ACPI tables and bring up PCI in the kernel before handing it off to userspace. Once these things are in place, the kernel is essentially ready to be used to write most drivers, and the focus will move to fleshing out Mercury and Venus, followed by a small version of Gaia that can at least support an interactive shell. There are some longer-term features which will be nice to have in the kernel at some point, though, such as SMP, IOMMU, or VT-x support.

Feel free to pull down the code and check it out, though remember my warning that it doesn’t do too much yet. You can download the latest ISO from the CI, if you want to reproduce the picture at the top of this post, and write it to a flash drive to stick in the x86_64 computer of your choice (boot via legacy BIOS). If you want to mess with the code, you could play around with the Vulcan system to get simple programs running in userspace. The kernel serial driver is write-only, but a serial driver written in userspace could easily be made to support interactive programs. If you’re feeling extra adventureous, it probably wouldn’t be too difficult to get a framebuffer online and draw some pixels — ping me in #helios on Libera Chat for a few words of guidance if you want to try it.

bleh

Mon, 30 May 2022 00:00:00 +0000

A few weeks ago, the maintainer of a project on SourceHut stepped down from their work, citing harassment over using SourceHut as their platform of choice. It was a difficult day when I heard about that.

Over the past few weeks, I have been enduring a bit of a depressive episode. It’s a complex issue rooted in several different problems, but I think a major source of it is the seemingly constant deluge of hate I find myself at the receiving end of online. I had to grow a thick skin a long time ago, but lately it has not been thick enough. I am finding it increasingly difficult to keep up with my work.

Perhaps it this has something to do with the backlash, not just against me and my work, but against others who use and participate in that work. It’s not enough to dislike my programming language, but the skeptics must publicly denounce it and discourage others from using it. It’s irresponsible, if not immoral, to design a language without a borrow checker in 2022. SourceHut’s email-oriented approach might not be for everyone, and instead of simply not using it, skeptics must harass any projects that do. This kind of harassment is something I hear about often from many maintainers of projects on SourceHut. It breaks my heart and I feel helpless to do anything about it.

I’m also often dealing with harassment directed at me alone. When I complained this week about being DDoSed by a company with over a billion dollars in annual revenue, it was portrayed as righteous retribution and a sign of incompetence. I can’t even count the number of times someone has said they would refuse to use SourceHut (and that you, too, dear reader, should avoid it) on the sole basis that I’m involved with it. There is a steady supply of vile comments about me based on “facts” delivered from the end of a game of telephone in which every participant hates my guts, all easily believable without further research because I’m such a villainous character. Every project I work on, every blog post I write, even many of the benign emails to public lists or GitHub issues I open — the response is just vitriol.

I have made no shortage of mistakes, and there are plenty of hurt feelings which can be laid at my feet. I am regretful for my mistakes, and I have worked actively to improve. I think that it has been working. Perhaps that’s arrogant of me to presume, but I’m not sure what else to do. Must I resign myself to my fate for stupid comments I made years ago? I’m sorry, and I’ve been working to do better. Can I have another chance?

For some I think the answer is “no”. Many of my detractors just want me to shut up. No more blog posts, no new projects. Just go away, Drew.

Well, I can’t say it’s not working. This stuff gets to me. At times like this I have very little motivation to work. If you’re looking for a strategy to get me to shut up, just ensure that I have a constant flow of toxic comments to read.

I love writing code, at least most of the time. I believe in my principles and I enjoy writing software that embodies them. I love doing it, and I’m really good at it, and thousands of people are depending on my work.

I’m doing the work that I believe in, and working with people who share those values. I have worked very hard for that privilege. I’m sorry that it’s not good enough for many people. I’m just trying to do my best. And if you must harass anyone over it, at least harass me, and not anyone else. My inbox is at drew@ddevault.org, and I promise that I will read your email and cry, so that no one else has to.

I’ll close by thanking those who have sent me positive notes. Some of these comments are very touching. If you’ve sent one of these, you have my thanks. Love you :)

Google has been DDoSing SourceHut for over a year

Wed, 25 May 2022 00:00:00 +0000

Just now, I took a look at the HTTP logs on git.sr.ht. Of the past 100,000 HTTP requests received by git.sr.ht (representing about 2½ hours of logs), 4,774 have been requested by GoModuleProxy — 5% of all traffic. And their requests are not cheap: every one is a complete git clone. They come in bursts, so every few minutes we get a big spike from Go, along with a constant murmur of Go traffic.

This has been ongoing since around the release of Go 1.16, which came with some changes to how Go uses modules. Since this release, following a gradual ramp-up in traffic as the release was rolled out to users, git.sr.ht has had a constant floor of I/O and network load for which the majority can be attributed to Go.

I started to suspect that something strange was going on when our I/O alarms started going off in February 2021 (we eventually had to tune these alarms up above the floor of I/O noise generated by Go), correlated with lots of activity from a Go user agent. I was able to narrow it down with some effort, but to the credit of the Go team they did change their User-Agent to make more apparent what was going on. Ultimately, this proved to be the end of the Go team’s helpfulness in this matter.

I did narrow it down: it turns out that the Go Module Mirror runs some crawlers that periodically clone Git repositories with Go modules in them to check for updates. Once we had narrowed this down, I filed a second ticket to address the problem.

I came to understand that the design of this feature is questionable. For a start, I never really appreciated the fact that Go secretly calls home to Google to fetch modules through a proxy (you can set GOPROXY=direct to fix this). Even taking the utility at face value, however, the implementation leaves much to be desired. The service is distributed across many nodes which all crawl modules independently of one another, resulting in very redundant git traffic.

140 8a42ab2a4b4563222b9d12a1711696af7e06e4c1092a78e6d9f59be7cb1af275
 57 9cc95b73f370133177820982b8b4e635fd208569a60ec07bd4bd798d4252eae7
 44 9e730484bdf97915494b441fdd00648f4198be61976b0569338a4e6261cddd0a
 44 80228634b72777eeeb3bc478c98a26044ec96375c872c47640569b4c8920c62c
 44 5556d6b76c00cfc43882fceac52537b2fdaa7dff314edda7b4434a59e6843422
 40 59a244b3afd28ee18d4ca7c4dd0a8bba4d22d9b2ae7712e02b1ba63785cc16b1
 40 51f50605aee58c0b7568b3b7b3f936917712787f7ea899cc6fda8b36177a40c7
 40 4f454b1baebe27f858e613f3a91dfafcdf73f68e7c9eba0919e51fe7eac5f31b

This is a sample from a larger set which shows the hashes of git repositories on the right (names were hashed for privacy reasons), and the number of times they were cloned over the course of an hour. The main culprit is the fact that the nodes all crawl independently and don’t communicate with each other, but the per-node stats are not great either: each IP address still clones the same repositories 8-10 times per hour. Another user hosting their own git repos noted a single module being downloaded over 500 times in a single day, generating 4 GiB of traffic.

The Go team holds that this service is not a crawler, and thus they do not obey robots.txt — if they did, I could use it to configure a more reasonable “Crawl-Delay” to control the pace of their crawling efforts. I also suggested keeping the repositories stored on-site and only doing a git fetch, rather than a fresh git clone every time, or using shallow clones. They could also just fetch fresh data when users request it, instead of pro-actively crawling the cache all of the time. All of these suggestions fell on deaf ears, the Go team has not prioritized it, and a year later I am still being DDoSed by Google as a matter of course.

I was banned from the Go issue tracker for mysterious reasons,¹ so I cannot continue to nag them for a fix. I can’t blackhole their IP addresses, because that would make all Go modules hosted on git.sr.ht stop working for default Go configurations (i.e. without GOPROXY=direct). I tried to advocate for Linux distros to patch out GOPROXY by default, citing privacy reasons, but I was unsuccessful. I have no further recourse but to tolerate having our little-fish service DoS’d by a 1.38 trillion dollar company. But I will say that if I was in their position, and my service was mistakenly sending an excessive amount of traffic to someone else, I would make it my first priority to fix it. But I suppose no one will get promoted for prioritizing that at Google.

Status update, May 2022

Mon, 16 May 2022 00:00:00 +0000

This was an exciting month: the Hare programming language is a secret no more! You can now try out the programming language I first teased over a year ago and tell me what you think. I hope you like it! I’m quite pleased with it so far.

One thing Hare has done is allow me to unshelve several projects which were blocked pending the availability of a suitable language to write them in. I have actually been working on several of these for a while now — and several more are to come later — but I couldn’t share them thanks to Hare’s policy of secrecy early in its development. Allow me to introduce you to a few projects!

Helios is a micro-kernel for x86_64, and ideally later for aarch64 and riscv64 as well (and possibly other targets as Hare grows additional ports). We have a few things working, such as paging and interrupts, and as of this morning we have entered userspace. Next up is rigging up syscalls and scheduling, then we’re going to start fleshing out an L4-inspired API and writing some drivers in userspace.

A screenshot showing Helios booting and entering userspace

Himitsu is a secret storage system. It can act as a password manager, but it also stores other arbitrary secret data, such as private keys. Each key is a set of key/value pairs, some of which can be secret. This allows you to store additional data alongside your password (such as your username or email for login), and also supports secret data other than passwords — like SSH keys. An extensible consent and agent protocols allow you to expand it to support a wide variety of use-cases for secure use of secrets.

btqd, or “bittorrent queue daemon”, is (going to be) a bittorrent daemon, but it is still very early in development. The design is essentially that of a process supervisor which manages a queue of torrents and fires up subprocesses to seed or leech for a set of active torrents. Each subprocess, such as btlc (bittorrent leech client), or btsc (bittorrent seed client), can also be used separately from the queue daemon. Further development is blocked on net::http, which is blocked on TLS support, for tracker announce requests. I may temporarily unblock this by shelling out to curl instead.

scheduled is also early in development. It is a replacement for crond (and also at(1)) which is redesigned from the ground up. I have never been thrilled with cron’s design — it’s very un-Unix like. scheduled will have better error handling and logging, a much more flexible and understandable approach to configuration, and a better approach to security, plus the ability to do ad-hoc scheduling from the command line. This was designed prior to date/time support landing in Hare, and was blocked for a while, but is now unblocked. However, it is not my highest priority.

Each of these projects will spawn more blog posts (or talks) going into greater depth on their design goals and rationale later on. For now, with the introductions out of the way, allow me to fill you in on the things which got done in this past month in particular.

I’ll keep the SourceHut news short, and expand upon it in the “what’s cooking” post later today. For my own part, I spent some time working on hut to add support for comprehensive account data import/export. This will allow you to easily take all of your data out of sourcehut and import it into another instance, or any compatible software — your git repos are just git repos and your mailing lists are just mbox files, so you could push them to GitHub or import them into GNU Mailman, for example. This work is also a step towards self-service account deletion and renaming, both prioritized for the beta.

Regarding Hare itself, there are many important recent developments. Over 300 commits landed this month, so I’ll have to leave some details out. An OpenBSD port is underway by Brian Callahan, and the initial patches have landed for the Hare compiler. The crypto module grew blowfish and bcrypt support, both useful mainly for legacy compatibility, as well as the more immediately useful x25519 and pem implementations. There is also a new encoding::json module,¹ and a number of fixes and improvements have been steadily flowing in for regex, bufio, net, net::uri, and datetime, along with dozens of others.

For Himitsu, I developed hare-ssh this month to facilitate the addition of himitsu-ssh, which provides SSH tooling that integrates with Himitsu (check out the video above for a demo). The “hissh-import” command decodes OpenSSH private keys and loads them into the Himitsu keystore, and the “hissh-agent” command runs an SSH agent that performs authentication with the private keys stored in Himitsu. Future additions will include “hissh-export”, for getting your private keys back out in a useful format, and “hissh-keygen”, for skipping the import/export step entirely. Presently only ed25519 keys are supported; more will be added as the necessary primitives are added to Hare upstream.

I did some work on Helios this weekend, following a brief hiatus. I wrote a more generalized page table implementation which can manage multiple page tables (necessary to have separate address spaces for each process), and started rigging up the kernel to userspace transition, which I briefly covered earlier in the post. As of this morning, I have some code running in userspace — one variant attempts to cli, causing a general protection fault (as expected), and another just runs a busy loop, which works without any faults. Next steps are syscalls and scheduling.

That’s all the news for today. Hare! Woo! Thanks for reading, and be sure to check out — and maybe contribute to? — some of these projects. Take care!

A Hare code generator for finding ioctl numbers

Sat, 14 May 2022 00:00:00 +0000

Modern Unix derivatives have this really bad idea called ioctl. It’s a function which performs arbitrary operations on a file descriptor. It is essentially the kitchen sink of modern Unix derivatives, particularly Linux, in which they act almost like a second set of extra syscalls. For example, to get the size of the terminal window, you use an ioctl specific to TTY file descriptors:

let wsz = rt::winsize { ... };
match (rt::ioctl(fd, rt::TIOCGWINSZ, &wsz: *void)) {
case let e: rt::errno =>
	switch (e: int) {
	case rt::EBADFD =>
		return errors::invalid;
	case rt::ENOTTY =>
		return errors::unsupported;
	case =>
		abort("Unexpected error from ioctl");
	};
case int =>
	return ttysize {
		rows = wsz.ws_row,
		columns = wsz.ws_col,
	};
};

This code performs the ioctl syscall against the provided file descriptor “fd”, using the “TIOCGWINSZ” operation, and setting the parameter to a pointer to a winsize structure. There are thousands of ioctls provided by Linux, and each of them is assigned a constant like TIOCGWINSZ (0x5413). Some constants, including this one, are assigned somewhat arbitrarily. However, some are assigned with some degree of structure.

Consider for instance the ioctl TUNSETOWNER, which is used for tun/tap network devices. This ioctl is assigned the number 0x400454cc, but this is not selected arbitrarily. It’s assigned with a macro, which we can find in /usr/include/linux/if_tun.h:

#define TUNSETOWNER   _IOW('T', 204, int)

The _IOW macro, along with similar ones like _IO, _IOR, and _IOWR, are defined in /usr/include/asm-generic/ioctl.h. They combine this letter, number, and parameter type (or rather its size), and the direction (R, W, WR, or neither), OR’d together into an unsigned 32-bit number:

#define _IOC_WRITE	1U

#define _IOC_TYPECHECK(t) (sizeof(t))

#define _IOC(dir,type,nr,size) \
	(((dir)  << _IOC_DIRSHIFT) | \
	 ((type) << _IOC_TYPESHIFT) | \
	 ((nr)   << _IOC_NRSHIFT) | \
	 ((size) << _IOC_SIZESHIFT))

#define _IOW(type,nr,size)	_IOC(_IOC_WRITE,(type),(nr),(_IOC_TYPECHECK(size)))

It would be useful to define ioctl numbers in a similar fashion for Hare programs. However, Hare lacks macros, so we cannot re-implement this in exactly the same manner. Instead, we can use code generation.

Hare is a new systems programming language I’ve been working on for a couple of years. Check out the announcement for more detail.

Again using the tun interface as an example, our goal is to turn the following input file:

type sock_filter = struct {
	code: u16,
	jt: u8,
	jf: u8,
	k: u32,
};

type sock_fprog = struct {
	length: u16,
	filter: *sock_filter,
};

def TUNSETNOCSUM: u32 = @_IOW('T', 200, int);
def TUNSETDEBUG: u32 = @_IOW('T', 201, int);
def TUNSETIFF: u32 = @_IOW('T', 202, int);
def TUNSETPERSIST: u32 = @_IOW('T', 203, int);
def TUNSETOWNER: u32 = @_IOW('T', 204, int);
def TUNSETLINK: u32 = @_IOW('T', 205, int);
def TUNSETGROUP: u32 = @_IOW('T', 206, int);
def TUNGETFEATURES: u32 = @_IOR('T', 207, uint);
def TUNSETOFFLOAD: u32 = @_IOW('T', 208, uint);
def TUNSETTXFILTER: u32 = @_IOW('T', 209, uint);
def TUNGETIFF: u32 = @_IOR('T', 210, uint);
def TUNGETSNDBUF: u32 = @_IOR('T', 211, int);
def TUNSETSNDBUF: u32 = @_IOW('T', 212, int);
def TUNATTACHFILTER: u32 = @_IOW('T', 213, sock_fprog);
def TUNDETACHFILTER: u32 = @_IOW('T', 214, sock_fprog);
def TUNGETVNETHDRSZ: u32 = @_IOR('T', 215, int);
def TUNSETVNETHDRSZ: u32 = @_IOW('T', 216, int);
def TUNSETQUEUE: u32 = @_IOW('T', 217, int);
def TUNSETIFINDEX: u32 = @_IOW('T', 218, uint);
def TUNGETFILTER: u32 = @_IOR('T', 219, sock_fprog);
def TUNSETVNETLE: u32 = @_IOW('T', 220, int);
def TUNGETVNETLE: u32 = @_IOR('T', 221, int);
def TUNSETVNETBE: u32 = @_IOW('T', 222, int);
def TUNGETVNETBE: u32 = @_IOR('T', 223, int);
def TUNSETSTEERINGEBPF: u32 = @_IOR('T', 224, int);
def TUNSETFILTEREBPF: u32 = @_IOR('T', 225, int);
def TUNSETCARRIER: u32 = @_IOW('T', 226, int);
def TUNGETDEVNETNS: u32 = @_IO('T', 227);

Into the following output file:

type sock_filter = struct {
	code: u16,
	jt: u8,
	jf: u8,
	k: u32,
};

type sock_fprog = struct {
	length: u16,
	filter: *sock_filter,
};

def TUNSETNOCSUM: u32 = 0x400454c8;
def TUNSETDEBUG: u32 = 0x400454c9;
def TUNSETIFF: u32 = 0x400454ca;
def TUNSETPERSIST: u32 = 0x400454cb;
def TUNSETOWNER: u32 = 0x400454cc;
def TUNSETLINK: u32 = 0x400454cd;
def TUNSETGROUP: u32 = 0x400454ce;
def TUNGETFEATURES: u32 = 0x800454cf;
def TUNSETOFFLOAD: u32 = 0x400454d0;
def TUNSETTXFILTER: u32 = 0x400454d1;
def TUNGETIFF: u32 = 0x800454d2;
def TUNGETSNDBUF: u32 = 0x800454d3;
def TUNSETSNDBUF: u32 = 0x400454d4;
def TUNATTACHFILTER: u32 = 0x401054d5;
def TUNDETACHFILTER: u32 = 0x401054d6;
def TUNGETVNETHDRSZ: u32 = 0x800454d7;
def TUNSETVNETHDRSZ: u32 = 0x400454d8;
def TUNSETQUEUE: u32 = 0x400454d9;
def TUNSETIFINDEX: u32 = 0x400454da;
def TUNGETFILTER: u32 = 0x801054db;
def TUNSETVNETLE: u32 = 0x400454dc;
def TUNGETVNETLE: u32 = 0x800454dd;
def TUNSETVNETBE: u32 = 0x400454de;
def TUNGETVNETBE: u32 = 0x800454df;
def TUNSETSTEERINGEBPF: u32 = 0x800454e0;
def TUNSETFILTEREBPF: u32 = 0x800454e1;
def TUNSETCARRIER: u32 = 0x400454e2;
def TUNGETDEVNETNS: u32 = 0x54e3;

I wrote the ioctlgen tool for this purpose, and since it demonstrates a number of interesting Hare features, I thought it would make for a cool blog post. This program must do the following things:

Scan through the file looking for @_IO* constructs
Parse these @_IO* constructs
Determine the size of the type specified by the third parameter
Compute the ioctl number based on these inputs
Write the computed constant to the output
Pass everything else through unmodified

The implementation begins thusly:

let ioctlre: regex::regex = regex::regex { ... };
let typedefre: regex::regex = regex::regex { ... };

@init fn init() void = {
	ioctlre = regex::compile(`@(_IO[RW]*)\((.*)\)`)!;
	typedefre = regex::compile(`^(export )?type `)!;
};

@fini fn fini() void = {
	regex::finish(&ioctlre);
	regex::finish(&typedefre);
};

This sets aside two regular expressions: one that identifies type aliases (so that we can parse them to determine their size later), and one that identifies our @_IO* pseudo-macros. I also defined some types to store each of the details necessary to compute the ioctl assignment:

type dir = enum u32 {
	IO = 0,
	IOW = 1,
	IOR = 2,
	IOWR = IOW | IOR,
};

type ioctl = (dir, rune, u32, const nullable *types::_type);

Hare’s standard library includes tools for parsing and analyzing Hare programs in the hare namespace. We’ll need to use these to work with types in this program. At the start of the program, we initialize a “type store” from hare::types, which provides a mechanism with which Hare types can be processed and stored. The representation of Hare types varies depending on the architecture (for example, pointer types have different sizes on 32-bit and 64-bit systems), so we have to specify the architecture we want. In the future it will be necessary to make this configurable, but for now I just hard-coded x86_64:

const store = types::store(types::x86_64, null, null);
defer types::store_free(store);

The two “null” parameters are not going to be used here, but are designed to facilitate evaluating expressions in type definitions, such as [8 * 16]int. Leaving them null is permissible, but disables the ability to do this sort of thing.

Following this, we enter a loop which processes the input file line-by-line, testing each line against our regular expressions and doing some logic on them if they match. Let’s start with the code for handling new types:

for (true) {
	const line = match (bufio::scanline(os::stdin)!) {
	case io::EOF =>
		break;
	case let line: []u8 =>
		yield strings::fromutf8(line);
	};
	defer free(line);

	if (regex::test(&typedefre, line)!) {
		bufio::unreadrune(os::stdin, '\n');
		bufio::unread(os::stdin, strings::toutf8(line));
		loadtype(store);
		continue;
	};

	// ...to be continued...

If we encounter a line which matches our type declaration regular expression, then we unread that line back into the (buffered) standard input stream, then call this “loadtype” function to parse and load it into the type store.

fn loadtype(store: *types::typestore) void = {
	const tee = io::tee(os::stdin, os::stdout);
	const lex = lex::init(&tee, "<ioctl>");
	const decl = match (parse::decl(&lex)) {
	case let err: parse::error =>
		fmt::fatal("Error parsing type declaration:",
			parse::strerror(err));
	case let decl: ast::decl =>
		yield decl;
	};

	const tdecl = decl.decl as []ast::decl_type;
	if (len(tdecl) != 1) {
		fmt::fatal("Multiple type declarations are unsupported");
	};
	const tdecl = tdecl[0];
	const of = types::lookup(store, &tdecl._type)!;
	types::newalias(store, tdecl.ident, of);
};

Hare includes a Hare lexer and parser in the standard library, which we’re making use of here. The first thing we do is use io::tee to copy any data the parser reads into stdout, passing it through to the output file. Then we set up a lexer and parse the type declaration. A type declaration looks something like this:

type sock_fprog = struct {
	length: u16,
	filter: *sock_filter,
};

The types::lookup call looks up the struct type, and newalias creates a new type alias based on that type with the given name (sock_filter). Adding this to the type store will let us resolve the type when we encounter it later on, for example in this line:

def TUNGETFILTER: u32 = @_IOR('T', 219, sock_fprog);

Back to the main loop, we have another regex test to check if we’re looking at a line with one of these pseudo-macros:

let groups = match (regex::find(&ioctlre, line)!) {
case void =>
	fmt::println(line)!;
	continue;
case let cap: []regex::capture =>
	yield cap;
};
defer free(groups);

const dir = switch (groups[1].content) {
case "_IO" =>
	yield dir::IO;
case "_IOR" =>
	yield dir::IOR;
case "_IOW" =>
	yield dir::IOW;
case "_IOWR" =>
	yield dir::IOWR;
case =>
	fmt::fatalf("Unknown ioctl direction {}", groups[1].content);
};
const ioctl = parseioctl(store, dir, groups[2].content);

Recall that the regex from earlier is @(_IO[RW]*)$(.*)$. This has two capture groups: one for “_IO” or “_IOW” and so on, and another for the list of “parameters” (the zeroth “capture group” is the entire match string). We use the first capture group to grab the ioctl direction, then we pass that into “parseioctl” along with the type store and the second capture group.

This “parseioctl” function is kind of neat:

fn parseioctl(store: *types::typestore, d: dir, params: str) ioctl = {
	const buf = bufio::fixed(strings::toutf8(params), io::mode::READ);
	const lex = lex::init(&buf, "<ioctl>");

	const rn = expect(&lex, ltok::LIT_RUNE).1 as rune;
	expect(&lex, ltok::COMMA);
	const num = expect(&lex, ltok::LIT_ICONST).1 as i64;

	if (d == dir::IO) {
		return (d, rn, num: u32, null);
	};

	expect(&lex, ltok::COMMA);
	const ty = match (parse::_type(&lex)) {
	case let ty: ast::_type =>
		yield ty;
	case let err: parse::error =>
		fmt::fatal("Error:", parse::strerror(err));
	};

	const ty = match (types::lookup(store, &ty)) {
	case let err: types::error =>
		fmt::fatal("Error:", types::strerror(err));
	case types::deferred =>
		fmt::fatal("Error: this tool does not support forward references");
	case let ty: const *types::_type =>
		yield ty;
	};

	return (d, rn, num: u32, ty);
};

fn expect(lex: *lex::lexer, want: ltok) lex::token = {
	match (lex::lex(lex)) {
	case let err: lex::error =>
		fmt::fatal("Error:", lex::strerror(err));
	case let tok: lex::token =>
		if (tok.0 != want) {
			fmt::fatalf("Error: unexpected {}", lex::tokstr(tok));
		};
		return tok;
	};
};

Here we’ve essentially set up a miniature parser based on a Hare lexer to parse our custom parameter list grammar. We create a fixed reader from the capture group string, then create a lexer based on this and start pulling tokens out of it. The first parameter is a rune, so we grab a LIT_RUNE token and extract the Hare rune value from it, then after a COMMA token we repeat this with LIT_ICONST to get the integer constant. dir::IO ioctls don’t have a type parameter, so can return early in this case.

Otherwise, we use hare::parse::_type to parse the type parameter, producing a hare::ast::_type. We then pass this to the type store to look up technical details about this type, such as its size, alignment, storage representation, and so on. This converts the AST type — which only has lexical information — into an actual type, including semantic information about the type.

Equipped with this information, we can calculate the ioctl’s assigned number:

def IOC_NRBITS: u32 = 8;
def IOC_TYPEBITS: u32 = 8;
def IOC_SIZEBITS: u32 = 14; // XXX: Arch-specific
def IOC_DIRBITS: u32 = 2; // XXX: Arch-specific

def IOC_NRSHIFT: u32 = 0;
def IOC_TYPESHIFT: u32 = IOC_NRSHIFT + IOC_NRBITS;
def IOC_SIZESHIFT: u32 = IOC_TYPESHIFT + IOC_TYPEBITS;
def IOC_DIRSHIFT: u32 = IOC_SIZESHIFT + IOC_SIZEBITS;

fn ioctlno(io: *ioctl) u32 = {
	const typesz = match (io.3) {
	case let ty: const *types::_type =>
		yield ty.sz;
	case null =>
		yield 0z;
	};
	return (io.0: u32 << IOC_DIRSHIFT) |
		(io.1: u32 << IOC_TYPESHIFT) |
		(io.2 << IOC_NRSHIFT) |
		(typesz: u32 << IOC_SIZESHIFT);
};

And, back in the main loop, print it to the output:

const prefix = strings::sub(line, 0, groups[1].start - 1);
fmt::printfln("{}0x{:x};", prefix, ioctlno(&ioctl))!;

Now we have successfully converted this:

type sock_filter = struct {
	code: u16,
	jt: u8,
	jf: u8,
	k: u32,
};

type sock_fprog = struct {
	length: u16,
	filter: *sock_filter,
};

def TUNATTACHFILTER: u32 = @_IOW('T', 213, sock_fprog);

Into this:

def TUNATTACHFILTER: u32 = 0x401054d5;

A quick C program verifies our result:

#include <linux/ioctl.h>
#include <linux/if_tun.h>
#include <stdio.h>

int main() {
	printf("TUNATTACHFILTER: 0x%lx\n", TUNATTACHFILTER);
}

And:

TUNATTACHFILTER: 0x401054d5

It works!

Critics may draw attention to the fact that we could have saved ourselves much of this work if Hare had first-class macros, but macros are not aligned with Hare’s design goals, so an alternative solution is called for. This particular program is useful only in a small set of specific circumstances (and mainly for Hare developers themselves, less so for most users), but it solves the problem pretty neatly given the constraints it has to work within.

I think this is a nice case study in a few useful features available from the Hare standard library. In addition to POSIX Extended Regular Expression support via the regex module, the hare namespace offers many tools to provide Hare programs with relatively deep insights into the language itself. We can use hare::lex to parse the custom grammar for our pseudo-macros, use hare::parse to parse type declarations, and use hare::types to compute the semantic details of each type. I also like many of the “little things” on display here, such as unreading data back into the buffered stdin reader, or using io::tee to copy data to stdout during parsing.

I hope you found it interesting!

When will we learn?

Thu, 12 May 2022 00:00:00 +0000

Congratulations to Rust for its first (but not its last) supply-chain attack this week! They join a growing club of broken-by-design package managers which publish packages uploaded by vendors directly, with no review step, and ship those packages directly to users with no further scrutiny.

Timeline of major incidents on npm/Crates/PyPI/etc

2022-05-10: Cargo: rustdecimal ships with malicious code
2022-05-09: npm: foreach is taken over via an expired email domain
2022-03-17: npm: node-ipc ships malware targeting Russia and Belarus
2022-01-09: npm: colors and faker are deliberately sabotaged
2021-11-19: PyPI: 11 malicious packages discovered
2021-11-04: npm: rc ships malicious code
2021-11-04: npm: coa steals your passwords
2021-10-22: npm: ua-parser-js ships malicious code
2021-10-11: PyPI: mitmproxy2 typo-squats mitmproxy with an added RCE
2021-07-30: PyPI: 8 malicious packages discovered
2020-12-16: RubyGems: pretty_color (and one other) steals bitcoin from victims
2020-09-11: npm: dozens of packages steal your user’s credit card number
2020-09-03: npm: bb-builder steals your password
2020-04-16: RubyGems: 760+ malicious packages found stealing bitcoin
2018-11-28: npm: event-stream ships with a bitcoin theft kit
2018-10-21: PyPI: colourama sneaks bitcoin addresses into your clipboard
2018-10-13: PyPI: more typo-squatting malware attempts various attacks
2018-07-12: npm: eslint-scope ships with malicious code
2018-07-08: AUR: acroread is compromised
2018-05-11: Snap: a 2048 clone ships a cryptocurrency miner
2017-09-09: PyPI: typo-squatted packages published by researchers
2016-07-22: npm: left-pad incident

There are hundreds of additional examples. I had to leave many of them out. Here’s a good source if you want to find more.

Timeline of similar incidents in official Linux distribution repositories

(this space deliberately left blank)

Why is this happening?

The correct way to ship packages is with your distribution’s package manager. These have a separate review step, completely side-stepping typo-squatting, establishing a long-term relationship of trust between the vendor and the distribution packagers, and providing a dispassionate third-party to act as an intermediary between users and vendors. Furthermore, they offer stable distributions which can be relied upon for an extended period of time, provide cohesive whole-system integration testing, and unified patch distribution and CVE notifications for your entire system.

For more details, see my previous post, Developers: Let distros do their job.

Can these package managers do it better?

I generally feel that overlay package managers (a term I just made up for npm et al) are redundant. However, you may feel otherwise, and wonder what they could do better to avoid these problems.

It’s simple: they should organize themselves more like a system package manager.

Establish package maintainers independent of the vendors
Establish a review process for package updates

There’s many innovations that system package managers have been working on which overlay package managers could stand to learn from as well, such as:

Universal package signatures and verification
Reproducible builds
Mirrored package distribution

For my part, I’ll stick to the system package manager. But if you think that the overlay package manager can do it better: prove it.

Implementing an SSH agent in Hare

Mon, 09 May 2022 00:00:00 +0000

Cross-posted from the Hare blog

In the process of writing an SSH agent for Himitsu, I needed to implement many SSH primitives from the ground up in Hare, now available via hare-ssh. Today, I’m going to show you how it works!

Important: This blog post deals with cryptography-related code. The code you’re going to see today is incomplete, unaudited, and largely hasn’t even seen any code review. Let me begin with a quote from the “crypto” module’s documentation in the Hare standard library:

Cryptography is a difficult, high-risk domain of programming. The life and well-being of your users may depend on your ability to implement cryptographic applications with due care. Please carefully read all of the documentation, double-check your work, and seek second opinions and independent review of your code. Our documentation and API design aims to prevent easy mistakes from being made, but it is no substitute for a good background in applied cryptography.

Do your due diligence before repurposing anything you see here.

Decoding SSH private keys

Technically, you do not need to deal with OpenSSH private keys when implementing an SSH agent. However, my particular use-case includes dealing with this format, so I started here. Unlike much of SSH, the OpenSSH private key format (i.e. the format of the file at ~/.ssh/id_ed25519) is, well, private. It’s not documented and I had to get most of the details from reverse-engineering the OpenSSH C code. The main area of interest is sshkey.c. I’ll spare you from reading it yourself and just explain how it works.

First of all, let’s just consider what an SSH private key looks like:

We can immediately tell that this is a PEM file (RFC 7468). The first step to read this file was to implement a decoder for the PEM format, which has been on our to-do list for a while now, and is also needed for many other use-cases. Similar to many other formats provided in the standard library, you can call pem::newdecoder to create a PEM decoder for an arbitrary I/O source, returning the decoder state on the stack. We can then call pem::next to find the next PEM header (-----BEGIN...), which returns a decoder for that specific PEM blob (this design accommodates PEM files which have several PEM segments concatenated together, or intersperse other data in the file alongside the PEM bits. This is common for other PEM use-cases). With this secondary decoder, we can simply read from it like any other I/O source and it decodes the base64-encoded data and returns it to us as bytes.

Based on this, we can examine the contents of this key with a simple program.

use encoding::hex;
use encoding::pem;
use fmt;
use io;
use os;

export fn main() void = {
	const dec = pem::newdecoder(os::stdin);
	defer pem::finish(&dec);

	for (true) {
		const reader = match (pem::next(&dec)) {
		case let reader: (str, pem::pemdecoder) =>
			yield reader;
		case io::EOF =>
			break;
		};
		const name = reader.0, stream = reader.1;
		defer io::close(&stream)!;

		fmt::printfln("PEM data '{}':", name)!;
		const bytes = io::drain(&stream)!;
		defer free(bytes);

		hex::dump(os::stdout, bytes)!;
	};
};

Running this program on our sample key yields the following:

PEM data 'OPENSSH PRIVATE KEY':
00000000  6f 70 65 6e 73 73 68 2d  6b 65 79 2d 76 31 00 00  |openssh-key-v1..|
00000010  00 00 0a 61 65 73 32 35  36 2d 63 74 72 00 00 00  |...aes256-ctr...|
00000020  06 62 63 72 79 70 74 00  00 00 18 00 00 00 10 d3  |.bcrypt.........|
00000030  22 6f f3 48 8e f3 78 70  2c e2 b2 17 68 e0 f5 00  |"o.H..xp,...h...|
00000040  00 00 10 00 00 00 01 00  00 00 33 00 00 00 0b 73  |..........3....s|
00000050  73 68 2d 65 64 32 35 35  31 39 00 00 00 20 4e ea  |sh-ed25519... N.|
00000060  ab fa 4c 93 d5 6b 46 ea  67 f6 3e 3f b4 d1 dc 94  |..L..kF.g.>?....|
00000070  99 ca 80 90 04 d6 97 d4  79 c4 4d 4f 57 c0 00 00  |........y.MOW...|
00000080  00 a0 47 31 ea 69 c9 98  59 58 5f 1d 5e 63 3b 70  |..G1.i..YX_.^c;p|
00000090  e9 26 79 ca e2 fa 4f 52  d6 7e 3f 9b a6 0e f3 35  |.&y...OR.~?....5|
000000a0  00 68 a8 95 b5 8f e4 35  ec f5 0c b5 78 fd c0 30  |.h.....5....x..0|
000000b0  ce 43 1f ba 38 4e c9 4f  f4 6b 2b 75 65 b6 9e 28  |.C..8N.O.k+ue..(|
000000c0  5e 61 48 e0 58 2c 61 b1  3d 45 b3 57 48 2d b1 bc  |^aH.X,a.=E.WH-..|
000000d0  b6 97 7e 99 66 c7 e7 80  3a 2a fd 51 36 46 4c 86  |..~.f...:*.Q6FL.|
000000e0  ce 1b 2d 5c 56 1a 3c 2f  30 cb 2e 8e 16 88 66 1b  |..-\V.</0.....f.|
000000f0  a7 1b 48 be 42 44 ef 67  f4 61 d4 e6 f1 1a cd 56  |..H.BD.g.a.....V|
00000100  42 e5 03 2e da c9 5b 85  72 8a b5 24 1a e3 20 06  |B.....[.r..$.. .|
00000110  bb 52 da 88 af d4 86 f1  be fa 00 b2 59 6f d8 cf  |.R..........Yo..|
00000120  73 70                                             |sp|

OpenSSH private keys begin with a magic string, “openssh-key-v1\0”, which we can see here. Following this are a number of binary encoded fields which are represented in a manner similar to the SSH wire protocol, most often as strings prefixed by their length, encoded as a 32-bit big-endian integer. In order, the fields present here are:

Cipher name (aes256-ctr)
KDF name (bcrypt)
KDF data
Public key data
Private key data (plus padding)

We parse this information like so:

export type sshprivkey = struct {
	cipher: str,
	kdfname: str,
	kdf: []u8,
	pubkey: []u8,
	privkey: []u8,
};

export fn decodesshprivate(in: io::handle) (sshprivkey | error) = {
	const pem = pem::newdecoder(in);
	const dec = match (pem::next(&pem)?) {
	case io::EOF =>
		return invalid;
	case let dec: (str, pem::pemdecoder) =>
		if (dec.0 != "OPENSSH PRIVATE KEY") {
			return invalid;
		};
		yield dec.1;
	};

	let magicbuf: [15]u8 = [0...];
	match (io::readall(&dec, magicbuf)?) {
	case size => void;
	case io::EOF =>
		return invalid;
	};
	if (!bytes::equal(magicbuf, strings::toutf8(magic))) {
		return invalid;
	};

	let key = sshprivkey { ... };
	key.cipher = readstr(&dec)?;
	key.kdfname = readstr(&dec)?;
	key.kdf = readslice(&dec)?;

	let buf: [4]u8 = [0...];
	match (io::readall(&dec, buf)?) {
	case size => void;
	case io::EOF =>
		return invalid;
	};
	const nkey = endian::begetu32(buf);
	if (nkey != 1) {
		// OpenSSH currently hard-codes the number of keys to 1
		return invalid;
	};

	key.pubkey = readslice(&dec)?;
	key.privkey = readslice(&dec)?;

	// Add padding bytes
	append(key.privkey, io::drain(&dec)?...);
	return key;
};

However, to get at the actual private key — so that we can do cryptographic operations with it — we first have to decrypt this inner data. Those three fields — cipher name, KDF name, and KDF data — are our hint. In essence, this data is encrypted by OpenSSH by using a variant of bcrypt as a key derivation function, which turns your password (plus a salt) into a symmetric encryption key. Then it uses AES 256 in CTR mode with this symmetric key to encrypt the private key data. With the benefit of hindsight, I might question these primitives, but that’s what they use so we’ll have to work with it.

Prior to starting this work, Hare already had support for AES and CTR, though they gained some upgrades during the course of this work, since using an interface for real-world code is the best way to evaluate its design. This leaves us to implement bcrypt.

bcrypt is a password hashing algorithm invented by OpenBSD based on the Blowfish cipher, and it is pretty badly designed. However, Blowfish was fairly straightforward to implement. I’ll spare you the details, but here’s the documentation and implementation for your consideration. I also implemented the standard bcrypt hash at crypto::bcrypt, whose implementation is here (for now). This isn’t especially relevant for us, however, since OpenSSH uses a modified form of bcrypt as a key derivation function.

The implementation the bcrypt KDF in Hare is fairly straightforward. To write it, I referenced OpenSSH portable’s vendored OpenBSD implementation at openbsd-compat/bcrypt_pbkdf.c, as well as the Go implementation in golang.org/x/crypto. Then, with these primitives done, we can implement the actual key decryption.

First, not all keys are encrypted with a passphrase, so a simple function tells us if this step is required:

// Returns true if this private key is encrypted with a passphrase.
export fn isencrypted(key: *sshprivkey) bool = {
	return key.kdfname != "none";
};

The “decrypt” function is used to perform the actual decryption. It begins by finding the symmetric key, like so:

export fn decrypt(key: *sshprivkey, pass: []u8) (void | error) = {
	assert(isencrypted(key));

	const cipher = getcipher(key.cipher)?;
	let ckey: []u8 = alloc([0...], cipher.keylen + cipher.ivlen);
	defer {
		bytes::zero(ckey);
		free(ckey);
	};

	let kdfbuf = bufio::fixed(key.kdf, io::mode::READ);
	switch (key.kdfname) {
	case "bcrypt" =>
		const salt = readslice(&kdfbuf)?;
		defer free(salt);
		const rounds = readu32(&kdfbuf)?;
		bcrypt_pbkdf(ckey, pass, salt, rounds);
	case =>
		return badcipher;
	};

The “KDF data” field I mentioned earlier uses a format private to each KDF mode, though at the present time the only supported KDF is this bcrypt one. In this case, it serves as the salt. The “getcipher” function returns some data from a static table of supported ciphers, which provides us with the required size of the cipher’s key and IV parameters. We allocate sufficient space to store these, create a bufio reader from the KDF field, read out the salt and hashing rounds, and hand all of this over to the bcrypt function to produce our symmetric key (and I/V) in the “ckey” variable.

We may then use these parameters to decrypt the private key area.

	let secretbuf = bufio::fixed(key.privkey, io::mode::READ);
	const cipher = cipher.init(&secretbuf,
		ckey[..cipher.keylen], ckey[cipher.keylen..]);
	defer cipher_free(cipher);

	let buf: []u8 = alloc([0...], len(key.privkey));
	defer free(buf);
	io::readall(cipher, buf)!;

	const a = endian::begetu32(buf[..4]);
	const b = endian::begetu32(buf[4..8]);
	if (a != b) {
		return badpass;
	};

	key.privkey[..] = buf[..];

	free(key.kdf);
	free(key.kdfname);
	free(key.cipher);
	key.kdfname = strings::dup("none");
	key.cipher = strings::dup("none");
	key.kdf = [];
};

The “cipher.init” function is an abstraction that allows us to support more ciphers in the future. For this particular cipher mode, it’s implemented fairly simply:

type aes256ctr = struct {
	st: cipher::ctr_stream,
	block: aes::ct64_block,
	buf: [aes::CTR_BUFSIZE]u8,
};

fn aes256ctr_init(handle: io::handle, key: []u8, iv: []u8) *io::stream = {
	let state = alloc(aes256ctr {
		block = aes::ct64(),
		...
	});
	aes::ct64_init(&state.block, key);
	state.st = cipher::ctr(handle, &state.block, iv, state.buf);
	return state;
};

Within this private key data section, once decrypted, are several fields. First is a random 32-bit integer which is written twice — comparing that these are equal to one another allows us to verify the user’s password. Once verified, we overwrite the private data field in the key structure with the decrypted data, and update the cipher and KDF information to indicate that the key is unencrypted. We could decrypt it directly into the existing private key buffer, without allocating a second buffer, but this would overwrite the encrypted data with garbage if the password was wrong — you’d have to decode the key all over again if the user wants to try again.

So, what does this private key blob look like once decrypted? The hare-ssh repository includes a little program at cmd/sshkey which dumps all of the information stored in an SSH key, and it provides us with this peek at the private data:

00000000  fb 15 e6 16 fb 15 e6 16  00 00 00 0b 73 73 68 2d  |............ssh-|
00000010  65 64 32 35 35 31 39 00  00 00 20 4e ea ab fa 4c  |ed25519... N...L|
00000020  93 d5 6b 46 ea 67 f6 3e  3f b4 d1 dc 94 99 ca 80  |..kF.g.>?.......|
00000030  90 04 d6 97 d4 79 c4 4d  4f 57 c0 00 00 00 40 17  |.....y.MOW....@.|
00000040  bf 87 74 0b 2a 74 d5 29  d0 14 10 3f 04 5d 88 c6  |..t.*t.)...?.]..|
00000050  32 fa 21 9c e9 97 b0 5a  e7 7e 5c 02 72 35 72 4e  |2.!....Z.~\.r5rN|
00000060  ea ab fa 4c 93 d5 6b 46  ea 67 f6 3e 3f b4 d1 dc  |...L..kF.g.>?...|
00000070  94 99 ca 80 90 04 d6 97  d4 79 c4 4d 4f 57 c0 00  |.........y.MOW..|
00000080  00 00 0e 73 69 72 63 6d  70 77 6e 40 74 61 69 67  |...sircmpwn@taig|
00000090  61 01 02 03 04 05 06 07  08 09 0a 0b 0c 0d 0e 0f  |a...............|

We can see upfront these two 32-bit verification numbers I mentioned, and following this are several fields in a similar format to earlier — length-prefixed strings. The fields are:

Key type (“ssh-ed25519” in this case)
Public key (in a format specific to each key type)
Private key (in a format specific to each key type)
Comment
Padding up to the cipher’s block size (16)

This is a little bit weird in my opinion — the public key field is redundant with the unencrypted data in this file, and the comment field is probably not so secret as to demand encryption. I think these are just consequences of the file format being private to OpenSSH’s implementation; not much thought has gone into it and implementation details (like the ability to call the same “dump private key” function here as OpenSSH uses elsewhere) have probably leaked through.

We can decode this data with the following Hare code:

export fn decodeprivate(src: *sshprivkey) (key | error) = {
	assert(!isencrypted(src));
	const buf = bufio::fixed(src.privkey, io::mode::READ);

	let verify: [8]u8 = [0...];
	io::read(&buf, verify)!;
	const a = endian::begetu32(verify[..4]);
	const b = endian::begetu32(verify[4..8]);
	if (a != b) {
		return badpass;
	};

	const keytype = readstr(&buf)?;
	defer free(keytype);

	switch (keytype) {
	case "ssh-ed25519" =>
		let key = ed25519key { ... };
		decode_ed25519_sk(&key, &buf)?;
		return key;
	case =>
		// TODO: Support additional key types
		return badcipher;
	};
};

// An ed25519 key pair.
export type ed25519key = struct {
	pkey: ed25519::publickey,
	skey: ed25519::privatekey,
	comment: str,
};

fn decode_ed25519_pk(key: *ed25519key, buf: io::handle) (void | error) = {
	const l = readu32(buf)?;
	if (l != ed25519::PUBLICKEYSZ) {
		return invalid;
	};
	io::readall(buf, key.pkey)?;
};

fn decode_ed25519_sk(key: *ed25519key, buf: io::handle) (void | error) = {
	decode_ed25519_pk(key, buf)?;

	const l = readu32(buf)?;
	if (l != ed25519::PRIVATEKEYSZ) {
		return invalid;
	};
	io::readall(buf, key.skey)?;

	// Sanity check
	const pkey = ed25519::skey_getpublic(&key.skey);
	if (!bytes::equal(pkey, key.pkey)) {
		return invalid;
	};

	key.comment = readstr(buf)?;
};

Fairly straightforward! Finally, we have extracted the actual private key from the file. For this SSH key, in base64, the cryptographic keys are:

Public key:  Tuqr+kyT1WtG6mf2Pj+00dyUmcqAkATWl9R5xE1PV8A=
Private key: F7+HdAsqdNUp0BQQPwRdiMYy+iGc6ZewWud+XAJyNXJO6qv6TJPVa0bqZ/Y+P7TR3JSZyoCQBNaX1HnETU9XwA==

Signing and verification with ed25519

Using these private keys, implementing signatures and signature verification are pretty straightforward. We can stop reading the OpenSSH code at this point — RFC 8709 standardizes this format for ed25519 signatures.

use crypto::ed25519;
use io;

// Signs a message using the provided key, writing the message signature in the
// SSH format to the provided sink.
export fn sign(
	sink: io::handle,
	key: *key,
	msg: []u8,
) (void | io::error) = {
	const signature = ed25519::sign(&key.skey, msg);
	writestr(sink, "ssh-ed25519")?;
	writeslice(sink, signature)?;
};

// Reads an SSH wire signature from the provided I/O handle and verifies that it
// is a valid signature for the given message and key. If valid, void is
// returned; otherwise [[badsig]] is returned.
export fn verify(
	source: io::handle,
	key: *key,
	msg: []u8,
) (void | error) = {
	const sigtype = readstr(source)?;
	defer free(sigtype);
	if (sigtype != keytype(key)) {
		return badsig;
	};
	const sig = readslice(source)?;
	defer free(sig);

	assert(sigtype == "ssh-ed25519"); // TODO: other key types
	if (len(sig) != ed25519::SIGNATURESZ) {
		return badsig;
	};

	const sig = sig: *[*]u8: *[ed25519::SIGNATURESZ]u8;
	if (!ed25519::verify(&key.pkey, msg, sig)) {
		return badsig;
	};
};

This implementation writes and reads signatures in the SSH wire format, which is generally how they will be most useful in this context. This code will be expanded in the future with additional keys, such as RSA, once the necessary primitives are implemented for Hare’s standard library.

The SSH agent protocol

The agent protocol is also standardized (albeit in draft form), so we refer to draft-miller-ssh-agent-11 from this point onwards. It’s fairly straightforward. The agent communicates over an unspecified protocol (Unix sockets in practice) by sending messages in the SSH wire format, which, again, mainly comes in the form of strings prefixed by their 32-bit length in network order.

The first step for implementing net::ssh::agent starts with adding types for all of the data structures and enums for all of the constants, which you can find in types.ha. Each message begins with its length, then a message type (one byte) and a message payload; the structure of the latter varies with the message type.

I started to approach this by writing some functions which, given a byte buffer that contains an SSH agent message, either parses it or asks for more data.

export fn parse(msg: []u8) (message | size | invalid) = {
	if (len(msg) < 5) {
		return 5 - len(msg);
	};
	const ln = endian::begetu32(msg[..4]);
	if (len(msg) < 4 + ln) {
		return 4 + ln - len(msg);
	};

	const mtype = msg[4];
	const buf = bufio::fixed(msg[5..], io::mode::READ);
	switch (mtype) {
	case messagetype::REQUEST_IDENTITIES =>
		return request_identities;
	case messagetype::SIGN_REQUEST =>
		return parse_sign_request(&buf)?;
	case messagetype::ADD_IDENTITY =>
		return parse_add_identity(&buf)?;
	// ...trimmed for brevity, and also because it's full of TODOs...
	case =>
		return invalid.
	};
};

Each individual message payload includes its own parser, except for some messages (such as REQUEST_IDENTITIES), which have no payload. Here’s what the parser for SIGN_REQUEST looks like:

fn parse_sign_request(src: io::handle) (sign_request | invalid) = {
	return sign_request {
		key = readslice(src)?,
		data = readslice(src)?,
		flags = readu32(src)?: sigflag,
	};
};

Pretty straightforward! A more complex one is ADD_IDENTITY:

fn parse_add_identity(src: io::handle) (add_identity | invalid) = {
	const keytype = readstr(src)?;
	// TODO: Support more key types
	const key: ssh::key = switch (keytype) {
	case "ssh-ed25519" =>
		let key = ssh::ed25519key { ... };
		const npub = readu32(src)?;
		if (npub != len(key.pkey)) {
			return invalid;
		};
		io::readall(src, key.pkey)!;
		const npriv = readu32(src)?;
		if (npriv != len(key.skey)) {
			return invalid;
		};
		io::readall(src, key.skey)!;
		yield key;
	case =>
		return invalid;
	};
	return add_identity {
		keytype = keytype,
		key = key,
		comment = readstr(src)?,
	};
};

One thing I’m not thrilled with in this code is memory management. In Hare, libraries like this one are not supposed to allocate memory if they can get away with it, and if they must, they should do it as conservatively as possible. This implementation does a lot of its own allocations, which is unfortunate. I might refactor it in the future to avoid this. A more subtle issue here is the memory leaks on errors — each of the readslice/readstr functions allocates data for its return value, but if they return an error, the ? operator will return immediately without freeing them. This is a known problem with Hare’s language design, and while we have some ideas for addressing it, we have not completed any of them yet. This is one of a small number of goals for Hare which will likely require language changes prior to 1.0.

We have a little bit more code in net::ssh::agent, which you can check out if you like, but this covers most of it — time to move onto the daemon implementation.

Completing our SSH agent

The ssh-agent command in the hare-ssh tree is a simple (and non-production) implementation of an SSH agent based on this work. Let’s go over its code to see how this all comes together to make it work.

First, we set up a Unix socket, and somewhere to store our application state.

let running: bool = true;

type identity = struct {
	comment: str,
	privkey: ssh::key,
	pubkey: []u8,
};

type state = struct {
	identities: []identity,
};

export fn main() void = {
	let state = state { ... };
	const sockpath = "./socket";

	const listener = unix::listen(sockpath)!;
	defer {
		net::shutdown(listener);
		os::remove(sockpath)!;
	};
	os::chmod(sockpath, 0o700)!;
	log::printfln("Listening at {}", sockpath);

We also need a main loop, but we need to clean up that Unix socket when we terminate, so we’ll also set up some signal handlers.

	signal::handle(signal::SIGINT, &handle_signal);
	signal::handle(signal::SIGTERM, &handle_signal);

	for (running) {
		// ...stay tuned...
	};

	for (let i = 0z; i < len(state.identities); i += 1) {
		const ident = state.identities[i];
		ssh::key_finish(&ident.privkey);
		free(ident.pubkey);
		free(ident.comment);
	};

	log::printfln("Terminated.");
};

// ...elsewhere...
fn handle_signal(sig: int, info: *signal::siginfo, ucontext: *void) void = {
	running = false;
};

The actual clean-up is handled by our “defer” statement at the start of “main”. The semantics of signal handling on Unix are complex (and bad), and beyond the scope of this post, so hopefully you already grok them. Our stdlib provides docs, if you care to learn more, but also includes this warning:

Signal handling is stupidly complicated and easy to get wrong. The standard library makes little effort to help you deal with this. Consult your local man pages, particularly signal-safety(7) on Linux, and perhaps a local priest as well. We advise you to get out of the signal handler as soon as possible, for example via the “self-pipe trick”.

We also provide signalfds on platforms that support them (such as Linux), which is less fraught with issues. Good luck.

Next: the main loop. This code accepts new clients, prepares an agent for them, and hands them off to a second function:

		const client = match (net::accept(listener)) {
		case errors::interrupted =>
			continue;
		case let err: net::error =>
			log::fatalf("Error: accept: {}", net::strerror(err));
		case let fd: io::file =>
			yield fd;
		};
		const agent = agent::new(client);
		defer agent::agent_finish(&agent);
		run(&state, &agent);

This is a really simple event loop for a network daemon, and comes with one major limitation: no support for serving multiple clients connecting at once. If you’re curious what a more robust network daemon looks like in Hare, consult the Himitsu code.

The “run” function simply reads SSH agent commands and processes them, until the client disconnects.

fn run(state: *state, agent: *agent::agent) void = {
	for (true) {
		const msg = match (agent::readmsg(agent)) {
		case (io::EOF | agent::error) =>
			break;
		case void =>
			continue;
		case let msg: agent::message =>
			yield msg;
		};
		defer agent::message_finish(&msg);

		const res = match (msg) {
		case agent::request_identities =>
			yield handle_req_ident(state, agent);
		case let msg: agent::add_identity =>
			yield handle_add_ident(state, &msg, agent);
		case let msg: agent::sign_request =>
			yield handle_sign_request(state, &msg, agent);
		case agent::extension =>
			const answer: agent::message = agent::extension_failure;
			agent::writemsg(agent, &answer)!;
		case => abort();
		};
		match (res) {
		case void => yield;
		case agent::error => abort();
		};
	};
};

Again, this is non-production code, and, among other things, is missing good error handling. The handlers for each message are fairly straightforward, however. Here’s the handler for REQUEST_IDENTITIES:

fn handle_req_ident(
	state: *state,
	agent: *agent::agent,
) (void | agent::error) = {
	let idents: agent::identities_answer = [];
	defer free(idents);

	for (let i = 0z; i < len(state.identities); i += 1) {
		const ident = &state.identities[i];
		append(idents, agent::identity {
			pubkey = ident.pubkey,
			comment = ident.comment,
		});
	};

	const answer: agent::message = idents;
	agent::writemsg(agent, &answer)!;
};

The first one to do something interesting is ADD_IDENTITY, which allows the user to supply SSH private keys to the agent to work with:

fn handle_add_ident(
	state: *state,
	msg: *agent::add_identity,
	agent: *agent::agent,
) (void | agent::error) = {
	let sink = bufio::dynamic(io::mode::WRITE);
	ssh::encode_pubkey(&sink, &msg.key)!;
	append(state.identities, identity {
		comment = strings::dup(msg.comment),
		privkey = msg.key,
		pubkey = bufio::buffer(&sink),
	});
	const answer: agent::message = agent::agent_success;
	agent::writemsg(agent, &answer)?;
	log::printfln("Added key {}", msg.comment);
};

With these two messages, we can start to get the agent to do something relatively interesting: accepting and listing keys.

$ hare run cmd/ssh-agent/
[2022-05-09 17:39:12] Listening at ./socket
^Z[1]+  Stopped                    hare run cmd/ssh-agent/
$ bg
[1] hare run cmd/ssh-agent/
$ export SSH_AUTH_SOCK=./socket
$ ssh-add -l
The agent has no identities.
$ ssh-add ~/.ssh/id_ed25519
Enter passphrase for /home/sircmpwn/.ssh/id_ed25519: 
Identity added: /home/sircmpwn/.ssh/id_ed25519 (sircmpwn@homura)
2022-05-09 17:39:31] Added key sircmpwn@homura
$ ssh-add -l
256 SHA256:kPr5ZKTNE54TRHGSaanhcQYiJ56zSgcpKeLZw4/myEI sircmpwn@homura (ED25519)

With the last message handler, we can upgrade from something “interesting” to something “useful”:

fn handle_sign_request(
	state: *state,
	msg: *agent::sign_request,
	agent: *agent::agent,
) (void | agent::error) = {
	let key: nullable *identity = null;
	for (let i = 0z; i < len(state.identities); i += 1) {
		let ident = &state.identities[i];
		if (bytes::equal(ident.pubkey, msg.key)) {
			key = ident;
			break;
		};
	};
	const key = match (key) {
	case let key: *identity =>
		yield key;
	case null =>
		const answer: agent::message = agent::agent_failure;
		agent::writemsg(agent, &answer)?;
		return;
	};

	let buf = bufio::dynamic(io::mode::WRITE);
	defer io::close(&buf)!;
	ssh::sign(&buf, &key.privkey, msg.data)!;

	const answer: agent::message = agent::sign_response {
		signature = bufio::buffer(&buf),
	};
	agent::writemsg(agent, &answer)?;
	log::printfln("Signed challenge with key {}", key.comment);
};

For performance reasons, it may be better to use a hash map in a production Hare program (and, as many commenters will be sure to point out, Hare does not provide a built-in hash map or generics). We select the desired key with a linear search, sign the provided payload, and return the signature to the client. Finally, the big pay-off:

$ ssh git@git.sr.ht
[2022-05-09 17:41:42] Signed challenge with key sircmpwn@homura
PTY allocation request failed on channel 0
Hi sircmpwn! You've successfully authenticated, but I do not provide an interactive shell. Bye!
Connection to git.sr.ht closed.

Incorporating it into Himitsu

Himitsu was the motivation for all of this work, and I have yet to properly introduce it to the public. I will go into detail later, but in essence, Himitsu is a key-value store that stores some keys in plaintext and some keys encrypted, and acts as a more general form of a password manager. One of the things it can do (at least as of this week) is store your SSH private keys and act as an SSH agent, via a helper called himitsu-ssh. The user can import their private key from OpenSSH’s private key format via the “hissh-import” tool, and then the “hissh-agent” daemon provides agent functionality via the Himitsu key store.

The user can import their SSH key like so:

$ hissh-import < ~/.ssh/id_ed25519
Enter SSH key passphrase: 
key proto=ssh type=ssh-ed25519 pkey=pF7SljE25sVLdWvInO4gfqpJbbjxI6j+tIUcNWzVTHU= skey! comment=sircmpwn@homura

# Query the key store for keys matching proto=ssh:
$ hiq proto=ssh
proto=ssh type=ssh-ed25519 pkey=pF7SljE25sVLdWvInO4gfqpJbbjxI6j+tIUcNWzVTHU= skey! comment=sircmpwn@homura

Then, when running the agent:

(Yes, I know that the GUI has issues. I slapped it together in C in an afternoon and it needs a lot of work. Help wanted!)

Ta-da!

What’s next?

I accomplished my main goal, which was getting my SSH setup working with Himitsu. The next steps for expanding hare-ssh are:

Expanding the supported key types and ciphers (RSA, DSA, etc), which first requires implementing the primitives in the standard library
Implement the SSH connection protocol, which requires primitives like ECDH in the standard library. Some required primitives, like ChaCha, are already supported.
Improve the design of the networking code. hare-ssh is one of a very small number of network-facing Hare libraries, and it’s treading new design ground here.

SSH is a relatively small target for a cryptography implementation to aim for. I’m looking forward to using it as a testbed for our cryptographic suite. If you’re interested in helping with any of these, please get in touch! If you’re curious about Hare in general, check out the language introduction to get started. Good luck!

Announcing the Hare programming language

Mon, 25 Apr 2022 00:00:00 +0000

The “secret programming language” I have been teasing for several months now is finally here! It is called Hare, and you can read about it on the Hare blog:

https://harelang.org/blog/2022-04-25-announcing-hare/

Check it out!

Status update, April 2022

Fri, 15 Apr 2022 00:00:00 +0000

This month marked my first time filing taxes in two countries, and I can assure you it is the worst. I am now a single-issue voter in the US: stop taxing expats! You can get some insight into the financials of SourceHut in the recently-published financial report. But let’s get right into the fun stuff: free software development news.

There was some slowdown from me this month thanks to all of the business and financial crap I had to put up with, but I was able to get some cool stuff done and many other contributors have been keeping things moving. I’ll start by introducing a new/old project: Himitsu.

Essentially, Himitsu is a secret storage system whose intended use-case is to provide features like password storage and SSH agent functionality. It draws much of its inspiration from Plan 9’s Factotum. You may have stumbled upon an early prototype on git.sr.ht which introduces the basic idea and included the start of an implementation in C. Ultimately I shelved this project for want of a better programming language to implement it with, and then I made a better programming language to implement it with. Over the past two weeks, I have implemented something similar to where the C codebase was left, in fewer than half the lines of code and much less than half the time. Here’s a little peek at what works now:

[12:18:31] taiga ~/s/himitsu $ ./himitsud 
Please enter your passphrase to unlock the keyring: 
[2022-04-15 12:18:56] himitsud running
^Z[1]+  Stopped                    ./himitsud
[12:18:57] taiga ~/s/himitsu $ bg
[1] ./himitsud
[12:18:58] taiga ~/s/himitsu $ nc -U ~/.local/state/himitsu/socket 
add proto=imap host=example.org user=drew@ddevault.org password!="Hello world!"
^C
[12:19:12] taiga ~/s/himitsu $ ls ~/.local/share/himitsu/
2849c1d5-61b3-4803-98cf-fc57fe5f69a6  index  key
[12:19:14] taiga ~/s/himitsu $ cat ~/.local/share/himitsu/index
YNfVlkORDX1GmXIfL8vOiiTgBJKh47biFsUaKrqzfMP2xfD4B9/lqSl2Y9OtIpVcYzrNjBBZOxcO81vNQdgnvxQ+xaCKaVpQS4Dh6DyaY0/lpq6rfowTY5GwcI155KkmTI4z1ABOVkL4z4XDsQ2DEiqClcQE5/+CxsQ/U/u9DthLJRjrjw==
[12:19:19] taiga ~/s/himitsu $ fg
./himitsud
^C[2022-04-15 12:19:22] himitsud terminated
[12:19:22] taiga ~/s/himitsu $ ./himitsud 
Please enter your passphrase to unlock the keyring: 
Loaded key proto=imap host=example.org user=drew@ddevault.org password!=2849c1d5-61b3-4803-98cf-fc57fe5f69a6
[2022-04-15 12:19:29] himitsud running
^C[2022-04-15 12:19:31] himitsud terminated
[12:19:33] taiga ~/s/himitsu $ find . -type f | xargs wc -l | tail -n1
  895 total

This project is progressing quite fast and I hope to have it working for some basic use-cases soon. I’ll do a dedicated blog post explaining how it works and why it’s important later on, though it will remain mostly under wraps until the language is released.

Speaking of the language, there were a number of exciting developments this month. Two major standard library initiatives were merged: regex and datetime. The regex implementation is simple, small, and fast, targeting POSIX ERE as a reasonably sane conservative baseline regex dialect. The datetime implementation is quite interesting as well, and it provides a pretty comprehensive API which should address almost all use-cases for timekeeping in our language with a robust and easy-to-use API. As a bonus, and a little flex at how robust our design is, we’ve included support for Martian time. I’m very pleased with how both of these turned out.

use datetime;
use fmt;
use os;
use time::chrono;

export fn main() void = {
	const now = datetime::in(chrono::MTC, datetime::now());
	fmt::printf("Current Martian coordinated time: ")!;
	datetime::format(os::stdout, datetime::STAMP, &now)!;
	fmt::println()!;
};

Other recent improvements include support for signal handling, glob, aes-ni, and net::uri. Work has slowed down on cryptography — please get in touch if you’d like to help. Many readers will be happy to know that there are rumblings about possibly going public soon; after a couple more milestones we’ll be having a meeting to nail down the most urgent priorities before going public and then we’ll get this language into your hands to play with.

I also started a bittorrent daemon in this language, but it’s temporarily blocked until we sort out HTTP/TLS. So, moving right along: SourceHut news? Naturally I will leave most of it for the “what’s cooking” post, but I’ll offer you a little tease of what we’ve been working on: GraphQL. We landed support this month for GraphQL-native webhooks in todo.sr.ht, as well as some new improvements to the pages.sr.ht GQL API. hg.sr.ht is also now starting to see some polish put on its GraphQL support, and some research is underway on GraphQL Federation. Very soon we will be able to put a bow on this work.

That’s all for today! Thanks again for reading and for your ongoing support. I appreciate you!

Announcing git snail-mail

Fri, 01 Apr 2022 00:00:00 +0000

You’ve heard of git-over-email thanks to git send-email — now you can enjoy git snail-mail: a new tool making it easier than ever to print out git commits on paper and mail them to your maintainers.
Running git snail-mail HEAD~2.. prepares the last two commits for post and sends them directly to the system’s default printer. Configuration options are available for changing printer settings, paper size, and options for faxing or printing envelopes automatically addressed to the maintainers based on address info stored in your git config. Be sure to help the maintainers review your work by including a return envelope and a stamp!
And for maintainers, code review has never been easier — just get out your red marker and write your feedback directly on the patch! When you’re ready to import the patch into your repository, just place it on your scanner and run git scan-mail.

A picture of a patch printed out on paper

At least, this is what I’d like to say, but I ended up cancelling the project before it was ready for April Fool’s. After my friend kline (a staffer at Libera Chat) came up with this idea, I actually did write a lot of the code! Git is mostly written in Perl, but I could not really rouse the enthusiasm for implementing this idea in Perl. I did the prototype in $secretlang instead, and got it mostly working, but decided not to try to do some sneaky half-private joke release while trying to maintain the secrecy of the language.

Essentially how it works is this: I have a TeX template for patches:

\documentclass{article}
\usepackage[
	a4paper,
	top=1cm,
	bottom=1cm,
	left=1cm,
	right=1cm,
]{geometry}
\usepackage{graphicx}
\usepackage{fancyvrb}
\pagenumbering{gobble}

\begin{document}

\section*{implement os::exec::peek\{,any\}}

From: Bor Grošelj Simić \textless{}bor.groseljsimic@telemach.net\textgreater{} \\
Date: Fri, 25 Feb 2022 01:46:13 +0100

\VerbatimInput{input.patch}

\newpage
Page 1 of 2 \\
\includegraphics[]{./output-1.png}

\newpage
Page 2 of 2 \\
\includegraphics[]{./output-2.png}

\end{document}

This is generated by my git snail-mail code and then run through pdflatex to produce a file like this. It pipes it into lp(1) to send it to your printer and ta-da!

I chose not to make the commit selection work like git send-email, because I think that’s one of the most confusing parts of git send-email. Instead I just use a standard revision selection, so to print a single commit, you just name it, and to print a range of commits you use “..”. Here’s a peek at how that works:

fn get_commits(
	data: *texdata,
	workdir: str,
	range: str,
) (void | exec::error | exec::exit_status | io::error | fs::error) = {
	const fmt = `--format=%H%x00%s%x00%aN%x00%aE%x00%aD`;

	const pipe = exec::pipe();
	const cmd = exec::cmd("git", "show", "-s", fmt, range)?;
	exec::addfile(&cmd, os::stdout_file, pipe.1);
	const proc = exec::start(&cmd)?;
	io::close(pipe.1);
	const pipe = pipe.0;
	defer io::close(pipe);

	static let buffer: [os::BUFSIZ]u8 = [0...];
	const pipe = &bufio::buffered(pipe, buffer[..], []);

	let path = path::init();

	for (true) {
		const line = match (bufio::scanline(pipe)?) {
		case let line: []u8 =>
			yield strings::fromutf8(line);
		case io::EOF =>
			break;
		};

		// XXX: This assumes git always does the thing
		const tok = strings::tokenize(line, "\0");
		let commit = commitdata {
			sha = strings::next_token(&tok) as str,
			subject = strings::next_token(&tok) as str,
			author = strings::next_token(&tok) as str,
			email = strings::next_token(&tok) as str,
			date = strings::next_token(&tok) as str,
			...
		};

		path::set(&path, workdir)!;
		path::add(&path, commit.sha)!;
		commit.diff = strings::dup(path::string(&path));
		append(data.commits, commit);

		const file = os::create(commit.diff, 0o644)?;
		defer io::close(file);
		const parent = strings::concat(commit.sha, "^");
		defer free(parent);

		const cmd = exec::cmd("git", "diff", parent, commit.sha)?;
		exec::addfile(&cmd, os::stdout_file, file);
		const proc = exec::start(&cmd)?;
		const status = exec::wait(&proc)?;
		exec::check(&status)?;
	};

	const status = exec::wait(&proc)?;
	exec::check(&status)?;
};

The --format argument provided to git at the start here allows me to change the format of git-show to use NUL delimited fields for easily picking out the data I want. Point of note: this is minimum-effort coding for a joke, so there’s a lot of missing error handling and other lazy design choices here.

Anyway, I would have liked to have rewritten this in Perl and pitched it to the git mailing list for inclusion upstream, but alas, after prototyping in $secretlang I could not bring myself to rewrite it in Perl, and the joke fell flat. Not every idea pans out, but they’re still worth trying, anyway. If you want to see some joke projects I’ve made that actually work, check these out:

shit: a git implementation in POSIX shell
bfbot: a working IRC bot written in brainfuck
classic6: a working Minecraft server written in 6 hours
evilpass: a password strength checker that detects password reuse
tw: a Wayland compositor for your terminal in 80 lines of “code”

Take care!

It is important for free software to use free software infrastructure

Tue, 29 Mar 2022 00:00:00 +0000

Disclaimer: I founded a project and a company that focuses on free software infrastructure. I will elect not to name them in this post, and will only recommend solutions I do not have a vested interest in.

Free and open source software (FOSS) projects need infrastructure. Somewhere to host the code, to facilitate things like code review, end-user support, bug tracking, marketing, and so on. A common example of this is the “forge” platform: infrastructure which pitches itself as a one-stop shop for many of the needs of FOSS projects in one place, such as code hosting and review, bug tracking, discussions, and so on. Many projects will also reach for additional platforms to provide other kinds of infrastructure: chat rooms, forums, social media, and more.

Many of these needs have non-free, proprietary solutions available. GitHub is a popular proprietary code forge, and GitLab, the biggest competitor to GitHub, is partially non-free. Some projects use Discord or Slack for chat rooms, Reddit as a forum, or Twitter and Facebook for marketing, outreach, and support; all of these are non-free. In my opinion, relying on these platforms to provide infrastructure for your FOSS project is a mistake.

When your FOSS project chooses to use a non-free platform, you give it an official vote of confidence on behalf of your project. In other words, you lend some of your project’s credibility and legitimacy to the platforms you choose. These platforms are defined by network effects, and your choice is an investment in that network. I would question this investment in and of itself, and the wisdom of offering these platforms your confidence and legitimacy, but there’s a more concerning consequence of this choice as well: an investment in a non-free platform is also a divestment from the free alternatives.

Again, network effects are the main driver of success in these platforms. Large commercial platforms have a lot of advantages in this respect: large marketing budgets, lots of capital from investors, and the incumbency advantage. The larger the incumbent platform, the more difficult the task of competing with it becomes. Contrast this with free software platforms, which generally don’t have the benefit of large amounts of investment or big marketing budgets. Moreover, businesses are significantly more likely to play dirty to secure their foothold than free software projects are. If your own FOSS projects compete with proprietary commercial options, you should be very familiar with these challenges.

FOSS platforms are at an inherent disadvantage, and your faith in them, or lack thereof, carries a lot of weight. GitHub won’t lose sleep if your project chooses to host its code somewhere else, but choosing Codeberg, for example, means a lot to them. In effect, your choice matters disproportionately to the free platforms: choosing GitHub hurts Codeberg much more than choosing Codeberg hurts GitHub. And why should a project choose to use your offering over the proprietary alternatives if you won’t extend them the same courtesy? FOSS solidarity is important for uplifting the ecosystem as a whole.

However, for some projects, what ultimately matters to them has little to do with the benefit of the ecosystem as a whole, but instead considers only the potential for their project’s individual growth and popularity. Many projects choose to prioritize access to the established audience that large commercial platforms provide, in order to maximize their odds of becoming popular, and enjoying some of the knock-on effects of that popularity, such as more contributions.¹ Such projects would prefer to exacerbate the network effects problem rather than risk some of its social capital on a less popular platform.

To me, this is selfish and unethical outright, though you may have different ethical standards. Unfortunately, arguments against most commercial platforms for any reasonable ethical standard are available in abundance, but they tend to be easily overcome by confirmation bias. Someone who may loudly object to the practices of the US Immigration and Customs Enforcement agency, for example, can quickly find some justification to continue using GitHub despite their collaboration with them. If this example isn’t to your tastes, there are many examples for each of many platforms. For projects that don’t want to move, these are usually swept under the rug.²

But, to be clear, I am not asking you to use inferior platforms for philosophical or altruistic reasons. These are only one of many factors which should contribute to your decision-making, and aptitude is another valid factor to consider. That said, many FOSS platforms are, at least in my opinion, functionally superior to their proprietary competition. Whether their differences are better for your project’s unique needs is something I must leave for you to research on your own, but most projects don’t bother with the research at all. Rest assured: these projects are not ghettos living in the shadow of their larger commercial counterparts, but exciting platforms in their own right which offer many unique advantages.

What’s more, if you need them to do something differently to better suit your project’s needs, you are empowered to improve them. You’re not subservient to the whims of the commercial entity who is responsible for the code, waiting for them to prioritize the issue or even to care about it in the first place. If a problem is important to you, that’s enough for you to get it fixed on a FOSS platform. You might not think you have the time or expertise to do so (though maybe one of your collaborators does), but more importantly, this establishes a mentality of collective ownership and responsibility over all free software as a whole — popularize this philosophy and it could just as easily be you receiving a contribution in a similar manner tomorrow.

In short, choosing non-free platforms is an individualist, short-term investment which prioritizes your project’s apparent access to popularity over the success of the FOSS ecosystem as a whole. On the other hand, choosing FOSS platforms is a collectivist investment in the long-term success of the FOSS ecosystem as a whole, driving its overall growth. Your choice matters. You can help the FOSS ecosystem by choosing FOSS platforms, or you can hurt the FOSS ecosystem by choosing non-free platforms. Please choose carefully.

Here are some recommendations for free software tools that facilitate common needs for free software projects:

Code forges: Codeberg, Gitea*, Gerrit*, GitLab†
Instant messaging: Matrix, Libera Chat³
Publishing: Codeberg pages, Write.as, PeerTube
Social media: Mastodon, Lemmy
Mailing lists: FreeLists, public-inbox*, Mailman*

* Self-hosted only
† Partially non-free, recommended only if no other solutions are suitable

P.S. If your project is already established on non-free platforms, the easiest time to revisit this choice is right now. It will only ever get more difficult to move as your project grows and gets further established on proprietary platforms. Please consider moving sooner rather than later.

The Netherlands so far

Thu, 24 Mar 2022 00:00:00 +0000

I moved to Amsterdam in July 2021, and now that I’ve had some time to settle in I thought I’d share my thoughts on how it’s been so far. In short: I love it here!

I did end up finding housing through the hacker community thanks to my earlier post, which was a great blessing. I am renting an apartment from a member of the Techinc hacker space, which I have joined as a member myself. One of my biggest fears was establishing a new social network here in the Netherlands, but making friends here has been easy. Through this hacker space and through other connections besides, I have quickly met many wonderful, friendly, and welcoming people, and I have never felt like a stranger in a strange land. For this I am very grateful.

There are many other things to love about this place. One of my favorite things about Amsterdam is getting around by bike. In Philadelphia, travelling by bicycle is signing up for a death wish. In the Netherlands, 27% of all trips utilize a bike, and in Amsterdam it’s as much as 38%. Cyclists enjoy dedicated cycling-first infrastructure, such as bike lanes separated entirely from the roads and dedicated bike-only longer-distance artery roads. The city is designed to reduce points of conflict between bikes and cars, and even when they have to share the road they’re almost always designed to slow cars down and give bikes priority. The whole country is very flat, too, though Dutch people will be quick to tell you about The Hill in their neighborhood, which is always no more than 2 meters tall.

Getting around without a bike is super pleasant as well. I have my choice of bus, tram, metro, long-distance train, or even free ferries across the river, all paid for with the same auto-recharging NFC card for a low price. Every line runs frequent stops, so during the day you’re generally not waiting more than 5 minutes to be picked up and at night you’re probably not going to be waiting more than 15 minutes at popular stops. When it gets really late, though, you might wait as much as 30 minutes. The inter-city trains are amazing — I can show up at any major station without a plan and there’s probably a train heading to where I want to go in less than 10 minutes. Compared to Amtrak, it’s simply mind boggling.

Little things no one here even thinks about have left an impression on me, too. I see street cleaners out all of the time, in a little squad where workers use leaf blowers and brooms to sweep trash and dirt from the sidewalks and squares into the streets where sweepers come through to pick it up. The trash and recycling bins are regularly collected, and when one of them in my neighborhood broke, it was replaced within days. There are some areas where trash does tend to accumulate, though, such as near benches in parks.

Isolated accumulations of trash aside, the parks are great. There’s a lot more of them throughout the city than you’d get in a typical American city. I live close to two large parks, Rembrandtpark and Vondelpark, plus the smaller Erasmuspark, all of which are less than 5 minutes of cycling away. I like to cycle there on cool summer days to read by the lakes or other water features, or on one of the lawns. These parks also typically have a lot of large cycling-only roads which act as little cycling highways throughout the city, which means many of my cycling routes take me through nature even for intra-city travel. Several of the parks also have public gym equipment available, with which you can get a pretty good outdoor work-out for free.

The layout of the neighborhoods is quite nice as well. I have not just one but four grocery stores within walking distance of my house, and I visit one multiple times per week to pick up food, just a 3 or 4 minute walk away from my place. Thanks to the ease of accessing good (and cheap) produce and other ingredients, my diet has improved quite a bit — something I didn’t expect when I moved here. I can’t get everything I want, though: finding genuinely spicy chili peppers is a challenge.

The infamous Dutch bureaucracy is not as bad as people made it out to be. Going through the immigration process was pretty stressful — as any process which could end with being kicked out of the country might be — but it was actually fairly straightforward for the kind of visa I wanted to get. Public servants here are more helpful and flexible than their reputation suggests.

Something which is proving to be a bit of a challenge, however, is learning Dutch. This surprised me given my existing background in languages; I thought it would be pretty easy to pick up. I was able to quickly learn the basics, and I can conduct many everyday affairs in Dutch, but I found it difficult to progress beyond this point with self-study alone. I enrolled in a formal class, which will hopefully help bridge that gap.

I could go on — experiences outside of Amsterdam and throughout the rest of Europe, the vibes of the FOSS community and other communities I’ve met, serendipitously meeting people I knew in America who also moved to Europe, and so on — but I think I’ll stop here for this post. Every time I’ve paused to reflect on my relocation abroad, I’ve come away smiling. So far, so good. Hopefully that doesn’t start to wear off!

Status update, March 2022

Tue, 15 Mar 2022 00:00:00 +0000

Greetings! The weather is starting to warm up again, eh? I’m a bit disappointed that we didn’t get any snow this winter. Yadda yadda insert intro text here. Let’s get down to brass tacks. What’s new this month?

I mainly focused on the programming language this month. I started writing a kernel, which you can see a screenshot of below. This screenshot shows a simulated page fault, demonstrating that we have a working interrupt handler, and also shows something mildly interesting: backtraces. I need to incorporate this approach into the standard library as well, so that we can dump useful stack traces on assertion failures and such. I understand that someone is working on DWARF support as well, so perhaps we’ll soon be able to translate function name + offset into a file name and line number.

A redacted screenshot of a kernel showing a simulated page fault

I also started working on a PNG decoder this weekend, which at the time of writing can successfully decode 77 of the 161 PNG test vectors. I am quite pleased with how the code turned out here: this library is a good demonstration of the strengths of the language. It has simple code which presents a comprehensive interface for the file format, has a strong user-directed memory management model, takes good advantage of features like slices, and makes good use of standard library features like compress::zlib and the I/O abstraction. I will supplement this later with a higher level image API which handles things like pixel format conversions and abstracting away format-specific details.

use bufio;
use bytes;
use compress::zlib;
use errors;
use io;

export type idat_reader = struct {
	st: io::stream,
	src: *chunk_reader,
	inflate: zlib::reader,
	decoder: *decoder,
};

// Returns a new IDAT reader for a [[chunk_reader]], from which raw pixel data
// may be read via [[io::read]]. The user must prepare a [[decoder]] object
// along with a working buffer to store the decoder state. For information about
// preparing a suitable decoder, see [[newdecoder]].
export fn new_idat_reader(
	cr: *chunk_reader,
	decoder: *decoder,
) (idat_reader | io::error) = {
	assert(cr.ctype == IDAT, "Attempted to create IDAT reader for non-IDAT chunk");
	return idat_reader {
		st = io::stream {
			reader = &idat_read,
			...
		},
		src = cr,
		inflate = zlib::decompress(cr)?,
		decoder = decoder,
	};
};

fn idat_read(
	st: *io::stream,
	buf: []u8,
) (size | io::EOF | io::error) = {
	let ir = st: *idat_reader;
	assert(ir.st.reader == &idat_read);
	let dec = ir.decoder;
	if (dec.buffered != 0) {
		return decoder_copy(dec, buf);
	};

	if (dec.filter is void) {
		const ft = match (bufio::scanbyte(&ir.inflate)) {
		case io::EOF =>
			return idat_finish(ir);
		case let b: u8 =>
			yield b: filter;
		};
		if (ft > filter::PAETH) {
			return errors::invalid;
		};
		dec.filter = ft;
	};

	// Read one scanline
	for (dec.read < len(dec.cr)) {
		match (io::read(&ir.inflate, dec.cr[dec.read..])?) {
		case io::EOF =>
			// TODO: The rest of the scanline could be in the next
			// IDAT chunk. However, if there is a partially read
			// scanline in the decoder and no IDAT chunk in the
			// remainder of the file, we should probably raise an
			// error.
			return idat_finish(ir);
		case let n: size =>
			dec.read += n;
		};
	};

	applyfilter(dec);
	dec.read = 0;
	dec.buffered = len(dec.cr);
	return decoder_copy(dec, buf);
};

fn idat_finish(ir: *idat_reader) (io::EOF | io::error) = {
	// Verify checksum
	if (io::copy(io::empty, ir.src)? != 0) {
		// Extra data following zlib stream
		return errors::invalid;
	};
	return io::EOF;
};

@test fn idat_reader() void = {
	const src = bufio::fixed(no_filtering, io::mode::READ);
	const read = newreader(&src) as reader;
	let chunk = nextchunk(&read) as chunk_reader;
	const ihdr = new_ihdr_reader(&chunk);
	const ihdr = ihdr_read(&ihdr)!;

	let pixbuf: []u8 = alloc([0...], decoder_bufsiz(&ihdr));
	defer free(pixbuf);
	let decoder = newdecoder(&ihdr, pixbuf);

	for (true) {
		chunk = nextchunk(&read) as chunk_reader;
		if (chunk_reader_type(&chunk) == IDAT) {
			break;
		};
		io::copy(io::empty, &chunk)!;
	};

	const idat = new_idat_reader(&chunk, &decoder)!;
	const pixels = io::drain(&idat)!;
	defer free(pixels);
	assert(bytes::equal(pixels, no_filtering_data));
};

In SourceHut news, I completed our migration to Alpine 3.15 this month after a brief outage, including an upgrade to our database server, which is upgraded on a less frequent cadance than the others. Thanks to Adnan’s work, we’ve also landed many GraphQL improvements, mainly refactorings and other like improvements, setting the stage for the next series of roll-outs. I plan on transitioning back from focusing on the language to focusing on SourceHut for the coming month, and I expect to see some good progress here.

That’s all I have to share for today. Until next time!

It takes a village

Mon, 14 Mar 2022 00:00:00 +0000

As a prolific maintainer of several dozen FOSS projects, I’m often asked how I can get so much done, being just one person. The answer is: I’m not just one person. I have enjoyed the help of thousands of talented people who have contributed to these works. Without them, none of the projects I work on would be successful.

I’d like to take a moment to recognize and thank all of the people who have participated in these endeavours. If you’ve enjoyed any of the projects I’ve worked on, you owe thanks to some of these wonderful people. The following is an incomplete list of authors who have contributed to one or more of the projects I have started:

A Mak
A. M. Joseph
a3v
Aaron Bieber
Aaron Holmes
Aaron Ouellette
Abdelhakim Qbaich
absrd
Ace Eldeib
Adam Kürthy
Adam Mizerski
Aditya Mahajan
Aditya Srivastava
Adnan Maolood
Adrusi
ael-code
agr
Aidan Epstein
Aidan Harris
Ajay R
Ajay Raghavan
Alain Greppin
Aleksa Sarai
Aleksander Usov
Aleksei Bavshin
Aleksis
Alessio
Alex Cordonnier
Alex Maese
Alex McGrath
Alex Roman
alex wennerberg
Alex Xu
Alexander ‘z33ky’ Hirsch
Alexander Bakker
Alexander Dzhoganov
Alexander Harkness
Alexander Johnson
Alexander Taylor
Alexandre Oliveira
Alexey Yerin
Aljaz Gantar
Alynx Zhou
Alyssa Ross
Amin Bandali
amingin
Amir Yalon
Ammar Askar
Ananth Bhaskararaman
Anders
Andreas Rammhold
Andres Erbsen
Andrew Conrad
Andrew Jeffery
Andrew Leap
Andrey Kuznetsov
Andri Yngvason
Andy Dulson
andyleap
Anirudh Oppiliappan
Anjandev Momi
Anthony Super
Anton Gusev
Antonin Décimo
aouelete
apt-ghetto
ARaspiK
Arav K
Ariadna Vigo
Ariadne Conill
Ariel Costas
Ariel Popper
Arkadiusz Hiler
Armaan Bhojwani
Armin Preiml
Armin Weigl
Arnaud Vallette d’Osia
Arsen Arsenović
Art Wild
Arthur Gautier
Arto Jonsson
Arvin Ignaci
ascent12
asdfjkluiop
Asger Hautop Drewsen
ash lea
Ashkan Kiani
Ashton Kemerling
athrungithub
Atnanasi
Aviv Eyal
ayaka
azarus
bacardi55
barfoo1
Bart Pelle
Bart Post
Bartłomiej Burdukiewicz
bbielsa
BearzRobotics
Ben Boeckel
Ben Brown
Ben Burwell
Ben Challenor
Ben Cohen
Ben Fiedler
Ben Harris
Benjamin Cheng
Benjamin Halsted
Benjamin Lowry
Benjamin Riefenstahl
Benoit Gschwind
berfr
bilgorajskim
Bill Doyle
Birger Schacht
Bjorn Neergaard
Björn Esser
blha303
bn4t
Bob Ham
bobtwinkles
boos1993
Bor Grošelj Simić
boringcactus
Brandon Dowdy
BrassyPanache
Brendan Buono
Brendon Smith
Brian Ashworth
Brian Clemens
Brian McKenna
Bruno Pinto
bschacht
BTD Master
buffet
burrowing-owl
Byron Torres
calcdude84se
Caleb Bassi
Callum Brown
Calvin Lee
Cameron Nemo
camoz
Campbell Vertesi
Cara Salter
Carlo Abelli
Cassandra McCarthy
Cedric Sodhi
Chang Liu
Charles E. Lehner
Charlie Stanton
Charmander
chickendude
chr0me
Chris Chamberlain
Chris Kinniburgh
Chris Morgan
Chris Morris
Chris Vittal
Chris Waldon
Chris Young
Christoph Gysin
Christopher M. Riedl
Christopher Vittal
chtison
Clar Charr
Clayton Craft
Clément Joly
cnt0
coderkun
Cole Helbling
Cole Mickens
columbarius
comex
Connor Edwards
Connor Kuehl
Conrad Hoffmann
Cormac Stephenson
Cosimo Cecchi
cra0zy
crondog
Cuber
curiousleo
Cyril Levis
Cédric Bonhomme
Cédric Cabessa
Cédric Hannotier
D.B
dabio
Dacheng Gao
Damien Tardy-Panis
Dan ELKOUBY
Dan Robertson
Daniel Bridges
Daniel De Graaf
Daniel Eklöf
Daniel Gröber
Daniel Kessler
Daniel Kondor
Daniel Lockyer
Daniel Lublin
Daniel Martí
Daniel Otero
Daniel P
Daniel Sockwell
Daniel V
Daniel V.
Daniel Vidmar
Daniel Xu
Daniil
Danilo
Danilo Spinella
Danny Bautista
Dark Rift
Darksun
Dave Cottlehuber
David Arnold
David Blajda
David Eklov
David Florness
David Hurst
David Kraeutmann
David Krauser
David Zero
David96
db
dbandstra
dece
delthas
Denis Doria
Denis Laxalde
Dennis Fischer
Dennis Schridde
Derek Smith
Devin J. Pohly
Devon Johnson
Dhruvin Gandhi
Di Ma
Dian M Fay
Diane
Diederik de Haas
Dillen Meijboom
Dimitris Triantafyllidis
Dizigma
Dmitri Kourennyi
Dmitry Borodaenko
Dmitry Kalinkin
dogwatch
Dominik Honnef
Dominique Martinet
Donnie West
Dorota Czaplejewicz
dudemanguy
Dudemanguy911
Duncaen
Dylan Araps
earnest ma
Ed Younis
EdOverflow
EIREXE
Ejez
Ekaterina Vaartis
Eli Schwartz
Elias Naur
Eloi Rivard
elumbella
Elyes HAOUAS
Elyesa
emersion
Emerson Ferreira
Emmanuel Gil Peyrot
Enerccio
Erazem Kokot
Eric Bower
Eric Drechsel
Eric Engestrom
Eric Molitor
Erik Reider
ernierasta
espkk
Ethan Lee
Euan Torano
EuAndreh
Evan Allrich
Evan Hanson
Evan Johnston
Evan Relf
Eyal Sawady
Ezra
Fabian Geiselhart
Fabio Alessandro Locati
Falke Carlsen
Fazlul Shahriar
Felipe Cardoso Resende
Fenveireth
Ferdinand Bachmann
FICTURE7
Filip Sandborg
finley
Flakebi
Florent de Lamotte
florian.weigelt
Francesco Gazzetta
Francis Dinh
Frank Smit
Franklin “Snaipe” Mathieu
Frantisek Fladung
François Kooman
Frode Aannevik
frsfnrrg
ftilde
fwsmit
Gabriel Augendre
Gabriel Féron
gabrielpatzleiner
Galen Abell
Garrison Taylor
Gauvain ‘GovanifY’ Roussel-Tarbouriech
Gaël PORTAY
gbear605
Genki Sky
Geoff Greer
Geoffrey Casper
George Craggs
George Hilliard
ggrote
Gianluca Arbezzano
gilbus
gildarts
Giuseppe Lumia
Gokberk Yaltirakli
Graham Christensen
Greg Anders
Greg Depoire–Ferrer
Greg Farough
Greg Hewgill
Greg V
Gregory Anders
Gregory Mullen
grossws
Grégoire Delattre
Guido Cella
Guido Günther
Guillaume Brogi
Guillaume J. Charmes
György Kurucz
Gökberk Yaltıraklı
Götz Christ
Haelwenn (lanodan) Monnier
Half-Shot
Hans Brigman
Haowen Liu
Harish Krupo
Harry Jeffery
Heghedus Razvan
Heiko Carrasco
heitor
Henrik Riomar
Honza Pokorny
Hoolean
Hristo Venev
Hubert Hirtz
hugbubby
Hugo Osvaldo Barrera
Humm
Hummer12007
Ian Fan
Ian Huang
Ian Moody
Ignas Kiela
Igor Sviatniy
Ihor Kalnytskyi
Ilia Bozhinov
Ilia Mirkin
Ilja Kocken
Ilya Lukyanov
Ilya Trukhanov
inwit
io mintz
Isaac Freund
Issam E. Maghni
Issam Maghni
István Donkó
Ivan Chebykin
Ivan Fedotov
Ivan Habunek
Ivan Mironov
Ivan Molodetskikh
Ivan Tham
Ivoah
ixru
j-n-f
Jaanus Torp
Jack Byrne
jack gleeson
Jacob Young
jajo-11
Jake Bauer
Jakub Kopański
Jakub Kądziołka
Jamelly Ferreira
James D. Marble
James Edwards-Jones
James Mills
James Murphy
James Pond
James Rowe
James Turner
Jan Beich
Jan Chren
Jan Palus
Jan Pokorný
Jan Staněk
JanUlrich
Jared Baldridge
Jarkko Oranen
Jasen Borisov
Jason Francis
Jason Miller
Jason Nader
Jason Phan
Jason Swank
jasperro
Jayce Fayne
jdiez17
Jeff Kaufman
Jeff Martin
Jeff Peeler
Jeffas
Jelle Besseling
Jente Hidskes
Jeremy Hofer
Jerzi Kaminsky
JerziKaminsky
Jesin
jhalmen
Jiri Vlasak
jman
Joe Jenne
johalun
Johan Bjäreholt
Johannes Lundberg
Johannes Schramm
John Axel Eriksson
John Chadwick
John Chen
John Mako
john muhl
Jon Higgs
Jonas Große Sundrup
Jonas Hohmann
Jonas Kalderstam
Jonas Karlsson
Jonas Mueller
Jonas Platte
Jonathan Bartlett
Jonathan Buch
Jonathan Halmen
Jonathan Schleußer
JonnyMako
Joona Romppanen
Joram Schrijver
Jorge Maldonado Ventura
Jose Diez
Josef Gajdusek
Josh Holland
Josh Junon
Josh Shone
Joshua Ashton
Josip Janzic
José Expósito
José Mota
JR Boyens
Juan Picca
Julian P Samaroo
Julian Samaroo
Julien Moutinho
Julien Olivain
Julien Savard
Julio Galvan
Julius Michaelis
Justin Kelly
Justin Mayhew
Justin Nesselrotte
Justus Rossmeier
Jøhannes Lippmann
k1nkreet
Kacper Kołodziej
Kaleb Elwert
kaltinril
Kalyan Sriram
Karl Rieländer
Karmanyaah Malhotra
Karol Kosek
Kenny Levinsen
kevin
Kevin Hamacher
Kevin Kuehler
Kevin Sangeelee
Kiril Vladimiroff
Kirill Chibisov
Kirill Primak
Kiëd Llaentenn
KoffeinFlummi
Koni Marti
Konrad Beckmann
Konstantin Kharlamov
Konstantin Pospelov
Konstantinos Feretos
kst
Kurt Kartaltepe
Kurt Kremitzki
kushal
Kévin Le Gouguec
Lane Surface
Langston Barrett
Lars Hagström
Laurent Bonnans
Lauri
lbonn
Leon Henrik Plickat
Leszek Cimała
Liam Cottam
Linus Heckemann
Lio Novelli
ljedrz
Louis Taylor
Lubomir Rintel
Luca Weiss
Lucas F. Souza
Lucas M. Dutra
Ludovic Chabant
Ludvig Michaelsson
Lukas Lihotzki
Lukas Märdian
Lukas Wedeking
Lukas Werling
Luke Drummond
Luminarys
Luna Nieves
Lyle Hanson
Lyndsy Simon
Lyudmil Angelov
M Stoeckl
M. David Bennett
Mack Straight
madblobfish
manio143
Manuel Argüelles
Manuel Mendez
Manuel Stoeckl
Marc Grondin
Marcel Hellwig
Marcin Cieślak
Marco Sirabella
Marian Dziubiak
Marien Zwart
Marius Orcsik
Mariusz Bialonczyk
Mark Dain
Mark Stosberg
Markus Ongyerth
MarkusVolk
Marten Ringwelski
Martijn Braam
Martin Dørum
Martin Hafskjold Thoresen
Martin Kalchev
Martin Michlmayr
Martin Vahlensieck
Matias Lang
Matrefeytontias
matrefeytontias
Matt Coffin
Matt Critchlow
Matt Keeter
Matt Singletary
Matt Snider
Matthew Jorgensen
Matthias Beyer
Matthias Totschnig
Mattias Eriksson
Matías Lang
Max Bruckner
Max Leiter
Maxime “pep” Buquet
mbays
MC42
meak
Mehdi Sadeghi
Mendel E
Merlin Büge
Miccah Castorina
Michael Anckaert
Michael Aquilina
Michael Forney
Michael Struwe
Michael Vetter
Michael Weiser
Michael Weiss
Michaël Defferrard
Michał Winiarski
Michel Ganguin
Michele Finotto
Michele Sorcinelli
Mihai Coman
Mikkel Oscar Lyderik
Mikkel Oscar Lyderik Larsen
Milkey Mouse
minus
Mitchell Kutchuk
mliszcz
mntmn
mnussbaum
Moelf
morganamilo
Moritz Buhl
Mrmaxmeier
mteyssier
Mukundan314
muradm
murray
Mustafa Abdul-Kader
mwenzkowski
myfreeweb
Mykola Orliuk
Mykyta Holubakha
n3rdopolis
Naglis Jonaitis
Nate Dobbins
Nate Guerin
Nate Ijams
Nate Symer
Nathan Rossi
Nedzad Hrnjica
NeKit
nerdopolis
ngenisis
Nguyễn Gia Phong
Niccolò Scatena
Nicholas Bering
Nick Diego Yamane
Nick Paladino
Nick White
Nicklas Warming Jacobsen
Nicolai Dagestad
Nicolas Braud-Santoni
Nicolas Cornu
Nicolas Reed
Nicolas Schodet
Nicolas Werner
NightFeather
Nihil Pointer
Niklas Schulze
Nils ANDRÉ-CHANG
Nils Schulte
Nixon Enraght-Moony
Noah Altunian
Noah Kleiner
Noah Loomans
Noah Pederson
Noam Preil
Noelle Leigh
NokiDev
Nolan Prescott
Nomeji
Novalinium
novenary
np511
nrechn
NSDex
Nuew
nyorain
nytpu
Nícolas F. R. A. Prado
oharaandrew314
Oleg Kuznetsov
Oliver Leaver-Smith
oliver-giersch
Olivier Fourdan
Ondřej Fiala
Orestis Floros
Oscar Cowdery Lack
Ossi Ahosalmi
Owen Johnson
Paco Esteban
Parasrah
Pascal Pascher
Patrick Sauter
Patrick Steinhardt
Paul Fenwick
Paul Ouellette
Paul Riou
Paul Spooren
Paul W. Rankin
Paul Wise
Pedro Côrte-Real
Pedro L. Ramos
Pedro Lucas Porcellis
Peroalane
Peter Grayson
Peter Lamby
Peter Rice
Peter Sanchez
Phil Rukin
Philip K
Philip Woelfel
Philipe Goulet
Philipp Ludwig
Philipp Riegger
Philippe Pepiot
Philz69
Pi-Yueh Chuang
Pierre-Albéric TROUPLIN
Piper McCorkle
pixelherodev
PlusMinus0
PoroCYon
ppascher
Pranjal Kole
ProgAndy
progandy
Przemyslaw Pawelczyk
psykose
punkkeks
pyxel
Quantum
Quentin Carbonneaux
Quentin Glidic
Quentin Rameau
R Chowdhury
r-c-f
Rabit
Rachel K
Rafael Castillo
rage 311
Ragnar Groot Koerkamp
Ragnis Armus
Rahiel Kasim
Raman Varabets
Ranieri Althoff
Ray Ganardi
Raymond E. Pasco
René Wagner
Reto Brunner
Rex Hackbro
Ricardo Wurmus
Richard Bradfield
Rick Cogley
rinpatch
Robert Günzler
Robert Johnstone
Robert Kubosz
Robert Sacks
Robert Vollmert
Robin Jarry
Robin Kanters
Robin Krahl
Robin Opletal
Robinhuett
robotanarchy
Rodrigo Lourenço
Rohan Kumar
Roman Gilg
ROMB
Ronan Pigott
ronys
Roosembert Palacios
roshal
Roshless
Ross L
Ross Schulman
Rostislav Pehlivanov
rothair
RoughB Tier0
Rouven Czerwinski
rpigott
Rune Morling
russ morris
Ryan Chan
Ryan Dwyer
Ryan Farley
Ryan Gonzalez
Ryan Walklin
Rys Sommefeldt
Réouven Assouly
S. Christoffer Eliesen
s0r00t
salkin-mada
Sam Newbold
Sam Whited
SatowTakeshi
Sauyon Lee
Scoopta
Scott Anderson
Scott Colby
Scott Leggett
Scott Moreau
Scott O’Malley
Scott Stevenson
sdilts
Sebastian
Sebastian Krzyszkowiak
Sebastian LaVine
Sebastian Noack
Sebastian Parborg
Seferan
Sergeeeek
Sergei Dolgov
Sergi Granell
sergio
Seth Barberee
Seán C McCord
sghctoma
Shaw Vrana
Sheena Artrip
Silvan Jegen
Simon Barth
Simon Branch
Simon Ruderich
Simon Ser
Simon Zeni
Siva Mahadevan
skip-yell
skuzzymiglet
Skyler Riske
Slowpython
Sol Fisher Romanoff
Solomon Victorino
somdoron
Sorcus
sourque
Spencer Michaels
SpizzyCoder
sqwishy
Srivathsan Murali
Stacy Harper
Steef Hegeman
Stefan Rakel
Stefan Schick
Stefan Tatschner
Stefan VanBuren
Stefan Wagner
Stefano Ragni
Stephan Hilb
Stephane Chauveau
Stephen Brennan
Stephen Brown II
Stephen Gregoratto
Stephen Paul Weber
Steve Jahl
Steve Losh
Steven Guikal
Stian Furu Øverbye
Streetwalrus Einstein
Stuart Dilts
Sudipto Mallick
Sumner Evans
Syed Amer Gilani
sykhro
Tadeo Kondrak
Taiyu
taiyu
taminaru
Tamir Zahavi-Brunner
Tancredi Orlando
Tanguy Fardet
Tarmack
Taryn Hill
tastytea
tcb
Teddy Reed
Tero Koskinen
Tharre
Thayne McCombs
The Depressed Milkman
TheAvidDev
TheMachine02
Theodor Thornhill
thermitegod
Thiago Mendes
thirtythreeforty
Thomas Bracht Laumann Jespersen
Thomas Hebb
Thomas Jespersen
Thomas Karpiniec
Thomas Merkel
Thomas Plaçais
Thomas Schneider
Thomas Weißschuh
Thomas Wouters
Thorben Günther
thuck
Till Hofmann
Tim Sampson
Tim Schumacher
Timidger
Timmy Douglas
Timothée Floure
Ting-Wei Lan
tiosgz
toadicus
Tobi Fuhrimann
Tobias Blass
Tobias Langendorf
Tobias Stoeckmann
Tobias Wölfel
Tom Bereknyei
Tom Lebreux
Tom Ryder
Tom Warnke
tomKPZ
Tommy Nguyen
Tomáš Čech
Tony Crisci
Torstein Husebø
Trannie Carter
Trevor Slocum
TriggerAu
Tudor Brindus
Tudor Roman
Tuomas Siipola
tuomas56
Twan Wolthof
Tyler Anderson
Uli Schlachter
Umar Getagazov
unlimitedbacon
unraised
User Name
v44r
Valentin
Valentin Hăloiu
Vasilij Schneidermann
Versus Void
vexhack
Vijfhoek
vil
vilhalmer
Vincent Gu
Vincent Vanlaer
Vinko Kašljević
Vitalij
Vitalij Mikhailov
Vlad Pănăzan
Vlad-Stefan Harbuz
Vyivel
w1ke
Wagner Riffel
wagner riffel
Wai Hon Law
wb9688
wdbw
Whemoon Jang
Wieland Hoffmann
Wiktor Kwapisiewicz
wil
Will Daly
Will Hunt
willakat
Willem Sonke
William Casarin
William Culhane
William Durand
William Moorehouse
William Wold
willrandship
Willy Goiffon
Wolf480pl
Wouter van Kesteren
Xaiier
xdavidwu
xPMo
y0ast
Yacine Hmito
yankejustin
Yasar
Yash Srivastav
Yong Joseph Bakos
Yorick van Pelt
yuiiio
yuilib
Yury Krivopalov
Yuya Nishihara
Yábir Benchakhtir
Yábir García
Zach DeCook
Zach Sisco
Zachary King
Zandr Martin
zccrs
Zetok Zalbavar
Zie
Zoltan Kalmar
Zuzana Svetlikova
Éloi Rivard
Érico Rolim
Štěpán Němec
наб
حبيب الامين

Each of these is a distinct person, with their own lives and aspirations, who took time out of those lives to help build some cool software. I owe everything to these wonderful, talented, dedicated people. Thank you, everyone. Let’s keep up the good work, together.

Why am I building a programming language in private?

Sun, 13 Mar 2022 00:00:00 +0000

As many readers are aware, I have been working on designing and implementing a systems programming language. This weekend, I’ve been writing a PNG file decoder in it, and over the past week, I have been working on a simple kernel with it as well. I’m very pleased with our progress so far — I recently remarked that this language feels like the language I always wanted, and that’s mission accomplished by any definition I care to consider.

I started the project on December 27th, 2019, just over two years ago, and I have kept it in a semi-private state since. Though I have not given its name in public, the git repos, mailing lists, and bug trackers use sourcehut’s “unlisted” state, so anyone who knows the URL can see them. The website is also public, though its domain name is also undisclosed, and it is full of documentation, tutorials, and resources for developers. People can find the language if they want to, though at this stage the community only welcomes contributors, not users or onlookers. News of the project nominally spreads by word of mouth and with calls-to-action on this blog, and to date a total of 30 people have worked on it over the course of 3,029 commits. It is a major, large-scale project, secret though it may be.

And, though we’ve invested a ton of work into this project together, it remains as-of-yet unfinished. There is no major software written in our language, though several efforts are underway. Several of our key goals have yet to be merged upstream, such as date/time support, TLS, and regular expressions, though, again, these efforts are well underway. Until we have major useful projects written in our language, we cannot be confident in our design, and efforts in these respects do a great deal to inform us regarding any changes which might be necessary. And some changes are already in the pipeline: we have plans to make several major revisions to the language and standard library design, which are certain to require changes in downstream software.

When our community is small and private, these changes are fairly easy to reckon with. Almost everyone who is developing a project based on our language is also someone who has worked on the compiler or standard library. Often, the person who implements a breaking change will also send patches to various downstreams updating them to be compatible with this change, for every extant software project written in the language. This is a task which can be undertaken by one person. We all understand the need for these changes, participate in the discussions and review the implementations, and have the expertise necessary to make the appropriate changes to our projects.

Moreover, all of these people are also understanding of the in-development nature of the project. All users of our language are equipped with the knowledge that they are expected to help fix the bugs they identify, and with the skills and expertise necessary to follow-up on this fact. We don’t have to think about users who stumble upon the project, spend a few hours trying to use it, then encounter an under-developed part of the language and run out of enthusiasm. We still lack DWARF support, so debugging is a chore. Sometimes the compiler segfaults or aborts without printing a useful error message. It’s a work-in-progress, after all. These kinds of problems can discourage new learners very fast, and often requires the developers to offer some of their precious bandwidth to provide expert assistance. With the semi-private model, there are, at any given time, a very small number of people involved who are new to the language and require more hands-on support to help them through their problems.

A new programming language is a major undertaking. We’re building one with an explicit emphasis on simplicity and we’re still not done after two years. When most people hear about the project for the first time, I don’t want them to find a half-completed language which they will fail to apply to their problem because it’s not fleshed out for their use-case. The initial release will have comprehensive documentation, a detailed specification, and stability guarantees, so it can be picked up and used in production by curious users on day one. I want to fast-forward to the phase where people study it to learn how to apply it to their problems, rather than to learn if they can apply it to their problems.

Even though it is under development in private, this project is both “free software” and “open source”, according to my strict understanding of those terms as defined by the FSF and OSI. “Open source” does not mean that the project has a public face. The compiler is GPL 3.0 licensed, the standard library is MPL 2.0, and the specification is CC-BY-ND (the latter is notably less free, albeit for good reasons), and these details are what matter. Every person who has worked on the project, and every person who stumbles upon it, possesses the right to lift the veil of secrecy and share it with the world. The reason they don’t is because I asked them not to, and we maintain a mutual understanding regarding the need for privacy.

On a few occasions, someone has discovered the project and taken it upon themselves to share it in public places, including Hacker News, Lemmy, and 4chan. While this is well within your rights, I ask you to respect our wishes and allow us to develop this project in peace. I know that many readers are excited to try it out, but please give us some time and space to ensure that you are presented with a robust product. At the moment, we anticipate going public early next year. Thank you for your patience.

Thank you for taking the time to read my thoughts as well. I welcome your thoughts and opinions on the subject: my inbox is always open. If you disagree, I would appreciate it if you reached out to me to discuss it before posting about the project online. And, if you want to get involved, here is a list of things we could use help with — email me to volunteer if you have both the time and expertise necessary:

Cryptography
Ports for new architectures or operating systems
Image & pixel formats/conversions
SQL database adapters
Signal handling
JSON parsing & encoding
Compression and decompression
Archive formats

If you definitely don’t want to wait for the language to go public, volunteering in one of our focus areas is the best way to get involved. Get in touch! If not, then the release will come around sooner than you think. We’re depending on your patience and trust.

Update 2022-03-14

This blog post immediately generated detailed discussions on Hacker News and Lobsters in which people posted the language’s website and started tearing into everything they don’t like about it.

It’s not done yet, and the current state of the language is not representative of the project goals. This post was not a marketing stunt. It was a heartfelt appeal to your better nature.

You know, I have a lot on my plate. All of it adds up to a lot of stress. I had hoped that you would help relieve some of that stress by taking me seriously when I explained my motivations and asked nicely for you to leave us be. I was wrong.

Open Source is defined by the OSI's Open Source Definition

Tue, 01 Mar 2022 00:00:00 +0000

The Open Source Initiative (OSI) publishes a document called the Open Source Definition (OSD), which defines the term “open source”. However, there is a small minority of viewpoints within the software community which wishes that this were not so. The most concerning among them are those who wish open source was more commercially favorable to themselves, and themselves alone, such as companies like Elastic.

I disagree with this perspective, and I’d like take a few minutes today to explore several of the most common arguments in favor of this view, and explain why I don’t agree with them. One of the most frustrating complications in this discussion is the context of motivated reasoning (relevant xkcd): most people arguing in favor of an unorthodox definition of “open source” have a vested interest in their alternative view.¹ This makes it difficult to presume good faith. For example, say someone wants to portray their software as open source even if it prohibits commercial use by third parties, which would normally disqualify it as such. Their interpretation serves to re-enforce their commercialization plans, providing a direct financial incentive not only for them to promote this definition of “open source”, but also for them to convince you that their interpretation is valid.

I find this argument to be fundamentally dishonest. Let me illustrate this with an analogy. Consider PostgreSQL. If I were to develop a new program called Postgres which was similar to PostgreSQL, but different in some important ways — let’s say it’s a proprietary, paid, hosted database service — that would be problematic. The industry understands that “Postgres” refers to the popular open source database engine, and by re-using their name I am diluting the brand of Postgres. It can be inferred that my reasoning for this comes from the desire to utilize their brand power for personal commercial gain. The terms “Postgres” and “PostgreSQL” are trademarked, but even if they were not, this approach would be dishonest and ethically wrong.

So too are the attempts to re-brand “open source” in a manner which is more commercially exploitable for an individual person or organization equally dishonest. The industry has an orthodox understanding of the meaning of “open source”, i.e. that defined by the Open Source Initiative, which is generally well-understood through the proliferation of software licenses which are compatible with the OSD. When a project describes itself as “open source”, this is a useful short-hand for understanding that the project adheres to a specific set of values and offers a specific set of rights to its users and contributors. When those rights are denied or limited, the OSD no longer applies and thus neither does the term “open source”. To disregard this in the interests of a financial incentive is dishonest, much like I would be dishonest for selling “cakes” and fulfilling orders with used car tires with “cake” written on them instead.

Critics of the OSD frequently point out that the OSI failed to register a trademark on the term “open source”, but a trademark is not necessary for this argument to hold. Language is defined by its usage, and the OSD is the popular usage of the term “open source”, without relying on the trademark system. The existence of a trademark on a specific term is not required for language which misuses that term to be dishonest.

As language is defined by its usage, some may argue that they are as entitled as anyone else to put forward an alternative usage. This is how language evolves. They are not wrong, though I might suggest that their alternative usage of “open source” requires a substantial leap in understanding which might not be as agreeable to those who don’t stand to benefit financially from that leap. Even so, I argue that the mainstream definition of open source, that forwarded by the OSI, is a useful term that is worth preserving in its current form. It is useful to quickly understand the essential values and rights associated with a piece of software as easily as stating that it is “open source”. I am not prepared to accept a new definition which removes or reduces important rights in service of your private financial interests.

The mainstream usage of “open source” under the OSD is also, in my opinion, morally just. You may feel a special relationship with the projects you start and invest into, and a sense of ownership with them, but they are not rightfully yours once you receive outside contributions. The benefit of open source is in the ability for the community to contribute directly to its improvements — and once they do, the project is the sum of your efforts and the efforts of the community. Thus, is it not right that the right to commercial exploitation of the software is shared with that community? In the absence of a CLA,² contributors retain their copyright as well, and the software is legally jointly owned by the sum of its contributors. And beyond copyright, the success of the software is the sum of its code along with the community who learns about and deploys it, offers each other support, writes blog posts and books about it, sells consulting services for it, and together helps to popularize it. If you wish to access all of these benefits of the open source model, you must play by the open source rules.

It’s not surprising that this would become a matter of contention among certain groups within the industry. Open source is not just eating the world, but has eaten the world. Almost all software developed today includes substantial open source components. The open source brand is very strong, and there are many interests who would like to leverage that brand without meeting its obligations. But the constraints of the open source definition are important, played a critical role in the ascension of open source in the software market, and worth preserving into the future.

That’s not to say that there isn’t room for competing ideologies. If you feel that the open source model does not work for you, then that’s a valid opinion to hold. I only ask that you market your alternative model honestly by using a different name for it. Software for which the source code is available, but which does not meet the requirements of the open source definition, is rightfully called “source available”. If you want a sexier brand for it, make one! “Open core” is also popular, though not exactly the same. Your movement has as much right to success as the open source movement, but you need to earn that success independently of the open source movement. Perhaps someday your alternative model will supplant open source! I wish you the best of luck in this endeavour.

A previous version of this blog post announced that I had submitted my candidacy for the OSI board. Due to unforseen circumstances, I will be postponing my candidacy until the next election. I apologise for the confusion.

Plaid is an evil nightmare product from Security Hell

Sat, 19 Feb 2022 00:00:00 +0000

Plaid is a business that has built a widget that can be embedded in any of their customer’s websites which allows their customers to configure integrations with a list of third-party service providers. To facilitate this, Plaid pops up a widget on their customer’s domain which asks the end-user to type in their username and password for the third-party service provider. If necessary, they will ask for a 2FA code. This is done without the third party’s permission, presumably through a browser emulator and a provider-specific munging shim, and collects the user’s credentials on a domain which is operated by neither the third party nor by Plaid.

The third-party service provider in question is the end-user’s bank.

What the actual fuck!

Plaid has weighed on my mind for a while, though I might have just ignored them if they hadn’t been enjoying a sharp rise in adoption across the industry. For decades, we have stressed the importance of double-checking the domain name and the little TLS “lock” icon before entering your account details for anything. It is perhaps the single most important piece of advice the digital security community has tried to bring into the public conciousness. Plaid wants to throw out all of those years of hard work and ask users to enter their freaking bank credentials into a third-party form.

The raison d’être for Plaid is that banks are infamously inflexible and slow on the uptake for new technology. The status quo which Plaid aims to disrupt (ugh), at least for US bank account holders, involves the user entering their routing number and account number into a form. The service provider makes two small (<$1) deposits, and when they show up on the user’s account statement a couple of days later, the user confirms the amounts with the service provider, the service provider withdraws the amounts again, and the integration is complete. The purpose of this dance is to provide a sufficiently strong guarantee that the account holder is same person who is configuring the integration.

This process is annoying. Fixing it would require banks to develop, deploy, and standardize on better technology, and, well, good luck with that. And, honestly, a company which set out with the goal of addressing this problem ethically would have a laudable ambition. But even so, banks are modernizing around the world, and tearing down the pillars of online security in exchange for a mild convenience is ridiculous.

A convincing argument can be made that this platform violates the Computer Fraud and Abuse Act. Last year, they paid out $58M in one of many lawsuits for scraping and selling your bank data. Plaid thus joins the ranks of Uber, AirBnB, and others like them in my reckoning as a “move fast and break laws” company. This platform can only exist if they are either willfully malignant or grossly incompetent. They’ve built something that they know is wrong, and are hoping that they can outrun the regulators.

This behavior is not acceptable. This company needs to be regulated into the dirt and made an example of. Shame on you Plaid, and shame on everyone involved in bringing this product to market. Shame on their B2B customers as well, who cannot, such as they may like to, offload ethical due-diligence onto their vendors. Please don’t work for these start-ups. I hold employees complicit in their employer’s misbehavior. You have options, please go make the world a better place somewhere else.

Status update, February 2022

Tue, 15 Feb 2022 00:00:00 +0000

Hello once again! Another month of free software development goes by with lots of progress in all respects.

I will open with some news about godocs.io: version 1.0 of our fork of gddo has been released! Big thanks to Adnan Maolood for his work on this. I’m very pleased that, following our fork, we were not only able to provide continuity for godoc.org, but also to simplify, refactor, and improve the underlying software considerably. Check out Adnan’s blog post for more details.

In programming language news, we have had substantial progress in many respects. One interesting project I’ve started is a Redis protocol implementation:

const conn = redis::connect()!;
defer redis::close(&conn);

fmt::println("=> SET foo bar EX 10")!;
redis::set(&conn, "foo", "bar", 10: redis::ex)!;

Another contributor has been working on expanding our graphics support, including developing a backend for glad to generate OpenGL bindings, and a linear algebra library ala glm for stuff like vector and matrix manipulation. Other new modules include a MIME database and encoding::base32. Cryptography progress continued with the introduction of XTS mode for AES, which is useful for full disk encryption implementations, but has slowed while we develop bigint support for future algorithms like RSA. I have also been rewriting the language introduction tutorial with a greater emphasis on practical usage.

Before we move on from the language project: I need your help! I am looking for someone to help develop terminal support. This is fairly straightforward, though laborsome: it involves developing libraries in our language which provide the equivalents of something like ncurses (or, better, libtickit), as well as the other end like libvterm offers. Please email me if you want to help.

In SourceHut news, we have hired our third full-time engineer: Conrad Hoffmann! Check out the blog post for details. The first major effort from Adnan’s NLnet-sponsored SourceHut work also landed yesterday, introducing GraphQL-native webhooks to git.sr.ht alongside a slew of other improvements. pages.sr.ht also saw some improvements that allow users to configure their site’s behavior more closely. Check out the “What’s cooking” post later today for all of the SourceHut news.

That’s all for today, thanks for reading!

Framing accessibility in broader terms

Sun, 13 Feb 2022 00:00:00 +0000

Upon hearing the term “accessibility”, many developers call to mind the HTML ARIA attributes and little else. Those who have done some real accessibility work may think of the WCAG guidelines. Some FOSS developers¹ may think of AT-SPI. The typical user of these accessibility features is, in the minds of many naive developers, a blind person. Perhaps for those who have worked with WCAG, a slightly more sophisticated understanding of the audience for accessibility tools may include users with a greater variety of vision-related problems, motor impairments, or similar needs.

Many developers² frame accessibility in these terms, as a list of boxes to tick off, or specific industry tools which, when used, magically create an accessible product. This is not the case. In truth, a much broader understanding of accessibility is required to create genuinely accessible software, and because that understanding often raises uncomfortable questions about our basic design assumptions, the industry’s relationship with accessibility borders on willful ignorance.

The typical developer’s relationship with accessibility, if they have one at all, is mainly concerned with making web pages work with screen readers. Even considering this very narrow goal, most developers have an even narrower understanding of the problem, and end up doing a piss-poor job of it. In essence, the process of doing accessibility badly involves making a web page for a sighted user, then using ARIA tags to hide cosmetic elements, adding alt tags, and making other surface-level improvements for users of screen readers. If they’re serious, they may reach for the WCAG guidelines and do things like considering contrast, font choices, and animations as well, but all framed within the context of adding accessibility band-aids onto a UI designed for sighted use.

A key insight here is that concerns like font choice and contrast involve making changes which are apparent to “typical” users as well, but we’ll expand on that in a moment. Instead of designing for people like you and then patching it up until it’s semi-functional for people who are not like you, a wise developer places themselves into the shoes of the person they’re designing for and builds something which speaks their design language. For visually impaired users, this might mean laying out information in a more logical sense than in a spatial sense.

Importantly, accessibility also means understanding that there are many other kinds of users who have accessibility needs.

For instance, consider someone who cannot afford a computer as nice as the one your developers are using. When your Electron ~~crapware~~ app eats up 8G of RAM, it may be fine on your 32G developer workstation, but not so much for someone who cannot afford anything other than a used $50 laptop from eBay. Waking up the user’s phone every 15 minutes to check in with your servers isn’t very nice for someone using a 5-year-old phone with a dying battery. Your huge JavaScript bundle, unoptimized images, and always-on network requirements are not accessible to users who are on low-bandwidth mobile connections or have a data cap — you’re essentially charging poorer users a tax to use your website.

Localization is another kind of accessibility, and it requires more effort than running your strings through gettext. Users in different locales speak not only different natural languages, but different design languages. Users of right-to-left languages like Arabic don’t just reverse their strings but also the entire layout of the page. Chinese and Japanese users are more familiar with denser UIs than the typical Western user. And subtitles and transcripts are important for Deaf users, but also useful for users who are consuming your content in a second language.

Intuitiveness is another important detail. Not everyone understands what your icons mean, for a start. They may not have the motor skill to hold their mouse over the button and read the tool-tip, either, and might not know that they can do that in the first place! Reliance on unfamiliar design language in general is a kind of inaccessible design. Remember the “save” icon? 💾 Flashing banner ads are also inaccessible for users with ADHD, and if we’re being honest, for everyone else, too. Software which is not responsive on many kinds of devices (touch, mouse and keyboard, different screen sizes, aspect ratios, orientations) is not accessible. Software which requires the latest and greatest technologies to use (such as a modern web browser) is also not accessible.

Adequate answers to these problems are often expensive and uncomfortable, so no one wants to think about them. Social-media-esque designs which are deliberately addictive are not accessible, and also not moral. The mountain of gross abstractions on which much software is built is cheap, but causes it to suck up all the user’s resources (RAM, CPU, battery, etc) on 10-year-old devices.³ And ads are inaccessible by design, but good luck explaining that to your boss.

It is a fool’s errand to aim for perfect accessibility for all users, but we need to understand that our design choices are excluding people from using our tools. We need to design our software with accessibility in mind from the ground up, and with a broad understanding of accessibility that acknowledges that simple, intuitive software is the foundation of accessibility which works for everyone, including you and me — and not retroactively adding half-assed tools to fundamentally unusable software. I want UI designers to be thinking in these terms, and less in terms of aesthetic properties, profitable designs, and dark patterns. Design with empathy first.

As someone who works exclusively in free software, I have to acknowledge the fact that free software is pretty pathetic when it comes to accessibility. In our case, this does not generally come from the perverse incentives that cause businesses to cut costs or even deliberately undermine accessibility for profit,⁴ but instead comes from laziness (or, more charitably, lack of free time and enthusiasm), and generally from free software’s struggles to build software for people who are not like its authors. I think that we can change this. We do not have the profit motive, and we can choose to take pride in making better software for everyone. Let’s do better.

Free software licenses explained: MIT

Mon, 07 Feb 2022 00:00:00 +0000

This is the first in a series of posts I intend to write explaining how various free and open source software licenses work, and what that means for you as a user or developer of that software. Today we’ll look at the MIT license, also sometimes referred to as the X11 or Expat license.

The MIT license is:

Both free software and open source
Permissive (and thus non-copyleft and non-viral)

This means that the license upholds the four essential freedoms of free software (the right to run, copy, distribute, study, change and improve the software) and all of the terms of the open source definition (largely the same). Further more, it is classified on the permissive/copyleft spectrum as a permissive license, meaning that it imposes relatively few obligations on the recipient of the license.

The full text of the license is quite short, so let’s read it together:

The MIT License (MIT)
Copyright (c)
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

The first paragraph of the license enumerates the rights which you, as a recipient of the software, are entitled to. It’s this section which qualifies the license as free and open source software (assuming the later sections don’t disqualify it). The key grants are the right to “use” the software (freedom 0), to “modify” and “merge” it (freedom 1), and to “distribute” and “sell” copies (freedoms 2 and 3), “without restriction”. We also get some bonus grants, like the right to sublicense the software, so you could, for instance, incorporate it into a work which uses a less permissive license like the GPL.

All of this is subject to the conditions of paragraph two, of which there is only one: you must include the copyright notice and license text in any substantial copies or derivatives of the software. Thus, the MIT license requires attribution. This can be achieved by simply including the full license text (copyright notice included) somewhere in your project. For a proprietary product, this is commonly hidden away in a menu somewhere. For a free software project, where the source code is distributed alongside the product, I often include it as a comment in the relevant files. You can also add your name or the name of your organization to the list of copyright holders when contributing to MIT-licensed projects, at least in the absence of a CLA.¹

The last paragraph sets the expectations for the recipient, and it is very important. This disclaimer of warranty is ubiquitous in nearly all free and open source software licenses. The software is provided “as is”, which is to say, in whatever condition you found it in, for better or worse. There is no expectation of warranty (that is to say, any support you receive is from the goodwill of the authors and not from a contractual obligation), and there is no guarantee of “merchantability” (that you can successfully sell it), fitness for a particular purpose (that you can successfully use it to solve your problem), or noninfringement (such as with respect to relevant patents). That last detail may be of particular importance: the MIT license disclaims all liability for patents that you might infringe upon by using the software. Other licenses often address this case differently, such as Apache 2.0.

MIT is a good fit for projects which want to impose very few limitations on the use or reuse of the software by others. However, the permissibility of the license permits behaviors you might not like, such as creating a proprietary commercial fork of the software and selling it to others without supporting upstream. Note that the right to sell the software is an inalienable requirement of the free software and open source definitions, but other licenses can level the playing field a bit with strategies like copyleft and virality, on the other end of the permissibility spectrum. I’ll cover some relevant licenses in the future.

Implementing a MIME database in XXXX

Fri, 28 Jan 2022 00:00:00 +0000

This is a (redacted) post from the internal blog of a new systems programming language we’re developing. The project is being kept under wraps until we’re done with it, so for this post I’ll be calling it xxxx. If you are interested in participating, send me an email with some details about your background and I’ll get back to you.

Recently, I have been working on implementing a parser for media types (commonly called MIME types) and a database which maps media types to file extensions and vice-versa. I thought this would be an interesting module to blog about, given that it’s only about 250 lines of code, does something useful and interesting, and demonstrates a few interesting xxxx concepts.

The format for media types is more-or-less defined by RFC 2045, specifically section 5.1. The specification is not great. The grammar shown here is copied and pasted from parts of larger grammars in older RFCs, RFCs which are equally poorly defined. For example, the quoted-string nonterminal is never defined here, but instead comes from RFC 822, which defines it but also states that it can be “folded”, which technically makes the following a valid Media Type:

text/plain;charset="hello
 world"

Or so I would presume, but the qtext terminal “cannot include CR”, which is the mechanism by which folding is performed in the first place, and… bleh. Let’s just implement a “reasonable subset” of the spec instead and side-step the whole folding issue.¹ This post will first cover parsing media types, then address our second goal: providing a database which maps media types to file extensions and vice versa.

Parsing Media Types

So, here’s what we’re going to implement today: we want to parse the following string:

text/plain; charset=utf-8; foo="bar baz"

The code for that I came up with for is as follows:

// Parses a Media Type, returning a tuple of the content type (e.g.
// "text/plain") and a parameter parser object, or [[errors::invalid]] if the
// input cannot be parsed.
//
// To enumerate the Media Type parameter list, pass the type_params object into
// [[next_param]]. If you do not need the parameter list, you can safely discard
// the object. Note that any format errors following the ";" token will not
// cause [[errors::invalid]] to be returned unless [[next_param]] is used to
// enumerate all of the parameters.
export fn parse(in: str) ((str, type_params) | errors::invalid) = {
	const items = strings::cut(in, ";");
	const mtype = items.0, params = items.1;
	const items = strings::cut(mtype, "/");
	if (len(items.0) < 1 || len(items.1) < 1) {
		return errors::invalid;
	};
	typevalid(items.0)?;
	typevalid(items.1)?;
	return (mtype, strings::tokenize(params, ";"));
};

This function accepts a string as input, then returns a tagged union which contains either a tuple of (str, type_params), or a syntax error.

I designed this with particular attention to the memory management semantics. xxxx uses manual memory management, and if possible it’s desirable to avoid allocating any additional memory so that the user of our APIs remains in control of the memory semantics. The return value is a sub-string borrowed from the “text/plain” part, as well as a tokenizer which is prepared to split the remainder of the string along the “;” tokens.

Inspiration for strings::cut comes from Go:

$ xxxxdoc strings::cut
// Returns a string "cut" along the first instance of a delimiter, returning
// everything up to the delimiter, and everything after the delimiter, in a
// tuple.
//
//      strings::cut("hello=world=foobar", "=") // ("hello", "world=foobar")
//      strings::cut("hello world", "=")        // ("hello world", "")
//
// The return value is borrowed from the 'in' parameter.
fn cut(in: str, delim: str) (str, str);

And strings::tokenize works like so:

$ xxxxdoc strings::tokenize
// Returns a tokenizer which yields sub-strings tokenized by a delimiter.
//
//      let tok = strings::tokenize("hello, my name is drew", " ");
//      assert(strings::next_token(tok) == "hello,");
//      assert(strings::next_token(tok) == "my");
//      assert(strings::next_token(tok) == "name");
//      assert(strings::remaining_tokens(tok) == "is drew");
fn tokenize(s: str, delim: str) tokenizer;

The RFC limits the acceptable characters for the media type and subtype, which we test with the typevalid function.

The user of this module often only cares about the media type and not its type parameters, so the tokenizer can be safely abandoned on the stack to get cleaned up when the stack frame exits if they don’t care about the rest.

This is enough to write a little test:

@test fn parse() void = {
	const res = parse("text/plain")!;
	assert(res.0 == "text/plain");

	const res = parse("image/png")!;
	assert(res.0 == "image/png");

	const res = parse("application/svg+xml; charset=utf-8; foo=\"bar baz\"")!;
	assert(res.0 == "application/svg+xml");
};

$ xxxx test mime::parse
mime::parse..................................... OK

1 passed; 0 failed; 1 tests completed in 0.10s

To handle the type parameters in the third case, we add this function:

// Returns the next parameter as a (key, value) tuple from a [[type_params]]
// object that was prepared via [[parse]], void if there are no remaining
// parameters, and [[errors::invalid]] if a syntax error was encountered.
export fn next_param(in: *type_params) ((str, str) | void | errors::invalid) = {
	const tok = match (strings::next_token(in)) {
	case let s: str =>
		if (s == "") {
			// empty parameter
			return errors::invalid;
		};
		yield s;
	case void =>
		return;
	};

	const items = strings::cut(tok, "=");
	// The RFC does not permit whitespace here, but whitespace is very
	// common in the wild. ¯\_(ツ)_/¯
	items.0 = strings::trim(items.0);
	items.1 = strings::trim(items.1);
	if (len(items.0) == 0 || len(items.1) == 0) {
		return errors::invalid;
	};

	if (strings::hasprefix(items.1, "\"")) {
		items.1 = quoted(items.1)?;
	};

	return (items.0, items.1);
};

This returns a (key, value) tuple and advances to the next parameter, or returns void if there are no further parameters (or, if necessary, an error). This is pretty straightforward: the tokenizer prepared by parse is splitting the string on ; tokens, so we first fetch the next token. We then use strings::cut again to split it over the = token, and after a quick trim to fix another RFC oversight, we can return it to the caller. Unless it’s using this pesky quoted-string terminal, which is where our implementation starts to show its weaknesses:

fn quoted(in: str) (str | errors::invalid) = {
	// We have only a basic implementation of quoted-string. It has a couple
	// of problems:
	//
	// 1. The RFC does not define it very well
	// 2. The parts of the RFC which are ill-defined are rarely used
	// 3. Implementing quoted-pair would require allocating a new string
	//
	// This implementation should handle most Media Types seen in practice
	// unless they're doing something weird and ill-advised with them.
	in = strings::trim(in, '"');
	if (strings::contains(in, "\\")
			|| strings::contains(in, "\r")
			|| strings::contains(in, "\n")) {
		return errors::invalid;
	};
	return in;
};

I think this implementation speaks for itself. It could be a bit faster if we didn’t do 3 × O(n) strings::contains calls, but someone will send a patch if they care. The completed test for this is:

@test fn parse() void = {
	const res = parse("text/plain")!;
	assert(res.0 == "text/plain");

	const res = parse("image/png")!;
	assert(res.0 == "image/png");

	const res = parse("application/svg+xml; charset=utf-8; foo=\"bar baz\"")!;
	assert(res.0 == "application/svg+xml");
	const params = res.1;
	const param = next_param(&params)! as (str, str);
	assert(param.0 == "charset" && param.1 == "utf-8");
	const param = next_param(&params)! as (str, str);
	assert(param.0 == "foo" && param.1 == "bar baz");
	assert(next_param(&params) is void);

	assert(parse("hi") is errors::invalid);
	assert(parse("text/ spaces ") is errors::invalid);
	assert(parse("text/@") is errors::invalid);

	const res = parse("text/plain;charset")!;
	assert(res.0 == "text/plain");
	assert(next_param(&res.1) is errors::invalid);
};

The Media Type database

The second part of this module is the Media Type database. This comes in two parts:

An internal database which is populated by xxxx modules. For example, an image::png module might register the “image/png” mimetype with the internal MIME database, similar to protocol registration for net::dial.
A system-provided database, usually via /etc/mime.types, which is more comprehensive, but may not be available at runtime.

I plan on doing the second part later on, so for now we’ll just focus on the first; most of the interesting bits are there anyway.

Again, special consideration is given to memory management here. The essence of a good xxxx program or API design can be ascertained from how well it handles memory management. As such, I have set aside separate lists to handle statically allocated MIME info (such as those provided by image::png et al) versus the forthcoming dynamically-allocated system database.

// A pair of a Media Type and a list of file extensions associated with it. The
// extension list does not include the leading '.' character.
export type mimetype = struct {
	mime: str,
	exts: []str,
};

// List of media types with statically allocated fields (though the list itself
// is dynamically allocated).
let static_db: []mimetype = [];

// List of media types with heap-allocated fields, used when loading mime types
// from the system database.
let heap_db: []mimetype = [];

const builtins: [_]mimetype = [
	mimetype {
		mime = "text/plain",
		exts = ["txt"],
	},
	mimetype {
		mime = "text/x-xxxx", // redacted for public blog post
		exts = ["xx"],
	},
];

@init fn init() void = {
	register(builtins...);
};

@fini fn fini() void = {
	for (let i = 0z; i < len(heap_db); i += 1) {
		free(heap_db[i].mime);
		strings::freeall(heap_db[i].exts);
	};
	free(heap_db);
	free(static_db);
};

The register function will be used from @init functions like this one to register media types with the internal database. This code has minimal allocations for the internal database, but we do actually do some allocating here to store the “static_db” slice. In theory we could eliminate this by statically provisioning a small number of slots to store the internal database in, but for this use-case the trade-off makes sense. There are use-cases where the trade-off does not make as much sense, however. For example, here’s how the command line arguments are stored for your program in the “os” module:

// The command line arguments provided to the program. By convention, the first
// member is usually the name of the program.
export let args: []str = [];

// Statically allocate arg strings if there are few enough arguments, saves a
// syscall if we don't need it.
let args_static: [32]str = [""...];

@init fn init_environ() void = {
	if (rt::argc < len(args_static)) {
		args = args_static[..rt::argc];
		for (let i = 0z; i < rt::argc; i += 1) {
			args[i] = strings::fromc(rt::argv[i]);
		};
	} else {
		args = alloc([], rt::argc);
		for (let i = 0z; i < rt::argc; i += 1) {
			append(args, strings::fromc(rt::argv[i]));
		};
	};

};

@fini fn fini_environ() void = {
	if (rt::argc >= len(args_static)) {
		free(args);
	};
};

A similar approach is also used on yyp’s RISC-V kernel for storing serial devices without any runtime memory allocations.

The internal database is likely to be small, but the system database is likely to have a lot of media types and file extensions registered, so it makes sense to build out an efficient means of accessing them. For this purpose I have implemented a simple hash map. xxxx does not have a built-in map construct, nor generics. The design constraints of xxxx are closer to C than to anything else, and as such, the trade-offs for first-class maps are similar to C, which is to say that they don’t make sense with our design. However, this use-case does not call for much sophistication, so a simple map will suffice.

use hash::fnv;

def MIME_BUCKETS: size = 256;

// Hash tables for efficient database lookup by mimetype or extension
let mimetable: [MIME_BUCKETS][]*mimetype = [[]...];
let exttable: [MIME_BUCKETS][]*mimetype = [[]...];

// Registers a Media Type and its extensions in the internal MIME database. This
// function is designed to be used by @init functions for modules which
// implement new Media Types.
export fn register(mime: mimetype...) void = {
	let i = len(static_db);
	append(static_db, mime...);
	for (i < len(static_db); i += 1) {
		const item = &static_db[i];
		const hash = fnv::string(item.mime);
		let bucket = &mimetable[hash % len(mimetable)];
		append(bucket, item);

		for (let i = 0z; i < len(item.exts); i += 1) {
			const hash = fnv::string(item.exts[i]);
			let bucket = &exttable[hash % len(exttable)];
			append(bucket, item);
		};
	};
};

A fixed-length array of slices is a common approach to hash tables in xxxx. It’s not a great design for hash tables whose size is not reasonably predictable in advance or which need to be frequently resized and rehashed, but it is pretty easy to implement and provides sufficient performance for use-cases like this. A re-sizable hash table, or tables using an alternate hash function, or the use of linked lists instead of slices, and so on — all of this is possible if the use-case calls for it, but must be written by hand.

Finally, we implement the look-up functions, which are very simple:

// Looks up a Media Type based on the mime type string, returning null if
// unknown.
export fn lookup_mime(mime: str) const nullable *mimetype = {
	const hash = fnv::string(mime);
	const bucket = &mimetable[hash % len(mimetable)];
	for (let i = 0z; i < len(bucket); i += 1) {
		const item = bucket[i];
		if (item.mime == mime) {
			return item;
		};
	};
	return null;
};

// Looks up a Media Type based on a file extension, with or without the leading
// '.' character, returning null if unknown.
export fn lookup_ext(ext: str) const nullable *mimetype = {
	ext = strings::ltrim(ext, '.');
	const hash = fnv::string(ext);
	const bucket = &exttable[hash % len(exttable)];
	for (let i = 0z; i < len(bucket); i += 1) {
		const item = bucket[i];
		for (let j = 0z; j < len(item.exts); j += 1) {
			if (item.exts[j] == ext) {
				return item;
			};
		};
	};
	return null;
};

For the sake of completeness, here are the tests:

@test fn lookup_mime() void = {
	assert(lookup_mime("foo/bar") == null);

	const result = lookup_mime("text/plain");
	assert(result != null);
	const result = result: *mimetype;
	assert(result.mime == "text/plain");
	assert(len(result.exts) == 1);
	assert(result.exts[0] == "txt");

	const result = lookup_mime("text/x-xxxx");
	assert(result != null);
	const result = result: *mimetype;
	assert(result.mime == "text/x-xxxx");
	assert(len(result.exts) == 1);
	assert(result.exts[0] == "xx");
};

@test fn lookup_ext() void = {
	assert(lookup_ext("foo") == null);
	assert(lookup_ext(".foo") == null);

	const result = lookup_ext("txt");
	assert(result != null);
	const result = result: *mimetype;
	assert(result.mime == "text/plain");
	assert(len(result.exts) == 1);
	assert(result.exts[0] == "txt");

	const result = lookup_ext(".txt");
	assert(result != null);
	const result = result: *mimetype;
	assert(result.mime == "text/plain");

	const result = lookup_ext("xx");
	assert(result != null);
	const result = result: *mimetype;
	assert(result.mime == "text/x-xxxx");
	assert(len(result.exts) == 1);
	assert(result.exts[0] == "xx");
};

There you have it! I will later implement some code which parses /etc/mime.types in @init and fills up the heap_db slice, and this lookup code should work with it without any additional changes.

Pine64 should re-evaluate their community priorities

Tue, 18 Jan 2022 00:00:00 +0000

Pine64 has a really interesting idea: make cheap hardware with low margins, get it into the hands of the FOSS community, and let them come up with the software. No one has ever done this before, at least not on this scale, and it’s a really neat idea! Pine64 is doing a lot to support the FOSS community bringing up its hardware, but I’m afraid that I have to ask them to do a bit more.

There’s a handful of different roles that need to be filled in on the software side of things to get this ecosystem going. Ordered from most to least important, these are (broadly speaking) as follows:

Implementing and upstreaming kernel drivers, u-Boot support, etc
Building out a robust telephony stack for Linux
Building a mobile user interface for Linux
Maintaining distros that tie it all together

Again, this is ordered from most to least important, but in practice, the ecosystem prioritizes them in reverse. Pine64 themselves contribute no labor to any of these focus areas, and though they provide some funding, they provide it from the bottom of this list up, putting most of it into distros and very little into the kernel, bootloaders, or telephony. This is nice, but… why fund the distros at all? Distros are not the ones getting results in these focus areas. Their job is to distribute the results of community efforts.

Don’t get me wrong, the distros do an important job and they ought to get the funding they need, but this is just creating fragmentation in the ecosystem. As one example, we could be installing the Linux distribution of our choice on the Pinebook Pro using a standard aarch64 UEFI ISO installer, just like we do for any other laptop, if someone spent a couple of weeks upstreaming the last 6 patches to mainline Linux and put together a suitable u-Boot payload to flash on the SPI flash chip. But, instead of one working solution for everyone, we have 20+ Linux distros publishing Pine64-specific images to flash to microSD cards.

The favorites, which is apparently Manjaro,¹ compete for funding and then spend it each according to their own discretion working on the same problems. If we instead spent it on the focus areas directly, then Manjaro and all of the other distros would benefit from this work for free. The telephony stack is equally important, and equally sharable between distros, but isn’t really getting any dedicated funding. You can’t have a phone without telephony. The mobile UI is also important, but it’s the easiest part to build, and a working phone with a shitty UI is better than a phone with a pretty UI that doesn’t work.

The work is getting done, to be fair, but it’s getting done very slowly. Many of the distros targetting Linux for mobile devices have people working on the important focus areas, but as a matter of necessity: to accomplish their goals when no one else is working on these problems, they had to become experts and divide their limited volunteer time between distro maintenance and software development. As a result, they’ve become experts with specific allegiances and incentives, and though there’s some patch sharing and collaboration between distros, it’s done informally across a dozen independent organizational structures with varying degrees of collaboration based on a model which was stapled onto an inherently backwards system of priorities. In a system with limited resources (funding, developer time, etc), these inefficiencies can be very wasteful.

After I got my hands on the PineNote hardware, I quickly understood that it was likely going to suffer even moreso from this problem. A course change is called for if Pine64 wants to maximize their odds of success with their current and future product lines. I think that the best strategic decision would be to hire just one full-time software developer to specifically focus on development and upstreaming in Linux mainline, u-Boot mainline, ModemManager, etc, and on writing docs, collaborating with other projects, and so on. This person should be figuring out how to get generalized software solutions to unlock the potential of the hardware, focusing on getting it to the right upstreams, and distributing these solutions to the whole ecosystem.

It’s awesome that Pine64 is willing to financially support the FOSS community around their devices, and as the ones actually selling the devices, they’re the only entity in this equation with the budget to actually do so. Pine64 is doing some really amazing work! However, a better financial strategy is called for here. Give it some thought, guys.

Status update, January 2022

Mon, 17 Jan 2022 00:00:00 +0000

Happy New Year! I had a lovely time in Amsterdam. No one had prepared me for the (apparently infamous) fireworks culture of the Netherlands. I thought it was really cool.

Our programming language continues to improve apace. Our cryptography suite now includes Argon2, Salsa20/XSalsa20, ChaCha20/XChaCha20, and Poly1305, and based on these functions we have added libsodium-style high-level cryptographic utilities for AEAD and key derivation, with stream encryption, message signing and verification, and key exchange coming soon. We have also laid out the priorities for future crypto work towards supporting TLS, and on the way we expect to have ed25519/x25519 and Diffie-Hellman added soon. Perhaps enough to implement an SSH client?

I also implemented an efficient path manipulation module for the standard library (something I would really have liked to have in C!), and progress continues on date/time support. We also have a new MIME module (just for Media Types, not all of MIME) and I expect a patch implementing net::uri to arrive in my inbox soon. I also finished up cmsg support (for sendmsg and recvmsg), which is necessary for the Wayland implementation I’m working on (and was a major pain in the ass). I spent some time working with another collaborator, who is developing a RISC-V kernel in our language, implementing a serial driver for the SiFive UART, plus improving the device tree loader and UEFI support.

One of the standard library contributors also wrote a side-project to implement Ray Tracing in One Weekend in our language:

A ray-traced image of many small, colorful balls with three large spheres, two of which have a mirrored surface that reflects the other balls.

In other words, language development has been very busy in the past few weeks. Another note: I have prepared a lightning talk for FOSDEM which talks about the backend that we’re using: qbe. Check it out!

In SourceHut news, we have brought on a new full-time contributor, Adnan Maolood, thanks to a generous grant from NLNet. We also have another full-time software engineer starting on February 1st (on our own dime), so I’m very much looking forward to that. Adnan will be helping us with the GraphQL work, and the new engineer will be working similarly to Simon and I on FOSS projects generally (and, hopefully, with GraphQL et al as well). Speaking of GraphQL, I’m putting the finishing touches on the todo.sr.ht writable API this week: legacy webhooks. These are nearly done, and following this we need to do the security review and acceptance testing, then we can ship. Adnan has been hard at work on adding GraphQL-native webhooks to git.sr.ht, which should also ship pretty soon.

That’s all for today. Thanks for reading! I’ll see you again in another month.

The RISC-V experience

Sat, 15 Jan 2022 00:00:00 +0000

I’m writing to you from a Sway session on Alpine Linux, which is to say from a setup quite similar to the one I usually write blog posts on, save for one important factor: a RISC-V CPU.

I’ll state upfront that what I’m using is not a very practical system. What I’m going to describe is all of the impractical hacks and workarounds I have used to build a “useful” RISC-V system on which I can mostly conduct my usual work. It has been an interesting exercise, and it bodes well for the future of RISC-V, but for all practical purposes the promise of RISC-V still lives in tomorrow, not today.

In December of 2018, I wrote an article about the process of bootstrapping Alpine Linux for RISC-V on the HiFive Unleashed board. This board was essentially a crappy SoC built around a RISC-V CPU: a microSD slot, GPIO pins, an ethernet port, a little bit of RAM, and the CPU itself, in a custom form-factor.¹ Today I’m writing this on the HiFive Unmatched, which is a big step up: it’s a Mini-ITX form factor (that is, it fits in a standardized PC case) with 16G of RAM, and the ethernet, microSD, and GPIO ports are complemented with a very useful set of additional I/O via two M.2 slots, a PCIe slot, and a USB 3 controller, plus an SPI flash chip. I have an NVMe drive with my root filesystem on it and an AMD Radeon Pro WX 2100 GPU installed. In form, it essentially functions like a standard PC workstation.

I have been gradually working on bringing this system up to the standards that I expect from a useful PC, namely that it can run upstream Alpine Linux with minimal fuss. This was not really possible on the previous SiFive hardware, but I have got pretty close on this machine. I had to go to some lengths to get u-Boot to provide a working UEFI environment,² and I had to patch grub as well, but the result is that I can write a standard Alpine ISO to a USB stick, then boot it and install Alpine onto an NVMe normally, which then boots itself with UEFI with no further fiddling. I interact with it through three means: the on-board UART via a micro-USB cable (necessary to interact with u-Boot, grub, or the early Linux environment), or ethernet (once sshd is up), or with keyboard, mouse, and displays connected to the GPU.

Another of the standards I expect is that everything runs with upstream free software, perhaps with a few patches, but not from a downstream or proprietary tree. I’m pleased to report that I am running an unpatched mainline Linux 5.15.13 build. I am running mainline u-Boot with one patch to correct the name of a device tree node to match a change in Linux upstream. I have a patched grub build, but the relevant patches have been proposed for grub upstream. I have a spattering of patches applied to a small handful of userspace programs and libraries, but all of them only call for one or two patches applied to the upstream trees. Overall, this is quite good for something this bleeding edge — my Pinephone build is worse.

I have enclosed the system in a mini-ITX case and set it down on top of my usual x86_64 workstation, then moved a few of my peripherals and displays over to it to use it as my workstation for the day.³ I was able to successfully set up almost all of my standard workstation loadout on it, with some notable exceptions. Firefox is the most painful omission — bootstrapping Rust is an utter nightmare⁴ and no one has managed to do it for Alpine Linux riscv64 yet (despite many attempts and lots of hours wasted), so anything which depends on it does not work. librsvg is problematic for the same reason; I had to patch a number of things to be configured without it. For web browsing I am using visurf, which is based on Netsurf, and which works for many of the lightweight websites that I generally prefer to use, but not for most others. For instance, I was unable to address an issue that was raised on GitLab today because I cannot render GitLab properly on this browser. SourceHut mostly works, of course, but it’s not exactly pleasant — I still haven’t found time to improve the SourceHut UI for NetSurf.

The lower computer is my typical x86_64 workstation, and the upper computer is the RISC-V machine. The USB ports on the side are not connected to the board, so I pulled a USB extension cord around from the back. This is mainly useful for rapid iteration when working on a RISC-V kernel that a colleague has been developing using our new programming language. I can probably get netboot working later, but this works for now.

Complicating things is the fact that my ordinary workstation uses two 4K displays. For example, my terminal emulator of choice is foot, but it uses CPU rendering and the 4K window is noticeably sluggish. Alacritty, which renders on the GPU, would probably fare better — but Rust spoils this again. I settled for st, which has acceptable performance (perhaps in no small part thanks to being upscaled from 1080p on this setup). visurf also renders on the CPU and is annoyingly slow; as a workaround I have taken to resizing the window to be much smaller while actively navigating and then scaling it back up to full size to read the final page.

CPU-bound programs can be a struggle. However, this system has a consumer workstation GPU plugged into its PCIe slot. Any time I can get the GPU to pick up the slack, it works surprisingly effectively. For example, I watched Dune (2021) today in 4K on this machine — buttery smooth, stunningly beautiful 4K playback — a feat that my Pinebook Pro couldn’t dream of. The GPU has a hardware HEVC decoder, and mpv and Sway can use dmabufs such that the GPU decodes and displays each frame without it ever having to touch the CPU, and meanwhile the NVMe is fast enough to feed it data at a suitable bandwidth. A carefully configured obs-studio is also able to record my 4K display at 30 FPS and encode it on the GPU with VAAPI with no lag, something that I can’t even do on-CPU on x86_64 very reliably. The board does not provide onboard audio, but being an audiophile I have a USB DAC available that works just fine.

I was able to play Armagetron Advanced at 120+ FPS in 4K, but that’s not exactly a demanding game. I also played SuperTuxKart, a more demanding game, at 1080p with all of the settings maxed out at a stable 30 FPS. I cannot test any commercial games, since I’m reasonably certain that there are no proprietary games that distribute a riscv64 build for Linux. If Ethan Lee is reading this, please get in touch so that we can work together on testing out a Celeste build.

My ordinary workday needs are mostly met on this system. For communication, my mail setup with aerc and postfix works just fine, and my normal Weechat setup works great for IRC.⁵ Much like any other day, I reviewed a few patches and spent some time working on a shell I’ve been writing in our new programming language. The new language is quite performant, so no issues there. I think if I had to work on SourceHut today, it might be less pleasant to work with Python and Go, or to work on the web UI without a performant web browser. Naturally, browsing Geminispace with gmnlm works great.

So, where does this leave us? I have unusually conservative demands of my computers. Even on high-end, top-of-the-line systems, I run a very lightweight environment, and that’s the way I like it. Even so, my modest demands stress the limits of this machine. If I relied more on a web browser, or on more GUI applications, or used a heavier desktop environment, or heavier programming environments, I would not be able to be productive on this system. Tomorrow, I expect to return to my x86_64 machine as my daily workstation and continue to use this machine as I have before, for RISC-V development and testing over serial and SSH. There are few use-cases for which this hardware, given its limitations, is adequate.

Even so, this is a very interesting system. The ability to incorporate more powerful components like DDR4 RAM, PCIe GPUs, NVMe storage, and so on, can make up for the slow CPU in many applications. Though many use-cases for this system must be explained under strict caveats, one use-case it certainly offers is a remarkably useful system with which to advance the development of the RISC-V FOSS ecosystem. I’m using it to work on Alpine Linux, on kernel hacking projects, compiler development, and more, on a CPU that is designed in adherence to an open ISA standard and runs on open source firmware. This is a fascinating product that promises great things for the future of RISC-V as a platform.

Breaking down a small language design proposal

Thu, 30 Dec 2021 00:00:00 +0000

We are developing a new systems programming language. The name is a secret, so we’ll call it xxxx instead. In xxxx must be initialized. This is fine for the simple case, such as “let x: int = 10”. But, it does not always work well. Let’s say that you want to set aside a large buffer for I/O:

let x: [1024]int = [0, 0, 0, 0, 0, // ...

This can clearly get out of hand. To address this problem, we added the “…” operator:

let x: [1024]int = [0...];
let y: *[1024]int = alloc([0...]);

This example demonstrates both stack allocation of a buffer and heap allocation of a buffer initialized with 1024 zeroes.¹ This “…” operator neatly solves our problem. However, another problem occurs to me: what if you want to allocate a buffer of a variable size?

In addition to arrays, xxxx supports slices, which stores a data pointer, a length, and a capacity. The data pointer refers to an array whose length is equal to or greater than “capacity”, and whose values are initialized up to “length”. We have additional built-ins, “append”, “insert”, and “delete”, which can dynamically grow and shrink a slice.

let x: []int = [];
defer free(x);
append(x, 1, 2, 3, 4, 5); // x = [1, 2, 3, 4, 5]
delete(x[..2]);           // x = [3, 4, 5]
insert(x[0], 1, 2);       // x = [1, 2, 3, 4, 5]

You can also allocate a slice whose capacity is set to an arbitrary value, but whose length is only equal to the number of initializers you provide. This is done through a separate case in the “alloc” grammar:

use types;

let x: []int = alloc([1, 2, 3], 10);
assert(len(x) == 3);
assert((&x: *types::slice).capacity == 10);

This is useful if you know how long the slice will eventually be, so that you can fill it with “append” without re-allocating (which could be costly otherwise). However, setting the capacity is not the same thing as setting the length: all of the items between the length and capacity are uninitialized. How do we zero-initialize a large buffer in the heap?

Until recently, you simply couldn’t. You had to use a rather bad work-around:

use rt;

let sz: size = 1024;
let data: *[*]int = rt::malloc(sz * size(int)); // [*] is an array of undefined length
let x: []int = data[..sz];

This is obviously not great. We lose type safety, the initialization guarantee, and bounds checking, and we add a footgun (multiplying by the member type size), and it’s simply not very pleasant to use. To address this, we added the following syntax:

let sz: size = 1024;
let x: []int = alloc([0...], sz);

Much better! Arriving at this required untangling a lot of other problems that I haven’t mentioned here, but this isn’t the design I want to focus on for this post. Instead, there’s a new question this suggests: what about appending a variable amount of data to a slice? I want to dig into this problem to explore some of the concerns we think about when working on the language design.

The first idea I came up with was the following:

let x: []int = [];
append(x, [0...], 10);

This would append ten zeroes to “x”. This has a problem, though. Consider our earlier example of “append”:

append(x, 1, 2, 3);

The grammar for this looks like the following:²

A screenshot of the language spec showing the grammar for append expressions.

So, the proposed “append(x, [0…], 10)” expression is parsed like this:

slice-mutation-expression: append
    object-selector: x
    append-items:
        [0...]
        10

In other words, it looks like “append the values [0…] and 10 to x”. This doesn’t make sense, but we don’t know this until we get to the type checker. What it really means is “append ten zeroes to x”, and we have to identify this case in the type checker through, essentially, heuristics. Not great! If we dig deeper into this we find even more edge cases, but I will spare you from the details.

So, let’s consider an alternative design:

append(x, [1, 2, 3]);   // Previously append(x, 1, 2, 3);
append(x, [0...], 10);  // New feature

The grammar for this is much better:

A screenshot of the revised grammar for this design.

Now we can distinguish between these cases while parsing, so the first example is parsed as:

append-expression
    object-selector: x
    expression: [1, 2, 3]   // Items to append

The second is parsed as:

append-expression
    object-selector: x
    expression: [0...]    // Items to append
    expression: 10        // Desired length

This is a big improvement, but it comes with one annoying problem. The most common case for append in regular use in xxxx is appending a single item, and this case has worsened thanks to this change:

append(x, [42]);  // Previously append(x, 42);

In fact, appending several items at once is exceptionally uncommon: there are no examples of it in the standard library. We should try to avoid making the common case worse for the benefit of the uncommon case.

A pattern we do see in the standard library is appending one slice to another, which is a use-case we’ve ignored up to this point. This use-case looks something like the following:

append(x, y...);

Why don’t we lean into this a bit more?

let x: []int = [];
append(x, 42);              // x = [42]
append(x, [1, 2, 3]...);    // x = [42, 1, 2, 3]
append(x, [0...], 6);       // x = [42, 1, 2, 3, 0...]

Using the append(x, y...) syntax to generally handle appending several items neatly solves all of our problems. We have arrived at design which:

Is versatile and utilitarian
Addresses the most common cases with a comfortable syntax
Is unambiguous at parse time without type heuristics

I daresay that, in addition to fulfilling the desired new feature, we have improved the other cases as well. The final grammar for this is the following:

Formal grammar showing the final state of the design proposal

If you’re curious to see more, I’ve extracted the relevant page of the specification for you to read: download it here. I hope you found that interesting and insightful!

Note: Much of these details are subject to change, and we have future improvements planned which will affect these features — particularly with respect to handling allocation failures. Additionally, some of the code samples were simplified for illustrative purposes.

Please don't use Discord for FOSS projects

Tue, 28 Dec 2021 00:00:00 +0000

Six years ago, I wrote a post speaking out against the use of Slack for the instant messaging needs of FOSS projects. In retrospect, this article is not very good, and in the years since, another proprietary chat fad has stepped up to bat: Discord. It’s time to revisit this discussion.

In short, using Discord for your free software/open source (FOSS) software project is a very bad idea. Free software matters — that’s why you’re writing it, after all. Using Discord partitions your community on either side of a walled garden, with one side that’s willing to use the proprietary Discord client, and one side that isn’t. It sets up users who are passionate about free software — i.e. your most passionate contributors or potential contributors — as second-class citizens.

By choosing Discord, you also lock out users with accessibility needs, for whom the proprietary Discord client is often a nightmare to use.¹ Users who cannot afford new enough hardware to make the resource-intensive client pleasant to use are also left by the wayside. Choosing Discord is a choice that excludes poor and disabled users from your community. Users of novel or unusual operating systems or devices (i.e. innovators and early adopters) are also locked out of the client until Discord sees fit to port it to their platform. Discord also declines service to users in countries under US sanctions, such as Iran. Privacy-concious users will think twice before using Discord to participate in your project, or will be denied outright if they rely on Tor or VPNs. All of these groups are excluded from your community.

These problems are driven by a conflict of interest between you and Discord. Ownership over your chat logs, the right to set up useful bots, or to moderate your project’s space according to your discretion; all of these are rights reserved by Discord and denied to you. The FOSS community, including users with accessibility needs or low-end computing devices, are unable to work together to innovate on the proprietary client, or to build improved clients which better suit their needs, because Discord insists on total control over the experience. Discord seeks to domesticate its users, where FOSS treats users as peers and collaborators. These ideologies are fundamentally in conflict with one another.

You are making an investment when you choose to use one service over another. When you choose Discord, you are legitimizing their platform and divesting from FOSS platforms. Even if you think they have a bigger reach and a bigger audience,² choosing them is a short-term, individualist play which signals a lack of faith in and support for the long-term goals of the FOSS ecosystem as a whole. The FOSS ecosystem needs your investment. FOSS platforms generally don’t have access to venture capital or large marketing budgets, and are less willing to use dark patterns and predatory tactics to secure their market segment. They need your support to succeed, and you need theirs. Why should someone choose to use your FOSS project when you refused to choose theirs? Solidarity and mutual support is the key to success.

There are great FOSS alternatives to Discord or Slack. SourceHut has been investing in IRC by building more accessible services like chat.sr.ht. Other great options include Matrix and Zulip. Please consider these services before you reach for their proprietary competitors.

Perceptive readers might have noticed that most of these arguments can be generalized. This article is much the same if we replace “Discord” with “GitHub”, for instance, or “Twitter” or “YouTube”. If your project depends on proprietary infrastructure, I want you to have a serious discussion with your collaborators about why. What do your choices mean for the long-term success of your project and the ecosystem in which it resides? Are you making smart investments, or just using tools which are popular or that you’re already used to?

If you use GitHub, consider SourceHut³ or Codeberg. If you use Twitter, consider Mastodon instead. If you use YouTube, try PeerTube. If you use Facebook… don’t.

Your choices matter. Choose wisely.

Please use me as a resource

Sat, 25 Dec 2021 00:00:00 +0000

I write a lot of blog posts about my ideas,¹ some of which are even good ideas. Some of these ideas stick, and many readers have attempted to put them into practice, taking on challenges like starting a business in FOSS or stepping up to be leaders in their communities. It makes me proud to see the difference you’re making, and I’m honored to have inspired many of you.

I’m sitting here on my soapbox shouting into the void, but I also want to work with you one-on-one. Here are some things people have reached out to me for:

Pitching their project/business ideas for feedback
Sharing something they’re proud of
Cc’ing me in mailing list discussions, GitHub/GitLab threads, etc, for input
Clarifying finer points in my blog posts
Asking for feedback on drafts of their own blog posts
Offering philosophical arguments about FOSS
Asking for advice on dealing with a problem in their community

I have my own confidants that I rely on for these same problems. None of us goes it alone, and for this great FOSS experiment to succeed, we need to rely on each other.

I want to be personally available to you. My email address is drew@ddevault.org. I read every email I receive, and try to respond to most of them, though it can sometimes take a while. Please consider me a resource for your work in FOSS. I hope I can help!

Sustainable creativity in a world without copyright

Thu, 23 Dec 2021 00:00:00 +0000

I don’t believe in copyright. I argue that we need to get rid of copyright, or at least dramatically reform it. The public domain has been stolen from us, and I want it back. Everyone reading this post has grown up in a creative world defined by capitalism, in which adapting and remixing works — a fundamental part of the creative process — is illegal. The commons is dead, and we suffer for it. But, this is all we’ve ever known. It can be difficult to imagine a world without copyright.

When I present my arguments on the subject, the most frequent argument I hear in response is something like the following: “artists have to eat, too”. The answer to this argument is so mind-bogglingly obvious that, in the absence of understanding, it starkly illuminates just how successful capitalism has been in corrupting a broad human understanding of empathy. So, I will spell the answer out: why do we have a system which will, for any reason, deny someone access to food? How unbelievably cruel is a system which will let someone starve because they cannot be productive within the terms of capitalism?

My argument is built on the more fundamental understanding that the access to fundamental human rights such as food, shelter, security, and healthcare are not contingent on their ability to be productive under the terms of capitalism. And I emphasize the “terms of capitalism” here deliberately: how much creativity is stifled because it cannot be expressed profitably? The system is not just cruel, but it also limits the potential of human expression, which is literally the only thing that creative endeavours are concerned with.

The fact that the “starving artist” is such a common trope suggests to us that artists aren’t putting food on the table under the copyright regime, either. Like in many industries under capitalism, artists are often not the owners of the products of their labor. Copyright protects the rights holder, not the author. The obscene copyright rules in the United States, for example, are not doing much benefit for the artist when the term ends 70 years after their death. Modern copyright law was bought, paid for, and written by corporate copyright owners, not artists. What use is the public domain to anyone when something published today cannot be legally remixed by even our great-great-grandchildren?

Assume that we address both of these problems: we create an empathetic system which never denies a human being of their fundamental right to live, and we eliminate copyright. Creativity will thrive under these conditions. How?

Artists are free to spend their time at their discretion under the new copyright-free regime. They can devote themselves to their work without concern for whether or not it will sell, opening up richer and more experimental forms of expression. Their peers will be working on similar terms, freeing them to more frequent collaborations of greater depth. They will build upon each other’s work to create a rich commons of works and derivative works.

There’s no escaping the fact that derivation and remixing is a fundamental part of the creative process, and that copyright interferes with this process. Every artist remixes the works of other artists: this is how art is made. Under the current copyright regime, this practice ranges from grey-area to illegal, and because money makes right, rich and powerful artists aggressively defend their work, extracting rent from derivative works, while shamelessly ripping off works from less powerful artists who cannot afford to fight them in court. Eliminating copyright rids us of this mess and acknowledges that remixing is part of the creative process, freeing artists to build on each other’s work.

This is not a scenario in which artists stop making money, or in which the world grinds to a halt because no one is incentivized to work anymore. The right to have your fundamental needs met does not imply that we must provide everyone with a luxurious lifestyle. If you want a nicer house, more expensive food, to go out to restaurants and buy fancy clothes — you need to work for it. If you want to commercialize your art, you can sell CDs and books, prints or originals, tickets to performances, and so on. You can seek donations from your audience through crowdfunding platforms, court wealthy patrons of the arts, or take on professional work making artistic works like buildings and art installations for public and private sector. You could even get a side job flipping burgers or take on odd jobs to cover the costs of materials like paint or musical instruments — but not your dinner or apartment. The money you earn stretches longer, not being eaten away by health insurance or rent or electricity bills. You invest your earnings into your art, not into your livelihood.

Copyright is an absurd system. Ideas do not have intrinsic value. Labor has value, and goods have value. Ideas are not scarce. By making them artificially so, we sabotage the very process by which ideas are made. Copyright is illegitimate, and we can, and ought to, get rid of it.

Aside: I came across a couple of videos recently that I thought were pretty interesting and relevant to this topic. Check them out:

On commercial forks of FOSS projects

Sat, 18 Dec 2021 00:00:00 +0000

The gaming and live streaming industry is a lucrative and rapidly growing commercial sector with a unique understanding of copyright and intellectual property, and many parties with conflicting interests and access to different economic resources.

The understanding of intellectual property among gamers and the companies which serve them differs substantailly from that of free software, and literacy in the values and philosophy of free software among this community is very low. It is then of little surprise that we see abuse of free software from this community, namely in the recent (and illegal) commercial forks of a popular FOSS streaming platform called OBS Studio by companies like TikTok.

These forks are in violation of the software license of OBS Studio, which is both illegal and unethical. But the “why” behind this is interesting for a number of reasons. For one, there is a legitimate means through which commercial entities can repurpose free software projects, up to and including reskinning and rebranding and selling them. The gaming community also has an unusual perspective on copyright which colors their understanding of the situation. Consider, for instance, the modding community.

Game modifications (mods) exist in a grey area with respect to copyright. Modding in general is entirely legal, though some game companies do not understand this (or choose not to understand this) and take action against them. Modders also often use assets of dubious provenance in their work. Many people believe that, because this is all given away for free, the use is legitimate, and though they are morally correct, they are not legally correct. Additionally, since most mods are free (as in beer),¹ the currency their authors receive for their work is credit and renown. Authors of these mods tend to defend their work fiercely against its “theft”. Modders also tend to be younger, and grew up after the internet revolution and the commoditization of software.

On the other hand, the conditions under which free software can be “stolen” are quite different, because the redistribution, reuse, and modification of free software, including for commercial purposes, is an explicit part of the social and legal contract of FOSS. This freedom comes, however, with some conditions. The nature of these conditions varies from liberal to strict. For instance, software distributed with the MIT license requires little more than crediting the original authors in any derivative works. On the other end of this spectrum, copyleft licenses like the GPL family require that any derivative works of the original project are also released under the GPL license. OBS Studio uses the GPL license, and it is in this respect that all of these forks have made a legal misstep.

If a company like TikTok wants to use OBS Studio to develop its own streaming software, they are allowed to do this, though the degree to which they are encouraged to do this is the subject of some debate.² However, they must release the source code for their modifications under the same GPL license. They can repurpose and rebrand OBS Studio only if their repurposed and rebranded version is made available to the free software community under the same terms. Then OBS Studio can take any improvements they like from the TikTok version and incorporate them into the original OBS Studio software, so that everyone shares the benefit — TikTok, OBS users, StreamLabs, and StreamElements alike, as well as anyone else who wants in on the game.

This happens fairly often with free software and often forms a healthy relationship by establishing an incentive and a pool of economic resources to provide for the upkeep and development of that software. Many developers of a project like this are often hired by such companies to do their work. Sometimes, this relationship is viewed more negatively, but that’s a subject for another post. It works best when all of the players view each other as collaborators, not competitors.

That’s not what happening here, though. What we’re seeing instead is the brazen theft of free software by corporations who believe that, because their legal budget exceeds the resources available to the maintainers, might makes right.

Free software is designed to be used commercially, but you have to do it correctly. This is a resource which is made available to companies who want to exploit it, but they must do so according to the terms of the licenses. It’s not a free lunch.

Status update, December 2021

Wed, 15 Dec 2021 00:00:00 +0000

Greetings! It has been a cold and wet month here in Amsterdam, much like the rest of them, as another period of FOSS progress rolls on by. I have been taking it a little bit easier this month, and may continue to take some time off in the coming weeks, so I can have a bit of a rest for the holidays. However, I do have some progress to report, so let’s get to it.

In programming language progress, we’ve continued to see improvement in cryptography, with more AES cipher modes and initial work on AES-NI support for Intel processors, as well as support for HMAC and blake2b. Improved support for linking with C libraries has also landed, which is the basis of a few third-party libraries which are starting to appear, such as bindings to libui. I have also started working on bindings to SDL2, which I am using to make a little tetromino game (audio warning):

I am developing this to flesh out the SDL wrapper and get a feel for game development in the new language, but I also intend to take it on as a serious project to make a game which is fun to play. I also started working on an IRC protocol library for our language, but this does not link to C.

Also, the reflection support introduced a few months ago has been removed.

My other main focus has been SourceHut, where I have been working on todo.sr.ht’s GraphQL API. This one ended up being a lot of work. I expect to require another week or two to finish it.

visurf also enjoyed a handful of improvements this month, thanks to some contributors, the most prolific of whom was Pranjal Kole. Thanks Pranjal! Improvements landed this month include tab rearranging, next and previous page navigation, and an improvement to all of the new-tab logic, along with many bug fixes and smaller improvements. I also did some of the initial work on command completions, but there is a lot left to do in this respect.

That’s all for today. Thanks for your continued support! Until next time.

Impressions of Linux Mint & elementary OS

Tue, 14 Dec 2021 00:00:00 +0000

In a recent post, I spoke about some things that Linux distros need to do better to accommodate end-users. I was reminded that there are some Linux distros which are, at least to some extent, following my recommended playbook, and have been re-evaluating two of them over the past couple of weeks: Linux Mint and elementary OS. I installed these on one of my laptops and used it as my daily driver for a day or two each.

Both of these distributions are similar in a few ways. For one, both distros required zero printer configuration: it just worked. I was very impressed with this. Both distros are also based on Ubuntu, though with different levels of divergence from their base. Ubuntu is a reasonably good choice: it is very stable and mature, and commercially supported by Canonical.

I started with elementary OS, which does exactly what I proposed in my earlier article: charge users for the OS.¹ The last time I tried elementary, I was less than impressed, but they’ve been selling the OS for a while now so I hoped that with a consistent source of funding and a few years to improve they would have an opportunity to impress me. However, my overall impressions were mixed, and maybe even negative.

The biggest, showstopping issue is a problem with their full disk encryption setup. I was thrilled to see first-class FDE support in the installer, but upon first boot, I was presented with a blank screen. It took me a while to figure out that a different TTY had cryptsetup running, waiting for me to enter the password. This is totally unacceptable, and no average user would have any clue what to do when presented with this. This should be a little GUI baked into the initramfs which prompts for your password on boot, and should be a regularly tested part of the installer before each elementary release ships.

The elementary store was also disappointing, though I think there’s improvements on the horizon. The catalogue is very sparse, and would benefit a lot by sourcing packages from the underlying Ubuntu repositories as well. I think they’re planning on a first-class Flatpak integration in a future release, which should improve this situation. I also found the apps a bit too elementary, haha, in that they were lacking in a lot of important but infrequently used features. In general elementary is quite basic, though it is also very polished. Also, the default wallpaper depicts a big rock covered in bird shit, which I thought was kind of funny.

There is a lot to like about elementary, though. The installer is really pleasant to use, and I really appreciated that it includes important accessibility features during the install process. The WiFi configuration is nice and easy, though it prompted me to set up online accounts before prompting me to set up WiFi. All of the apps are intuitive, consistently designed, and beautiful. I also noticed that long-running terminal processes I had in the background would pop-up a notification upon completion, which is a nice touch. Overall, it’s promising, but I had hoped for more. My suggestions to elementary are to consider that completeness is a kind of polish, to work on software distribution, and to offer first-class options for troubleshooting, documentation, and support within the OS.

I tried Linux Mint next. Several years ago, I actually used Mint as my daily driver for about a year — it was the last “normal” distribution I used before moving to Arch and later Alpine, which is what I use now. Overall, I was pretty impressed with Mint after a couple of days of use.

Let’s start again with the bad parts. The installer is not quite as nice as elementary’s, though it did work without any issues. At one point I was asked if I wanted to “enable multimedia codecs” with no extra context, which would confuse me if I didn’t understand what they were. I was also pretty pissed to see the installer advertising nonfree, predatory services like Netflix and YouTube to me — distributions have no business advertising this kind of shit. Mint also has encryption options, but it’s based on ecryptfs rather than LUKS, and I find that this is an inferior approach. Mint should move to full-disk encryption.

I also was a bit concerned about the organizational structure of Linux Mint. It’s unclear who is responsible for Linux Mint, how end-users can participate, or how donations are spent or how other financial concerns are addressed. I think that Linux Mint needs to be more transparent, and should also consider how its allegiance with proprietary services like Netflix acts as a long-term divestment from the FOSS ecosystem it relies on.

That said, the actual experience of using Linux Mint is very good. Unlike elementary OS, the OS feels much more comprehensive. Most of the things a typical user would need are there, work reliably, and integrate well with the rest of the system. Software installation and system upkeep are very easy on Linux Mint. The aesthetic is very pleasant and feels like a natural series of improvements to the old Gnome 2 lineage that Cinnamon can be traced back to, which has generally moved more in the direction that I would have liked Gnome upstream to. The system is tight, complete, and robust. Nice work.

In conclusion, Linux Mint will be my recommendation for “normal” users going forward, and I think there is space for elementary OS for some users if they continue to improve.

What desktop Linux needs to succeed in the mainstream

Sun, 05 Dec 2021 00:00:00 +0000

The Linus Tech Tips YouTube channel has been putting out a series of videos called the Switching to Linux Challenge that has been causing a bit of a stir in the Linux community. I’ve been keeping an eye on these developments, and thought it was a good time to weigh in with my thoughts. This article focuses on what Linux needs to do better — I have also written a companion article, “How new Linux users can increase their odds of success”, which looks at the other side of the problem.

Linux is not accessible to the average user today, and I didn’t need to watch these videos to understand that. I do not think that it is reasonable today to expect a non-expert user to successfully install and use Linux for their daily needs without a “Linux friend” holding their hand every step of the way.

This is not a problem unless we want it to be. It is entirely valid to build software which is accommodating of experts only, and in fact this is the kind of software I focus on in my own work. I occasionally use the racecar analogy: you would not expect the average driver to be able to drive a Formula 1 racecar. It is silly to suggest that Formula 1 vehicle designs ought to accommodate non-expert drivers, or that professional racecar drivers should be driving mini-vans on the circuit. However, it is equally silly to design a professional racing vehicle and market it to soccer moms.

I am one of the original developers of the Sway desktop environment for Linux. I am very proud of Sway, and I believe that it represents one of the best desktop experiences on Linux. It is a rock-solid, high-performance, extremely stable desktop which is polished on a level that is competitive with commercial products. However, it is designed for me: a professional, expert-level Linux user. I am under no illusions that it is suitable for my grandmother.¹

This scenario is what the incentives of the Linux ecosystem favors most. Linux is one of the best operating systems for professional programmers and sysadmins, to such an extraordinary degree that most programmers I know treat Windows programmers and sysadmins as the object of well-deserved ridicule. Using Windows for programming or production servers is essentially as if the race car driver from my earlier metaphor did bring a mini-van to the race track. Linux is the operating system developed by programmers, for programmers, to suit our needs, and we have succeeded tremendously in this respect.

However, we have failed to build an operating system for people who are not like us.

If this is not our goal, then that’s fine. But, we can build things for non-experts if we choose to. If we set “accessible to the average user” as a goal, then we must take certain steps to achieve it. We need to make major improvements in the following areas: robustness, intuitiveness, and community.

The most frustrating moments for a user is when the software they’re using does something inexplicable, and it’s these moments that they will remember the most vividly as part of their experience. Many Linux desktop and distribution projects are spending their time on shiny new features, re-skins, and expanding their scope further and further. This is a fool’s errand when the project is not reliable at its current scope. A small, intuitive, reliable program is better than a large, unintuitive, unreliable program. Put down the paint brush and pick up the polishing stone. I’m looking at you, KDE.²

A user-friendly Linux desktop system should not crash. It should not be possible to install a package which yeets gnome-desktop and dumps them into a getty. The smallest of interactions must be intuitive and reliable, so that when Linus drags files from the decompression utility into a directory in Dolphin, it does the right thing. This will require a greater degree of cooperation and unity between desktop projects. Unrelated projects with common goals need to be reaching out to one another and developing robust standards for achieving those goals. I’m looking at you, Gnome.

Linux is a box of loosely-related tools held together with staples and glue. This is fine when the user understands the tools and is holding the glue bottle, but we need to make a more cohesive, robust, and reliable system out of this before it can accommodate average end-users.

We also have a lot of work to do in the Linux community. The discussion on the LTT video series has been exceptionally toxic and downright embarrassing. There is a major problem of elitism within the Linux community. Given a hundred ways of doing things on Linux (✓), there will be 99 assholes ready to tell you that your way sucks (✓). Every Linux user is responsible for doing better in this regard, especially the moderators of Linux-adjacent online spaces. Wouldn’t it be better if we took pride in being a friendly, accessible community? Don’t flame the noobs.

Don’t flame the experts, either. When Pop!_OS removed gnome-desktop upon installing Steam, the Linux community rightly criticised them for it. This was a major failure mode of the system in one of its flagship features, and should have never shipped. It illuminates systemic failures in the areas I have drawn our attention to in this article such as robustness and intuitiveness, and Pop!_OS is responsible for addressing the problem. None of that excuses the toxic garbage which was shoveled into the inboxes of Pop!_OS developers and users. Be better people.

Beyond the toxicity, there are further issues with the Linux community. There are heaps and heaps of blogs shoveling out crappy non-solutions to problems noobs might be Googling, most of which will fuck up their Linux system in some way or another. It’s very easy to find bad advice for Linux, and very hard to find good advice for Linux. The blog spammers need to cut it out, and we need to provide better, more accessible resources for users to figure out their issues. End-user-focused Linux distributions need to take responsibility for making certain that their users understand the best ways to get help for any issues they run into, so they don’t go running off to the Arch Linux forums blindly running terminal commands which will break their Ubuntu installation.

End-user software also needs to improve in this respect. In the latest LTT video, Luke wanted to install OBS, and the right thing to do was install it from their package manager. However, the OBS website walks them through installing a PPA instead, and has a big blue button for building it from source, which is definitely not what an average end-user should be doing.

One thing that we do not need to do is “be more like Windows”, or any other OS. I think that this is a common fallacy found in end-user Linux software. We should develop a system which is intuitive in its own right without having to crimp off of Windows. Let’s focus on what makes Linux interesting and useful, and try to build a robust, reliable system which makes those interesting and useful traits accessible to users. Chasing after whatever Windows does is not the right thing to do. Let’s be prepared to ask users to learn things like new usability paradigms if it illuminates a better way of doing things.

So, these are the goals. How do we achieve them?

I reckon that we could use a commercial, general-purpose end-user Linux distro. As I mentioned earlier, the model of developers hacking in their spare time to make systems for themselves does not create incentives which favor the average end-user. You can sell free software — someone ought to do so! Build a commercial Linux distro, charge $20 to download it or mail an install CD to the user, and invest that money in developing a better system and offer dedicated support resources. Sure, it’s nice that Linux is free-as-in-beer, but there’s no reason it has to be. I’ve got my own business to run, so I’ll leave that as an exercise for the reader. Good luck!

How new Linux users can increase their odds of success

Sun, 05 Dec 2021 00:00:00 +0000

The Linus Tech Tips YouTube channel has been putting out a series of videos called the Switching to Linux Challenge that has been causing a bit of a stir in the Linux community. I’ve been keeping an eye on these developments, and thought it was a good time to weigh in with my thoughts. This article focuses on how new Linux users can increase their odds for success — I have also written a companion article, “What desktop Linux needs to succeed in the mainstream”, which looks at the other side of the problem.

Linux is, strictly speaking, an operating system kernel, which is a small component of a larger system. However, in the common usage, Linux refers to a family of operating systems which are based on this kernel, such as Ubuntu, Fedora, Arch Linux, Alpine Linux, and so on, which are referred to as distributions. Linux is used in other contexts, such as Android, but the common usage is generally limited to this family of Linux “distros”. Several of these distros have positioned themselves for various types of users, such as office workers or gamers. However, the most common Linux user is much different. What do they look like?

The key distinction which sets Linux apart from more common operating systems like Windows and macOS is that Linux is open source. This means that the general public has access to the source code which makes it tick, and that anyone can modify it or improve it to suit their needs. However, to make meaningful modifications to Linux requires programming skills, so, consequentially, the needs which Linux best suits are the needs of programmers. Linux is the preeminent operating system for programmers and other highly technical computer users, for whom it can be suitably molded to purpose in a manner which is not possible using other operating systems. As such, it has been a resounding success on programmer’s workstations, on servers in the cloud, for data analysis and science, in embedded workloads like internet-of-things, and other highly technical domains where engineering talent is available and a profound level of customization is required.

The Linux community has also developed Linux as a solution for desktop users, such as the mainstream audience of Windows and macOS. However, this work is mostly done by enthusiasts, rather than commercial entities, so it can vary in quality and generally any support which is available is offered on a community-run, best-effort basis. Even so, there have always been a lot of volunteers interested in this work — programmers want a working desktop, too. Programmers also want to play games, so there has been interest in getting a good gaming setup working on Linux. In the past several years, there has also been a commercial interest with the budget to move things forward: Valve Software. Valve has been instrumental in developing more sophisticated gaming support on Linux, and uses Linux as the basis of a commercial product, the Steam Deck.¹

Even so, I must emphasize the following point:

The best operating system for gaming is Windows.

Trying to make Linux do all of the things you’re used to from Windows or macOS is not going to be a successful approach. It is possible to run games on Linux, and it is possible to run some Windows software on Linux, but it is not designed to do these things, and you will likely encounter some papercuts on the way. Many advanced Linux users with a deep understanding of the platform and years of experience under their belt can struggle for days to get a specific game running. However, thanks to Valve, and the community at large, many games — but not all games — run out-of-the-box with much less effort than was once required of Linux gamers.

Linux users are excited about improved gaming support because it brings gaming to a platform that they already want to use for other reasons. Linux is not Windows, and offers an inferior gaming experience to Windows, but it does offer a superior experience in many other regards! If you are trying out Linux, you should approach it with an open mind, prepared to learn about what makes Linux special and different from Windows. You’ll learn about new software, new usability paradigms, and new ways of using your computer. If you just want to do all of the same things on Linux that you’re already doing on Windows, why switch in the first place? The value of Linux comes from what it can do differently. Given time, you will find that there are many things that Linux can do that Windows cannot. Leave your preconceptions at the door and seek to learn what makes Linux special.

I think that so-called “power users” are especially vulnerable to this trap, and I’ve seen it happen many times. A power user is someone who deeply understands the system that they’re using, knows about every little feature, knows all of the keyboard shortcuts, and integrates all of these details into their daily workflow. Naturally, it will take you some time to get used to a new system. You can be a power user on Linux — I am one such user myself — but you’re essentially starting from zero, and you will learn about different features, different nuances, and different shortcuts, all of which ultimately sums to an entirely different power user.

The latest LTT video in the Linux series shows the team going through a set of common computer tasks on Linux. However, these tasks do little to nothing to show off what makes Linux special. Watching a 4K video is nice, sure, and you can do it on Linux, but how does that teach you anything interesting about Linux?

Let me offer a different list of challenges for a new Linux user to attempt, hand-picked to show off the things which set Linux apart in my opinion.

Learn how to use the shell. A lot of new Linux users are intimidated by the terminal, and a lot of old Linux users are understandably frustrated about this. The terminal is one of the best things about Linux! We praise it for a reason, intimidating as it may be. Here’s a nice tutorial to start with.
Find and install packages from the command line. On Linux, you install software by using a “package manager”, a repository of software controlled by the Linux distribution. Think of it kind of like an app store, but non-commercial and without malware, adware, or spyware. If you are downloading Linux software from a random website, it’s probably the wrong thing to do. See if you can figure out the package manager instead!
Try out a tiling window manager, especially if you consider yourself a power user. I would recommend sway, though I’m biased because I started this project. Tiling window managers change the desktop usability paradigm by organizing windows for you and letting you navigate and manipulate them using keyboard shortcuts alone. These are big productivity boosters.
Compile a program from source. This generally is not how you will usually find and use software, but it is an interesting experience that you cannot do on Windows or Mac. Pick something out and figure out where the source code is and how to compile it yourself. Maybe you can make a little change to it, too!
Help someone else out online. Linux is a community of volunteers supporting each other. Take what you’ve learned to /r/linuxquestions or your distro’s chat rooms, forums, wikis, or mailing lists, and make them a better place for everyone else. The real magic of Linux comes from the collaborative, grassroots nature of the project, which is something you really cannot get from Windows or Mac.

Bonus challenge: complete all of the challenges from the LTT video, but only using the command line.

All of these tasks might take a lot longer than 15 minutes to do, but remember: embrace the unfamiliar. You don’t learn anything by doing the things you already know how to do. If you want to know why Linux is special, you’ll have to step outside of your comfort zone. Linux is free, so there’s no risk in trying 🙂 Good luck, and do not be afraid to ask for help if you get stuck!

postmarketOS revolutionizes smartphone hacking

Fri, 26 Nov 2021 00:00:00 +0000

I briefly mentioned postmarketOS in my Pinephone review two years ago, but after getting my Dutch SIM card set up in my Pinephone and having another go at using postmarketOS, I reckon they deserve special attention.

Let’s first consider the kind of ecosystem into which postmarketOS emerged: smartphone hacking in the XDA Forums era. This era was dominated by amateur hackers working independently for personal prestige, with little to no regard for the values of free software or collaboration. It was common to see hacked-together binary images shipped behind adfly links in XDA forum threads in blatant disregard of the GPL, with pages and pages of users asking redundant questions and receiving poor answers to the endless problems caused by this arrangement.

The XDA ecosystem is based on Android, which is a mess in and of itself. It’s an enormous, poorly documented ball of Google code, mixed with vendor drivers and private kernel trees, full of crappy workarounds and locked-down hardware. Most smart phones are essentially badly put-together black boxes and most smart phone hackers are working with their legs cut off. Not to mention that the software ecosystem which runs on the platform is full of scammers and ads and theft of private user information. Android may be Linux in implementation, but it’s about as far from the spirit of free software as you can get.

postmarketOS, on the other hand, is based on Alpine Linux, which happens to be my favorite Linux distribution. Instead of haphazard forum threads collecting inscrutable ports for dozens of devices, they have a single git repository where all of their ports are maintained under version control, complete with issue trackers and merge requests, plus a detailed centralized wiki providing a wealth of open technical info on their supported platforms. And, by virtue of being a proper Linux distribution, they essentially opt-out of the mess of predatory mobile apps and instead promote a culture of trusted applications which respect the user and are built by and for the community instead of by and for a corporation.

Where we once had to live with illegally closed-source forks of the Linux kernel, we now have a git repository in which upstream Linux releases are tracked with a series of auditable patches for supporting various devices, many of which are making their way into upstream Linux. Where we once had a forum thread with five wrong answers to the same question on page 112, we now have a bug report on GitLab with a documented workaround and a merge request pending review. Instead of begging my vendor to unlock my bootloader and using janky software reminiscent of old keygen hacks to flash a dubious Android image, I can build postmarketOS’s installer, pop it onto a microSD card, and two minutes I’ll have Linux installed on my Pinephone.

pmOS does not seek to elevate the glories of tiny individual hackers clutching their secrets close to their chest, instead elevating the glory of the community as a whole. It pairs perfectly with Pine64, the only hardware vendor working closely with upstream developers with the same vision and ideals. There is a promise for hope in the future of smart phones in their collaboration.

However, the path they’ve chosen is a difficult one. Android, for all of its faults, presents a complete solution for a mobile operating system, and upstream Linux does not. In my review, I said that software would be the biggest challenge of the Pinephone, and 2 years later, that remains the case. Work reverse engineering the Pine64 hardware is slow, there is not enough cooperation between project silos, and there needs to be much better prioritization of the work. To complete their goals, the community will have to work more closely together and narrow their attention in on the key issues which stand between the status quo and the completion of a useful Linux smartphone. It will require difficult, boring engineering work, and will need the full attention and dedication of the talented people working on these projects.

If they succeed in spite of these challenges, the results will be well worth it. postmarketOS and pine64 represent the foundations of a project which could finally deliver Linux on smartphones and build a robust mobile platform that offers freedom to its users for years to come.

My philosophy for productive instant messaging

Wed, 24 Nov 2021 00:00:00 +0000

We use Internet Relay Chat (IRC) extensively at sourcehut for real-time group chats and one-on-one messaging. The IRC protocol is quite familiar to hackers, who have been using it since the late 80’s. As chat rooms have become more and more popular among teams of both hackers and non-hackers in recent years, I would like to offer a few bites of greybeard wisdom to those trying to figure out how to effectively use instant messaging for their own work.

For me, IRC is a vital communication tool, but many users of ¹ find it frustrating, often to the point of resenting the fact that they have to use it at all. Endlessly catching up on discussions they missed, having their workflow interrupted by unexpected messages, searching for important information sequestered away in a discussion which happened weeks ago… it can be overwhelming and ultimately reduce your productivity and well-being. Why does it work for me, but not for them? To find out, let me explain how I think about and use IRC.

The most important trait to consider when using IM software is that it is ephemeral, and must be treated as such. You should not “catch up” on discussions that you missed, and should not expect others to do so, either. Any important information from a chat room discussion must be moved to a more permanent medium, such as an email to a mailing list,² a ticket filed in a bug tracker, or a page updated on a wiki. One very productive use of IRC for me is holding a discussion to hash out the details of an issue, then writing up a summary up for a mailing list thread where the matter is discussed in more depth.

I don’t treat discussions on IRC as actionable until they are shifted to another mode of discussion. On many occasions, I have discussed an issue with someone on IRC, and once the unknowns are narrowed down and confirmed to be actionable, ask them to follow-up with an email or a bug report. If the task never leaves IRC, it also never gets done. Many invalid or duplicate tasks are filtered out by this approach, and those which do get mode-shifted often have more detail than they otherwise might, which improves the signal-to-noise ratio on my bug trackers and mailing lists.

I have an extensive archive of IRC logs dating back over 10 years, tens of gigabytes of gzipped plaintext files. I reference these logs perhaps only two or three times a year, and often for silly reasons, like finding out how many swear words were used over some time frame in a specific group chat, or to win an argument about who was the first person to say “yeet” in my logs. I almost never read more than a couple dozen lines of the backlog when starting up IRC for the day.

Accordingly, you should never expect anyone to be in the know for a discussion they were not present at. This also affects how I use “highlights”.³ Whenever I highlight someone, I try to include enough context in the message so that they can understand why they were mentioned without having to dig through their logs, even if they receive the notification hours later.

Bad:

<sircmpwn> minus: ping
<sircmpwn> what is the best way to frob foobars?

Good:

<sircmpwn> minus: do you know how to frob foobars?

I will also occasionally send someone a second highlight un-pinging them if the question was resolved and their input is no longer needed. Sometimes I will send a vague “ping ” example when I actually want them to participate in the discussion right now, but if they don’t answer immediately then I will usually un-ping them later.⁴

This draws attention to another trait of instant messaging: it is asynchronous. Not everyone is online at the same time, and we should adjust our usage of it in consideration of this. For example, when I send someone a private message, rather than expecting them to engage in a real-time dialogue with me right away, I dump everything I know about the issue for them to review and respond to in their own time. This could be hours later, when I’m not available myself!

Bad:

<sircmpwn> hey emersion, do you have a minute?
*8 hours later*
<emersion> yes?
*8 hours later*
<sircmpwn> what is the best way to frob foobars?
*8 hours later*
<emersion> did you try mongodb?

Good:⁵

<sircmpwn> hey emersion, what's the best way to frob foobars?
<sircmpwn> I thought about mongodb but they made it non-free
*10 minutes later*
<sircmpwn> update: considered redis, but I bet they're one bad day away from making that non-free too
*8 hours later*
<emersion> good question
<emersion> maybe postgresql? they seem like a trustworthy bunch
*8 hours later*
<sircmpwn> makes sense. Thanks!

This also presents us a solution to the interruptions problem: just don’t answer right away, and don’t expect others to. I don’t have desktop or mobile notifications for IRC. I only use it when I’m sitting down at my computer, and I “pull” notifications from it instead of having it “push” them to me — that is, I glance at the client every now and then. If I’m in the middle of something, I don’t read it.

With these considerations in mind, IRC has been an extraordinarily useful tool for me, and maybe it can be for you, too. I’m not troubled by interruptions to my workflow. I never have to catch up on a bunch of old messages. I can communicate efficiently and effectively with my team, increasing our productivity considerably, without worrying about an added source of stress. I hope that helps!

Python: Please stop screwing over Linux distros

Tue, 16 Nov 2021 00:00:00 +0000

Linux distributions? Oh, those things we use to bootstrap our Docker containers? Yeah, those are annoying. What were you complaining about again?

The Python community is obsessed with reinventing the wheel, over and over and over and over and over and over again. distutils, setuptools, pip, pipenv, tox, flit, conda, poetry, virtualenv, requirements.txt, setup.py, setup.cfg, pyproject.toml… I honestly can’t even list all of the things you have to deal with. It’s a disaster.

An xkcd comic showing a convoluted graph of competing Python environments

This comic is almost 4 years old and it has become much worse since. Python is a mess. I really want to like Python. I have used it for many years and in many projects, including SourceHut, which was predominantly developed in Python. But I simply can’t handle it anymore, and I have been hard at work removing Python from my stack.

This has always been a problem with Python, but in the past few years everyone and their cousin decided to “solve” it by building another mess which is totally incompatible with all of the others, all of the “solutions” enjoying varying levels of success in the community and none of them blessed as the official answer.

I manage my Python packages in the only way which I think is sane: installing them from my Linux distribution’s package manager. I maintain a few dozen Python packages for Alpine Linux myself. It’s from this perspective that, throughout all of this turmoil in Python’s packaging world, I have found myself feeling especially put out.

Every one of these package managers is designed for a reckless world in which programmers chuck packages wholesale into ~/.pip, set up virtualenvs and pin their dependencies to 10 versions and 6 vulnerabilities ago, and ship their computers directly into production in Docker containers which aim to do the minimum amount necessary to make their user’s private data as insecure as possible.

None of these newfangled solutions addresses the needs of any of the distros, despite our repeated pleas. They all break backwards compatibility with our use-case and send our complaints to /dev/null. I have seen representatives from every Linux distro making repeated, desperate pleas to Python to address their concerns, from Debian to Arch to Alpine to NixOS, plus non-Linux distros like FreeBSD and Illumos. Everyone is frustrated. We are all struggling to deal with Python right now, and Python is not listening to us.

What is it about Linux distros that makes our use-case unimportant? Have we offered no value to Python over the past 30 years? Do you just feel that it’s time to shrug off the “legacy” systems we represent and embrace the brave new world of serverless cloud-scale regulation-arbitrage move-fast-and-break-things culture of the techbro startup?

Distros are feeling especially frustrated right now, but I don’t think we’re alone. Everyone is frustrated with Python packaging. I call on the PSF to sit down for some serious, sober engineering work to fix this problem. Draw up a list of the use-cases you need to support, pick the most promising initiative, and put in the hours to make it work properly, today and tomorrow. Design something you can stick with and make stable for the next 30 years. If you have to break some hearts, fine. Not all of these solutions can win. Right now, upstream neglect is destroying the Python ecosystem. The situation is grave, and we need strong upstream leadership right now.

P.S. PEP-517 and 518 are a start, but are very disappointing in how little they address distro problems. These PEPs are designed to tolerate the proliferation of build systems, which is exactly what needs to stop. Python ought to stop trying to avoid hurting anyone’s feelings and pick one. Maybe their decision-making framework prevents this, if so, the framework needs to be changed.

P.P.S. There are a lot of relevant xkcds that I wanted to add. Here’s the ones I left out:

Further reading: Developers: Let distros do their job

I will pay you cash to delete your npm module

Tue, 16 Nov 2021 00:00:00 +0000

npm’s culture presents a major problem for global software security. It’s grossly irresponsible to let dependency trees grow to thousands of dependencies, from vendors you may have never heard of and likely have not critically evaluated, to solve trivial tasks which could have been done from scratch in mere seconds, or, if properly considered, might not even be needed in the first place.

We need to figure out a way to curb this reckless behavior, but how?

I have an idea. Remember left-pad? That needs to happen more often.

A LaTeX rendering of an equation which sets a reward (in dollars) to the logarithm of weekly downloads over lines of code in base 10 times one hundred

I’ll pay you cold hard cash to delete your npm module. The exact amount will be determined on this equation, which is designed to offer higher payouts for modules with more downloads and fewer lines of code. A condition of this is that you must delete it without notice, so that everyone who depends on it wakes up to a broken build.

Let’s consider an example: isArray. It has only four lines of code:

var toString = {}.toString;

module.exports = Array.isArray || function (arr) {
  return toString.call(arr) === '[object Array]';
};

With 51 million downloads this week, this works out to a reward of $710.

To prevent abuse, we’ll have to agree to each case in advance. I’ll review your module to make sure it qualifies, and check for any funny business like suspicious download figures or minified code. We must come to an agreement before you delete the module, since I will not be able to check the line counts or download numbers after it’s gone.

I may also ask you to wait to delete your module, so that the chaos from each deletion is separated by a few weeks to maximize the impact. Also, the reward is capped at $1,000, so that I can still pay rent after this.

Do we have a deal? Click here to apply →

Alright, the gig is up: this is satire. I’m not actually going to pay you to delete your npm module, nor do I want to bring about a dark winter of chaos in the Node ecosystem. Plus, it wouldn’t actually work.

I do hope that this idea strikes fear in the hearts of any Node developers that read it, and in other programming language communities which have taken after npm. What are you going to do if one of your dependencies vanishes? What if someone studies the minified code on your website, picks out an obscure dependency they find there, then bribes the maintainers?

Most Node developers have no idea what’s in their dependency tree. Most of them are thousands of entries long, and have never been audited. This behavior is totally reckless and needs to stop.

Most of my projects have fewer than 100 dependencies, and many have fewer than 10. Some have zero. This is by design. You can’t have a free lunch, I’m afraid. Adding a dependency is a serious decision which requires consensus within the team, an audit of the new dependency, an understanding of its health and long-term prospects, and an ongoing commitment to re-audit them and be prepared to change course as necessary.

isArray license:

Copyright (c) 2013 Julian Gruber <julian@juliangruber.com>.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Status update, November 2021

Mon, 15 Nov 2021 00:00:00 +0000

Hello again! Following a spooky month, we find ourselves again considering the progress of our eternal march towards FOSS world domination.

I’ll first address SourceHut briefly: today is the third anniversary of the opening of the public alpha! I have written a longer post for sourcehut.org which I encourage you to read for all of the details.

In other news, we have decided to delay the release of our new programming language, perhaps by as much as a year. We were aiming for February ‘22, but slow progress on some key areas such as cryptography and the self-hosting compiler, plus the looming necessity of the full-scale acceptance testing of the whole language and standard library, compound to make us unsure about meeting the original release plans. However, progress is slow but moving. We have incorporated the first parts of AES support in our cryptography library, and ported the language to FreeBSD. A good start on date/time support has been under development and I’m pretty optimistic about the API design we’ve come up with. Things are looking good, but it will take longer than expected.

visurf has enjoyed quite a bit of progress this month, thanks in large part to the help of a few new contributors. Nice work, everyone! We could still use more help, so please swing by the #netsurf channel on Libera Chat if you’re interested in participating. Improvements which landed this month include configuration options, url filtering via awk scripts, searching through pages, and copying links on the page with the link following tool.

Projects which received minor updates this month include scdoc, gmni, kineto, and godocs.io. That’s it for today! My focus for the next month will be much the same as this month: SourceHut GraphQL work and programming language work. See you in another month!

Breaking down Apollo Federation's anti-FOSS corporate gaslighting

Fri, 05 Nov 2021 00:00:00 +0000

Gather around, my friends, for there is another company which thinks we are stupid and we enjoy having our faces spat in. Apollo Federation¹ has announced that they will switch to a non-free license. Let’s find out just how much the Elastic license really is going to “protect the community” like they want you to believe.

Let’s start by asking ourselves, objectively, what practical changes can we expect from a switch from the MIT license to the Elastic License? Both licenses are pretty short, so I recommend quickly reading them yourself before we move on.

I’ll summarize the difference between these licenses. First, the Elastic license offers you (the recipient of the software) one benefit that MIT does not: an explicit license for any applicable patents. However, it also has many additional restrictions, such as:

No sublicensing (e.g. incorporating part of it into your own program)
No resale (e.g. incorporating it into Red Hat and selling support)
No modifications which circumvent the license key activation code
No use in a hosted or managed service

This is an objective analysis of the change. How does Apollo explain the changes?

Why the new license?
The Apollo developer community is at the heart of everything we do. As stewards of our community, we have a responsibility to prevent harm from anyone who intends to exploit our work without contributing back. We want to continue serving you by funding the development of important open-source graph technology for years to come. To honor that commitment, we’re moving Apollo Federation 2 to the Elastic License v2 (ELv2).

Taking them at their word, this change was motivated by their deep care for their developer community. They want to “honor their commitment”, which is to “fund the development of important open-source graph technology” and “prevent harm from anyone who intends to exploit our work without contributing back”.

This is a very misleading statement. The answer to the question stated by the header is “funding the development”, but they want us to first think that they’re keeping the community at the heart of this decision — a community that they have just withheld several rights from. Their wording also seeks to link the community with the work, “our work”, when the change is clearly motivated from a position where Apollo believes they have effective ownership over the software, sole right to its commercialization, and a right to charge the community a rent — enforced via un-circumventable license key activation code. The new license gives Apollo exclusive right to commercial exploitation of the software — so they can “exploit our work”, but the community itself cannot.

What’s more, the change does not fund “open-source graph technology” as advertised, because after this change, Apollo Federation is no longer open source. The term “open source” is defined by the Open Source Definition², whose first clause is:

[The distribution terms of open-source software] shall not restrict any party from selling or giving away the software as a component of an aggregate software distribution containing programs from several different sources. The license shall not require a royalty or other fee for such sale.

The OSD elaborates later:

The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.
The rights attached to the program must apply to all to whom the program is redistributed without the need for execution of an additional license by those parties.

The Elastic license clearly does not meet this criteria.

Reading the Apollo announcement further, it continues to peddle this and other lies. The next paragraph attempts to build legitimacy for its peers in this anti-FOSS gaslighting movement:

Open-source licensing is evolving with the cloud. Many successful companies built on open-source technology (such as Elastic, MongoDB, and Confluent) have followed the path we’re taking to protect their communities and combine open, collaborative development with the benefits of cloud services that are easy to adopt and manage.

They continue to use “open-source” language throughout, and misleads us into believing that they’ve made this change to protect the community and empower developers.

When the Elastic License v2 was released, Elastic CEO Shay Banon called upon open-source companies facing a similar decision to “coalesce around a smaller number of licenses.” We’re excited to be part of this coalition of modern infrastructure companies building businesses that empower developers. […] Moving the Apollo Federation libraries and gateway to ELv2 helps us focus on our mission: empowering all of you.

It should be evident by now that this is complete horseshit. Let me peel away the bullshit and explain what is actually going on here in plain English.

Free and open source software can be commercialized — this is an essential requirement of the philosophy! However, it cannot be exclusively commercialized. Businesses which participate in the FOSS ecosystem must give up their intellectual property monopoly, and allow the commercial ecosystem to flourish within their community — not just within their own ledger. They have to make their hosted version better than the competitors, or seek other monetization strategies: selling books, support contracts, consulting, early access to security patches, and so on.

The community, allegedly at the heart of everything Apollo does, participates in the software’s development, marketing, and growth, and they are rewarded with the right to commercialize it. The community is incentivized to contribute back because they retain their copyright and the right to monetize the software. 634 people have contributed to Apollo, and the product is the sum of their efforts, and should belong to them — not just to the business which shares a name with the software. The community built their projects on top of Apollo based on the open source social contract, and gave their time, effort, and copyright for their contributions to it, and Apollo pulled the rug out from under them. In the words of Bryan Cantrill, this shameful, reprehensible behavior is shitting in the pool of open source.

The smashing success of the free and open source software movement, both socially and commercially, has attracted the attention of bad actors like Apollo, who want to capitalize on this success without meeting its obligations. This wave of nonfree commercial gaslighting is part of a pattern where a company builds an open-source product, leverages the open-source community to build a market for it and to directly improve the product via their contributions, then switches to a nonfree license and steals the work for themselves, fucking everyone else over.

Fuck Matt DeBergalis, Shay Banon, Jay Kreps, and Dev Ittycheria. These are the CEOs and CTOs responsible for this exploitative movement. They are morally bankrupt assholes and rent-seekers who gaslight and exploit the open source community for personal gain.

This is a good reminder that this is the ultimate fate planned by any project which demands a copyright assignment from contributors in the form of a Contributor License Agreement (CLA). Do not sign these! Retain your copyright over your contributions and contribute to projects which are collectively owned by their community — because that’s how you honor your community.

Previously:

If you are an Apollo Federation user who is affected by this change, I have set up a mailing list to organize a community-maintained fork. Please send an email to this list if you are interested in participating in such a fork.

GitHub stale bot considered harmful

Tue, 26 Oct 2021 00:00:00 +0000

Disclaimer: I work for a GitHub competitor.

One of GitHub’s “recommended” marketplace features is the “stale” bot. The purpose of this bot is to automatically close GitHub issues after a period of inactivity, 60 days by default. You have probably encountered it yourself in the course of your work.

This is a terrible, horrible, no good, very bad idea.

A screenshot of an interaction with this bot

I’m not sure what motivates maintainers to install this on their repository, other than the fact that GitHub recommends it to them. Perhaps it’s motivated by a feeling of shame for having a lot of unanswered issues? If so, this might stem from a misunderstanding of the responsibilities a maintainer has to their project. You are not obligated to respond to every issue, implement every feature request, or fix every bug, or even acknowledge them in any way.

Let me offer you a different way of thinking about issues: a place for motivated users to collaborate on narrowing down the problem and planning a potential fix. A space for the community to work, rather than an action item for you to deal with personally. It gives people a place to record additional information, and, ultimately, put together a pull request for you to review. It does not matter if this process takes days or weeks or years to complete. Over time, the issue will accumulate details and workarounds to help users identify and diagnose the problem, and to provide information for the person that might eventually write a patch/pull request.

It’s entirely valid to just ignore your bug tracker entirely and leave it up to users to deal with themselves. There is no shame in having a lot of open issues — if anything, it signals popularity. Don’t deny your users access to an important mutual support resource, and a crucial funnel to bring new contributors into your project.

This is the approach I would recommend on GitHub, but for illustrative purposes I’ll also explain a slightly modified approach I encourage for SourceHut users. sr.ht provides mailing lists (and, soon, IRC chat rooms), which are recommended for first-line support and discussion about your project, including bug reports, troubleshooting, and feature requests, instead of filing a ticket (our name for issues). The mailing list gives you a space to refine the bug report, solicit extra details or point out an existing ticket, or clarifying and narrowing down feature requests. This significantly improves the quality of bug reports, eliminates duplicates, and better leverages the community for support, resulting in every single ticket representing a unique, actionable item.

I will eventually ask the user to file a ticket when the bug or feature request is confirmed. This does not imply that I will follow up with a fix or implementation on any particular time frame. It just provides this space I discussed before: somewhere to collect more details, workarounds, and additional information for users who experience a bug or want a feature, and to plan for its eventual implementation at an undefined point in the future, either from a SourceHut maintainer or from the community.

How SmarterEveryDay's 4privacy can, and cannot, meet its goals

Fri, 22 Oct 2021 00:00:00 +0000

I don’t particularly find myself to be a fan of the SmarterEveryDay YouTube channel, simply for being outside of Destin’s target audience most of the time. I understand that Destin, the channel’s host, is a friendly person and a great asset to his peers, and that he generally strives to do good. When I saw that he was involved in a Kickstarter to develop a privacy product, it piqued my interest. As a privacy advocate and jaded software engineer, I set out to find out what it’s all about.

You can watch the YouTube video here, and a short follow-up here.

There are several things to praise here. I honestly thought that Destin’s coverage of the topic of privacy for the layman was really well presented, and took some notes to use the next time I’m explaining privacy issues to my friends. The coverage of the history of wiretapping and the pivotal role played by 9/11, complete with an empathetic view of the mindset of American adults contemporary to it that many find hard to express, along with great drone shots of Big Tech’s mysterious datacenters, this is all great stuff. For the right project, Destin is a valuable asset with a large audience and a lot of experience in making complex issues digestible for the every-person, and 4privacy is lucky to have access to him.

A lot of the buzzwords and things found on their technology page are promising as well. The focus on end-to-end encryption and zero-knowledge principles, and the commitment to open source, are absolutely necessary and are great to see here. A lot of the tech described, although briefly, seems like it’s on the right track. The ability to use your own service provider, and the focus on decentralization and federation, is very good.

I do have some concerns, however. Let’s break them down into these categories:

Incentives and economics
Responsibilities and cultivating trust
Ambitions and feasibility

Given the value ($$$) associated with private user information, it’s important to know that the trove of private information overseen by a company like this is safe from threats from the robber-barons of tech. 4privacy is looking for investors, which is a red flag: investors demand a return, and if the product isn’t profitable, user data is the first thing up for auction. So, how will 4privacy make money? We need to know. They might say that the E2EE prevents them from directly monetizing user data, and they’re right, but that’s only for today. If they become a market incumbent, they will have the power to change the technology in a way which compromises privacy faster than we can move to another system, and we need to understand that this will not happen.

Growing consumer awareness in privacy issues over the past decade, combined with a generally low level of technology literacy in the population, has allowed a lot of grifters to arise. One of the common forms these grifts take is seen in the rise of VPN companies, which prey on consumer fear and often use YouTube as a marketing channel, including on Destin’s previous videos. Another giant, flaming red flag appears whenever cryptocurrency is involved. In general terms, the privacy space is thoroughly infested with bad actors, which makes matters of trust very difficult. 4privacy needs to be prepared to be very honest and transparent with not only their tech, but their financial structure and incentives. With SourceHut, I had to engineer our incentives to suit stated goals, and I communicate this to users so that they can make informed choices about us. 4privacy would be wise to take similar steps, in full view of the public.

Empowering users to make informed choices leads me into our next point: is 4privacy ready to bear the burden of responsibility for this system? As far as I can glean from their mock-ups, they plan to be handling your government IDs, passwords, healthcare information, confidential attorney/client communications, and so on. The consequences of having this information compromised are grave, and this demands world-class security. It’s also extremely important for 4privacy to be honest with their users about what their security model can, and cannot, make promises about.

You must be honest with your users, and help them to understand how the system works, and when it doesn’t work, so that they can make informed choices about how to trust it. This can be difficult when the profit motive is involved, because they might conclude that they don’t want to use your service. It’s even more difficult when you exist in a space full of grifters that are happy to tell sweet lies to your users about fixing all of their problems. However, it must be done.

Privacy tools are relied upon by vulnerable people facing challenging situations. If you promise something you cannot deliver on, and they depend on you to keep their information private in impossible conditions, when the other shoe drops there could be dramatic consequences for their lives. If a journalist in a war-torn country depends on you to keep their documents private, and you fail, they could end up in prison or a labor camp or splattered on the wall of a dark alley, and it’ll have been your fault. You must be forthright and realistic with users about how your system can and cannot keep them safe. I hope Destin’s future videos in the privacy series will cover how the system works in more detail, including its limitations. He is skilled at explaining complicated topics in a comprehensible manner for everyday people to understand, and I hope he will leverage these skills here.

I have already noticed one place where they have failed to be honest in their limitations, however, and it presents a major concern for me. Much of their marketing speaks of the ability to revoke access to your private information after a third-party has been provided access to it. This is, frankly, entirely impossible, and I think it is extraordinarily irresponsible to design your application in a manner that suggests that it can be done. To keep things short, I’ll refute the idea as briefly as possible: what’s to stop someone from taking a picture of the phone while it’s displaying your private info? Or writing it down? When you press the “revoke” button in the app, and it dutifully disappears from their phone screen,¹ the private information is still written on a piece of paper in their desk drawer and you’re none the wiser. The application has given you a false sense of security, which is a major problem for a privacy-oriented tool.

You can work in this problem space, albeit under severely limited constraints. For example, consider how the SSH agent works: an application which wants to use your private keys to sign something can ask the agent for help, but the agent will not provide the cryptographic keys for it to use directly — the agent will do the cryptographic operation on the application’s behalf and send the results to the application to use. These constraints limit the use-cases significantly, such that, for example, you could not send someone your social security number using this system. You could, however, design a protocol in which an organization which needs to verify your identity can ask, in programmatic terms, “is this person who they say they are?”, and 4privacy answers, possibly consulting their SSN, “yes” or “no”. This does not seem to be what they’re aiming for, however.

So, with all of this in mind, how ambitious is their idea as a whole? Is it feasible? What kind of resources will they need to pull it off?

In short, this idea is extraordinarily ambitious. They are designing a novel cryptosystem, which is an immediate red flag: designing a secure cryptosystem is one of the most technologically challenging feats a programming team can undertake. Furthermore, they’re building a distributed, federated system, which is itself a highly complex and challenging task, even more so when the system is leveraged to exchange sensitive information. It can be done, but it takes an extraordinarily talented team with hard-core technical chops and a lot of experience.

What’s more, if they were to do this well, it would involve developing and standardizing open protocols. This requires a greater degree of openness and community participation than they are planning to do. Furthermore, they need to get others to agree to implement these protocols, which involves solving social and political problems — both in technical and non-technical senses. For instance, the Dutch government stores much of my personal information in the DigiD system. Will they be able to convince the Netherlands to work with their protocols? How about every other country? And, if they want me to store my health insurance in the app, how are they going to convince my doctor to use the app to receive it? And how about every other doctor? And what about all of the other domains they want to be involved in outside of healthcare data? Will they interoperate with legacy systems to achieve the market penetration they need? Will those legacy systems provide for their end-to-end encryption needs, and if not, will users understand the consequences?

I’m not saying that any of this is impossible — only that it is extraordinarily difficult to pull off. Extraordinary projects require extraordinary resources. They will need multiple highly talented engineering teams working in parallel, and the support staff necessary to keep them going.

Their goal on Kickstarter, which was quickly met and exceeded, is $175,000. This is nowhere near enough, so either they aren’t going to pull it off, or they have more money from somewhere else. Destin is acknowledged as an investor, and they are seeking more investments on their website — how much money, and from whom, now and in the future? By taking the lion’s share from entities other than their users, they have set up concerning incentives in which the entities responsible for private data have millions on the line and are itchy to get returns, and the entities whom the private data concerns haven’t been invited to the negotiating table.

In short, I would urge them to do the following:

Make clear their funding sources, incentive model, and plans for monetization. Tell everyone the pitch they tell to private investors.
Publish their whitepaper draft and invite public comment now, rather than when it’s “finished”. Consider doing the same with the source code.
Work to inform potential users about how the technology works, to the extent that they can make informed choices about it. Destin would be a great help for this.

4privacy should generally institute a policy of greater transparency and openness by default, preferring to keep private only what they absolutely must. There is no shame in iterating on an incomplete product in the view of the public. On the contrary, I am quite proud that my business works in this manner.

The fundraising campaign quickly met its goal and will presumably only continue to grow in the coming weeks — it’s reasonably certain that it will close with at least $1M raised. Having met their goal, the product will presumably ship, and we’ll see the answers to these questions eventually. The team has a lot of work ahead of them: good luck.

Software developers have stopped caring about reliability

Sun, 17 Oct 2021 00:00:00 +0000

Of all the principles of software engineering which has fallen by the wayside in the modern “move fast and break things” mentality of ~~assholes~~ modern software developers, reliability is perhaps the most neglected, along with its cousin, robustness. Almost all software that users encounter in $CURRENTYEAR is straight-up broken, and often badly.

Honestly, it’s pretty embarassing. Consider all of the stupid little things you’ve learned how to do in order to work around broken software. Often something as simple as refreshing the page or rebooting the program to knock some sense back into it — most users can handle that. There are much stupider problems, however, and they are everywhere. Every morning, I boot, then immediately hard-reboot, my workstation, because it seems to jigger my monitors into waking up properly to do their job. On many occasions, I have used the browser dev tools to inspect a broken web page to figure out how to make it do the thing I want to do, usually something complicated like submitting a form properly (a solved problem since 1993).¹

When the average person (i.e. a non-nerd) says they “don’t get computers”, I believe them. It’s not because they’re too lazy to learn, or because they’re backwards and outdated, or can’t keep with the times. It’s because computers are hard to understand. They are enigmatic and unreliable. I know that when my phone suddenly stops delivering SMS messages mid-conversation, it’s not because I’ve been abandoned by my friend, but because I need to toggle airplane mode to reboot the modem. I know that when I middle click a link and “javascript:;” opens in a new tab, ~~an asshole~~ a developer wants me to left click it instead. Most people don’t understand this! You and I, dear reader, have built up an incredible amount of institutional knowledge about how to deal with broken computers. We’ve effectively had to reverse engineer half the software we’ve encountered to figure out just where to prod it to make it do the thing you asked. If you don’t have this background, then computers are a nightmare.

It’s hard to overstate just how much software developers have given the finger to reliability in the past 10 years or so. It’s for the simplest, silliest reasons, too, like those web forms. My web browser has been perfectly competent at submitting HTML forms for the past 28 years, but for some stupid reason some ~~asshole~~ developer decided to reimplement all of the form semantics in JavaScript, and now I can’t pay my electricity bill without opening up the dev tools. Imagine what it’s like to not know how to do that. Imagine if you were blind.

Folks, this is not okay. Our industry is characterized by institutional recklessness and a callous lack of empathy for our users. It’s time for a come-to-jesus moment. This is our fault, and yes, dear reader, you are included in that statement. We are personally responsible for this disaster, and we must do our part to correct it.

This is what you must do.

You must prioritize simplicity. You and I are not smart enough to be clever, so don’t try. As the old saying goes, there are two kinds of programs: those simple enough to obviously have no bugs, and those complicated enough to have no obvious bugs. It is by no means easier to make the simpler kind, in fact, it’s much more difficult. However, the simpler the system is, the easier it is to reason about all of its states and edge cases. You do not need a JavaScript-powered custom textbox widget. YOU DO NOT NEED A JAVASCRIPT-POWERED CUSTOM TEXTBOX WIDGET.

On the subject of state, state is the language of robustness. When something breaks, it’s because a state occured that you didn’t plan for. Think about your program in terms of this state. Design data structures that cannot represent invalid states (within reason), and then enumerate each of those possible states and check that your application does something reasonable in that situation.

Identify your error cases, plan for them, implement that plan, and then test it. Sometimes things don’t work! Most languages give you tools to identify error cases and handle them appropriately, so use them. And again, for the love of god, test it. If you commit and push a line of code that you have not personally watched run and work as expected, you have failed to do your job properly.

Prefer to use proven technologies. If you use unproven technologies, you must use them scarcely, and you must personally understand them at an intimate level. If you haven’t read the source code for the brand-new database engine you heard about on HN two weeks ago, you shouldn’t be putting it into production.²

Finally, stop letting economics decide everything you do. Yes, developers have finite time, and that time costs. Yes, users with annoying needs like accessibility and internationalization are more expensive to support than the returns they produce. You need to pay for it anyway. It’s the right thing to do. We can be profitable and empathetic. Don’t think about rushing to market first, and instead prioritize getting a good product into your user’s hands. Our users are not cattle. It is not our job to convert attention into money at their expense. We need to treat users with respect, and that means testing our goddamn code before we ship it.

Do an exercise with me. Grab a notepad and make a note every time you encounter some software bug in production (be it yours or someone else’s), or need to rely on your knowledge as a computer expert to get a non-expert system to work. Email me your list in a week.

Status update, October 2021

Fri, 15 Oct 2021 00:00:00 +0000

On this dreary morning here in Amsterdam, I’ve made my cup of coffee and snuggled my cat, and so I’m pleased to share some FOSS news with you. Some cool news today! We’re preparing for a new core product launch at sr.ht, cool updates for our secret programming language, plus news for visurf.

Simon Ser has been hard at work on expanding his soju and gamja projects for the purpose of creating a new core sourcehut product: chat.sr.ht. We’re rolling this out in a private beta at first, to seek a fuller understanding of the system’s performance characteristics, to make sure everything is well-tested and reliable, and to make plans for scaling, maintenance, and general availability. In short, chat.sr.ht is a hosted IRC bouncer which is being made available to all paid sr.ht users, and a kind of webchat gateway which will be offered to unpaid and anonymous users. I’m pretty excited about it, and looking forward to posting a more detailed announcement in a couple of weeks. In other sourcehut news, work on GraphQL continues, with paste.sr.ht landing and todo.sr.ht’s writable API in progress.

Our programming langauge project grew some interesting features this month as well, the most notable of which is probably reflection. I wrote an earlier blog post which goes over this in some detail. There’s also ongoing work to develop the standard library’s time and date support, riscv64 support is essentially done, and we’ve overhauled the grammar for switch and match statements to reduce a level of indentation for typical code. In the coming weeks, I hope to see date/time support and reflection fleshed out much more, and to see some more development on the self-hosted compiler.

Work has also continued apace on visurf, which is a project I would love to have your help with — drop me a note on #netsurf on libera.chat if you’re interested. Since we last spoke, visurf has gained support for readline-esque keybindings on the exline, a “follow” mode for keyboard navigation, Wayland clipboard support, and a few other features besides. Please help! This project will need a lot of work to complete, and much of that work is very accessible to programmers of any skill level.

Also on the subject of Netsurf and Netsurf-adjacent work, I broke ground on antiweb this month. The goal of this project is to provide a conservative CSS toolkit which allows you to build web interfaces which are compatible with marginalized browsers like Netsurf and Lynx. I should be able to migrate my blog to this framework in the foreseeable future, and ultimately the sourcehut frontend will be overhauled with this framework.

And a collection of minor updates:

I have been working on Alpine Linux for RISC-V again, and have upstreamed the necessary patches to get u-Boot to bootstrap UEFI into grub for a reasonably sane boot experience. Next up will be getting this installed onto the onboard SPI flash so that it works more like a native firmware.
I have tagged versions 1.0 of gmnisrv and gmni.
Adnan Maolood has been hard at work on godocs.io and we should soon expect a 1.0 of our gddo fork as well, which should make it more or less plug-and-play to get a working godocs instance on localhost from your local Go module cache.

That’s all for today! Take care, and thank you as always for your continued support. I’ll see you next month!

How reflection works in Hare

Tue, 05 Oct 2021 00:00:00 +0000

Note: this is a redacted copy of a blog post published on the internal development blog of a new systems programming language. The name of the project and further details are deliberately being kept in confidence until the initial release. You may be able to find it if you look hard enough — you have my thanks in advance for keeping it to yourself. For more information, see “We are building a new systems programming language”.

I’ve just merged support for reflection in Hare. Here’s how it works!

Background

“Reflection” refers to the ability for a program to examine the type system of its programming language, and to dynamically manipulate types and their values at runtime. You can learn more at Wikipedia.

Reflection from a user perspective

Let’s start with a small sample program:

use fmt;
use types;

export fn main() void = {
    const my_type: type = type(int);
    const typeinfo: *types::typeinfo = types::reflect(my_type);
    fmt::printfln("int\nid: {}\nsize: {}\nalignment: {}",
        typeinfo.id, typeinfo.sz, typeinfo.al)!;
};

Running this program produces the following output:

int
id: 1099590421
size: 4
alignment: 4

This gives us a simple starting point to look at. We can see that “type” is used as the type of the “my_type” variable, and initialized with a “type(int)” expression. This expression returns a type value for the type given in the parenthesis — in this case, for the “int” type.

To learn anything useful, we have to convert this to a “types::typeinfo” pointer, which we do via types::reflect. The typeinfo structure looks like this:

type typeinfo = struct {
	id: uint,
	sz: size,
	al: size,
	flags: flags,
	repr: repr,
};

The ID field is the type’s unique identifier, which is universally unique and deterministic, and forms part of Hare’s ABI. This is derived from an FNV-32 hash of the type information. You can find the ID for any type by modifying our little example program, or you can use the helper program in the cmd/hare/type directory of the Hare source tree.

Another important field is the “repr” field, which is short for “representation”, and it gives details about the inner structure of the type. The repr type is defined as a tagged union of all possible type representations in the Hare type system:

type repr = (alias | array | builtin | enumerated | func | pointer | slice | struct_union | tagged | tuple);

In the case of the “int” type, the representation is “builtin”:

type builtin = enum uint {
  BOOL, CHAR, F32, F64, I16, I32, I64, I8, INT, NULL, RUNE, SIZE, STR, U16, U32,
  U64, U8, UINT, UINTPTR, VOID, TYPE,
};

builtin::INT, in this case. The structure and representation of the “int” type is defined by the Hare specification and cannot be overridden by the program, so no further information is necessary. The relevant part of the spec is:

“The precision of ‘int’ and ‘uint’ are implementation-defined. ‘int’ shall be signed, and ‘uint’ shall be unsigned. Both types shall be at least 32-bits in precision. The precision in bits shall be a power of two.”

A table from the specification showing the precision ranges of each integer type

More information is provided for more complex types, such as structs.

use fmt;
use types;

export fn main() void = {
	const my_type: type = type(struct {
		x: int,
		y: int,
	});
	const typeinfo: *types::typeinfo = types::reflect(my_type);
	fmt::printfln("id: {}\nsize: {}\nalignment: {}",
		typeinfo.id, typeinfo.sz, typeinfo.al)!;
	const st = typeinfo.repr as types::struct_union;
	assert(st.kind == types::struct_kind::STRUCT);
	for (let i = 0z; i < len(st.fields); i += 1) {
		const field = st.fields[i];
		assert(field.type_ == type(int));
		fmt::printfln("\t{}: offset {}", field.name, field.offs)!;
	};
};

The output of this program is:

id: 2617358403
size: 8
alignment: 4
	x: offset 0
	y: offset 4

Here the “repr” field provides the “types::struct_union” structure:

type struct_union = struct {
	kind: struct_kind,
	fields: []struct_field,
};

type struct_kind = enum {
	STRUCT,
	UNION,
};

type struct_field = struct {
	name: str,
	offs: size,
	type_: type,
};

Makes sense? Excellent. So how does it all work?

Reflection internals

Let me first draw the curtain back from the magic “types::reflect” function:

// Returns [[typeinfo]] for the provided type.
export fn reflect(in: type) const *typeinfo = in: *typeinfo;

It simply casts the “type” value to a pointer, which is what it is. When the compiler sees an expression like let x = type(int), it statically allocates the typeinfo data structure into the program and returns a pointer to it, which is then wrapped up in the opaque “type” meta-type. The “reflect” function simply converts it to a useful pointer. Here’s the generated IR for this:

%binding.4 =l alloc8 8
storel $rt.builtin_int, %binding.4

A clever eye will note that we initialize the value to a pointer to “rt.builtin_int”, rather than allocating a typeinfo structure here and now. The runtime module provides static typeinfos for all built-in types, which look like this:

export const @hidden builtin_int: types::typeinfo = types::typeinfo {
	id = 1099590421,
	sz = 4, al = 4, flags = 0,
	repr = types::builtin::INT,
};

These are an internal implementation detail, hence “@hidden”. But many types are not built-in, so the compiler is required to statically allocate a typeinfo structure:

export fn main() void = {
	let x = type(struct { x: int, y: int });
};

data $strdata.7 = section ".data.strdata.7" { b "x" }

data $strdata.8 = section ".data.strdata.8" { b "y" }

data $sldata.6 = section ".data.sldata.6" {
  l $strdata.7, l 1, l 1, l 0, l $rt.builtin_int,
  l $strdata.8, l 1, l 1, l 4, l $rt.builtin_int,
}

data $typeinfo.5 = section ".data.typeinfo.5" {
  w 2617358403, z 4,
  l 8,
  l 4,
  w 0, z 4,
  w 5555256, z 4,
  w 0, z 4,
  l $sldata.6, l 2, l 2,
}

export function section ".text.main" "ax" $main() {
@start.0
	%binding.4 =l alloc8 8
@body.1
	storel $typeinfo.5, %binding.4
@.2
	ret
}

This has the unfortunate effect of re-generating all of these typeinfo structures every time someone uses type(struct { x: int, y: int }). We still have one trick up our sleeve, though: type aliases! Most people don’t actually use anonymous structs like this often, preferring to use a type alias to give them a name like “coords”. When they do this, the situation improves:

type coords = struct { x: int, y: int };

export fn main() void = {
	let x = type(coords);
};

data $strdata.1 = section ".data.strdata.1" { b "coords" }

data $sldata.0 = section ".data.sldata.0" { l $strdata.1, l 6, l 6 }

data $strdata.4 = section ".data.strdata.4" { b "x" }

data $strdata.5 = section ".data.strdata.5" { b "y" }

data $sldata.3 = section ".data.sldata.3" {
  l $strdata.4, l 1, l 1, l 0, l $rt.builtin_int,
  l $strdata.5, l 1, l 1, l 4, l $rt.builtin_int,
}

data $typeinfo.2 = section ".data.typeinfo.2" {
  w 2617358403, z 4,
  l 8,
  l 4,
  w 0, z 4,
  w 5555256, z 4,
  w 0, z 4,
  l $sldata.3, l 2, l 2,
}

data $type.1491593906 = section ".data.type.1491593906" {
  w 1491593906, z 4,
  l 8,
  l 4,
  w 0, z 4,
  w 3241765159, z 4,
  l $sldata.0, l 1, l 1,
  l $typeinfo.2
}

export function section ".text.main" "ax" $main() {
@start.6
	%binding.10 =l alloc8 8
@body.7
	storel $type.1491593906, %binding.10
@.8
	ret
}

The declaration of a type alias provides us with the perfect opportunity to statically allocate a typeinfo singleton for it. Any of these which go unused by the program are automatically stripped out by the linker thanks to the --gc-sections flag. Also note that a type alias is considered a distinct representation from the underlying struct type:

type alias = struct {
	ident: []str,
	secondary: type,
};

This explains the differences in the structure of the “type.1491593906” global. The struct { x: int, y: int } type is the “secondary” field of this type.

Future improvements

This is just the first half of the equation. The next half is to provide useful functions to work with this data. One such example is “types::strenum”:

// Returns the value of the enum at "val" as a string. Aborts if the value is
// not present. Note that this does not work with enums being used as a flag
// type, see [[strflag]] instead.
export fn strenum(ty: type, val: *void) str = {
	const ty = unwrap(ty);
	const en = ty.repr as enumerated;
	const value: u64 = switch (en.storage) {
	case builtin::CHAR, builtin::I8, builtin::U8 =>
		yield *(val: *u8);
	case builtin::I16, builtin::U16 =>
		yield *(val: *u16);
	case builtin::I32, builtin::U32 =>
		yield *(val: *u32);
	case builtin::I64, builtin::U64 =>
		yield *(val: *u64);
	case builtin::INT, builtin::UINT =>
		yield switch (size(int)) {
		case 4 =>
			yield *(val: *u32);
		case 8 =>
			yield *(val: *u64);
		case => abort();
		};
	case builtin::SIZE =>
		yield switch (size(size)) {
		case 4 =>
			yield *(val: *u32);
		case 8 =>
			yield *(val: *u64);
		case => abort();
		};
	case => abort();
	};

	for (let i = 0z; i < len(en.values); i += 1) {
		if (en.values[i].1.u == value) {
			return en.values[i].0;
		};
	};

	abort("enum has invalid value");
};

This is used like so:

use types;
use fmt;

type watchmen = enum {
	VIMES,
	CARROT,
	ANGUA,
	COLON,
	NOBBY = -1,
};

export fn main() void = {
	let officer = watchmen::ANGUA;
	fmt::println(types::strenum(type(watchmen), &officer))!; // Prints ANGUA
};

Additional work is required to make more useful tools like this. We will probably want to introduce a “value” abstraction which can store an arbitrary value for an arbitrary type, and helper functions to assign to or read from those values. A particularly complex case is likely to be some kind of helper for calling a function pointer via reflection, which we I may cover in a later article. There will also be some work to bring the “types” (reflection) module closer to the hare::* namespace, which already features hare::ast, hare::parse, and hare::types, so that the parser, type checker, and reflection systems are interopable and work together to implement the Hare type system.

Want to help us build this language? We are primarily looking for help in the following domains:

Architectures or operating systems, to help with ports
Compilers & language design
Cryptography implementations
Date & time implementations
Unix

If you’re an expert in a domain which is not listed, but that you think we should know about, then feel free to reach out. Experts are perferred, motivated enthusiasts are acceptable. Send me an email if you want to help!

Developers: Let distros do their job

Mon, 27 Sep 2021 00:00:00 +0000

I wrote a post some time ago titled Developers shouldn’t distribute their own software, and after a discussion on the sr.ht IRC channel today, the topic seems worthy of renewed mention. Let’s start with this: what exactly is a software distribution, anyway?

I use “software distribution” here, rather than “Linux distribution”, because it generalizes better. For example, all of the major BSD systems, plus Illumos and others besides, are software distributions, but don’t involve Linux. Some differ further still, sitting on top of another operating system, such as Nix or pkgsrc. What these systems all have in common is that they concern themselves with the distribution of software, and thus are a software distribution.

An important trait of these systems is that they function independently of the development of the software they distribute, and are overseen by a third party. For the purpose of this discussion, I will rule out package repositories which are not curated by the third-party in question, such as npm or PyPI. It is no coincidence that such repositories often end up distributing malware.

Software distributions are often volunteer-run and represent the interests of the users; in a sense they are a kind of union of users. They handle building your software for their system, and come to the table with domain-specific knowledge about the concerns of the platform that they’re working with. There are hundreds of Linux distros and each does things differently — the package maintainers are the experts who save you the burden of learning how all of them work. Instead of cramming all of your files into /opt, they will carefully sort it into the right place, make sure all of your dependencies are sorted upon installation, and make the installation of your software a single command (or click) away.

They also serve an important role as the user’s advocate. If an update ships which breaks a bunch of other packages, they’ll be in the trenches dealing with it so that the users don’t face the breakage themselves. They are also the first line of defense preventing the installation of malware on the user’s system. Many sideloaded packages for Linux include ~~telemetry~~ spyware or adware from the upstream distributor, which is usually patched out by the distribution.

Distributions are also working on innovative projects at the scale of the entire software ecosystem, and are dealing with bigger picture things than you need to concern yourself with. Here are some things which they have already solved:

Automatic updates and dependency management
Universal cryptographic signatures for all packages
Worldwide distribution and bandwidth sharing via mirrors
System-wide audits of software installed on your machine
CVE management and patch distribution
Long-term support

There are several areas of open research, too, such as reproducible builds or deterministic whole-system configuration like Nix and Guix are working on. You can take advantage of all of this innovation and research for the low price of zero dollars by standing back and letting distros handle the distribution of your software. It’s what they’re good at.

There are a few things you can do to make this work better.

Ship your software as a simple tarball. Don’t ship pre-built binaries and definitely don’t ship a “curl | bash” command. Naive users will mess up their systems when they use them.
Use widely adopted, standard build systems and methodologies. Use the standard approach for your programming language. They have already been through the gamut of distros and their operating modes are well-understood by packagers.
Ship good release notes. Distro packagers read them! Give them a head’s up about any important changes which might affect their distro.
Be picky with your dependencies and try to avoid making huge dependency trees. Bonus: this leads to better security and maintainability!
Maintain a friendly dialogue with distro maintainers if and when they come asking questions. They’re the expert on their distro, but you’re the expert on your software, and sometimes you will meet to compare notes.

One thing you shouldn’t do is go around asking distros to add your program to their repos. Once you ship your tarballs, your job is done. It’s the users who will go to their distro and ask for a new package. And users — do this! If you find yourself wanting to use some cool software which isn’t in your distro, go ask for it, or better yet, package it up yourself. For many packages, this is as simple as copying and pasting a similar package (let’s hope they followed my advice about using an industry-standard build system), making some tweaks, and building it.

Distros are quite accessible projects, packaging is usually not that difficult. Distributions always need more volunteers, and there are plenty of friendly experts at your local distro who would be pleased to help you figure out the finer details, assuming you’re prepared to stand up and do the work yourself. Once you get used to it, making and submitting a new package can take as little as 10 or 15 minutes for a simple one.

Oh, and if you are in the developer role — you are presumably also a user of both your own software and some kind of software distribution. This puts you in a really good position to champion it for inclusion in your own distro :)

P.S. Systems which invert this model, e.g. Flatpak, are completely missing the point.

Nitter and other Internet reclamation projects

Thu, 23 Sep 2021 00:00:00 +0000

The world wide web has become an annoying, ultra-commercialized space. Many websites today are prioritizing the interests of the company behind the domain, at the expense of the user’s experience and well-being. This has been a frustrating problem for several years, but lately there’s been a heartwarming trend of users fighting back against the corporate web and stepping up to help and serve each other’s needs in spite of them, through what I’ve come to think of as Internet reclamation projects.

I think the first of these which appeared on my radar was Invidious, which scrapes information off of a YouTube page and presents it in a more pleasant, user-first interface— something which NewPipe also does well for Android. These tools pry data out of YouTube’s hands and present it on a simple UI, designed for users first, with no ads or spyware, and with nice features YouTube would never add, like download links, audio mode, and offline viewing. It shows us what users want, but YouTube refuses to give.

Another project which has been particularly successful recently is Nitter, which does something similar for Twitter. Twitter’s increasingly draconian restrictions on who can access what data, and their attitude towards logged-out users in particular, has been a great annoyance to anyone who does not have, and does not want, a Twitter account, but who may still encounter Twitter links around the web. Nitter has been quite helpful in de-crapifying Twitter for these folks. I have set up an automatic redirect in my browser which takes me straight to Nitter, and I never have to see the shitty, user-hostile Twitter interface again.

Bibliogram is another attempt which has done its best to fix Instagram, but they have encountered challenges with Instagram’s strict rate limits and anti-scraping measures. Another project, Teddit, is attempting to fix Reddit’s increasingly anti-user interface, and Libreddit has similar ambitions.

All of these services are more useful, more accessible, and more inclusive than their corporate counterparts. They work better on older browsers and low-end devices. They have better performance. They aren’t spying on you. In short, they are rejecting the domestication of their users that the platforms they interact with have been trying to do. Their efforts are part of an inspiring trend of internet activism which rejects the corporate shells and walled gardens without giving up the useful data they have stolen away inside.

Here are some more services full of user-hostile behavior I’d like to see replaced with user-first, high performance, FOSS frontends:

Facebook
GitLab and GitHub
~~Medium et al~~ 2021-11-08: Check out scribe.rip!

I would be happy to redirect myself away from any of these services for a faster, lighter weight, more inclusive, user-first experience. Any others you’d like to see?

Status update, September 2021

Wed, 15 Sep 2021 00:00:00 +0000

It’s a quiet, foggy morning here in Amsterdam, and here with my fresh mug of coffee and a cuddly cat in my lap, I’d like to share the latest news on my FOSS efforts with you. Grab yourself a warm drink and a cat of your own and let’s get started.

First, a new project: visurf. I announced this a few days ago, but the short of it is that I am building a minimal Wayland-only frontend for the NetSurf web browser which uses vi-inspired keybindings. Since the announcement there has been some good progress: touch support, nsvirc, tabs, key repeat, and so on. Some notable medium-to-large efforts ahead of us include a context menu on right click, command completion and history, kinetic scrolling via touch, pinch-to-zoom, clipboard support, and a readability mode. Please help! It’s pretty easy to get involved: join the IRC channel at #netsurf on libera.chat and ask for something to do.

The programming language is also doing well. Following the codegen rewrite we have completed some long-pending refactoring to parts of the language design, which we intend to keep working on with further refinements in the coming weeks and months. We also developed a new frontend for reading the documentation in your terminal:

Other improvements include the addition of parametric format modifiers (fmt::printfln("{%}", 10, &fmt::modifiers { base = strconv::base::HEX, ... })), fnmatch, and (WIP) design improvements to file I/O, the latter relying on new struct subtyping semantics. I’m hoping that we’ll have improvements to the grammar and semantics of match expressions and tagged unions in the near future, and we are also looking into some experiments with reflection.

Many improvements have landed for SourceHut. lists.sr.ht now has a writable GraphQL API, along with the first implementation of GraphQL-native webhooks. Thanks to a few contributors, you can also now apply custom sorts to your search results on todo.sr.ht, and builds.sr.ht has grown Rocky Linux support. More details to follow in the “What’s cooking” post for the SourceHut blog.

That’s all for today! Thanks for tuning in for this update, and thanks for continuing to support our efforts. Have a great day!

visurf, a web browser based on NetSurf

Sat, 11 Sep 2021 00:00:00 +0000

I’ve started a new side project that I would like to share with you: visurf. visurf, or nsvi, is a NetSurf frontend which provides vi-inspired key bindings and a lightweight Wayland UI with few dependencies. It’s still a work-in-progress, and is not ready for general use yet. I’m letting you know about it today in case you find it interesting and want to help.

NetSurf is a project which has been on my radar for some time. It is a small web browser engine, developed in C independently of the lineage of WebKit and Gecko which defines the modern web today. It mostly supports HTML4 and CSS2, plus only a small amount of HTML5 and CSS3. Its JavaScript support, while present, is very limited. Given the epidemic of complexity in the modern web, I am pleased by the idea of a small browser, more limited in scope, which perhaps requires the cooperation of like-minded websites to support a pleasant experience.

I was a qutebrowser user for a long time, and I think it’s a great project given the constraints that it’s working in — namely, the modern web. But I reject the modern web, and qute is just as much a behemoth of complexity as the rest of its lot. Due to stability issues, I finally ended up abandoning it for Firefox several months ago.

The UI paradigm of qutebrowser’s modal interface, inspired by vi, is quite nice. I tried to use Tridactyl, but it’s a fundamentally crippled experience due to the limitations of Web Extensions on Firefox. Firefox has more problems besides — it may be somewhat more stable, but it’s ultimately still an obscenely complex, monsterous codebase, owned by an organization which cares less and less about my needs with each passing day. A new solution is called for.

Here’s where visurf comes in. Here’s a video of it in action:

I hope that this project will achieve these goals:

Create a nice new web browser
Drive interest in the development of NetSurf
Encourage more websites to build with scope-constrained browsers in mind

The first goal will involve fleshing out this web browser, and I could use your help. Please join #netsurf on irc.libera.chat, browse the issue tracker, and send patches if you are able. Some features I have in mind for the future are things like interactive link selection, a built-in readability mode to simplify the HTML of articles around the web, and automatic redirects to take advantage of tools like Nitter. However, there’s also more fundamental features to do, like clipboard support, command completion, even key repeat. There is much to do.

I also want to get people interested in improving NetSurf. I don’t want to see it become a “modern” web browser, and frankly I think that’s not even possible, but I would be pleased to see more people helping to improve its existing features, and expand them to include a reasonable subset of the modern web. I would also like to add Gemini support. I don’t know if visurf will ever be taken upstream, but I have been keeping in touch with the NetSurf team while working on it and if they’re interested it would be easy to see that through. Regardless, any improvements to visurf or to NetSurf will also improve the other.

To support the third goal, I plan on overhauling sourcehut’s frontend¹, and in the course of that work we will be building a new HTML+CSS framework (think Bootstrap) which treats smaller browsers like NetSurf as a first-class target. The goal for this effort will be to provide a framework that allows for conservative use of newer browser features, with suitable fallbacks, with enough room for each website to express its own personality in a manner which is beautiful and useful on all manner of web browsers.

Status update, August 2021

Sun, 15 Aug 2021 00:00:00 +0000

Greetings! It’s shaping up to be a beautiful day here in Amsterdam, and I have found the city much to my liking so far. If you’re in Amsterdam and want to grab a beer sometime, send me an email! I’ve been making a lot of new friends here. Meanwhile, I’ve also enjoyed a noticable increase in my productivity levels. Let’s go over the month’s accomplishments.

First, I have spent most of my time on the programming language project. I mentioned in the last update that we broke ground on a codegen rewrite, and yesterday all of our tests finally passed and I merged it. The new design is much better, and we should be able to simplify it even further still when we write the hosted compiler in the near future. This will also give us a better basis for a small number of experiments we’d like to do before finalizing the language design. Some other improvements include fleshing out our floating point math support library, a base64 module, a poll module, and parallel DNS resolution.

In SourceHut news, we shipped the lists.sr.ht GraphQL API. Future work will expand support for thread parsing and implement write operations. Presently, I am also working on a design for GraphQL-native webhooks, targetting meta.sr.ht for the initial release. sr.ht packages for Alpine 3.14 were now made available, and planned maintenance two weeks ago was the first of two fleet-wide rollouts of the upgrades to sr.ht hosted — the next is scheduled for tomorrow.

These two projects are my primary focus right now, and they’re both making good progress. In the coming month, I hope to address a few language design questions and build a more sophisticated I/O abstraction for the standard library. On sr.ht, I plan on expanding the GraphQL-native webhooks prototype and hopefully shipping it to one of the GQL APIs, along with starting on another major GQL support movement — either write support for lists.sr.ht, or the initial paste.sr.ht GQL API.

That’s all I have to share today! Thanks for tuning in.

Tips for debugging your new programming language

Wed, 11 Aug 2021 00:00:00 +0000

Say you’re building a new (compiled) programming language from scratch. You’ll inevitably have to debug programs written in it, and worse, many of these problems will lead you into deep magic, as you uncover problems with your compiler or runtime. And as you find yourself diving into the arcane arts, your tools may be painfully lacking: how do you debug code written in a language for which debuggers and other tooling simply has not been written yet?

In the implementation of my own programming language, I have faced this problem many times, and developed, by necessity, some skills around debugging with crippled tools that may lack an awareness of your language. Of course, the ultimate goal is to build out first-class debugging support, but we must have a language in the first place before we can write tools to debug it. If you find yourself in this situation, here are my recommendations.

First, I’ll echo the timeless words of Brian Kernighan:

The most effective debugging tool is still careful thought, coupled with judiciously placed print statements.

— Unix for Beginners (1979)

Classic debugging techniques are of heightened importance in this environment: first seek to isolate the problem code, then to understand the problem code, then form, and test, a hypothesis — usually with a thoughtful print statement. Often, this is enough.

Unfortunately, you may have to fire up gdb. gdb is often painful in the best of situations, but if you have to use it without debug symbols, you may find yourself shutting off the computer and seeking out rural real estate on which you can establish a new career in farming. If you can stomach it, I can offer some advice.

First, you’re going to be working in assembly, so make sure you’re familiar with how it works. I would recommend keeping the ISA manual and your ABI specification handy. If you’re smart and your language sets up stack frames properly (this is easy, do it early), you should at least have a backtrace, breakpoints at functions, and globals, though all of these will be untyped. You can write C casts to add some ad-hoc types to examine data in your process, like “print *(int *)$rdi”.

You’ll also get used to the ‘x’ command, which eXamines memory. The command format is “x/NT”, where N is the number of objects, and T is the object type: w for word (int), g for giantword (long), and h and b for halfword (short) and byte, respectively: “x/8g $rdi” will interpret rdi as an address where 8 longs are stored and print them out in hexadecimal. Of particular use is the “i” format, for “instruction”, which will disassemble from the given address:

(gdb) x/8i $rip
=> 0x5555555565c8 <rt.memcpy+4>:	mov    $0x0,%eax
   0x5555555565cd <rt.memcpy+9>:	cmp    %rdx,%rax
   0x5555555565d0 <rt.memcpy+12>:	jae    0x5555555565df <rt.memcpy+27>
   0x5555555565d2 <rt.memcpy+14>:	movzbl (%rsi,%rax,1),%ecx
   0x5555555565d6 <rt.memcpy+18>:	mov    %cl,(%rdi,%rax,1)
   0x5555555565d9 <rt.memcpy+21>:	add    $0x1,%rax
   0x5555555565dd <rt.memcpy+25>:	jmp    0x5555555565cd <rt.memcpy+9>
   0x5555555565df <rt.memcpy+27>:	leave

You can set breakpoints on the addresses you find here (e.g. “b *0x5555555565d0”), and step through one instruction at a time with the “si” command.

I also tend to do some silly workarounds to avoid having to read too much assembly. If I want to set a breakpoint in some specific place, I might do the following:

fn _break() void = void;

export fn main() void = {
    // ...some code...

    // Point of interest
    let x = y[z * q];
    _break();
    somefunc(x);

    // ...some code...
};

Then I can instruct gdb to “b _break” to break when this function is called, use “finish” to step out of the call frame, and I’ve arrived at the point of interest without having to rely on line numbers being available in my binary.

Overall, this is a fairly miserable process which can take 5-10× longer than normal debugging, but with these tips you should at least find your problems solvable. Good motivation to develop better debugging tools for your new language, eh? A future blog post might go over some of this with DWARF and possibly how to teach gdb to understand a new language natively. In the meantime, good luck!

Police to begin regular, warrant-free searches of homes for child abuse material

Tue, 10 Aug 2021 00:00:00 +0000

The Federal Bureau of Investigations announced a new initiative today to combat the proliferation of child sexual abuse materials (CSAM) in the United States. Starting next year, police will be conducting regular searches of US homes, as often as once or twice per week per home, to find child sexual abuse materials. This initiative will bring more child abusers to justice and help abuse victims to find solace in the knowledge that records of their abuse are not being shared in perpetuity.

To facilitate frequent and convenient searches, the FBI will be working with lock manufacturers to institute a new standard for home locks in the United States which permits their use via a “master key”, to be held securely by authorized government employees only. These new locks will become mandatory for all new homes next year, and a gradual process of retrofitting will take place in existing homes with the goal of having the program up to its full throughput no later than 2024.

In response to questions raised by ~~child abuse apologists~~ concerned privacy advocates, the director of the FBI stated in a press conference:

Of course, for citizens who do not possess images of child sexual abuse, there is no cause for concern. Search operatives will undergo a mandatory 2 hour training course, and will be instructed to disregard anything they find or learn in the course of their searches which does not involve CSAM. Through our partnership with industry leaders in home security, we will make the process as safe and convenient as possible, so that authorized officers may enter your home at any time and quietly conduct their business without disturbing your day. We are excited about this unprecedented opportunity to curb the distribution of child abuse material in this country.

In short, government officials are confident that the possibility of having your home searched at any time will ultimately pose little to no inconvenience to Americans, particularly with respect to the things they choose to do, people they choose to associate with, and things they say to their family and friends in ~~the privacy of~~ their homes.

The director also noted the numerous jobs which will be created to fill the increased demand for officers, and petitioned congress for the appropriate increase to their budget.

…wait. This is happening, but I got some of the details wrong.

It’s not homes which are being searched, but the digital devices we use for all of our communication and information storage and retrieval needs in contemporary life.

And it’s not lock manufacturers that are making it possible, but Apple. And the government didn’t ask: they volunteered.

And it’s not police officers, but a proprietary machine learning algorithm that no one understands.

Oh, and it’s not happening one or two times a week, but on an ongoing basis, every time you use your device.

I did get a few things, right though. The only thing which limits the scope of searches will be whichever things Apple chooses to search or not to search. And whatever Congress demands they repurpose the system to use. Ah — and it is warrant-free.

Won’t you think of the children?

source

proxy.golang.org allows many Go packages to be silently broken

Fri, 06 Aug 2021 00:00:00 +0000

GOPROXY (or proxy.golang.org) is a service through which all “go get” commands (and other module downloads) are routed. It may speed up some operations by providing a cache, and it publishes checksums and an “index” of all Go packages; but this is done at the cost of sending details of all of your module downloads to Google and imposing extra steps when using Go packages from an intranet.

This cache never expires, which can cause some problems: you can keep fetching a module from proxy.golang.org long after the upstream version has disappeared. The upstream author probably had a good reason for removing a version! Because I set GOPROXY=direct in my environment,¹ which bypasses the proxy, I’ve been made aware of a great number of Go packages which have broken dependencies and are none the wiser. They generally can’t reproduce the problem without GOPROXY=direct, which can make it a challenge to rouse up the enthusiasm for upstream to actually fix the issue. Caching modules forever can encourage bitrot.

Packages which have these issues cannot be built unless Google keeps the cache valid forever and can be trusted to treat the personal data associated with the request with respect. Furthermore, as soon as a debugging session finds its way to an absent module, you could be surprised to find that upstream is gone and that fetching or patching the code may be a challenge. This has created ticking time bombs throughout the Go ecosystem, which go undetected because GOPROXY hides the problem from developers.

If you want to check if your packages are affected by this, just set GOPROXY=direct in your environment, blow away your local cache, and build your packages again. You might uncover an unpleasant surprise.

It may be worth noting that I already have a poor opinion of the Go module mirror — it’s been DDoS’ing my servers since February.² Since I reported this, the Go team has been very opaque and non-communicative, and none of their mitigations have had a meaningful improvement. Most of the traffic is redundant — many modules are downloaded over and over again in short time intervals. I have the option of blocking their traffic, of course, but that would also block all Go programmers from fetching modules from my service. I hope they adopt my recommendation of allowing admins to configure the crawl parameters via robots.txt.

But, to be honest, the Go module mirror might not need to exist at all.

P.S. Do you have feedback on this post?

I said, in Cryptocurrency is an abject disaster, that I wanted to make my blog more constructive. As it necessarily required a critical tone, this post might have broken this promise. Taking extra care to avoid this, I made an effort to use measured, reasonable language, to address specific problems rather than making generalizations, and to avoid flamebait, and I sought second opinions on the article before publishing.

I would welcome your feedback on the results. Was this post constructive? Should I instead refrain from this kind of criticism in general? Do you have any other thoughts to share? Please email me if so.

In praise of PostgreSQL

Thu, 05 Aug 2021 00:00:00 +0000

After writing Praise for Alpine Linux, I have decided to continue writing more articles in praise of good software. Today, I’d like to tell you a bit about PostgreSQL.

Many people don’t understand how old Postgres truly is: the first release¹ was in July of 1996. It used this logo:

A “logo” which depicts the word “PostgreSQL” in a 3D chrome font bursting through a brick wall from space. No, seriously.

After 25 years of persistence, and a better logo design, Postgres stands today as one of the most significant pillars of profound achievement in free software, alongside the likes of Linux and Firefox. PostgreSQL has taken a complex problem and solved it to such an effective degree that all of its competitors are essentially obsolete, perhaps with the exception of SQLite.

For a start, Postgres is simply an incredibly powerful, robust, and reliable piece of software, providing the best implementation of SQL.² It provides a great deal of insight into its own behavior, and allows the experienced operator to fine-tune it to achieve optimal performance. It supports a broad set of SQL features and data types, with which I have always been able to efficiently store and retrieve my data. SQL is usually the #1 bottleneck in web applications, and Postgres does an excellent job of providing you with the tools necessary to manage that bottleneck.

Those tools are also exceptionally well-documented. The PostgreSQL documentation is incredibly in-depth. It puts the rest of us to shame, really. Not only do they have comprehensive reference documentation which exhaustively describes every feature, but also vast amounts of prose which explains the internal design, architecture, and operation of Postgres, plus detailed plain-English explanations of how various high-level tasks can be accomplished, complete with the necessary background to understand those tasks. There’s essentially no reason to ever read a blog post or Stack Overflow answer about how to do something with Postgres — the official docs cover every aspect of the system in great depth.

The project is maintained by a highly disciplined team of engineers. I have complete confidence in their abilities to handle matters of performance, regression testing, and security. They publish meticulously detailed weekly development updates, as well as thorough release notes that equips you with sufficient knowledge to confidently run updates on your deployment. Their git discipline is also legendary — here’s the latest commit at the time of writing:

postgres_fdw: Fix issues with generated columns in foreign tables.

postgres_fdw imported generated columns from the remote tables as plain
columns, and caused failures like "ERROR: cannot insert a non-DEFAULT
value into column "foo"" when inserting into the foreign tables, as it
tried to insert values into the generated columns.  To fix, we do the
following under the assumption that generated columns in a postgres_fdw
foreign table are defined so that they represent generated columns in
the underlying remote table:

* Send DEFAULT for the generated columns to the foreign server on insert
  or update, not generated column values computed on the local server.
* Add to postgresImportForeignSchema() an option "import_generated" to
  include column generated expressions in the definitions of foreign
  tables imported from a foreign server.  The option is true by default.

The assumption seems reasonable, because that would make a query of the
postgres_fdw foreign table return values for the generated columns that
are consistent with the generated expression.

While here, fix another issue in postgresImportForeignSchema(): it tried
to include column generated expressions as column default expressions in
the foreign table definitions when the import_default option was enabled.

Per bug #16631 from Daniel Cherniy.  Back-patch to v12 where generated
columns were added.

Discussion: https://postgr.es/m/16631-e929fe9db0ffc7cf%40postgresql.org

They’re all like this.

Ultimately, PostgreSQL is a technically complex program which requires an experienced and skilled operator to be effective. Learning to use it is a costly investment, even if it pays handsomely. Though Postgres has occasionally frustrated or confused me, on the whole my feelings for it are overwhelmingly positive. It’s an incredibly well-made product and its enormous and still-growing successes are very well-earned. When I think of projects which have made the most significant impacts on the free software ecosystem, and on the world at large, PostgreSQL has a place on that list.

My wish-list for the next YAML

Wed, 28 Jul 2021 00:00:00 +0000

YAML is both universally used, and universally reviled. It has a lot of problems, but it also is so useful in solving specific tasks that it’s hard to replace. Some new kids on the block (such as TOML) have successfully taken over a portion of its market share, but it remains in force in places where those alternatives show their weaknesses.

I think it’s clear to most that YAML is in dire need of replacement, which is why many have tried. But many have also failed. So what are the key features of YAML which demonstrate its strengths, and key weaknesses that could be improved upon?

Let’s start with some things that YAML does well, which will have to be preserved.

Hierarchical relationships emphasized with whitespace. There is no better way of representing a hierarchical data structure than by actually organizing your information visually. Note that semantically meaningful whitespace is not actually required — the use of tokens like { is acceptable — so long as, by convention, hierarchies are visually apparent.
Defined independently of its implementation. There should not be a canonical implementation of the format (though a reference implementation is, perhaps, acceptable). It should not be defined as “a config library for $language”. Interoperability is key. It must have a specification.

Easily embeds documents written in other formats. This is the chief reason that YAML still dominates in CI configuration: the ability to trivially write scripts directly into config file, without escaping anything or otherwise molesting the script.

tasks:
- configure: |
    jit_flags=""
    if [ "$(uname -m)" != "x86_64" ]
    then
        jit_flags=--without-jit
    fi
    ./configure \
        --prefix=/usr \
        $jit_flags
- build: |
    make
- test: |
    make check

Both machine- and human-editable. It’s very useful for both humans and machines to collaborate on a YAML file. For instance, humans write build manifests for their git.sr.ht repos, and then the project hub adds steps to download and apply patches from mailing lists before submitting them to the build driver. For the human’s part, the ability to easily embed scripts (see above) and write other config parameters conveniently is very helpful — everyone hates config.json.
Not a programming language. YAML entities are a problem, but we’ll talk about that separately. In general, YAML files are not programs. They’re just data. This is a good thing. If you want, you can use a separate pre-processor, like jsonnet.

What needs to be improved upon?

A much simpler grammar. No more billion laughs, please. Besides this, 90% of YAML’s features go un-used, which increases the complexity of implementations, not to mention their attack surface, for little reason.
A means of defining a schema, which can influence the interpretation of the input. YAML does this poorly. Consider the following YAML list:
```
items:
- hello
- 24
- world
```
Two of these are strings, and one is a number. Representing numbers and strings plainly like this makes it easier for humans to write, though requiring humans to write their values in a format which provides an unambiguous type is not so inconvenient as to save this trait from the cutting room floor. Leaving the ambiguity in place, without any redress, provides a major source of bugs in programs that consume YAML.
I don’t care about JSON interoperability. Being a superset of JSON is mildly useful, but not so much so as to compromise any other features or design. I’m prepared to yeet it at the first sign of code smells.

Someday I may design something like this myself, but I’m really hoping that someone else does it instead. Good luck!

Status update, July 2021

Thu, 15 Jul 2021 00:00:00 +0000

Hallo uit Nederland! I’m writing to you from a temporary workstation in Amsterdam, pending the installation of a better one that I’ll put together after I visit a furniture store today. I’ve had to slow a few things down somewhat while I prepare for this move, and I’ll continue to be slower for some time following it, but things are moving along regardless.

One point of note is that the maintainer for aerc, Reto Brunner, has stepped down from his role. I’m looking for someone new to fill his shoes; please let me know if you are interested.

As far as the language project is concerned, there has been some significant progress. We’ve broken ground on the codegen rewrite, and it’s looking much better than its predecessor. I expect progress on this front to be fairly quick. In the meanwhile, a new contributor has come onboard to help with floating-point math operations, and I merged their first patch this morning — adding math::abs, math::copysign, etc. Another contributor has been working in a similar space, and sent in an f32-to-string function last week. I implemented DNS resolution and a “dial” function as well, which you can read about in my previous post about a finger client.

I also started writing some POSIX utilities in the new language for fun:

use fmt;
use fs;
use getopt;
use io;
use main;
use os;

export fn utilmain() (io::error | fs::error | void) = {
	const cmd = getopt::parse(os::args);
	defer getopt::finish(&cmd);

	if (len(cmd.args) == 0) {
		io::copy(os::stdout, os::stdin)?;
		return;
	};

	for (let i = 0z; i < len(cmd.args); i += 1z) {
		const file = match (os::open(cmd.args[i])) {
			err: fs::error => fmt::fatal("Error opening '{}': {}",
				cmd.args[i], fs::strerror(err)),
			file: *io::stream => file,
		};
		defer io::close(file);
		io::copy(os::stdout, file)?;
	};
};

We’re still looking for someone to contribute in cryptography, and in date/time support — please let me know if you want to help.

In SourceHut news, I have mostly been focused on writing the GraphQL API for lists.sr.ht. I have made substantial progress, and I had hoped to ship the first version before publishing today’s status updates, but I was delayed due to concerns with the move abroad. I hope to also have sr.ht available for Alpine 3.14 in the near future.

Is GitHub a derivative work of GPL'd software?

Sun, 04 Jul 2021 00:00:00 +0000

GitHub recently announced a tool called Copilot, a tool which uses machine learning to provide code suggestions, inciting no small degree of controversy. One particular facet of the ensuing discussion piques my curiosity: what happens if the model was trained using software licensed with the GNU General Public License?

Disclaimer: I am the founder of a company which competes with GitHub.

The GPL is among a family of licenses considered “copyleft”, which are characterized by their “viral” nature. In particular, the trait common to copyleft works is the requirement that “derivative works” are required to publish their new work under the same terms as the original copyleft license. Some weak copyleft licenses, like the Mozilla Public License, only apply to any changes to specific files from the original code. Stronger licenses like the GPL family affect the broader work that any GPL’d code has been incorporated into.

A recent tweet by @mitsuhiko notes that Copilot can be caused to produce, verbatim, the famous fast inverse square root function from Quake III Arena: a codebase distributed under the GNU GPL 2.0 license. This raises an interesting legal question: is the work produced by a machine learning system, or even the machine learning system itself, a derivative work of the inputs to the model? Another tweet suggests that, if the answer is “no”, GitHub Copilot can be used as a means of washing the GPL off of code you want to use without obeying its license. But, what if the answer is “yes”?

I won’t take a position on this question¹, but I will point out something interesting: if the answer is “yes, machine learning models create derivative works of their inputs”, then GitHub may itself now be considered a derivative work of copyleft software. Consider this statement from GitHub’s blog post on the subject:

During GitHub Copilot’s early development, nearly 300 employees used it in their daily work as part of an internal trial.

— Albert Ziegler: A first look at rote learning in GitHub Copilot suggestions

If 300 GitHub employees used Copilot as part of their daily workflow, they are likely to have incorporated the output of Copilot into nearly every software property of GitHub, which provides network services to users. If the model was trained on software using the GNU Affero General Public License (AGPL), and the use of this model created a derivative work, this may entitle all GitHub users to receive a copy of GitHub’s source code under the terms of the AGPL, effectively forcing GitHub to become an open source project. I’m normally against GPL enforcement by means of pulling the rug out from underneath someone who made an honest mistake², but in this case it would certainly be a fascinating case of comeuppance.

Following the Copilot announcement, many of the ensuing discussions hinted to me at a broader divide in the technology community with respect to machine learning. I’ve seen many discussions having to wrestle with philosophical differences between participants, who give different answers to more fundamental questions regarding the ethics of machine learning: what rights should be, and are, afforded to the owners of the content which is incorporated into training data for machine learning? If I want to publish a work which I don’t want to be incorporated into a model, or which, if used for a model, would entitle the public to access to that model, could I? Ought I be allowed to? What if the work being used is my personal information, collected without my knowledge or consent? What if the information is used against me, for example in making lending decisions? What if it’s used against society’s interests at large?

The differences of opinion I’ve seen in the discussions born from this announcement seem to suggest a substantial divide over machine learning, which the tech community may have yet to address, or even understand the depth of. I predict that GitHub Copilot will mark one of several inciting events which start to rub some of the glamour off of machine learning technology and gets us thinking about the ethical questions it presents.³

How does IRC's federation model compare to ActivityPub?

Sat, 03 Jul 2021 00:00:00 +0000

Today’s federated revolution is led by ActivityPub, leading to the rise of services like Mastodon, PeerTube, PixelFed, and more. These new technologies have a particular approach to federation, which is coloring perceptions on what it actually means for a system to be federated at all. Today’s post will explain how Internet Relay Chat (IRC), a technology first introduced in the late 1980’s, does federation differently, and why.

As IRC has aged, many users today have only ever used a few networks, such as Liberachat (or Freenode, up until several weeks ago), which use a particular IRC model which does not, at first glance, appear to utilize federation. After all, everyone types “irc.libera.chat” into their client and they all end up on the same network and in the same namespace. However, this domain name is backed by a round-robin resolver which will connect you to any of several dozen servers, which are connected to each other¹ and exchange messages on behalf of the users who reside on each. This is why we call them IRC networks — each is composed of a network of servers that work together.

But why can’t I send messages to users on OFTC from my Libera Chat session? Well, IRC networks are federated, but they are typically a closed federation, such that each network forms a discrete graph of servers, not interconnected with any of the others. In ActivityPub terms, imagine a version of Mastodon where, instead of automatically federating with new instances, server operators whitelisted each one, forming a closed graph of connected instances. Organize these servers under a single named entity (“Mastonet” or something), and the result is an “ActivityPub network” which operates in the same sense as a typical “IRC network”.

In contrast to Mastodon’s open federation, allowing any server to peer with any others without prior agreement between their operators, most IRC networks are closed. The network’s servers may have independent operators, but they operate together under a common agreement, rather than the laissez-faire approach typical of² ActivityPub servers. The exact organizational and governance models vary, but many of these networks have discrete teams of staff which serve as moderators³, often unrelated to the people responsible for the servers. The social system can be designed independently of the technology.

Among IRC networks, there are degrees of openness. Libera Chat, the largest network, is run by a single governing organization, using servers donated by (and in the possession of) independent sponsors. Many smaller networks are run on as few as one server, and some larger networks (particularly older ones) are run by many independent operators acting like more of a cooperative. EFnet, the oldest network, is run in this manner — you can even apply to become an operator yourself.

We can see from this that the idea of federation is flexible, allowing us to build a variety of social and operational structures. There’s no single right answer — approaches like IRC are able to balance many different benefits and drawbacks of their approach, such as balancing a reduced level of user mobility with a stronger approach to moderation and abuse reduction, while simultaneously enjoying the cost and scalability benefits of a federated design. Other federations, like Matrix, email, and Usenet, have their own set of tradeoffs. What unifies them is the ability to scale to a large size without expensive infrastructure, under the social models which best suit their users’ needs, without a centralizing capital motive.

You can't capture the nuance of my form fields

Sun, 27 Jun 2021 00:00:00 +0000

Check out this text box:

Here are some of the nuances of using this text box on my operating system (Linux) and web browser (Firefox):

Double clicking selects a word, and triple-clicking selects the whole line. If I double- or triple-click-and-hold, I can drag the mouse to expand the selection word-wise or line-wise, not just character-wise. This works with the paragraphs of text in the body of this blog post, too.
Holding control and pressing right will move move word-wise through the file. It always moves to the start or end of the next or prior word, so pressing “control+left, control+left, control+right” will end up in a different position than “control+left” alone. Adding “shift” to any of these will mutate the text selection.
Clicking any of the whitespace after the end of the text will put the cursor after the last character, even if you click to the left of the last character. This makes it easy to start appending text to the end.
Clicking and dragging from any point, I can drag the mouse straight upward, exceeding the bounds of the text box or even the entire web browser, to select all text from that point to the start of the text box. (Thanks minus for mentioning this one)
Selecting text and middle clicking anywhere will paste the text at the clicked location. This uses a separate, distinct clipboard from the one accessed with ctrl+c/ctrl+v. I can also use shift+insert to paste text from this secondary clipboard (this is called the “primary selection”).

I rely on all of these nuances when I use form controls in my everyday life. This is just for English, by the way. I often type in Japanese, which has an entirely alien set of nuances. Here’s what that looks like on Android (mobile is another beast entirely, too!):

If you're seeing this, your browser doesn't support HTML5 video, or webm, idk.

Here’s another control:

There’s an invisible edit buffer, so I can type “Pennsylvania” (or just P) to select what I want. I can type “New” and then press down to select “New Jersey”. If I make a mistake and I’ve kept track of what I’ve typed in my head, I can use backspace to make a correction, and it just works. I have lived in both of these places, and worked both of these keystrokes into my muscle memory. Filling out a form with my address on it and using an input box like this to select my state of residence takes me less than a second.

You cannot capture all of this nuance in a home-grown form control, or even anything close to it, but many JavaScript programmers do it anyway. Whenever I encounter a custom form control, the time required to complete the form increases from under a second to as much as a minute.

For myself, this is just very annoying. Imagine the same situation if you were blind. The standard form inputs work everywhere, and are designed with accessibility in mind, so you’re used to them and can easily fill in forms which use the standard browser controls. But, when you hit a JavaScript-powered organic cage-free non-GMO text box, you’re screwed.

There are hundreds of little nuances that users learn to use their computers efficiently. The exact features a user relies on will vary between operating systems, browsers, hardware, natural languages, physical ability, and personal preferences and experience. There are dozens of tiny workflows that people depend on every day that have never even occurred to you.

Making a custom form control with JavaScript is going to make life worse for a lot of people. Just don’t do it. The browser’s built-in controls are quite sufficient.

A finger client

Thu, 24 Jun 2021 00:00:00 +0000

This is a short follow-up to the io_uring finger server article posted about a month ago. In the time since, we have expanded our language with a more complete networking stack, most importantly by adding a DNS resolver. I have used these improvements to write a small client implementation of the finger protocol.

use fmt;
use io;
use net::dial;
use os;
use strings;

@init fn registersvc() void = dial::registersvc("tcp", "finger", [], 79);
@noreturn fn usage() void = fmt::fatal("Usage: {} <user>[@<host>]", os::args[0]);

export fn main() void = {
	if (len(os::args) != 2) usage();

	const items = strings::split(os::args[1], "@");
	defer free(items);
	if (len(items) == 0) usage();

	const user = items[0];
	const host = if (len(items) == 1) "localhost"
		else if (len(items) == 2) items[1]
		else usage();

	match (execute(user, host)) {
		err: dial::error => fmt::fatal(dial::strerror(err)),
		err: io::error => fmt::fatal(io::strerror(err)),
		void => void,
	};
};

fn execute(user: str, host: str) (void | dial::error | io::error) = {
	const conn = dial::dial("tcp", host, "finger")?;
	defer io::close(conn);
	fmt::fprintf(conn, "{}\r\n", user)?;
	io::copy(os::stdout, conn)?;
};

Technically, we could do more, but I chose to just address the most common use-case for finger servers in active use today: querying a specific user. Expanding this with full support for all finger requests would probably only grow this code by 2 or 3 times.

Our language now provides a net::dial module, inspired by Go’s net.Dial and the Plan 9 dial function Go is derived from. Our dial actually comes a bit closer to Plan 9 by re-introducing the service parameter — Plan 9’s “tcp!example.org!http” becomes net::dial(“tcp”, “example.org”, “http”) in our language — which we use to find the port (unless you specify it in the address). The service parameter is tested against a small internal list of known services, and against /etc/services. We also automatically perform an SRV lookup for “_finger._tcp.example.org”, so most programs written in our language will support SRV records with no additional effort.

In our client code, we can see that the @init function adds “finger” to the list of known internal services. @init functions run on start-up, and this one just lets dial know about our protocol. Our network stack is open to extension in other respects, too — unlike Go, third-party libraries can define new protocol handlers for dial as well, perhaps opening it up in the future to networks like AF_BLUETOOTH, AF_AX25, and so on, complete with support for network-appropriate addresses and resolver functionality.

The rest is pretty straightforward! We just parse the command line, dial the server, write the username to it, and splice the connection into stdout. Much simpler than the server. Future improvements might rewrite the CRLF to LF, but that’s not particularly important.

Status update, June 2021

Tue, 15 Jun 2021 00:00:00 +0000

Hiya! Got another status update for you. First, let me share this picture that my dad and I took on our recent astronomy trip (click for full res):

A long-exposure picture of the night sky. Thousands of stars are visible, as well as the band of the milky way.

Bonus Venus:

A bright white circle against a dark background

So, what’s new? With SourceHut, there are a few neat goings-on. For one, thanks to Michael Forney putting the finishing touches on the patchset, the long-awaited NetBSD image is now available for builds.sr.ht. Also, the initial lists.sr.ht GraphQL API design is in place, and Simon Ser is working on a new and improved implementation of email discussion parsing for us to use. I’ve also redesigned the registration & onboarding flow based on a maintainer/contributor distinction, which should help people understand how sourcehut works a bit better. Also, as promised, the writable GraphQL API for builds.sr.ht is now available.

I had been working on a new feature for the secret programming language, but in the course of implementing it, it became clear to me that we need to take a step back and do some deep refactoring in the compiler. This will probably occupy us for a couple of months. Even so, some improvements in the standard library have been made and shall continue to be made. You may have seen a few weeks ago that I wrote a finger server in the new language, and there’s a bunch of code for you to read there if you’re interested in learning more.

I also spent some time this month on Simon’s gamja and soju projects. Libera.chat is running an experimental instance of gamja for their webchat, and I’ve helped Simon incorporate some of their feedback and apply a layer of polish to the client. I’m also working on generalizing soju a bit so that we can eventually utilize it to offer a hosted IRC bouncer for sr.ht users.

That’s all I have to share for now. My foci have been on sourcehut and the secret language, and will continue to be those. I plan on advancing the work on the GraphQL APIs for sr.ht and ideally shipping an initial version of the lists.sr.ht API in a few weeks. I’ll share more news about the new language when it’s ready. Until next time!

Provided "as is", without warranty of any kind

Mon, 14 Jun 2021 00:00:00 +0000

The MIT license contains the following text, in all uppercase no less:

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.

The BSD licenses, GPL family of licenses, Apache 2.0, Mozilla Public License, and likely any other license you’d care to name, have similar clauses of their own. It’s worth taking a moment to consider the implications of this statement and what it says about the social aspects of free and open source software.

Many people who rely on free and open source software feel entitled to some degree of workitude or support from the developers, or think that the developers have a responsibility to provide good maintenance, or any maintenance at all, for their work. This is simply not true. All free and open source software disclaims all responsibility for your use of them for any purpose, often in all capital letters.

Some maintainers will allow you to negotiate additional terms with them, for example through the sale of a support contract, for which you may receive such a guarantee. If you have not made such an agreement with your maintainers, they have no responsibility to provide you with any support or assurance of quality. That means that they do not have to solve your bug reports or answer your questions. They do not have to review and apply your patch. They do not have to write documentation. They do not have to port it to your favorite platform. You are not entitled to the blood, sweat, and tears of the maintainers of the free & open source software you use.

It is nice when a maintainer offers you their time, but by no means are they required to. FOSS is what you make of it. You have the right to make the changes you need from the software yourself, and you are the only person that you can reliably expect to do it. You aren’t entitled to the maintainer’s time, but you are, per the open source definition and free software definition, entitled to change the software, distribute your changes to others, and to sell the software with or without those changes.

Though this idea is important for users of free software to understand, it’s equally important that maintainers understand this as well. We have a problem with burn-out in the free software community, wherein a maintainer, feeling pressured into accepting greater responsibility over their work from a community that increasingly depends on them, will work themselves half to death for little or no compensation. You should not do this! That wasn’t part of the deal!

As a maintainer, you need to be prepared to say “no”. Working on your project should never feel like a curse. You started it for a reason — remember that reason. Was it to lose your sanity? Or was it to have fun? Was it to solve a specific problem you had? Or was it to solve problems for someone you’ve never met? Remember these goals, and stay true to them. If you’re getting stressed out, stop. You can always walk away. You don’t owe anything to anyone.

If you enjoy the work, and you enjoy helping others, that’s great! Of course, you are allowed to help your users out if you so choose. However, I recommend that you manage their expectations, and make sure you’re spending time cultivating a healthy relationship between you, your colleagues, and your users. FOSS projects are made out of people, and maintaining that social graph is as important as maintaining the code. Make sure everyone understands the rules and talk about your frustrations with each other. Having an active dialogue can prevent problems before they happen in the first place.

I will be moving to the Netherlands

Mon, 07 Jun 2021 00:00:00 +0000

I had been planning a move to the Netherlands for a while, at least until a large COVID-shaped wrench was thrown into the gears. However, I was fully vaccinated by early April, and there are signs of the border opening up now, so my plans have been slowly getting back on track. I sent off my visa application today, and assuming I can navigate the pandemic-modified procedures, I should be able to make my move fairly soon. It’s a little bit intimidating, but I am looking forward to it!

Quick note: I am looking for temporary housing in NL; somewhere I can stay for 6-12 months with a permanent mailing address for receiving immigration-related documents. I would prefer to rent out a room than to use some kind of commercial accommodation, to be certain that I can receive mail from the immigration services for the duration of the process. Please shoot me an email if you’ve got a lead! I’d rather meet someone through the FOSS community than dig through ~~Craigslist~~ Marktplaats from overseas.

I have felt a kind of dissonance with my home country of the United States for a long time now, and I have found it very difficult to resolve. I am not of one mind with my peers in this country on many issues; social, economic, and political. Even limiting this inquiry to matters related to FOSS, it’s quite clear that the FOSS community in Europe is much stronger than in America. In the United States, capitalism is the secular religion, and my values, in FOSS and otherwise, are incompatible with the American ethos.¹

Leaving the US is a selfish choice. I could stay here to get involved in solving these problems, but I chose to leave for a place which has already made much more progress on them. Ultimately, this is the only life I’m gonna get, and I have decided not to spend it on politics. I’ll spare you from the rest of the details. I’ll also acknowledge that I’m very privileged to even have this choice at all. Because I know how difficult it is to leave, for reasons unique to each person’s own situation, I don’t hold anyone who stays behind accountable for their country’s cruelties.²

So, why the Netherlands? I considered many options. For instance, I am fluent in Japanese, have an existing social network there, and understand their immigration laws and process. However, as much as I love to visit, I’m not on their cultural wavelength. Integration would pose a challenge. That said, I have also spent a lot of time in the EU, which is a hot spot for the FOSS ecosystem. Access to any EU country with a path to citizenship opens up access to the rest of the EU, making it a very strong choice with lots of second choices easily available.

The Netherlands is an attractive place in these respects. It is relatively easy for me to obtain a visa there, for one, but it also ranks very highly in numerous respects: social, economic, political, and basic happiness. I have many friends in Europe and I won’t have to worry too much about establishing a new social network there.

There are also some risks. Housing is expensive and only getting more so. Also, like the rest of the world, how NL will emerge from the crises of the pandemic remains to be seen, and many countries are likely to suffer from long-term consequences in all aspects of life. They are also already dealing with an influx of immigrants, and it’s quite possible that I will face some social and legal challenges in the future.

Despite these and other risks, I am optimistic about this change. The path to citizenship takes only five years, and after many careful inquiries into the process, I believe my plans for getting there are on very solid footing. I have been studying Dutch throughout much of the pandemic, and I’m not having much trouble with it — I intend to achieve fluency. Integration is well within my grasp. I expect to look back on this transition with confidence in a decision well-made.

Leuk jullie te ontmoeten, Nederland!

Oh, and yes: I will be re-locating SourceHut, the incorporated entity, to the Netherlands, and gradually moving our infrastructure over the pond. Details regarding these plans will eventually appear on the ops wiki. Users can expect little to no disruption from the change.

Build your project in our new language

Sun, 30 May 2021 00:00:00 +0000

Do you have a new systems programming project on your todo list? If you’re feeling adventurous, I would like you to give it a crack in our new systems programming language, and to use it to drive improvements in the less-developed areas of our standard library.

Note: we have enough projects on board now. Keep an eye on the blog, I’ll publish another announcement when we’re ready for more.

Are you making a new coreutils implementation? A little OS kernel? A new shell? A GUI toolkit? Database system? Web server? Whatever your systems programming use-case, we think that our language is likely to be a good fit for you, and your help in proving that, and spurring development to rise to meet your needs, would be quite welcome.

Here’s our pitch:

XXXX is a systems programming language designed to be simple and robust. XXXX uses a static type system, manual memory management, and a minimal runtime. It is well-suited to writing operating systems, system tools, compilers, networking software, and other low-level, high performance tasks.

You can get a peek at how it feels by reading about the finger server I wrote with it.

Sounds interesting? Please tell me about your project idea!

Using io_uring to make a high-performance... finger server

Mon, 24 May 2021 00:00:00 +0000

I’m working on adding a wrapper for the Linux io_uring interface to my secret programming language project. To help learn more about io_uring and to test out the interface I was designing, I needed a small project whose design was well-suited for the value-add of io_uring. The Finger protocol is perfect for this! After being designed in the 70’s and then completely forgotten about for 50 years, it’s the perfect small and simple network protocol to test drive this new interface with.

In short, finger will reach out to a remote server and ask for information about a user. It was used back in the day to find contact details like the user’s phone number, office address, email address, sometimes their favorite piece of ASCII art, and, later, a summary of the things they were working on at the moment. The somewhat provocative name allegedly comes from an older usage of the word to mean “a snitch” or a member of the FBI. The last useful RFC related to Finger is RFC 1288, circa 1999, which will be our reference for this server. If you want to give it a test drive, try this to ping the server we’ll be discussing today:

printf 'drew\r\n' | nc drewdevault.com 79

You might also have the finger command installed locally (try running “finger drew@drewdevault.com”), and you can try out the Castor browser by sourcehut user ~julienxx for a graphical experience.

And what is io_uring? It is the latest interface for async I/O on Linux, and it’s pretty innovative and interesting. The basic idea is to set up some memory which is shared between the kernel and the userspace program, and stash a couple of ring buffers there that can be updated with atomic writes. Userspace appends submission queue entries (SQEs) to the submission queue (SQ), and the kernel processes the I/O requests they describe and then appends completion queue events (CQEs) to the completion queue (CQ). Interestingly, both sides can see this happening without entering the kernel with a syscall, which is a major performance boost. It more or less solves the async I/O problem for Linux, which Linux (and Unix at large) has struggled to do for a long time.

With that the background in place, I’m going to walk you through my finger server’s code. Given that this is written in an as-of-yet unreleased programming language, I’ll do my best to help you decipher the alien code.

A quick disclaimer

This language, the standard library, and the interface provided by linux::io_uring, are all works in progress and are subject to change. In particular, this program will become obsolete when we design a portable I/O bus interface, which on Linux will be backed by io_uring but on other systems will use kqueue, poll, etc.

As a rule of thumb, anything which uses rt:: or linux:: is likely to change or be moved behind a portable abstraction in the future.

Let’s start with the basics:

use fmt;
use getopt;
use net::ip;
use strconv;
use unix::passwd;

def MAX_CLIENTS: uint = 128;

export fn main() void = {
	let addr: ip::addr = ip::ANY_V6;
	let port = 79u16;
	let group = "finger";

	const cmd = getopt::parse(os::args,
		"finger server",
		('B', "addr", "address to bind to (default: all)"),
		('P', "port", "port to bind to (default: 79)"),
		('g', "group", "user group enabled for finger access (default: finger)"));
	defer getopt::finish(&cmd);

	for (let i = 0z; i < len(cmd.opts); i += 1) {
		const opt = cmd.opts[i];
		switch (opt.0) {
			'B' => match (ip::parse(opt.1)) {
				a: ip::addr => addr = a,
				ip::invalid => fmt::fatal("Invalid IP address"),
			},
			'P' => match (strconv::stou16(opt.1)) {
				u: u16 => port = u,
				strconv::invalid => fmt::fatal("Invalid port"),
				strconv::overflow => fmt::fatal("Port exceeds range"),
			},
			'g' => group = opt.1,
		};
	};

	const grent = match (passwd::getgroup(group)) {
		void => fmt::fatal("No '{}' group available", group),
		gr: passwd::grent => gr,
	};
	defer passwd::grent_finish(grent);
};

None of this code is related to io_uring or finger, but just handling some initialization work. This is the daemon program, and it will accept some basic configuration via the command line. The getopt configuration shown here will produce the following help string:

$ fingerd -h
fingerd: finger server

Usage: ./fingerd [-B <addr>] [-P <port>] [-g <group>]

-B <addr>: address to bind to (default: all)
-P <port>: port to bind to (default: 79)
-g <group>: user group enabled for finger access (default: finger)

The basic idea is to make finger access opt-in for a given Unix account by adding them to the “finger” group. The “passwd::getgroup” lookup fetches that entry from /etc/group to identify the list of users for whom we should be serving finger access.

let serv = match (net::listen(addr, port,
		256: net::backlog, net::reuseport)) {
	err: io::error => fmt::fatal("listen: {}", io::strerror(err)),
	l: *net::listener => l,
};
defer net::shutdown(serv);
fmt::printfln("Server running on :{}", port)!;

Following this, we set up a TCP listener. I went for a backlog of 256 connections (overkill for a finger server, but hey), and set reuseport so you can achieve CLOUD SCALE by running several daemons at once.

Next, I set up the io_uring that we’ll be using:

// The ring size is 2 for the accept and sigfd read, plus 2 SQEs for
// each of up to MAX_CLIENTS: either read/write plus a timeout, or up to
// two close SQEs during cleanup.
static assert(MAX_CLIENTS * 2 + 2 <= io_uring::MAX_ENTRIES);

let params = io_uring::params { ... };
let ring = match (io_uring::setup(MAX_CLIENTS * 2 + 2, &params)) {
	ring: io_uring::io_uring => ring,
	err: io_uring::error => fmt::fatal(io_uring::strerror(err)),
};
defer io_uring::finish(&ring);

If we were running this as root (and we often are, given that fingerd binds to port 79 by default), we could go syscall-free by adding io_uring::setup_flags::SQPOLL to params.flags, but this requires more testing on my part so I have not added it yet. With this configuration, we’ll need to use the io_uring_enter syscall to submit I/O requests.

We also have to pick a queue size when setting up the uring. I planned this out so that we can have two SQEs in flight for every client at once — one for a read/write request and its corresponding timeout, or for the two “close” requests used when disconnecting the client — plus two extra entries, one for the “accept” call, and another to wait for signals from a signalfd.

Speaking of signalfds:

let mask = rt::sigset { ... };
rt::sigaddset(&mask, rt::SIGINT)!;
rt::sigaddset(&mask, rt::SIGTERM)!;
rt::sigprocmask(rt::SIG_BLOCK, &mask, null)!;
let sigfd = signalfd::signalfd(-1, &mask, 0)!;
defer rt::close(sigfd)!;

const files = [net::listenerfd(serv) as int, sigfd];
io_uring::register_files(&ring, files)!;

const sqe = io_uring::must_get_sqe(&ring);
io_uring::poll_add(sqe, 1, rt::POLLIN: uint, flags::FIXED_FILE);
io_uring::set_user(sqe, &sigfd);

We haven’t implemented a high-level signal interface yet, so this is just using the syscall wrappers. I chose to use a signalfd here so I can monitor for SIGINT and SIGTERM with my primary I/O event loop, to (semi-)gracefully¹ terminate the server.

This also happens to show off our first SQE submission. “must_get_sqe” will fetch the next SQE, asserting that there is one available, which relies on the math I explained earlier when planning for our queue size. Then, we populate this SQE with a “poll_add” operation, which polls on the first fixed file descriptor. The “register” call above adds the socket and signal file descriptors to the io_uring’s list of “fixed” file descriptors, and so with “flags::FIXED_FILE” this refers to the signalfd.

We also set the user_data field of the SQE with “set_user”. This will be copied to the CQE later, and it’s necessary that we provide a unique value in order to correlate the CQE back to the SQE it refers to. We can use any value, and the address of the signalfd variable is a convenient number we can use for this purpose.

There’s one more step — submitting the SQE — but that’ll wait until we set up more I/O. Next, I have set up a “context” structure which will store all of the state the server needs to work with, to be passed to functions throughout the program.

type context = struct {
	users: []str,
	clients: []*client,
	uring: *io_uring::io_uring,
};

// ...

const ctx = context {
	users = grent.userlist,
	uring = &ring,
	...
};

The second “...” towards the end is not for illustrative purposes - it sets all of the remaining fields to their default values (in this case, clients becomes an empty slice).

Finally, this brings us to the main loop:

let accept_waiting = false;
for (true) {
	const peeraddr = rt::sockaddr { ... };
	const peeraddr_sz = size(rt::sockaddr): uint;
	if (!accept_waiting && len(ctx.clients) < MAX_CLIENTS) {
		const sqe = io_uring::must_get_sqe(ctx.uring);
		io_uring::accept(sqe, 0, &peeraddr, &peeraddr_sz,
			0, flags::FIXED_FILE);
		io_uring::set_user(sqe, &peeraddr);
		accept_waiting = true;
	};

	io_uring::submit(&ring)!;

	let cqe = match (io_uring::wait(ctx.uring)) {
		err: io_uring::error => fmt::fatal("Error: {}",
			io_uring::strerror(err)),
		cqe: *io_uring::cqe => cqe,
	};
	defer io_uring::cqe_seen(&ring, cqe);

	const user = io_uring::get_user(cqe);
	if (user == &peeraddr) {
		accept(&ctx, cqe, &peeraddr);
		accept_waiting = false;
	} else if (user == &sigfd) {
		let si = signalfd::siginfo { ... };
		rt::read(sigfd, &si, size(signalfd::siginfo))!;
		fmt::errorln("Caught signal, terminating")!;
		break;
	} else for (let i = 0z; i < len(ctx.clients); i += 1) {
		let client = ctx.clients[i];
		if (user == client) {
			dispatch(&ctx, client, cqe);
			break;
		};
	};
};

At each iteration, assuming we have room and aren’t already waiting on a new connection, we submit an “accept” SQE to fetch the next incoming client. This SQE accepts an additional parameter to write the client’s IP address to, which we provide via a pointer to our local peeraddr variable.

We call “submit” at the heart of the loop to submit any SQEs we have pending (including both the signalfd poll and the accept call, but also anything our future client handling code will submit) to the io_uring, then wait the next CQE from the kernel.

When we get one, we defer a “cqe_seen”, which will execute at the end of the current scope (i.e. the end of this loop iteration) to advance our end of the completion queue, then figure out what I/O request was completed. The code earlier sets up SQEs for the accept and signalfd, which we check here. If a signal comes in, we read the details to acknowledge it and then terminate the loop. We also check if the user data was set to the address of any client state data, which we’ll use to dispatch for client-specific I/O later on. If a new connection comes in:

fn accept(ctx: *context, cqe: *io_uring::cqe, peeraddr: *rt::sockaddr) void = {
	const fd = match (io_uring::result(cqe)) {
		err: io_uring::error => fmt::fatal("Error: accept: {}",
			io_uring::strerror(err)),
		fd: int => fd,
	};
	const peer = net::ip::from_native(*peeraddr);
	const now = time::now(time::clock::MONOTONIC);
	const client = alloc(client {
		state = state::READ_QUERY,
		deadline = time::add(now, 10 * time::SECOND),
		addr = peer.0,
		fd = fd,
		plan_fd = -1,
		...
	});
	append(ctx.clients, client);
	submit_read(ctx, client, client.fd, 0);
};

This is fairly self-explanatory, but we do see the first example of how to determine the result from a CQE. The result field of the CQE structure the kernel fills in is set to what would normally be the return value of the equivalent syscall, and “linux::io_uring::result” is a convenience function which translates negative values (i.e. errno) into a more idiomatic result type.

We choose a deadline here, 10 seconds from when the connection is established, for the entire exchange to be completed by. This helps to mitigate Slowloris attacks, though there are more mitigations we could implement for this.

Our client state is handled by a state machine, which starts in the “READ_QUERY” state. Per the RFC, the client will be sending us a query, followed by a CRLF. Our initial state is prepared to handle this. The full client state structure is as follows:

type state = enum {
	READ_QUERY,
	OPEN_PLAN,
	READ_PLAN,
	WRITE_RESP,
	WRITE_ERROR,
};

type client = struct {
	state: state,
	deadline: time::instant,
	addr: ip::addr,
	fd: int,
	plan_fd: int,
	plan_path: *const char,
	xbuf: [2048]u8,
	buf: []u8,
};

Each field will be explained in due time. We add this to our list of active connections and call “submit_read”.

fn submit_read(ctx: *context, client: *client, fd: int, offs: size) void = {
	const sqe = io_uring::must_get_sqe(ctx.uring);
	const maxread = len(client.xbuf) / 2;
	io_uring::read(sqe, fd, client.xbuf[len(client.buf)..]: *[*]u8,
		maxread - len(client.buf), offs: u64, flags::IO_LINK);
	io_uring::set_user(sqe, client);

	let ts = rt::timespec { ... };
	time::instant_to_timespec(client.deadline, &ts);
	const sqe = io_uring::must_get_sqe(ctx.uring);
	io_uring::link_timeout(sqe, &ts, timeout_flags::ABS);
};

I’ve prepared two SQEs here. The first is a read, which will fill half of the client buffer with whatever they send us over the network (why half? I’ll explain later). It’s configured with “flags::IO_LINK”, which will link it to the following request: a timeout. This will cause the I/O to be cancelled if it doesn’t complete before the deadline we set earlier. “timeout_flags::ABS” specifies that the timeout is an absolute timestamp rather than a duration computed from the time of I/O submission.

I set the user data to the client state pointer, which will be used the next time we have a go-around in the main event loop (feel free to scroll back up if you want to re-read that bit). The event loop will send the CQE to the dispatch function, which will choose the appropriate action based on the current client state.

fn dispatch(ctx: *context, client: *client, cqe: *io_uring::cqe) void = {
	match (switch (client.state) {
		state::READ_QUERY => client_query(ctx, client, cqe),
		state::OPEN_PLAN => client_open_plan(ctx, client, cqe),
		state::READ_PLAN => client_read_plan(ctx, client, cqe),
		state::WRITE_RESP, state::WRITE_ERROR =>
			client_write_resp(ctx, client, cqe),
	}) {
		err: error => disconnect_err(ctx, client, err),
		void => void,
	};
};

What’s the difference between match and switch? The former works with types, and switch works with values. We might attempt to merge these before the language’s release, but for now the distinction simplifies our design.

I’ve structured the client state machine into four states based on the kind of I/O they handle, plus a special case for error handling:

Reading the query from the client
Opening the plan file for the requested user
Reading from the plan file
Forwarding its contents to the client

Each circle in this diagram represents a point where we will submit some I/O to our io_uring instance and return to the event loop. If any I/O resulted in an error, we’ll follow the dotted line to the error path, which transmits the error to the user (and if an error occurs during error transmission, we’ll immediately disconnect them, but that’s not shown here).

I need to give a simplified introduction to error handling in this new programming language before we move on, so let’s take a brief detour. In this language, we require the user to explicitly do something about errors. Generally speaking, there are three somethings that you will do:

Some context-appropriate response to an error condition
Bumping the error up to the caller to deal with
Asserting that the error will never happen in practice

The latter two options have special operators (”?” and “!”, respectively, used as postfix operators on expressions which can fail), and the first option is handled manually in each situation as appropriate. It’s usually most convenient to use ? to pass errors up the stack, but the buck has got to stop somewhere. In the code we’ve seen so far, we’re in or near the main function — the top of the call stack — and so have to handle these errors manually, usually by terminating the program with “!”. But, when a client causes an error, we cannot terminate the program without creating a DoS vulnerability. This “dispatch” function sets up common client error handling accordingly, allowing later functions to use the “?” operator to pass errors up to it.

To represent the errors themselves, we use a lightweight approach to tagged unions, similar to a result type. Each error type, optionally with some extra metadata, is enumerated, along with any possible successful types, as part of a function’s return type. The only difference between an error type and a normal type is that the former is denoted with a “!” modifier — so you can store any representable state in an error type.

I also wrote an “errors” file which provides uniform error handling for all of the various error conditions we can expect to occur in this program. This includes all of the error conditions that we define ourselves, as well as any errors we expect to encounter from modules we depend on. The result looks like this:

use fs;
use io;
use linux::io_uring;

type unexpected_eof = !void;
type invalid_query = !void;
type no_such_user = !void;
type relay_denied = !void;
type max_query = !void;
type error = !(
	io::error |
	fs::error |
	io_uring::error |
	unexpected_eof |
	invalid_query |
	no_such_user |
	relay_denied |
	max_query
);

fn strerror(err: error) const str = {
	match (err) {
		err: io::error => io::strerror(err),
		err: fs::error => fs::strerror(err),
		err: io_uring::error => io_uring::strerror(err),
		unexpected_eof => "Unexpected EOF",
		invalid_query => "Invalid query",
		no_such_user => "No such user",
		relay_denied => "Relay access denied",
		max_query => "Maximum query length exceeded",
	};
};

With an understanding of error handling, we can re-read the dispatch function’s common error handling for all client issues:

fn dispatch(ctx: *context, client: *client, cqe: *io_uring::cqe) void = {
	match (switch (client.state) {
		state::READ_QUERY => client_query(ctx, client, cqe),
		state::OPEN_PLAN => client_open_plan(ctx, client, cqe),
		state::READ_PLAN => client_read_plan(ctx, client, cqe),
		state::WRITE_RESP, state::WRITE_ERROR =>
			client_write_resp(ctx, client, cqe),
	}) {
		err: error => disconnect_err(ctx, client, err),
		void => void,
	};
};

Each dispatched-to function returns a tagged union of (void | error), the latter being our common error type. If they return void, we do nothing, but if an error occurred, we call “disconnect_err”.

fn disconnect_err(ctx: *context, client: *client, err: error) void = {
	fmt::errorfln("{}: Disconnecting with error: {}",
		ip::string(client.addr), strerror(err))!;

	const forward = match (err) {
		(unexpected_eof | invalid_query | no_such_user
			| relay_denied | max_query) => true,
		* => false,
	};
	if (!forward) {
		disconnect(ctx, client);
		return;
	};

	client.buf = client.xbuf[..];
	const s = fmt::bsprintf(client.buf, "Error: {}\r\n", strerror(err));
	client.buf = client.buf[..len(s)];
	client.state = state::WRITE_ERROR;
	submit_write(ctx, client, client.fd);
};

fn disconnect(ctx: *context, client: *client) void = {
	const sqe = io_uring::must_get_sqe(ctx.uring);
	io_uring::close(sqe, client.fd);
	if (client.plan_fd != -1) {
		const sqe = io_uring::must_get_sqe(ctx.uring);
		io_uring::close(sqe, client.plan_fd);
	};

	let i = 0z;
	for (i < len(ctx.clients); i += 1) {
		if (ctx.clients[i] == client) {
			break;
		};
	};
	delete(ctx.clients[i]);
	free(client);
};

We log the error here, and for certain kinds of errors, we “forward” them to the client by writing them to our client buffer and going into the “WRITE_RESP” state. For other errors, we just drop the connection.

The disconnect function, which disconnects the client immediately, queues io_uring submissions to close the open file descriptors associated with it, and then removes it from the list of clients.

Let’s get back to the happy path. Remember the read SQE we submitted when the client established the connection? When the CQE comes in, the state machine directs us into this function:

fn client_query(ctx: *context, client: *client, cqe: *io_uring::cqe) (void | error) = {
	const r = io_uring::result(cqe)?;
	if (r <= 0) {
		return unexpected_eof;
	};

	const r = r: size;
	if (len(client.buf) + r > len(client.xbuf) / 2) {
		return max_query;
	};

	client.buf = client.xbuf[..len(client.buf) + r];

	// The RFC requires queries to use CRLF, but it is also one of the few
	// RFCs which explicitly reminds you to, quote, "as with anything in the
	// IP protocol suite, 'be liberal in what you accept'", so we accept LF
	// as well.
	let lf = match (bytes::index(client.buf, '\n')) {
		z: size => z,
		void => {
			if (len(client.buf) == len(client.xbuf) / 2) {
				return max_query;
			};
			submit_read(ctx, client, client.fd, 0);
			return;
		},
	};
	if (lf > 0 && client.buf[lf - 1] == '\r': u8) {
		lf -= 1; // CRLF
	};
	const query = match (strings::try_fromutf8(client.buf[..lf])) {
		* => return invalid_query,
		q: str => q,
	};

	fmt::printfln("{}: finger {}", ip::string(client.addr), query)!;
	const plan = process_query(ctx, query)?;
	defer free(plan);

	client.plan_path = strings::to_c(plan);

	const sqe = io_uring::must_get_sqe(ctx.uring);
	io_uring::openat(sqe, rt::AT_FDCWD, client.plan_path, rt::O_RDONLY, 0);
	io_uring::set_user(sqe, client);
	client.state = state::OPEN_PLAN;
};

The first half of this function figures out if we’ve received a full line, including CRLF. The second half parses this line as a finger query and prepares to fulfill the enclosed request.

The read operation behaves like the read(2) syscall, which returns 0 on EOF. We aren’t expecting an EOF in this state, so if we see this, we boot them out. We also have a cap on our buffer length, so we return the max_query error if it’s been exceeded. Otherwise, we look for a line feed. If there isn’t one, we submit another read to get more from the client, but if a line feed is there, we trim off a carriage return (if present) and decode the completed query as a UTF-8 string.

We call “process_query” (using the error propagation operator to bubble up errors), which returns the path to the requested user’s ~/.plan file. We’ll look at the guts of that function in a moment. The return value is heap allocated, so we defer a free for later.

Strings in our language are not null terminated, but io_uring expects them to be. This is another case which will be addressed transparently once we build a higher-level, portable interface. For now, though, we need to call “strings::to_c” ourselves, and stash it on the client struct. It’s heap allocated, so we’ll free it in the next state when the I/O submission completes.

Speaking of which, we finish this process after preparing the next I/O operation — opening the plan file — and setting the client state to the next step in the state machine.

Before we move on, though, I promised that we’d talk about the process_query function. Here it is in all of its crappy glory:

use path;
use strings;
use unix::passwd;

fn process_query(ctx: *context, q: str) (str | error) = {
	if (strings::has_prefix(q, "/W") || strings::has_prefix(q, "/w")) {
		q = strings::sub(q, 2, strings::end);
		for (strings::has_prefix(q, " ") || strings::has_prefix(q, "\t")) {
			q = strings::sub(q, 1, strings::end);
		};
	};
	if (strings::contains(q, '@')) {
		return relay_denied;
	};

	const user = q;
	const pwent = match (passwd::getuser(user)) {
		void => return no_such_user,
		p: passwd::pwent => p,
	};
	defer passwd::pwent_finish(pwent);

	let enabled = false;
	for (let i = 0z; i < len(ctx.users); i += 1) {
		if (user == ctx.users[i]) {
			enabled = true;
			break;
		};
	};
	if (!enabled) {
		return no_such_user;
	};

	return path::join(pwent.homedir, ".plan");
};

The grammar described in RFC 1288 is pretty confusing, but most of it is to support features I’m not interested in for this simple implementation, like relaying to other finger hosts or requesting additional information. I think I’ve “parsed” most of the useful bits here, and ultimately I’m aiming to end up with a single string: the username whose details we want. I grab the user’s passwd entry and check if they’re a member of the “finger” group we populated way up there in the first code sample. If so, we pull the path to their homedir out of the passwd entry, join it with “.plan”, and send it up the chain.

At this point we’ve received, validated, and parsed the client’s query, and looked up the plan file we need. The next step is to open the plan file, which is where we left off at the end of the last function. The I/O we prepared there takes us here when it completes:

fn client_open_plan(
	ctx: *context,
	client: *client,
	cqe: *io_uring::cqe,
) (void | error) = {
	free(client.plan_path);

	client.plan_fd = io_uring::result(cqe)?;
	client.buf = client.xbuf[..0];
	client.state = state::READ_PLAN;
	submit_read(ctx, client, client.plan_fd, -1);
};

By now, this should be pretty comprehensible. I will clarify what the “[..0]” syntax does here, though. This language has slices, which store a pointer to an array, a length, and a capacity. In our client state, xbuf is a fixed-length array which provides the actual storage, and “buf” is a slice of that array, which acts as a kind of cursor, telling us what portion of the buffer is valid. The result of this expression is to take a slice up to, but not including, the 0th item of that array — in other words, an empty slice. The address and capacity of the slice still reflect the traits of the underlying array, however, which is what we want.

We’re now ready to read data out of the user’s plan file. We submit a read operation for that file descriptor, and when it completes, we’ll end up here:

fn client_read_plan(
	ctx: *context,
	client: *client,
	cqe: *io_uring::cqe,
) (void | error) = {
	const r = io_uring::result(cqe)?;
	if (r == 0) {
		disconnect(ctx, client);
		return;
	};

	client.buf = client.xbuf[..r];

	// Convert LF to CRLF
	//
	// We always read a maximum of the length of xbuf over two so that we
	// have room to insert these.
	let seencrlf = false;
	for (let i = 0z; i < len(client.buf); i += 1) {
		switch (client.buf[i]) {
			'\r' => seencrlf = true,
			'\n' => if (!seencrlf) {
				static insert(client.buf[i], '\r');
				i += 1;
			},
			* => seencrlf = false,
		};
	};

	client.state = state::WRITE_RESP;
	submit_write(ctx, client, client.fd);
};

Again, the read operation for io_uring behaves similarly to the read(2) syscall, so it returns the number of bytes read. If this is zero, or EOF, we can terminate the state machine and disconnect the client (this is a nominal disconnect, so we don’t use disconnect_err here). If it’s nonzero, we set our buffer slice to the subset of the buffer which represents the data io_uring has read.

The Finger RFC requires all data to use CRLF for line endings, and this is where we deal with it. Remember earlier when I noted that we only ever used half of the read buffer? This is why: if we read 1024 newlines from the plan file, we will need another 1024 bytes to insert carriage returns. Because we’ve planned for and measured out our memory requirements in advance, we can use “static insert” here. This built-in works similarly to how insert normally works, but it will never re-allocate the underlying array. Instead, it asserts that the insertion would not require a re-allocation, and if it turns out that you did the math wrong, it aborts the program instead of buffer overflowing. But, we did the math and it works out, so it saves us from an extra allocation.

Capping this off, we submit a write to transmit this buffer to the client. “submit_write” is quite similar to submit_read:

fn submit_write(ctx: *context, client: *client, fd: int) void = {
	const sqe = io_uring::must_get_sqe(ctx.uring);
	io_uring::write(sqe, fd, client.buf: *[*]u8, len(client.buf),
		0, flags::IO_LINK);
	io_uring::set_user(sqe, client);

	let ts = rt::timespec { ... };
	time::instant_to_timespec(client.deadline, &ts);
	const sqe = io_uring::must_get_sqe(ctx.uring);
	io_uring::link_timeout(sqe, &ts, timeout_flags::ABS);
};

Ideally, this should not require explanation. From here we transition to the WRITE_RESP state, so when the I/O completes we end up here:

fn client_write_resp(
	ctx: *context,
	client: *client,
	cqe: *io_uring::cqe,
) (void | error) = {
	const r = io_uring::result(cqe)?: size;
	if (r < len(client.buf)) {
		client.buf = client.buf[r..];
		submit_write(ctx, client, client.fd);
		return;
	};

	if (client.state == state::WRITE_ERROR) {
		disconnect(ctx, client);
		return;
	};

	client.buf = client.xbuf[..0];
	client.state = state::READ_PLAN;
	submit_read(ctx, client, client.plan_fd, -1);
};

First, we check if we need to repeat this process: if we have written less than the size of the buffer, then we advance the slice by that much and submit another write.

We can arrive at the next bit for two reasons: because “client.buf” includes a fragment of a plan file which has been transmitted to the client, which we just covered, or because it is the error message buffer prepared by “disconnect_err”, which we discussed earlier. The dispatch function will bring us here for both the normal and error states, and we distinguish between them with this second if statement. If we’re sending the plan file, we submit a read for the next buffer-ful of plan. But, our error messages always fit into one buffer, so if we ran out of buffer then we can just disconnect in the error case.

And that’s it! That completes our state machine, and I’m pretty sure we’ve read the entire program’s source code by this point. Pretty neat, huh? io_uring is quite interesting. I plan on using this as a little platform upon which I can further test our io_uring implementation and develop a portable async I/O abstraction. We haven’t implemented a DNS resolver for the stdlib yet, but I’ll also be writing a finger client (using synchronous I/O this time) once we do.

If you really wanted to max out the performance for a CLOUD SCALE WEB 8.0 XTREME PERFORMANCE finger server, we could try a few additional improvements:

Adding an internal queue for clients until we have room for their I/O in the SQ
Using a shared buffer pool with the kernel, with io_uring ops like READ_FIXED
Batching requests for the same plan file by only answering requests for it every Nth millisecond (known to some as the “data loader” pattern)
More slow loris mitigations, such as limiting open connections per IP address

It would also be cool to handle SIGHUP to reload our finger group membership list without rebooting the daemon. I would say “patches welcome”, but I won’t share the git repo until the language is ready. And the code is GPL’d, but not AGPL’d, so you aren’t entitled to it if you finger me!